Apparent time instability after enabling daylight-saving

fblanchard · March 17, 2022, 9:28pm

Hi,
Our project depends on a reliable clock that can works with timezones and daylight-saving Linux features.
The PLCnext Engineer IDE gives the possibility to set up the clock with an absolute numeric offset, fair enough for simulating timezone, but not compatible with the Linux daylight-saving feature. Anyways, we want our client to adjust the clock without using the IDE.
I am aware on how the „date“ versus „hwclock“ comands basically works but using them had led us in some situations where the system clock became unstable (especially after a power toggle).
It is probably my fault, but I wonder if its is not something special with your RTC implementation, or a particular limitation that make you says in the OS release notes (between 2021.0 LTS and 2021.9.0):
Local time zone setting****Setting local time zones is not fully supported.
Can you be more specific on this?
Can you gives me guidelines on how using the command line „date“ and „hwclock“?
Thanks
F.Blanchard
Using PLCnext AFC 2152, OS 2022.0.3 LTS, IDE 2021.0.LTS

Example:

// Set the date
root@axcf2152:~# date --set „Wed Mar 16 16:19:10 CDT 2022“
Wed Mar 16 16:19:10 CDT 2022

// Set the local time to UTC timezone
root@axcf2152:~# ln -sf /usr/share/zoneinfo/UTC /etc/localtime root@axcf2152:~# echo -e „UTC“ > /etc/timezoneroot@axcf2152:~# date
Wed Mar 16 21:20:30 UTC 2022

// Write this change to the hardware clock
root@axcf2152:~# hwclock --systohc

//Validation
root@axcf2152:~# date; hwclock
Wed Mar 16 21:24:59 UTC 2022
Wed Mar 16 21:25:00 2022 0.000000 seconds

// Set the local time zone back to its original setting.
root@axcf2152:~# ln -sf /usr/share/zoneinfo/CST6CDT /etc/localtime root@axcf2152:~# echo -e „CST6CDT“ > /etc/timezoneroot@axcf2152:~# date; hwclock
Wed Mar 16 16:25:52 CDT 2022
Wed Mar 16 21:25:52 2022 0.000000 seconds

// Power toggle or reboot here

root@axcf2152:~# date; hwclock Wed Mar 16 16:03 :25 CDT 2022
Wed Mar 16 15:03 :26 2022 0.000000 seconds

// Reboot here

root@axcf2152:~# date; hwclock
Wed Mar 16 16:06:01 CDT 2022
Wed Mar 16 21:06:02 2022 0.000000 seconds
root@axcf2152:~# echo

// Power toggle here (11 minutes OFF)

root@axcf2152:~# date; hwclock
Wed Mar 16 16:09:18 CDT 2022
Wed Mar 16 21:09:19 2022 0.000000 seconds

// Power toggle here (11 minutes OFF)

root@axcf2152:~# date; hwclock Wed Mar 16 16 :14:19 CDT 2022
Wed Mar 16 15 :14:20 2022 0.000000 seconds

// Power toggle here (11 minutes OFF)

root@axcf2152:~# date; hwclock
Wed Mar 16 16:19:59 CDT 2022
Wed Mar 16 21:20:00 2022 0.000000 seconds
root@axcf2152:~#
root@axcf2152:~#
root@axcf2152:~# date; hwclock Wed Mar 16 16 :25:23 CDT 2022
Wed Mar 16 15 :25:24 2022 0.000000 seconds
root@axcf2152:~#

martin-plcnext · March 18, 2022, 10:32am

Firstly, this comment in the release notes:
Local time zone settingSetting local time zones is not fully supported… only means that we do not provide users with a simple way of setting the time zone on the device (e.g. through PLCnext Engineer or WBM). There are ways to set the time zone, as you have done, but there is no guarantee that this will work properly with all PLCnext Runtime components, or after a firmware upgrade. While we do not officially support timezones on the device, setting the timezone should not lead to instability in the system clock or the hardware clock.
I haven’t been able to reproduce any problems with the hwclock and system time.
Here is my sequence of commands on an AXC F 2152 FW 2022.0.3:

# No time zone set
admin@axcf2152:~$ ls -la /etc/localtime
lrwxrwxrwx 1 root root 29 Mar  9  2018 /etc/localtime -> /usr/share/zoneinfo/Universal
admin@axcf2152:~$ date; date -u; hwclock; hwclock -u
Fri Mar 18 09:15:04 UTC 2022
Fri Mar 18 09:15:04 UTC 2022
Fri Mar 18 09:15:05 2022  0.000000 seconds
Fri Mar 18 09:15:05 2022  0.000000 seconds
# Set the date/time, specifying that it is North America Central Daylight Savings Time (UTC-5)
root@axcf2152:/opt/plcnext/#  date --set "Fri Mar 18 04:25:00 CDT 2022"
Fri Mar 18 09:25:00 UTC 2022
# The date/time is correct (5 hours ahead of CDT).
root@axcf2152:/opt/plcnext/# date; date -u; hwclock; hwclock -u
Fri Mar 18 09:25:10 UTC 2022
Fri Mar 18 09:25:10 UTC 2022
Fri Mar 18 09:23:05 2022  0.000000 seconds
Fri Mar 18 09:23:05 2022  0.000000 seconds
# Set the time zone on the device to North America Central Time
root@axcf2152:/opt/plcnext/# ln -sf /usr/share/zoneinfo/CST6CDT /etc/localtime
# The date/time is correct.
root@axcf2152:/opt/plcnext/# date; date -u; hwclock; hwclock -u
Fri Mar 18 04:34:47 CDT 2022
Fri Mar 18 09:34:47 UTC 2022
Fri Mar 18 09:32:41 2022  0.000000 seconds
Fri Mar 18 04:32:41 2022  0.000000 seconds
# hwclock is stored in UTC, so hwclock -u converts UTC to local time.
# Reboot
# The date/time is correct.
root@axcf2152:/opt/plcnext/# date; date -u; hwclock; hwclock -u
Fri Mar 18 04:48:54 CDT 2022
Fri Mar 18 09:48:54 UTC 2022
Fri Mar 18 09:48:54 2022  0.000000 seconds
Fri Mar 18 04:48:54 2022  0.000000 seconds
# Set the hwclock from the system clock.
root@axcf2152:/opt/plcnext/# hwclock --systohc
# Because no options were specified, hwclock is now set to the local time.
# The date/time is correct.
root@axcf2152:/opt/plcnext/# date; date -u; hwclock; hwclock -u
Fri Mar 18 04:56:01 CDT 2022
Fri Mar 18 09:56:01 UTC 2022
Fri Mar 18 04:56:01 2022  0.000000 seconds
Thu Mar 17 23:56:01 2022  0.000000 seconds
# hwclock now stores the local time, so telling hwclock that it is in UTC (-u) gives a meaningless result.
# Reboot
# The date/time is correct.
root@axcf2152:/opt/plcnext/# date; date -u; hwclock; hwclock -u
Fri Mar 18 05:04:59 CDT 2022
Fri Mar 18 10:04:59 UTC 2022
Fri Mar 18 10:05:00 2022  0.000000 seconds
Fri Mar 18 05:05:00 2022  0.000000 seconds
# hwclock has now been updated to the correct UTC time, so the -u option give the correct local time.
# Reboot
# The date/time is correct.
admin@axcf2152:~$ date; date -u; hwclock; hwclock -u
Fri Mar 18 05:13:59 CDT 2022
Fri Mar 18 10:13:59 UTC 2022
Fri Mar 18 10:14:00 2022  0.000000 seconds
Fri Mar 18 05:14:00 2022  0.000000 seconds
# Set the value of /etc/timezone
root@axcf2152:/opt/plcnext/# echo -e "CST6CDT" > /etc/timezone
root@axcf2152:/opt/plcnext/# more /etc/timezone
CST6CDT
# The date/time is correct.
admin@axcf2152:~$ date; date -u; hwclock; hwclock -u
Fri Mar 18 05:16:52 CDT 2022
Fri Mar 18 10:16:52 UTC 2022
Fri Mar 18 10:16:53 2022  0.000000 seconds
Fri Mar 18 05:16:53 2022  0.000000 seconds
# Reboot
# The date/time is correct.
admin@axcf2152:~$ date; date -u; hwclock; hwclock -u
Fri Mar 18 05:20:41 CDT 2022
Fri Mar 18 10:20:41 UTC 2022
Fri Mar 18 10:20:42 2022  0.000000 seconds
Fri Mar 18 05:20:42 2022  0.000000 seconds

The problem on your PLC may have happened when the hwclock was set to the local time (not the UTC time). The hwclock is changed back to UTC time after a reboot. This might explain the time variations you were seeing. I can’t see a reason for you to set the hwclock explicitly with the --systohc option, but if you do then it is probably best to make sure it is always set to UTC time, not local time.
If you are still finding unexplained behaviour after doing this, please let us know.

martin-plcnext · March 18, 2022, 11:16am

You can also see some interesting information in the comments in this init.d script:
/etc/init.d/hwclock.sh
Specifically:

                                          * The system clock is set from the hwclock during startup.
                                          * During a "normal" shutdown or restart, the hwclock is set from the system clock.

For anyone who is interested, the „Notes“ section of this page gives a good general background on these two clocks.
https://linux.die.net/man/8/hwclock

fblanchard · March 18, 2022, 3:50pm

Thanks Martin for the detailed example. It is very appreciated.
I exactly repeated your test sequence and I got the same result as you did.
However, after waiting a few minutes, I saw the system clock going nuts, as illustrated below:

// At the beginning, everything looks good
root@axcf2152:~# date; date -u; hwclock; hwclock -u
Fri Mar 18 05:25:18 CDT 2022
Fri Mar 18 10:25:18 UTC 2022
Fri Mar 18 10:25:18 2022 0.000000 seconds
Fri Mar 18 05:25:18 2022 0.000000 seconds

// Then after a few minutes the hardware clock gets bad.
root@axcf2152:~# date; date -u; hwclock; hwclock -u
Fri Mar 18 05:30:35 CDT 2022
Fri Mar 18 10:30:35 UTC 2022
Fri Mar 18 04:30:36 2022 0.000000 seconds
Thu Mar 17 23:30:36 2022 0.000000 seconds

// Then the system time gets also bad.
root@axcf2152:~# date; date -u; hwclock; hwclock -u
Fri Mar 18 06:04:34 CDT 2022
Fri Mar 18 11:04:34 UTC 2022
Fri Mar 18 05:04:35 2022 0.000000 seconds
Fri Mar 18 00:04:35 2022 0.000000 seconds

// Then no more flipping changes
root@axcf2152:~# date; date -u; hwclock; hwclock -u
Fri Mar 18 06:41:10 CDT 2022
Fri Mar 18 11:41:10 UTC 2022
Fri Mar 18 05:41:11 2022 0.000000 seconds
Fri Mar 18 00:41:11 2022 0.000000 seconds
root@axcf2152:~#

Can you repeat a similar behaviour?

Observations:
This example showed that the power interruption is not necessary to repeat the problem.
In the documentation you mentioned (hwclock man page), they talk about a file that I did not find on the PLCnext system: /etc/adjtime. Since the problem arrives after a certain time, it looks like it is related to the periodic hardware clock time adjustment. The period looks like the classic 11 minutes.
Is it possible that my many experiments could have cause the time adjustment mechanism to get disbalanced? I am referring to this text on the man page:
" Every time you calibrate (set) the clock (using --set or --systohc), hwclock recalculates the systematic drift rate based on how long it has been since the last calibration, how long it has been since the last adjustment, what drift rate was assumed in any intervening adjustments, and the amount by which the clock is presently off".
If it is the case, is there a way to clear the time update history so that the time adjustment starts from fresh?
Thanks again

fblanchard · March 21, 2022, 1:27pm

Update:
· Mistake in my previous comment: The system clock is not affected, only the hardware clock (RTC) after a few minutes after reboot.
· Changing the time using the IDE is working well. Is it possible to get a code same (whatever the language)? It might be a good pointer to the effective way to change the clock.
· Despite the IDE way (which I cannot use to help our customer update the system time), there still a very visible and reproducible flaw in the PLCnext: Just after power up the hardware clock is good, but a few minutes later, the same clock gets wrong:

// Just after powerup
root@axcf2152:~# date; date -u; hwclock;
Mon Mar 21 09:02:12 EDT 2022
Mon Mar 21 13:02:12 UTC 2022
Mon Mar 21 13:02 :12 2022 0.000000 seconds

// After a few minutes
root@axcf2152:~# date; date -u; hwclock;
Mon Mar 21 09:07:32 EDT 2022
Mon Mar 21 13:07:32 UTC 2022
Mon Mar 21 08:07 :33 2022 0.000000 seconds

martin-plcnext · March 21, 2022, 2:07pm

I saw similar strange behaviour when I started my AXC F 2152 this morning.
I still had the same time zone settings from last week (North America Central Time as the local time zone).
After a bit of messing, I think this has fixed the problem for me:
Set the hwclock to anything, but making sure to use the --utc option. i.e. 2022-03-21_14-34-58.png
* hwclock --systohc --utc
* Set the system (linux) time to the correct date/time, e.g. date --set „03/21/2022 08:30“
* Restart the device.
I think this is relevant:

In my case it looks like the hwclock now contains the local time, even though it was last set using the --utc option. But it seems to be stable. Unfortunately the BusyBox version of hwclock that is used in the current version of the firmware does not give information about what option the hwclock is currently using (utc or localtime).
Hopefully this will work on your device. I will keep watching the time on my device to see if it misbehaves.

fblanchard · March 21, 2022, 9:34pm

Yes indeed, we need the „–utc“ argument in „hwclock -w –utc“, but the reboot apparently does not change anything.
I made more PLCnext tests today like this:
1. Factory reset
2. Set timezone (command line)
3. Set system clock (command line)
4. Synch the hardware clock (command line)
5. (Optional reboot)
6. Power toggle the PLCnext
7. Check the sysclock and hwclock consistency for the next 12 minutes.
8. Toggle and check a second time.
That’s it, not even installing an HMI application.
I got the following results:
With OS 2019 LTS, no problem, except for a moment at bootup when the hwclock temporally lags one hour. But after that all is fine for both system and hardware clocks. So, the risk of corruption by a following power toggle theoretically only occurs at bootup (not tested).
With OS 2022 LTS, the hardware clock corruption always occurs between 1 to 12 minutes after bootup. Therefore, it is guarantied that a power toggle after that period will automatically cause a corruption of both the system and hardware clock. This is perfectly repeatable.
As our product rely heavily on the system clock, and that power toggle may occur anytime in the field, this behavior is totally unacceptable. We badly need a fix to the 2022 LTS as soon as possible.

martin-plcnext · March 22, 2022, 1:09pm

I note that disabling the NTP daemon „solves“ this problem, since the NTP daemon resets the HW clock every 11 minutes. However this is not an acceptable solution, since any application that wants to maintain an accurate time will probably be using NTP.
I have tried a few configuration options, like SYNC_HWCLOCK=no in the ntpd.conf file, but this didn’t seem to make any difference.
It seems like the longer the device is powered off, the further the clock drifts from the correct time.
I have sent this to the developers for investigation.

fblanchard · March 22, 2022, 4:05pm

Thanks Martin
I attached my final test procedure and results.
Update: The problem still persist if I use the PLCnext Engineer IDE or a custom HMI program to set the system date.
NWC3-TST-008-01-PlcNextRtcFailure.pdfFrederic

NWC3-TST-008-01-PlcNextRtcFailure.pdf

fblanchard · March 22, 2022, 8:08pm

Update:
We got no problem with the previous OS 2019.0 LTS. Therefore, its a regression.
We also tested OS 2020.0.1LTS, and only found that the hardware clock (RTC) will get late while power is OFF (especially the first time), like with 2022.0.3 LTS.
Finally, with 2022.0.3 LTS, when the described problem occurs, the PLCnext will also spontaneously reboot after a certain amount of time.

fblanchard · March 30, 2022, 5:35pm

Update:

                                          * The new OS 2022.0.4 LTS have the same problem.
                                          * Like suggested by Martin, disabling the NTP deamon and then rebooting the system provides a good temporary bypass.

Thanks.

martin-plcnext · March 31, 2022, 6:10am

Thanks for the feedback.
The developers are aware of this issue and I will update this thread when there is progress.

martin-plcnext · April 14, 2022, 10:58am

Conclusion:
The developers have reproduced and investigated this issue, and implemented a fix that is currently scheduled to be included in firmware version 2022.6.

fblanchard · July 27, 2022, 1:28pm

The firmware 2022.6 is released, but without a ReleaseNote. Does it finally fix our problem?
Thanks

martin-plcnext · August 9, 2022, 11:57am

The problem is fixed in firmware version 2022.6.
I have not been able to find out why this was not mentioned in the release notes for that version (sorry).

fblanchard · February 24, 2023, 2:32pm

I upgraded one of our test bench with OS 2023.0 LTS. We observed no RTC corruption this time after running the test described previously in this thread. So, the problem is solved on that regard.
But, the annoying side effect of the temporary workaround you suggested (disabling the NTP daemon), that we used with OS 2021.6, is also observable natively in 2023.0 LTS: The hardware clock is drifting significatively, and it is only resynchronized with the system clock at the start of the reboot sequence.
The not so funny part occurs during a power toggle: The shutdown clock resynch script could not be executed and at the following bootup the system clock is initialized with the drifted hardware clock! More the power toogles are far away, more the problem can become serious.
Here is an example of how important the drift is:

Starting the test with a system clock in synch with an external time reference:
admin@axcf2152:~$ date; date --utc; hwclock
Mon Feb 13 13:13:00 EST 2023
Mon Feb 13 18:13:00 UTC 2023
Mon Feb 13 18:13:00 2023 0.000000 seconds

After three days, at 16 :25 :00
admin@axcf2152:~$ date; date --utc; hwclock
Thu Feb 16 16:25:00 EST 2023 → Delta: 0s
Thu Feb 16 21:25:00 UTC 2023
Thu Feb 16 21:24:55 2023 0.000000 seconds → Delta: -5s

After nine days, at 09:13:00
admin@axcf2152:~$ date; date --utc; hwclock
Wed Feb 22 09:13:00 EST 2023 → Delta: 0s
Wed Feb 22 14:13:00 UTC 2023
Wed Feb 22 14:12:45 2023 0.000000 seconds → Delta: -15s

Note: On a normal Linux system, the resynch of the hardware clock by the system clock is typically executed every 11 minutes.

martin-plcnext · February 28, 2023, 12:27pm

Thanks for the update.
These findings have been passed to the developers, and they will investigate this as part of their work on future firmware versions. It will probably be some time before there are any updates, but I will report them here when I get them.