Thanks Martin, I’ll wait for your answer. In the mean time, my PLC stopped again, but this time it didn’t give me a Red Cross as Severity but several Grey and Yellow Exclamation Marks with the following notifications, what do they mean? Plc state changed: None ==> Ready Plc state changed: Ready ==> Stop () system watchdog state changed: new state: UnacknowledgedWatchdog plc startup is: forbidden SystemManager state changed: Stop, error=false, warning=false SystemManager state changed: Running, error=false, warning=false Plc state changed: Stop () ==> Stop ()
Hi Alex, I have branched this to a separate thread, since this is a new question. I hope that’s OK. (original thread is https://www.plcnext-community.net/en/discussions-2-offcanvas/arp-plc-esm-exception.html) Those notifications you can see in PLCnext Engineer come from various system components, via the Notification Manager. You can see all the messages in your listing (and much more) here: https://www.plcnext.help/te/Service_Components/Notifications/Notifications_by_Runtime.htm In your case it looks like the System Watchdog has been triggered. This is entirely different to a Task Watchdog (the subject of the earlier discussion), and usually results from an error in a user’s C++ program or component. After a System Watchdog event, the whole system reboots, but after that the “PLC” component of the PLCnext Runtime will not start (hence the “forbidden” message). This is to prevent an endless cycle of watchdog → reboot → watchdog → reboot → … When the user manually restarts the PLC (hopefully after fixing the error that caused the System Watchdog), everything should continue as normal. After the PLC has been running OK for 60 minutes, you will see a message related to the System Watchdog in the Output.log file, similar to this:
03.09.20 10:43:56.765 Arp.System.Watchdog.Internal.SystemWatchdogRebootCounter INFO - WaitForResetRebootCounterThreadBody reboot counter resetted, more than 60 minutes in run modeWe are planning to add more information on the System Watchdog to the Info Center in the near future. ~ Martin.
Thanks Martin, this information is very useful. We are using blocks with C# code, so this could be causing the System Watchdog error? How would be go about determining what section of the program is causing the error? The Task that many times indicates that TimeOut only executes once and then we used a Jump instruction avoid execution of the code, so I don’t understand why it happens. In particular, the outputs about FAN and SD card I don’t understand, what do they mean? 11.09.20 14:42:10.531 Arp.Plc.Esm.Internal.WatchdogHandler ERROR - Watchdog of task ‘Cyclic1s’ triggered. (watchdog time 1000000us / time after task start 1099836us) 11.09.20 14:43:18.008 Arp.Device.Interface.Internal.DeviceStatus INFO - Fan module not supported! 11.09.20 14:43:18.016 Arp.Device.Interface.Internal.DeviceStatus WARN - Adding an external sd-card during runtime is unsupported. To use the card for the overlay filesystem the controller must be rebooted.
Hi Alex,
In particular, the outputs about FAN and SD card I don’t understand, what do they mean?
The message about the fan is an information message, it just means that the PLC doesn’t support the external fan module that can be added to some PLCnext Control devices (e.g. RFC 4072S). I also noticed that strange SD card warning a while back. I don’t think it was meant to appear during every startup sequence, and it seems to be gone in FW version 2020.6.1. If you still see this warning in FW 2020.6.1, please let us know. Re. the watchdog - it would be unusual for a C# FB to trigger the System watchdog, in my experience. A C# FB is more likely to trigger a Task watchdog, like the one shown in your listing above, because it is executing inside a real-time ESM task. Unfortunately there is no simple way to find out what is holding up the task long enough to trigger a Task watchdog. I usually add debugging code to my program to see how far it gets before it hangs, or how long suspect operation(s) take to execute. If you disable the task watchdog completely, it might help to identify where the code gets stuck. ~ Martin.
Thanks Martin, Today the PLC crashed again, but this time I got a little bit more information, it tells me what program, but what is Synchronized Abort? 12.09.20 07:41:47.503 Arp.Plc.Esm.Internal.WatchdogHandler ERROR - Watchdog of task ‘Cyclic1s’ triggered. (watchdog time 1500000us / time after task start 1599841us) 12.09.20 07:41:47.761 Arp.Plc.Eclr.Internal.IecProgram ERROR - Program Mcv_Learn1 synchronized Abort failed 12.09.20 07:41:47.861 Arp.Plc.Esm.Internal.Tasks.TaskBase WARN - Task ‘Cyclic1s’ was hard terminated. Also noticed that on PLC Engineer I don’t get all those notirications, I had to go to the Output.log And lastly, how do I disable the Task WatchDog? Thanks
Hi Alex, Without checking with the FW devs, my guess is that, after the Watchdog is tripped, the ESM tries to terminate the program nicely - a “synchronised abort” - and when that fails (for whatever reason) it does a not-so-nice hard termination. I don’t think we can tell much from the failed synchronised abort - either way, there’s something in the code that blocks the program long enough to trip the Watchdog. You can disable the task watchdog by setting the task watchdog time to zero. This, together with some debug code, should help you find where the code is blocking. ~ Martin.
Thanks Martin, One more question. I haven’t disabled the Watchdog, I just increased it to a very long time (5 seconds), and still crashed. I just noticed that just after the crash the Data Memory is at 100% and in pink color! That might be causing the crashes. The notifications were (among many others): 16.09.20 14:44:33.041 Arp.Plc.Esm.Internal.WatchdogHandler ERROR - Watchdog of task ‘Cyclic1s’ triggered. (watchdog time 5500000us / time after task start 5549836us)
16.09.20 14:44:33.257 Arp.Plc.Eclr.Internal.IecProgram ERROR - Program Mcv_Learn1 synchronized Abort failed
16.09.20 14:44:33.357 Arp.Plc.Esm.Internal.Tasks.TaskBase WARN - Task ‘Cyclic1s’ was hard terminated.
16.09.20 14:44:33.360 Arp.Plc.Domain.Internal.PlcManager ERROR - Fatal exception occured: stopping Plc asynchronously. After rebooting the PLC the memory went back to 1%. So it has to be something related to that. Can I email you the Output.log so you can help me diagnose? Thanks!
Hi Alex, You should have received an email yesterday - please include the Output.log file in the reply. If you did not receive the email, please check your spam box. The email was from notify-ticket@phoenixcontact.com ~ Martin.