Unexpected restart during stop of firmware

mathias.hatlestad

Hello, 

I'm trying to setup automatic update of the application we are running, this require that we shutdown the firmware (which is responsible for running our component).

However, after i call the /etc/init.d/plcnext stop command the system will shutdown automatically, this before I'm able to transfer the required files etc. 

Console output: 

Stopping service plcnext
Warning: Application with pid 1301 does not terminate within 20 seconds.

Output.log output: 

Jan 9 16:52:52 axcf2152 daemon.err watchdog[1490]: pinging process 1301 (/opt/plcnext/data/System/Watchdog/mainProcess.pid) gave errno = 3 = 'No such process'
Jan 9 16:52:52 axcf2152 daemon.alert watchdog[1490]: shutting down the system because of error 3 = 'No such process'

 

Could this issue be related to my application not being able to shutdown correctly on a stop(), or do you have any pointers on what the problem could be? 

mathias.hatlestad

I also get the following messages when the controller is rebooted. Note, that "Run" and "Fail" light is flashing at 2Hz intervals on the controller.  

Jan 13 11:43:38 axcf2152 authpriv.notice sudo: plcnext_firmware : TTY=unknown ; PWD=/var/volatile/tmp ; USER=root ; COMMAND=/usr/sbin/watchdog --config-file /opt/plcnext/data/System/Watchdog/WatchdogDaemon.config
Jan 13 11:43:38 axcf2152 daemon.notice watchdog[1478]: starting daemon (5.15):
Jan 13 11:43:38 axcf2152 daemon.info watchdog[1478]: int=1s realtime=yes sync=no load=0,0,0 soft=no
Jan 13 11:43:38 axcf2152 daemon.info watchdog[1478]: memory not checked
Jan 13 11:43:38 axcf2152 daemon.info watchdog[1478]: ping: no machine to check
Jan 13 11:43:38 axcf2152 daemon.info watchdog[1478]: file: no file to check
Jan 13 11:43:38 axcf2152 daemon.info watchdog[1478]: pidfile: /opt/plcnext/data/System/Watchdog/mainProcess.pid
Jan 13 11:43:38 axcf2152 daemon.info watchdog[1478]: pidfile: /opt/plcnext/data/System/Watchdog/ExternalIoProcess.pid
Jan 13 11:43:38 axcf2152 daemon.info watchdog[1478]: pidfile: /opt/plcnext/data/System/Watchdog/LocalIoProcess.pid
Jan 13 11:43:38 axcf2152 daemon.info watchdog[1478]: interface: no interface to check
Jan 13 11:43:38 axcf2152 daemon.info watchdog[1478]: temperature: no sensors to check
Jan 13 11:43:38 axcf2152 daemon.info watchdog[1478]: no test binary files
Jan 13 11:43:38 axcf2152 daemon.info watchdog[1478]: no repair binary files
Jan 13 11:43:38 axcf2152 daemon.info watchdog[1478]: error retry time-out = 0 seconds
Jan 13 11:43:38 axcf2152 daemon.info watchdog[1478]: repair attempts = 1
Jan 13 11:43:38 axcf2152 daemon.info watchdog[1478]: alive=/dev/watchdog1 heartbeat=[none] to=root no_act=no force=no
Jan 13 11:43:38 axcf2152 daemon.err watchdog[1478]: cannot set timeout 60 (errno = 95 = 'Operation not supported')
Jan 13 11:43:38 axcf2152 daemon.info watchdog[1478]: hardware watchdog identity: Watchdog Driver for axc2152
Jan 13 11:43:39 axcf2152 user.info [1302]: Arp.Services.OpcUAServer.Internal.Config.OpcUAConfigManager INFO - Using the self-signed certificate.
Jan 13 11:43:39 axcf2152 user.info [1302]: OpcServer INFO - OPC UA server opened endpoint URI opc.tcp://axc-f-2152-1:4840 listening at opc.tcp://0.0.0.0:4840
Jan 13 11:43:39 axcf2152 user.info [1302]: Arp.Services.OpcUAServer.OpcUAServerComponent INFO - Started OPC UA Server successfully.
Jan 13 11:43:39 axcf2152 user.info [1302]: Arp.Services.OpcUAServer.Internal.Security.SessionManager INFO - Applying configuration for client authentication to 1 endpoints: trustAll=true, checkCRLs=false, checkIssuerCRLs=false, ignoreValidity=false
Jan 13 11:43:39 axcf2152 user.info [1302]: Arp.Plc.Domain.Internal.PlcManager INFO - Cannot start Plc: Plc is suspended by system watchdog.

Martin [PLCnext Technology Team]

Hi Mathias,

This is all related to the System Watchdog feature that was introduced with firmware version 2020.0.

There is currently no user documentation on this feature (!), so I am gathering information that explains what you are seeing, and what you can do about it. I hope to have something for you shortly.

~ Martin.

Martin [PLCnext Technology Team]

Hi Mathias,

An update for you:

• The System Watchdog feature uses the watchdog daemon to monitor the plcnext process. If the plcnext process behaves unexpectedly, the watchdog is triggered.
• This watchdog can be triggered if the plcnext process takes too long to shut down after you issue the /etc/init.d/plcnext stop or the /etc/init.d/plcnext restart command.
• Once the watchdog is triggered, the PLC reboots.
• After reboot, the plcnext runtime starts, but the "PLC" component (i.e. the part of the PLCnext runtime that implements real-time PLC programs) remains in the "Ready" state, and can only be started using a "Cold Start" command. This explains the final message in your log file listing above.
• In order to perform a Cold Start of the PLC ... you must use PLCnext Engineer, or else call a method on the relevant RSC service from one of your C++ components. Unfortunately this is not very convenient if you are not using PLCnext Engineer.
• After you perform the cold start, everything should go back to normal ... until you issue the the next /etc/init.d/plcnext stop command, when the same problem is likely to happen again.

Because of the inconvenience this causes during development, there is a way to disable this behaviour. Please note, however, that you do so at your own risk, and this behaviour should be re-enabled before deploying your PLC in a production environment.

To disable this behaviour:

1. Create the following directory(s) on the PLC: /opt/plcnext/config/System/Watchdog/
2. In that directory, create a file named: watchdog.primary.config
3. Edit the file and paste the text from the attached PDF into the file (ignore the PDF document title!).
4. Close the file and restart the PLC.

The System Watchdog will see this configuration file, and it should then not require a Cold Start of the PLC when the watchdog is triggered.

Again, please remember to restore this System Watchdog function before using the PLC in a production environment.

~ Martin.

 

 

mathias.hatlestad

Hello,

Thanks for the answer. 

However, the system will not boot up in normal operation after the watchdog is triggered, even if i add the file as you suggested. 

Furthermore, the only thing that helps is a Warm Start (not a cold one), and even if I do that I'm not able to get my program to read the GDS. I guess this could be due to the order of startup, and since the plc is disabled the GDS is not setup properly? 

Note that after a full reset, I'm able to read the GDS as i expect. However, on the next boot / restart of plcnext I'm not longer able to read the input channels. DI16 module. 

 

I have currently put the configuration to start my component in the /opt/plcnext/projects/Default/Default.acf.config, is this correct given that I need the GDS to be setup correctly to read the input channels etc? 

 

Martin [PLCnext Technology Team]

Hi Mathias,

I have currently put the configuration to start my component in the /opt/plcnext/projects/Default/Default.acf.config, is this correct ...

That should work, at least until you next upgrade the firmware.

I guess yours is a an ACF component (i.e. inheriting from ComponentBase or MetaComponentBase)? If so, which one of the standard Component methods are you using to trigger the calls to the GDS?

Can you please send me an Output.log file showing a startup when the problem happens? (I think you have my email address :-). That will help us diagnose the problem.

~ Martin.

Martin [PLCnext Technology Team]

Hi Mathias,

Further to my comment above - please try disabling the System Watchdog feature altogether, as described in this post. The environment variable that you should set to "false" is ARP_SYSTEM_WATCHDOG_SUPPORT.

If that doesn't help, then please feel free to send more info to my email address.

~ Martin.