PLCnext crash after a few days

[fblanchard]

Hello, We got a problem on the field with our PLCnexts that simply stop working after 7 days or less. We then cannot connect by any means like ssh, telnet or the hmi. The system only responds to pings. The expected system behavior is stopped. The /opt/plcnext/logs/Output.log file does not contain any relevant information (just a blank “NULL NULL NULL …” string at the time of the event). We also observed once the appearance of zombie processes before the crash. In my investigation, I found two interesting items in the release notes that may explain theses crashes (our PLCnext have FW 2020.0.1):

                                             * Release note: “System files modified with “root“ access could prevent proper reboot after a firmware update.”

We actually did modify the nginx.conf files using root access before upgrading the OS from 2019.9 to 2020.0.1, but we never encountered any booting problem. How can we get around this problem? What if we change (as root user) the owner of this file to “admin:plcnext” before making the firmware update? Any other suggestions?
* Release note: “The PLCnext CLI version used must match the current SDK for this version. Downward compatibility cannot be guaranteed”
We actually did compile our C++ library using SDK 2019.9 and PLCnCLI 2020. Is this possible compatibility problem will lead to simple compiler errors/warnings or instead lead to real system instabilities? - → Also, in your documentation, you are asking to be logged as root before installing dotnet. Is this could lead to the same kind of problem after a firmware update? - → Finally, does the PLCnext uses a hardware or software watchdog ? We are afraid that zombie processes can block a software watchdog. Yes, the “System Watchdog” parameter “Restart after system watchdog” is set to “Allowed” Thanks Frédéric ********************************************************************************** Update: Attached extra PDF and PNG files with diagnostic informations. **********************************************************************************

[Oliver_PLCnext_Team]

Hello Frédéric,

your local sales person should have contacted you by now for detailed support.

Thank you very much for sharing these debuging Information.
The Watchdog is not triggering in in this case because all runtime processes are still running.
Even thou you have some network or IO issues happening on the PLC.
We are investgating this case and having a look at how to further improve the Watchdog mechanisms.

If some zombie process with a high priority blocks the Watchdog that will trigger the Watchdog.
Also if parts of the PLCnext Runtime process die the Watchdog will be triggered.
The watchdog will even trigger if you have the IEC Watchdog disabled and infiit loops on both ESMs.

----------------
1. Question:

                                             * _Release note: “System files modified with “root“ access could prevent proper reboot after a firmware update.”_

_
We actually did modify the nginx.conf files using root access before upgrading the OS from 2019.9 to 2020.0.1, but we never encountered any booting problem._
How can we get around this problem?
What if we change (as root user) the owner of this file to “admin:plcnext” before making the firmware update?
Any other suggestions?

1. Answer:
nginx.conf should not make a problem. (as long as the nginx.conf is not modified by the FW update as well.)

This warning mainly refers to files located at “/etc/plcnext”

What causes these issues is the overlay file system.
When you update the FW. the read only layer of the Overlay gets modified and updated.
The userdata located at the “uppderdir” will not be touched.
So what happens when you modify a file that is part of the FW is as follows:
A copy of the file with your modification gets created in the “upperdir”.
The System will always use the “upperdir”-version of a file first.
If that does not exist it will use the “lowerdir” version.
So if the new version of the FW provides files with syntax changes or additional parameters (for example to enable new features), these will not be loaded.
changed permissions can be an issue as well but are not the main issue here.

----------------
2. Question:

                                             * _Release note: “The PLCnext CLI version used must match the current SDK for this version. Downward compatibility cannot be guaranteed”_

We actually did compile our C++ library using SDK 2019.9 and PLCnCLI 2020. Is this possible compatibility problem will lead to simple compiler errors/warnings or instead lead to real system instabilities?

2. Answer:
Some of the templates or generated code by a later PLCnCLI versions can change.

For example:
by default the “New” project template changed between 2019 and 2020 PLCNCLI. SDK prior 2020.0:
In each component .cpp file replace the line ‘MetaComponentBase::SetupSettings();’
with ‘this->RegisterComponentPorts();’ and the line
‘MetaComponentBase::Dispose();’ with ‘this->dataInfoProvider.Reset();’ Thats why it is recommended to always stick to the bundles. -- In this case there should be no issues in continuing to use the old project template(2019) with a plcncli 2020.

---------------

3. Question:
- → Also, in your documentation, you are asking to be logged as root before installing dotnet.
Is this could lead to the same kind of problem after a firmware update?
3.Answer:
Installing dotnet manually might cause issues in the future IF dotnet gets included into the FW by default. (i am not aware of any plans regarding that thou…)
I do not think that it directly modifies any Firmware Files and is fairly standalone.
If the installation deploys some binaries (in different versions) then the FW uses it might obviously cause issues.
( I do not think thats the case with dotnet core thou.)

[fblanchard]

We just finished two long term testing (15 days) and it look like we found the recipes to avoid the problem: Every time we upgrade the PLCnext firmware on the field, from 2019.0 or previous versions, to 2020.0.1, we start by a factory reset. It is cumbersome to do it (must reinstall and reconfigure everything), but the resulting system is stable. I read in the release notes that a FW upgrade issue have been resolved in 2020.6. We will analyse this fix when we will move on to the future 2021.0 LTS version. I will keep you informed if anything new comes up. Frédéric

[fblanchard]

Moving to 2020.6.1 solved the problem.

[zmandy159]

I came across this thread looking for some insight. We are experiencing the exact same issue but are running firmware 22.9.0.33. Did you guys every diagnose which caused the issue?