Runtime Down with no obvious reason in logs

PLCnext Technology & PLCnext Controls
1

Hi,
We have been having another separate issue wherein the plcnext runtime on an AXC F 3152 (2023 LTS firmware) seems to go down with no obvious problems in the logs (logs are set to DEBUG, and there are no seg-faults or watchdog errors reported). Has happened 2-3 times in the last day or so. Some other relevant information :

                              * We are running C++ programs.
                              * Following this event, we confirmed that the runtime is not running with htop. 
                              * Nothing obviously wrong/bad in dmesg or syslog. 
                              *

From here, I am wondering what next steps to debug are :
We will be attaching gdb going forward a bit more religiously.
* Is there anything we can do to bump up dmesg/syslog to see what signal killed the plcnext process ?
* Any obvious reasons for why this might be happening ? Only thing that springs to mind/google is linux OOM killer, but we should still have plenty of memory.
* Any other ideas etc… ? (some other application to log the state of the plcnext runtime).

Cheers,
Lindsay

2

It would be very unusual for the PLCnext Runtime to just be killed silently during normal operation. I can’t remember seeing that situation before.
There are sample scripts published here:
GitHub - PLCnext/UtilityScripts: Collection of usefull scripts to debug and modify PLCnext Controll settings. Collection of usefull scripts to debug and modify PLCnext Controll settings. - GitHub - PLCnext/UtilityScripts: Collection of usefull scripts to debug and modify PLCnext Controll settings. … and some of those should help with diagnostics. In particular, the one that logs memory usage could be relevant. You will probably need to modify the script(s) for your situation.

3

Hi Martin,
Agreed it is unusual (have never had it happen in other projects). Thanks again for the prompt response and the link to the utils (should come in handy). With GDB attached we found that the problem was a SEG FAULT with an external C++ library we were using to communicate with some robotic manipulators. We wrapped the library in an Arp::Thread, so likely that this could help explain why there is no stack trace in the logs. Will let you know if we find anything else.
Cheers,
Lindsay