Embedded systems are autonomous. This simple fact is a driving force in the design of embedded systems which cannot afford the luxury of an operator to press a reset button or even a remote sysadmin to check what happened. Monitoring an application in an embedded system is a complex problem that must deal with the various ways an application can fail, detect them and restart the application if need be. Systemd provides a comprehensive toolbox for the embedded developer to diagnose, monitor and restart the main application of an embedded system. Especially if the embedded application is a black-box software. This talk will review the tools provided by systemd for process monitoring and discuss how to easily deploy them in an embedded system. Jérémy Rosen – Smile-Embedded and connected systems

1. Process monitoring with systemd
And why use it on an embedded system
Jérémy Rosen
September 25, 2017

2. systemd on an embedded system
systemd is a very interesting tool for embedded systems
• Disk cost: around 6Mb disk (size of complete yocto package)
• Integrated in Yocto and Buildroot
• Makes it easy to secure processes (capabilities)
• Makes it easy to contol ressources (cgroups)
• Makes it easy to monitor processes
systemd requires some work to understand, but once you get it, it greatly reduces the
work needed to have a reliable, autonomous system.

3. Focus on: Monitoring of embedded application
Fine grained control on how to kill a unit
• Command to stop
• Signal to send if service doesn’t stop (and to whom)
• Command to execute after the service stops (cleanup)
• Service to start on failure
• Fine grained definition of normal/abnormal stops (signals, exit code)
• Fine grained decision on restart (rate-limit, delay before restart, only on some exit
statuses)
• Reboot/Poweroff on service failure
• Core-dump management (including /proc/pid, tracebacks, disk space
management)
• Soft watchdog monitoring by PID1
• PID1 uses the harware watchodog for self-monitoring

4. Focus on: Monitoring of embedded application
Fine grained control of service interaction
• Services that must be started/stoped together
• Services that must be started after the initialisation phase of other services
• React to hardware changes (dbus events)
• React to filesystem changes (spool directories, data on socket, inotify)