This document provides a history of Unix and Linux service management from the early days of /etc/init through the development of systemd. It describes the issues with early init systems like limitations in flexibility, lack of monitoring, and inconsistencies. It then discusses how various operating systems attempted to address these problems through tools like SMF, launchd, upstart, and others. Finally, it provides an overview of how systemd comprehensively solves the issues through features like unit files, control groups, journald logging, and integration with the Linux kernel.

1. From /etc/init to systemd
A journey through the years of
UNIX
and
Linux
service management
Lubomir Rintel <lkundrak@v3.sk>
BTC: 15wvWxN5QMpreKR37pYb7VBu8xLu4TiNR2

2. Sixth Edition UNIX (1976)
●
/etc/init
(219 lines)
●
●
gettys from /etc/ttys
●
●
Single user shell
utmp & wtmp
/etc/rc
(2 lines)
rm -f /etc/mtab
/etc/update

3. UNIX System III (1980)
●
/etc/init
●
(486 lines)
/etc/inittab "states" controlling gettys on
terminals
●
/etc/rc
(67 lines)
●
State aware, single user mode
●
acct
●
errdemon
●
cron
●
"edit to add umounts"

4. 2.9 BSD UNIX (1983)
●
/sbin/init
(593 lines)
●
●
●
Single user shell
gettys from /etc/ttys
/etc/rc
(53 lines)
●
fsck, quotacheck
●
mount -a, from /etc/fstab
●
savecore, ex/vi recovery
●
clear /tmp, locks in /usr/spool
●
update, cron, acct
●
hostname

5. UNIX System V Release 4 (1983)
●
/etc/init
●
●
/etc/inittab with runlevels, respawns
/etc/rc?
●
Modular init system, /etc/rc.d
●
Per-daemon init scripts (8-56 lines, avg. 26)
–
–
●
●
start & stop arguments
pid from ps, stop = kill -TERM
Enablement/disablement by linking into level
dir
Ordered by numbers

6. #ident
"@(#)/etc/init.d/cron.sl 1.1 4.0 10/15/90 8479 AT&T-SF"
#
cron control
pid=`/usr/bin/ps -e | /usr/bin/grep cron |
/usr/bin/sed -e 's/^ *//' -e 's/ .*//'`
case $1 in
'start')
if [ "${pid}" = "" ]
then
/usr/bin/rm -f /etc/cron.d/FIFO
if [ -x /usr/bin/cron ]
then
/usr/bin/cron
elif [ -x /usr/sbin/cron ]
then
/usr/sbin/cron
fi
fi
;;
'stop')
if [ "${pid}" != "" ]
then
/usr/bin/kill ${pid}
fi
;;
*)
echo "usage: /etc/init.d/cron {start|stop}"
;;
esac

7. 2.11 BSD UNIX (1986)
●
/sbin/init
(782 lines)
●
●
●
Single user
gettys from /etc/gettytab
/etc/rc
(165 lines)
●
●
●
Everything 2.9BSD had
Network, inetd, routed, named, lpd, rwhod
/etc/rc.local
●
Editable for starting local daemons

8. Red Hat Enterprise Linux 5 (2007)
●
SVR4-like init
●
●
Some BSD elements: /etc/rc, /etc/rc.local
Init scripts LSB compliant
●
61-584 lines, avg. 128
●
start, stop, status, restart, condrestart
●
Pidfiles in /var/run
●
Subsystem locks in /var/lock/subsys
●
/etc/rc.d/functions library
●
/etc/sysconfig init script configuration

9. #!/bin/bash
#
# sendmail
This shell script takes care of starting and stopping
#
sendmail.
#
# chkconfig: 2345 80 30
# description: Sendmail is a Mail Transport Agent, which is the program
#
that moves mail from one machine to another.
# processname: sendmail
# config: /etc/mail/sendmail.cf
# pidfile: /var/run/sendmail.pid
if [ -x /usr/bin/make -a -f /etc/mail/Makefile ]; then
make all -C /etc/mail -s > /dev/null
else
for i in virtusertable access domaintable mailertable ; do
if [ -f /etc/mail/$i ] ; then
makemap hash /etc/mail/$i < /etc/mail/$i
fi
done
fi
daemon /usr/sbin/sendmail $([ "x$DAEMON" = xyes ] && echo -bd)
$([ -n "$QUEUE" ] && echo -q$QUEUE)
RETVAL=$?
killproc sendmail -HUP
RETVAL=$?
echo
if [ $RETVAL -eq 0 -a -f /var/run/sm-client.pid ]; then
echo -n $"reloading sm-client: "
killproc sm-client -HUP
RETVAL=$?
echo
fi
return $RETVAL
# Source function library.
. /etc/rc.d/init.d/functions
# Source networking configuration.
[ -f /etc/sysconfig/network ] && . /etc/sysconfig/network
# Source sendmail configureation.
if [ -f /etc/sysconfig/sendmail ] ; then
. /etc/sysconfig/sendmail
else
DAEMON=no
QUEUE=1h
fi
[ -z "$SMQUEUE" ] && SMQUEUE="$QUEUE"
[ -z "$SMQUEUE" ] && SMQUEUE=1h
}
stop() {
# Stop daemons.
if test -f /var/run/sm-client.pid ; then
echo -n $"Shutting down sm-client: "
killproc sm-client
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && rm -f /var/run/sm-client.pid
[ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/sm-client
fi
echo -n $"Shutting down $prog: "
killproc sendmail
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/sendmail
return $RETVAL
# Check that networking is up.
[ "${NETWORKING}" = "no" ] && exit 0
[ -f /usr/sbin/sendmail ] || exit 0
RETVAL=0
prog="sendmail"
start() {
# Start daemons.
echo -n $"Starting $prog: "
if test -x /usr/bin/make -a -f /etc/mail/Makefile ; then
make all -C /etc/mail -s > /dev/null
else
for i in virtusertable access domaintable mailertable ; do
if [ -f /etc/mail/$i ] ; then
makemap hash /etc/mail/$i < /etc/mail/$i
fi
done
fi
/usr/bin/newaliases > /dev/null 2>&1
daemon /usr/sbin/sendmail $([ "x$DAEMON" = xyes ] && echo -bd)
$([ -n "$QUEUE" ] && echo -q$QUEUE) $SENDMAIL_OPTARG
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && touch /var/lock/subsys/sendmail
if ! test -f /var/run/sm-client.pid ; then
echo -n $"Starting sm-client: "
touch /var/run/sm-client.pid
chown smmsp:smmsp /var/run/sm-client.pid
if [ -x /usr/sbin/selinuxenabled ] && /usr/sbin/selinuxenabled; then
/sbin/restorecon /var/run/sm-client.pid
fi
daemon --check sm-client /usr/sbin/sendmail -L sm-msp-queue -Ac
-q$SMQUEUE $SENDMAIL_OPTARG
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && touch /var/lock/subsys/sm-client
fi
return $RETVAL
reload() {
# Stop daemons.}
echo -n $"reloading $prog: "
/usr/bin/newaliases > /dev/null 2>&1
}
# See how we were called.
case "$1" in
start)
start
;;
stop)
stop
;;
reload)
reload
RETVAL=$?
;;
restart)
stop
start
RETVAL=$?
;;
condrestart)
if [ -f /var/lock/subsys/sendmail ]; then
stop
start
RETVAL=$?
fi
;;
status)
status sendmail
RETVAL=$?
;;
*)
echo $"Usage: $0 {start|stop|restart|condrestart|status}"
exit 1
esac
exit $RETVAL

10. ··
ddish):
hket (Yi
far∙pot∙s
pronoun
a result
ruined as
letely
erfection
1. comp
inor imp
o f ix a m
ttempts t
of a

11. Issues
●
inittab
●
service control inflexible
Not actually used very much
●
No way to monitor services
●
Error output is lost
●
Inflexible startup ordering
●
Single-threaded startup
●
●
●
Slow
Malfunctional startup script can lock up boot
Limited to single instance of a service

12. Issues (cont.)
●
No way to track processes belonging to a service
●
●
No way to reliably terminate a service
●
No way to tell whether the service is operational
●
●
They double-fork to daemonize
Resource hogging
Inconsistent
●
Who drops privileges?
●
Who writes PID files?
●
Who chroots?

13. SVR4 & LSB init scripts
●
Unbelievably shitty
●
Unreliably work around issues mentioned
●
Very long and ugly
●
Lots of code duplication
●
●
Races, improper
subsystem locks
Insecure PID
determination

14. How did UNIX address this
●
Sun Solaris 10: SMF
●
●
Multiple service instances
Apple Darwin: launchd
●
●
On demand startup of network daemons (inetd)
●
●
System and User sessions
Job scheduler
Common:
●
●
Dependency management, service ordering
●
●
Parallel startup
Monitoring/restarts of services
Various others: upstart, daemontools, Monit, ...

15. Linux: systemd
●
Heavily inspired by concepts from launchd
●
Already used in most modern Linux distributions
●
Solves all of the mentioned issues!
●
Leverages Linux inovations -- a lot more powerful
●
Control groups, Namespaces
●
Seccomp, Capabilities, SELinux
●
Auditing
●
Automounter
●
DBus API
●
Structured kernel messaging

16. systemd Unit types
Unit type
Description/equivalent
service
A daemon (SVR4 init script)
socket
A network or UNIX socket (inetd)
device
UDev device instance
mount
Mount point (fstab)
target
Runlevel
swap
Swap space (fstab)
automount
Autofs
path
Inotify watch
timer
Crond
snapshot
Dynamically created target

17. httpd.service
├─system.slice
└─basic.target
├─fedora-loadmodules.service
├─paths.target
├─sockets.target
│ ├─cups.socket
│ ├─rpcbind.socket
│ ├─systemd-initctl.socket
│ └─systemd-udevd-kernel.socket
├─sysinit.target
│ ├─kmod-static-nodes.service
│ ├─systemd-udev-trigger.service
│ ├─local-fs.target
│ │ ├─-.mount
│ │ ├─fedora-import-state.service
│ │ ├─home.mount
│ │ ├─systemd-fsck-root.service
│ │ └─tmp.mount
│ └─swap.target
│
└─dev-disk-byx2dlabel-yolo.swap
└─timers.target
└─systemd-tmpfiles-clean.timer

18. systemd Units
●
Defined from unit files
●
●
●
/lib/systemd
/etc/systemd
Generated automatically
●
●
●
Compatibility or dynamic changes
device unit appears as device appears in
udev
mount units generated from /etc/fstab

19. Service units
●
Service runs in its own control group
●
Isolated from the rest of system to some extent
●
A process can't escape
●
Freezer control group assures reliable shutdown
●
Service should not double-fork (launchd)
●
systemd-journald takes care of logging
●
Can depend on socket units for activation (inetd)
●
DBus activation also possible

20. User sessions
●
Manages processes for a user session (e.g. tty
or GNOME desktop)
●
Session runs in separate control group
●
systemd-logind replaces ConsoleKit
●
Multiseat
●
Reliable log-off

21. Essential tools
systemctl --all
systemctl stop sshd.service
systemctl status sshd
systemctl disable sshd
systemd-cgtop
systemd-cgls
journalctl -f
systemd-analyze blame
systemd-analyze critical-chain

22. What else
●
Takes care of system-wide events
●
●
Laptop lid close
●
●
Pinging watchdog
Shutdown, kexec
Documentation
●
●
Well written manual pages for everything
Lightweight virtualization (LXC)
●
clone()s all namespaces

23. Thanks for listening!
Found this useful? My Bitcoin address:
15wvWxN5QMpreKR37pYb7VBu8xLu4TiNR2