The document discusses different methods for managing services and daemons at system startup. It begins by explaining that traditional init systems launched processes in a specific order, but that event-driven systems are now more common. It then provides examples of init systems including launchd, upstart, and systemd. Launchd configurations in macOS are defined through XML property list files that can start daemons, sockets, periodic jobs, and monitor directories for changes.

1. COSCUP 2016
The Origin: Init
Tzung-Bi Shih
<penvirus@gmail.com>

2. COSCUP 2016
Introduction
• some particular jobs are required to execute during
system boot up
• more or less in a specific order
• service: bind on I/O ports for serving requests
• daemon: "never-dead" background process
➡ How does OS manage services/daemons to be
launched at appropriate time?
2

3. COSCUP 2016
Glance at Booting
• CPU executes the first instruction at a fixed address after
reset
• in the past, BIOS[1] / MBR
• initialize peripherals
• POST
• beep
• now, UEFI[2] / GPT
• attempt to find boot loader
3

4. COSCUP 2016
Boot Loader Example
Grub2 in Linux[3]
• grub2
• mount temporary root file system
• load vmlinuz
• with kernel parameters[4]
• load initrd
• execute /init in initrd
• initrd[5]
• initialize essential environment, e.g., sysfs, procfs, device drivers
• remount real root file system
• /sbin/init, /etc/init, /bin/init, /bin/sh
4
set root='hd0,msdos1'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1
--hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1
86b2cdf5-17d5-4d82-9f7b-933f1329209c
else
search --no-floppy --fs-uuid --set=root 86b2cdf5-17d5-4d82-9f7b-933f1329209c
fi
linux /vmlinuz-4.2.0-16-generic root=UUID=a06139f7-ad29-45fd-a759-882c7cdbe488 ro
initrd /initrd.img-4.2.0-16-generic
init has been responsible for run control (rc) for 20+ years
# blkid
/dev/sda1: UUID="86b2cdf5-17d5-4d82-9f7b-933f1329209c" TYPE="ext4" PARTUUID="2ef8dba4-01"
/dev/sda2: UUID="a06139f7-ad29-45fd-a759-882c7cdbe488" TYPE="ext4" PARTUUID="2ef8dba4-02"
# mount | grep sda
/dev/sda2 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/sda1 on /boot type ext4 (rw,relatime,data=ordered)

5. COSCUP 2016
History
Run Control in Init[6][7][8][9]
• Research UNIX
• /etc/rc
• 4.3BSD
• /etc/rc.local
• NetBSD 1.5, FreeBSD 5.0
• /etc/rc.conf, /etc/rc.d/
• SysVInit
• run level
• /etc/inittab
5

6. COSCUP 2016
RC Example
inittab[10][11]
6
# Note, BusyBox init doesn't support runlevels. The runlevels field is
# completely ignored by BusyBox init. If you want runlevels, use sysvinit.
#
# Format for each entry: <id>:<runlevels>:<action>:<process>
::sysinit:/etc/init.d/rcS
# Start an "askfirst" shell on the console (whatever that may be)
::askfirst:-/bin/sh
# Start an "askfirst" shell on /dev/tty2-4
tty2::askfirst:-/bin/sh
tty3::askfirst:-/bin/sh
# /sbin/getty invocations for selected ttys
tty4::respawn:/sbin/getty 38400 tty5
tty5::respawn:/sbin/getty 38400 tty6
# Example of how to put a getty on a serial line (for a terminal)
#::respawn:/sbin/getty -L ttyS0 9600 vt100
#::respawn:/sbin/getty -L ttyS1 9600 vt100
# Example how to put a getty on a modem line.
#::respawn:/sbin/getty 57600 ttyS2
# Stuff to do before rebooting
::ctrlaltdel:/sbin/reboot
::shutdown:/bin/umount -a -r
::shutdown:/sbin/swapoff -a
id:3:initdefault:
si::sysinit:/etc/rc.d/rc.sysinit
l0:0:wait:/etc/rc.d/rc 0
l1:1:wait:/etc/rc.d/rc 1
l2:2:wait:/etc/rc.d/rc 2
l3:3:wait:/etc/rc.d/rc 3
l4:4:wait:/etc/rc.d/rc 4
l5:5:wait:/etc/rc.d/rc 5
l6:6:wait:/etc/rc.d/rc 6
ca::ctrlaltdel:/sbin/shutdown -t3 -r now
1:2345:respawn:/sbin/mingetty tty1
2:2345:respawn:/sbin/mingetty tty2
3:2345:respawn:/sbin/mingetty tty3
4:2345:respawn:/sbin/mingetty tty4
5:2345:respawn:/sbin/mingetty tty5
6:2345:respawn:/sbin/mingetty tty6
x:5:once:/etc/X11/prefdm -nodaemon
busybox SysV

7. COSCUP 2016
RC Example
BSD rc.d
7
# cat /etc/rc.d/cron
#!/bin/sh
#
# $FreeBSD: releng/10.2/etc/rc.d/cron 240336 2012-09-11 05:04:59Z obrien $
#
# PROVIDE: cron
# REQUIRE: LOGIN FILESYSTEMS
# BEFORE: securelevel
# KEYWORD: shutdown
. /etc/rc.subr
name="cron"
rcvar="cron_enable"
command="/usr/sbin/${name}"
pidfile="/var/run/${name}.pid"
load_rc_config $name
if checkyesno cron_dst
then
cron_flags="$cron_flags -s"
fi
run_rc_command "$1"
should be defined in /etc/rc.conf or /etc/defaults/rc.conf
specify dependencies
Note: service will not be re-spawned if got crashed

8. COSCUP 2016
# ls -l /etc/rc1.d/
lrwxrwxrwx 1 root root 14 Mar 23 2015 K20atop -> ../init.d/atop*
lrwxrwxrwx 1 root root 15 Sep 5 2014 K20rsync -> ../init.d/rsync*
lrwxrwxrwx 1 root root 24 Sep 5 2014 K20screen-cleanup -> ../init.d/screen-cleanup*
lrwxrwxrwx 1 root root 27 May 30 2015 K80nfs-kernel-server -> ../init.d/nfs-kernel-server*
-rw-r--r-- 1 root root 369 Mar 13 2014 README
lrwxrwxrwx 1 root root 19 Sep 5 2014 S30killprocs -> ../init.d/killprocs*
lrwxrwxrwx 1 root root 19 Sep 5 2014 S70dns-clean -> ../init.d/dns-clean*
lrwxrwxrwx 1 root root 18 Sep 5 2014 S70pppd-dns -> ../init.d/pppd-dns*
lrwxrwxrwx 1 root root 16 Sep 5 2014 S90single -> ../init.d/single*
RC Example
Linux init.d
8
# ls -ld /etc/rc*.d
drwxr-xr-x 2 root root 4096 May 30 2015 /etc/rc0.d/
drwxr-xr-x 2 root root 4096 May 30 2015 /etc/rc1.d/
drwxr-xr-x 2 root root 4096 May 30 2015 /etc/rc2.d/
drwxr-xr-x 2 root root 4096 May 30 2015 /etc/rc3.d/
drwxr-xr-x 2 root root 4096 May 30 2015 /etc/rc4.d/
drwxr-xr-x 2 root root 4096 May 30 2015 /etc/rc5.d/
drwxr-xr-x 2 root root 4096 May 30 2015 /etc/rc6.d/
drwxr-xr-x 2 root root 4096 Sep 5 2014 /etc/rcS.d/
hybrid of SysV and BSD style
helper utilities (e.g. chkconfig, service, update-rc.d) manipulate the symbolic links directly
Note: service will not be re-spawned if got crashed

9. COSCUP 2016
We've Got a Problem
Traditional Init is Insufficient[12][13][14]
• boot process is time-consumed
• some jobs can be launched simultaneously
• event-driven is now a common discipline
• especially in mobile devices and laptops
• e.g., PnP of USB drive
• too many similar but not comprehensive subsystems
• e.g., inetd, at, cron
9

10. COSCUP 2016
Modern Init
• Mac OS X
• launchd[15][16][17]
• Ubuntu
• upstart[18]
• start from 15.04, Ubuntu defaults to systemd
• other mainstream Linux distributions
• systemd[19][20][21][22]
10

11. COSCUP 2016
launchd (1/9)
Introduction
# launchctl load /Library/LaunchDaemons/com.example.xxx.plist
# launchctl start /Library/LaunchDaemons/com.example.xxx.plist
• pid 1 of OS X 10.4 and later
• XML configuration file format
• daemon: runs on behalf of root
• agent: runs on behalf of the logged in user
11

12. COSCUP 2016
launchd (2/9)
Traditional Daemon
12
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.example.hello</string>
<key>ProgramArguments</key>
<array>
<string>hello</string>
<string>world</string>
</array>
<key>KeepAlive</key>
<true/>
</dict>
</plist>
unconditionally running at all time, i.e., will
be re-spawned if the process got crashed
argv

13. COSCUP 2016
<key>SockServiceName</key>
<string>23</string>
launchd (3/9)
On-demand Network Socket
13
<key>Sockets</key>
<dict>
<key>Listeners</key>
<dict>
<key>SockServiceName</key>
<string>bootps</string>
<key>SockType</key>
<string>dgram</string>
<key>SockFamily</key>
<string>IPv4</string>
</dict>
</dict>
port number
from /etc/services

14. COSCUP 2016
launchd (4/9)
Debugging
14
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
[snip]
<key>StandardOutPath</key>
<string>/var/log/myjob.log</string>
<key>StandardErrorPath</key>
<string>/var/log/myjob.log</string>
<key>Debug</key>
<true/>
<key>SoftResourceLimits</key>
<dict>
<key>Core</key>
<integer>9223372036854775807</integer>
</dict>
<key>HardResourceLimits</key>
<dict>
<key>Core</key>
<integer>9223372036854775807</integer>
</dict>
</dict>
</plist>
set rlimits for core dumps
redirect stdout and stderr

15. COSCUP 2016
launchd (5/9)
Periodic Job
15
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.example.touchsomefile</string>
<key>ProgramArguments</key>
<array>
<string>touch</string>
<string>/tmp/helloworld</string>
</array>
<key>StartInterval</key>
<integer>300</integer>
</dict>
</plist>
in seconds

16. COSCUP 2016
launchd (6/9)
Cron Job
16
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.example.touchsomefile</string>
<key>ProgramArguments</key>
<array>
<string>touch</string>
<string>/tmp/helloworld</string>
</array>
<key>StartCalendarInterval</key>
<dict>
<key>Minute</key>
<integer>45</integer>
<key>Hour</key>
<integer>13</integer>
<key>Day</key>
<integer>7</integer>
</dict>
</dict>
</plist>
wildcards if unspecified

17. COSCUP 2016
launchd (7/9)
Monitor Directory for Changing
17
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.example.watchhostconfig</string>
<key>ProgramArguments</key>
<array>
<string>syslog</string>
<string>-s</string>
<string>-l</string>
<string>notice</string>
<string>somebody touched /etc/hostconfig</string>
</array>
<key>WatchPaths</key>
<array>
<string>/etc/hostconfig</string>
</array>
</dict>
</plist>
target directories

18. COSCUP 2016
launchd (8/9)
Monitor Directory for Emptiness
18
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.example.mailpush</string>
<key>ProgramArguments</key>
<array>
<string>my_custom_mail_push_tool</string>
</array>
<key>QueueDirectories</key>
<array>
<string>/var/spool/mymailqdir</string>
</array>
</dict>
</plist>
• start the job whenever the directories are non-empty
• keep running as long as the directories are not empty

19. COSCUP 2016
launchd (9/9)
Inetd
19
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.example.telnetd</string>
<key>ProgramArguments</key>
<array>
<string>/usr/libexec/telnetd</string>
</array>
<key>inetdCompatibility</key>
<dict>
<key>Wait</key>
<false/>
</dict>
<key>Sockets</key>
<dict>
<key>Listeners</key>
<dict>
<key>SockServiceName</key>
<string>telnet</string>
<key>SockType</key>
<string>stream</string>
</dict>
</dict>
</dict>
</plist>
This flag corresponds to the "wait" or "nowait"
option of inetd.
• If true, then the listening socket is passed
via the standard in/out/error file descriptors.
• If false, then accept(2) is called on behalf of
the job, and the result is passed via the
standard in/out/error descriptors.

20. COSCUP 2016
upstart (1/2)
Introduction
• pid 1 of Ubuntu 6.10~14.10
• non-standard configuration file format
• intended to be default init for all Linux distributions
• has frozen release and replaced by systemd[22][23]
20
upstart has obsoleted; no need to pay too much attention on it

21. COSCUP 2016
upstart (2/2)
Example: cron
21
# cat /etc/init/cron.conf
# cron - regular background program processing daemon
#
# cron is a standard UNIX program that runs user-specified programs at
# periodic scheduled times
description "regular background program processing daemon"
start on runlevel [2345]
stop on runlevel [!2345]
expect fork
respawn
exec cron
# start cron
# stop cron
# status cron
# initctl status cron
# initctl emit xxx
the process will call fork exactly once
dependency events
respawn if the process has crashed
target executable
symbolic links to initctl

22. COSCUP 2016
systemd (1/6)
Introduction
• pid 1 of most Linux distributions
• young but huge project
• ini configuration file format
• everything is an "unit"
• e.g., target, service, timer, socket
• run level has replaced by target unit
22
# systemctl start xxx
# systemctl stop xxx
# systemctl enable xxx
# systemctl disable xxx
# systemctl status xxx
# journalctl

23. COSCUP 2016
systemd (2/6)
Service Unit
23
# cat /lib/systemd/system/sshd.service
[Unit]
Description=OpenBSD Secure Shell server
After=network.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run
[Service]
EnvironmentFile=-/etc/default/ssh
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify
[Install]
WantedBy=multi-user.target
Alias=sshd.service
specify belonging target
associated executable
the service will be restarted when the
process exits with a non-zero exit code

24. COSCUP 2016
systemd (3/6)
Socket Unit
24
# cat /lib/systemd/system/sshd.socket
[Unit]
Description=OpenSSH Server Socket
Documentation=man:sshd(8) man:sshd_config(5)
Conflicts=sshd.service
[Socket]
ListenStream=22
Accept=yes
[Install]
WantedBy=sockets.target
If a unit has a Conflicts= setting on
another unit, starting the former will stop
the latter and vice versa.
something like placeholder
# systemctl list-unit-files | grep ssh
ssh.service enabled
ssh@.service static
sshd.service enabled
ssh.socket disabled

25. COSCUP 2016
systemd (4/6)
Target Unit
25
# cat /lib/systemd/system/multi-user.target
[Unit]
Description=Multi-User System
Documentation=man:systemd.special(7)
Requires=basic.target
Conflicts=rescue.service rescue.target
After=basic.target rescue.service rescue.target
AllowIsolate=yes
# systemctl isolate multi-user.target
# systemctl isolate graphical.target
used for grouping units and as well-known synchronization points during start-up

26. COSCUP 2016
# cat /lib/systemd/system/systemd-tmpfiles-clean.service
[Unit]
Description=Cleanup of Temporary Directories
Documentation=man:tmpfiles.d(5) man:systemd-tmpfiles(8)
DefaultDependencies=no
Conflicts=shutdown.target
After=local-fs.target time-sync.target
Before=shutdown.target
[Service]
Type=oneshot
ExecStart=/bin/systemd-tmpfiles --clean
IOSchedulingClass=idle
systemd (5/6)
Timer Unit
26
# cat /lib/systemd/system/systemd-tmpfiles-clean.timer
[Unit]
Description=Daily Cleanup of Temporary Directories
Documentation=man:tmpfiles.d(5) man:systemd-tmpfiles(8)
[Timer]
OnBootSec=15min
OnUnitActiveSec=1d

27. COSCUP 2016
# cat /lib/systemd/system/systemd-networkd-resolvconf-update.service
[Unit]
Description=Update resolvconf for networkd DNS
ConditionPathIsSymbolicLink=/etc/resolv.conf
ConditionPathExists=/run/resolvconf/enable-updates
After=resolvconf.service
[Service]
Type=oneshot
StartLimitBurst=20
# we might be triggered several times in short succession during restarting networkd, so
wait until we get a DNS entry
ExecStart=/bin/sh -c 'for timeout in `seq 30`; do out=$(sed -n "/^DNS=/ { s/^DNS=/
nameserver /; p}" /run/systemd/netif/state); [ -z "$out" ] || break; sleep 1; done; echo
"$out" | /sbin/resolvconf -a networkd'
systemd (6/6)
Path Unit
27
# cat /lib/systemd/system/systemd-networkd-resolvconf-update.path
[Unit]
Description=Trigger resolvconf update for networkd DNS
ConditionPathIsSymbolicLink=/etc/resolv.conf
[Path]
PathChanged=/run/systemd/netif/state

28. COSCUP 2016
Conclusion
• launchd is excellent
• systemd covers a lot of business
• much more than what we mentioned before
• systemd won the war
• although it is controversial[25]
• start to learn and use systemd
• whether you like it or not
• If systemd is not qualified, it shall be replaced again soon. But you need
to use it before judging systemd.
28

29. COSCUP 2016
References
29
[1]: https://en.wikipedia.org/wiki/BIOS
[2]: https://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface
[3]: https://en.wikipedia.org/wiki/Linux_startup_process
[4]: https://www.kernel.org/doc/Documentation/kernel-parameters.txt
[5]: https://www.kernel.org/doc/Documentation/initrd.txt
[6]: http://aplawrence.com/Basics/unix-startup-scripts-1.html
[7]: http://aplawrence.com/Basics/unix-startup-scripts-2.html
[8]: http://linuxmafia.com/faq/Admin/init.html
[9]: https://en.wikipedia.org/wiki/Init
[10]: http://linuxembedded.blogspot.tw/2006/11/understanding-busybox-inittab.html
[11]: http://linux.vbird.org/linux_basic/0510osloader/0510osloader-fc4.php#startup_init
[12]: http://aplawrence.com/Basics/unix-startup-scripts-3.htm
[13]: http://www.tecmint.com/systemd-replaces-init-in-linux/
[14]: http://www.pcworld.com/article/2841873/meet-systemd-the-controversial-project-taking-over-a-linux-distro-near-you.html
[15]: http://launchd.info/
[16]: http://paul.annesley.cc/2012/09/mac-os-x-launchd-is-cool/
[17]: https://developer.apple.com/library/mac/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html
[18]: http://upstart.ubuntu.com/
[19]: https://www.certdepot.net/rhel7-get-started-systemd/
[20]: http://www.freedesktop.org/wiki/Software/systemd/
[21]: https://fedoraproject.org/wiki/Systemd
[22]: https://wiki.archlinux.org/index.php/Systemd
[23]: http://www.zdnet.com/article/debian-init-decision-further-isolates-ubuntu/
[24]: http://www.zdnet.com/article/after-linux-civil-war-ubuntu-to-adopt-systemd/
[25]: http://without-systemd.org/wiki/index.php/Arguments_against_systemd