This document discusses CoreOS, an open source operating system focused on automation, updates, and security for containers and microservices. It provides CoreOS features like CoreUpdate for seamless automatic updates, etcd for service discovery and configuration, and Fleet for managing applications across clusters using systemd. Fleet builds on systemd and etcd to schedule applications and replicate state across machines. The document also covers related tools like Flannel for networking containers across hosts and SkyDNS for service discovery.

1. CoreOS
@superstructor / joukou.com
Commanding the Fleet

2. Warning: Live demos

6. 1st Solution: Ubuntu Server
Internet is full of
servers running years-
old software with
dozens of
vulnerabilities
(Shellshock,
Heartbleed etc)

7. 2nd Solution: Ubuntu + Docker
● Isolation of services
● Rapid service
deployment
● Portable
● Version control
● Reuse
● Lightweight & fast

8. Docker = Standardized Deployment of Services

9. etcd

10. CoreOS Cluster Dev Environment
or
git clone https://github.com/coreos/coreos-vagrant.git

11. CoreUpdate
● Make upgrading the default, seamless option
o Regular
o Reliable
o Automatic
● The “Google Chrome” of server operating
systems

12. CoreUpdate: How does it work ?
● Omaha protcol (check-in/retrieval)
o Simple XML-over-HTTP protocol developed by
Google to facilitate polling and pulling updates from
a server
● Active/passive read-only rootfs partitions
o One for running live system, one for updates

13. CoreUpdate: Active/passive rootfs

14. CoreUpdate: Active/passive rootfs

15. CoreUpdate: Active/passive rootfs

16. CoreUpdate: Active/passive rootfs

17. CoreUpdate: Active/passive rootfs /usr
● /usr is a single image containing most of the
OS
o mounted read-only
o rootfs / is mounted read-write on top (persistent
data)
o parts of /etc dynamically generated at boot
o a lot of work moving default config from /etc to /usr

18. Atomic Updates
● Entire OS is a single read-only image
o e.g. “touch /usr/bin/foo”
● Easy to verify cryptographically
● No chance of inconsistencies due to partial
upgrades

20. Cloud Config
Automate customizing a CoreOS machine
through user-data.
● YAML
● configure etcd, fleet...
● e.g. EC2 user-data

21. Example Cloud Config

22. Etcd
● key-value store
● basis for
o shared configuration
o service discovery
o distributed locking
● REST API
● Raft algorithm:
raftconsensus.github.io

23. systemd is great!
● Automatically handles:
o Process daemonization
o Service inter-dependencies
o Resource isolation/containment (cgroups)
 e.g. MemoryLimit=512M
o Health-checking, restarting failed services
o Logging (journal)
 applications can just write to stdout, systemd
adds metadata
o Timers, socket activation...

24. systemd (and docker) are… not so great
● Problem: poor integration with Docker
o Docker is de-facto standard
o Docker and systemd do not always place nice
together
o Both Docker and systemd manage cgroups and
processes, so when the two are trying to manage
the same thing the results are mixed

25. Example: sending signals to a container
Given a simple container...
[Service]
ExecStart=/usr/bin/docker run busybox /bin/bash -c “while true; do echo Hello
World; sleep 1; done”
● Try to kill it with systemctl kill hello.service
● … Nothing happens
● Kill command sends SIGTERM, but bash in a Docker container has PID1,
which happily ignores the signal...

26. Example: sending signals to a container
● OK, SIGTERM didn’t work, so escalate to SIGKILL:
systemctl kill -s SIGKILL hello.service
● Now the systemd service is gone:
hello.service main process exited, code=killed, status=9/KILL
● But… the Docker container still exists ?
# docker ps
CONTAINER ID COMMAND STATUS NAMES
feb755848a9a /bin/sh -c ‘whi Up 31 seconds hello
# ps -ef|grep ‘[d]ocker run’
root 24231 1 0 03:49 ? 00:00:00 /usr/bin/docker run -name hello….

27. Why ?
● Docker client does not run containers itself; it
just sends a command to the Docker
daemon
● systemd expects processes to fork directly
● Since Docker daemon’s cgroup is entirely
seperate systemd cannot keep track of the
forked container

28. systemd-cgls

29. Solution: systemd-docker
● systemd-docker - small app that moves
cgroups of Docker containers back under
systemd’s cgroup
● Uses Docker for images, but systemd-
nspawn for runtime
● (proposed) Docker standalone mode: client
starts container directly rather than via
daemon

30. Fleet: systemd for the cluster

31. Fleet Overview

32. Fleet + systemd
● systemd takes care of things so fleet doesn’t
have to
● fleet configuration is just systemd unit files
● fleet extends systemd to the cluster-level,
and adds some features of its own (using [X-
Fleet])
o Template units (run n identical copies of a unit)

33. Fleet Scheduling
● Global (run a unit everywhere in the cluster)
● MachineMetadata (run only on certain
machines)
● Conflicts (prevent a unit from being
collocated)
● MachineOf (run only on a machine that hosts
a specific unit)

34. Fleet + etcd
● fleet needs a consistent view of the cluster to
make scheduling decisions: etcd provides
this view
o What units exist in the cluster ?
o What machines exist in the cluster ?
o What are their current states ?
● All unit files, unit state, machine state and
scheduling information is stored in etcd

35. Fleet Reconciler Model
In a loop, run periodically until stopped:
1. Retrieve current state (how the world is) and
desired state (how the world should be) from
etcd
2. Determine necessary actions to transform
current state -> desired state

36. Fleet Unit & fleetctl
Examples

37. Service Annoucement & Discovery
+ SkyDNS v2

38. Flannel
Intra-machine Docker overlay network backed
by etcd.
● one subnet per machine
● get rid of Docker port mapping
● containers talk to each other directly
● pluggable transport backends:
o encapsulate packets in UDP
o VXLAN implementation coming

40. Questions ?

41. ● Everything is open source under Apache 2.0 - join us!
○ https://github.com/joukou
○ https://github.com/coreos
● Any more questions, feel free to
○ email isaac.johnston@joukou.com
○ or tweet @superstructor