The document presents an overview of CoreOS, a container-optimized operating system designed for running applications as containers with a focus on security and automation. It details the components and architecture of CoreOS, including systemd for process management, etcd for configuration synchronization, and fleet for scheduling, alongside its update mechanisms and installation methods. Additionally, it highlights CoreOS's integration with container technologies like Docker and Kubernetes, and its release cycle structure.

1. COREOS OVERVIEW AND
CURRENT STATUS
Presenter Name: Sreenivas Makam
Presented at: Open source Meetup Bangalore
Presentation Date: April 16, 2016

2. About me
• Senior Engineering Manager at
Cisco Systems Data Center group
• Personal blog can be found at
https://sreeninet.wordpress.com/
and my hacky code at
https://github.com/smakam
• Author of “Mastering CoreOS”
book, published on Feb 2016.
(https://www.packtpub.com/netw
orking-and-servers/mastering-
coreos )
• You can reach me on LinkedIn at
https://in.linkedin.com/in/sreeniva
smakam, Twitter handle -
@srmakam

3. Container Optimized OS - Characteristics
• All applications run as Containers on top of OS.
• Base OS should be small and fast to bootup.
• Services are managed by Init system. eg: Systemd
• OS need to have auto-update strategy like web
browsers.
• Systems needs to be managed as a cluster rather than
individual node.
• Built-in support should be present for Service
discovery, Container networking and Orchestration.
• Examples – CoreOS, Rancher OS, Redhat Atomic,
Ubuntu snappy, Mesos DCOS, VMWare Photon

4. CoreOS
• First Container optimized OS. First release was done in July 2013.
• Linux based and based on concepts from ChromeOS.
• OS is Security focused.
• Auto Update OS with A/B partition.
• OS is Open sourced. Along with OS, CoreOS has following
components:
– Systemd as Init system
– Etcd as distributed database
– Fleet as Cluster service scheduler
– Flannel for Container networking
– Docker, Rkt for Containers
• Has commercial components like Tectonic, Quay.
• CoreOS integrates well with Kubernetes

5. CoreOS Architecture
Kernel
Systemd, cloud-init
Etcd, fleet, flannel, update-service
Docker, Rkt
Docker
Container
Rkt
Container
Kubernetes
Tectonic, Quay
CoreOS
Applications
Orchestration
Platform

6. CoreOS Automatic Update
• The CoreOS update mechanism is based on Google's open source Omaha protocol
(https://code.google.com/p/omaha/) that is used in the Chrome browser.
• Dual partition scheme is used to achieve automatic update and to handle upgrade
related failures.
• Critical CoreOS services to handle Upgrade are update-engine and locksmith service.
• Etcd is used to serialize upgrades in a CoreOS cluster.
• User can control upgrades using different schemes like etcd-lock, reboot, best effort.

7. Cluster Architecture - Development
• Single cluster to run critical services
and application Containers.
• Odd cluster size gives better fault
tolerance.
• Used for development purposes.
• etcd is run in a separate node and application
Containers run in Worker nodes.
• Etcd in Worker nodes proxy to main etcd node.
Cluster 1
Cluster 2
Picture from https://coreos.com/os/docs/latest/cluster-architectures.html

8. Cluster architecture - Production
Cluster 3
Fault Tolerance• This cluster architecture is suited for
production
• There is redundancy available for core
services as well as for application Containers.
• Services are scheduled based on node
metadata(eg: services, worker)
Picture from https://coreos.com/os/docs/latest/cluster-architectures.html

9. CoreOS Release cycle
• Alpha, Beta, and Stable are release channels within
CoreOS.
• CoreOS releases progress through each channel in this
order: Alpha->Beta->Stable.
• All releases get started as Alpha, but the promotion to
Beta and Stable happens on the basis of testing.
• The major version number (for example, 1000 in
1000.0.0) is the number of days from July 13, 2013,
which was the CoreOS epoch.
core@core-01 ~ $ cat /etc/os-release
NAME=CoreOS
ID=coreos
VERSION=1000.0.0
VERSION_ID=1000.0.0
BUILD_ID=2016-03-27-0652
PRETTY_NAME="CoreOS 1000.0.0 (MoreOS)"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"
core@core-01 ~ $ uname -a
Linux core-01 4.5.0-coreos-r1 #2 SMP Thu Apr 7 03:52:11 UTC
2016 x86_64 Intel(R) Core(TM) i7-4800MQ CPU @ 2.70
GHz GenuineIntel GNU/Linux

10. Systemd
• Systemd is an Init system used by CoreOS to
start, stop, and manage processes.
• Dependencies between services can be specified
easily inside the unit files.
• All processes inside a single Systemd unit run in
1 cgroup and this includes forked processes.
• Systemd can monitor health of service and
restart service if it dies.
• Common Systemd unit types—service, socket,
device, and mount.

11. Systemd unit file example
Fleet.service:
[Unit]
Description=fleet daemon
After=etcd.service
After=etcd2.service
Wants=fleet.socket
After=fleet.socket
[Service]
User=fleet
Environment=GOMAXPROCS=1
ExecStart=/usr/bin/fleetd
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target
ordering
dependency
Service
restartability
Unit
grouping

12. Systemd unit execution
Hello1.service:
[Unit]
Description=My Service
After=docker.service
[Service]
TimeoutStartSec=0
KillMode=none
ExecStartPre=-/usr/bin/docker kill hello1
ExecStartPre=-/usr/bin/docker rm hello1
ExecStartPre=/usr/bin/docker pull busybox
ExecStart=/usr/bin/docker run --name hello1
busybox /bin/sh -c "while true; do echo Hello
World; sleep 1;
done"
Restart=always
RestartSec=30s
ExecStop=/usr/bin/docker stop hello1
[Install]
WantedBy=multi-user.target
Execute service:
1. Copy hello1.service as sudo to /etc/systemd/system.
2. Enable the service:
sudo systemctl enable hello1.service
3. Start hello1.service:
sudo systemctl start hello1.service
Hello1.service Status:
core@core-01 /etc/systemd/system $ systemctl status
hello1.service
? hello1.service - My Service
Loaded: loaded (/etc/systemd/system/hello1.service; disabled;
vendor preset: disabled)
Active: active (running) since Mon 2016-04-04 15:38:29 UTC;
2min 38s ago
Process: 1069 ExecStartPre=/usr/bin/docker pull busybox
(code=exited, status=0/SUCCESS)
Process: 1060 ExecStartPre=/usr/bin/docker rm hello1
(code=exited, status=1/FAILURE)
Process: 961 ExecStartPre=/usr/bin/docker kill hello1
(code=exited, status=1/FAILURE)
Main PID: 1093 (docker)
Memory: 33.1M
CPU: 634ms
CGroup: /system.slice/hello1.service
└─1093 /usr/bin/docker run --name hello1 busybox /bin/sh
-c while true; do echo Hello World;...
Apr 04 15:40:59 core-01 docker[1093]: Hello World

13. etcd
• Distributed key, value store used by CoreOS nodes.
• Uses Raft consensus algorithm to maintain highly
available cluster.
• Etcd is used outside CoreOS by companies like Cloud
foundry.
• Fleet, Flannel, Kubernetes uses etcd to communicate
configuration and monitoring data.
• Etcd can be used for Service discovery.
• Etcd exposes REST api and also provides etcdctl for CLI
access.

14. Etcd examples
Member list:
core@core-01 ~ $ etcdctl member list
156d6ef020d72caa: name=a4562fd1d8d346b79e80ca467d82b7ee peerURLs=http://172.17.8.101:2380 clientURLs=http://172.17.8.101:2379
2ba42fe58ee8dc90: name=d29b150786c04239937794f87985cd76 peerURLs=http://172.17.8.102:2380 clientURLs=http://172.17.8.102:2379
72ab4a0a6ba49052: name=7a895214d7274c0f8c362188c3bf8eb8 peerURLs=http://172.17.8.103:2380 clientURLs=http://172.17.8.103:2379
Set, get, delete example:
core@core-01 ~ $ etcdctl set /message hello
hello
core@core-02 ~ $ etcdctl get /message
Hello
core@core-01 ~ $ etcdctl rm /message
PrevNode.Value: hello
Get all keys:
core@core-01 ~ $ etcdctl ls / --recursive
/coreos.com
/coreos.com/updateengine
/coreos.com/updateengine/rebootlock
/coreos.com/updateengine/rebootlock/semaphore
/message
TTL expiry:
core@core-01 ~ $ etcdctl set /expirymessage "I will expire soon"
--ttl 30
I will expire soon
core@core-02 ~ $ etcdctl get /expirymessage
I will expire soon
core@core-02 ~ $ etcdctl get /expirymessage
Error: 100: Key not found (/expirymessage) [51952]
Watch:
Node 1:
core@core-01 ~ $ etcdctl set /didichange "not changed"
not changed
core@core-01 ~ $ etcdctl get /didichange
not changed
core@core-01 ~ $ etcdctl watch /didichange –recursive
----
Node 2:
core@core-02 ~ $ etcdctl set /didichange "changed"
Changed
----
Node 1:
[set] /didichange
changed

15. Fleet
• Init system for the CoreOS cluster
• Fleet uses etcd for communication across the cluster
• Fleet was originally planned as Orchestrator. After
Kubernetes took the role of Orchestrator, Fleet is used
only for scheduling system services. For example,
Kubernetes services are scheduled using Fleet.
• Fleet is not under active development and is mostly
under the maintenance mode.
• Fleet service exposes REST api and CLI can be
accessed using fleetctl.

16. Fleet Architecture
• One master Fleet engine is selected
in the CoreOS cluster using etcd.
• Fleet master talks to Fleet agents of
each node and schedules the Jobs.
• Fleet also takes care of Service high
availability at cluster level.
• Fleet global units gets scheduled on
all nodes in the cluster.
• Fleet can schedule units based on
metadata onto the appropriate
node in the cluster.

17. Fleet global unit example
helloglobals.service:
[Unit]
Description=My Service
After=docker.service
[Service]
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill hello
ExecStartPre=-/usr/bin/docker rm hello
ExecStartPre=/usr/bin/docker pull busybox
ExecStart=/usr/bin/docker run --name hello
busybox /bin/sh -c "while true; do echo Hello
World; sleep 1; d
one"
ExecStop=/usr/bin/docker stop hello
[X-Fleet]
Global=true
Start Unit:
fleetctl start helloglobal.service
Fleet Status:
core@core-01 ~ $ fleetctl list-units
UNIT MACHINE ACTIVE SUB
helloglobal.service 7a895214.../172.17.8.103 active running
helloglobal.service a4562fd1.../172.17.8.101 active running
helloglobal.service d29b1507.../172.17.8.102 active running
Systemd status:
core@core-01 ~ $ systemctl status helloglobal.service
? helloglobal.service - My Service
Loaded: loaded (/run/fleet/units/helloglobal.service; linked-
runtime; vendor preset: disabled)
Active: active (running) since Mon 2016-04-04 17:28:17 UTC; 1min
44s ago
Process: 7702 ExecStartPre=/usr/bin/docker pull busybox
(code=exited, status=0/SUCCESS)
Process: 7693 ExecStartPre=/usr/bin/docker rm hello (code=exited,
status=1/FAILURE)
Process: 7686 ExecStartPre=/usr/bin/docker kill hello (code=exited,
status=1/FAILURE)
Main PID: 7716 (docker)
Memory: 9.4M
CPU: 444ms
CGroup: /system.slice/helloglobal.service
└─7716 /usr/bin/docker run --name hello busybox /bin/sh -c
while true; do echo Hello World; ...

18. Flannel
• Flannel uses an Overlay network to allow Containers
across different hosts to talk to each other.
• Both Docker and Rkt Containers in CoreOS uses
Flannel to do Container networking.
• Projects like Kubernetes use Flannel for Container
networking.
• Flannel can be implemented as a
Container networking
interface(CNI) plugin like shown in
the diagram.

19. Flannel Architecture
Flannel Control path
Flannel Data path
• Flannel agent in each node communicate to each other using etcd.
• Flannel agent of each node gets an individual subnet and Containers of that node
gets IP address from the subnet.
• Inter-host Container traffic gets encapsulated into either UDP or VXLAN by Flannel
bridge of each node.

20. Rkt
• Rkt is the Container runtime developed by CoreOS.
• CoreOS created Rkt to have a better security model for
Containers and to have an industry standard.
• Rkt uses the Application Container image (ACI) image
format, which is according to the APPC specification.
• Rkt does not have a daemon(Like Docker) and is managed
by Systemd.
• Rkt uses multiple stages to run Containers. This is done so
that some stages can be swapped with other
implementations.
– First stage does Container image discovery and retrieval.
– Second stage sets the Container runtime environment like
filesystem, cgroups.
– Third stage runs the Container.

21. Rkt Examples
Run Docker image as Rkt Container:
core@core-02 ~ $ sudo rkt run --insecure-options=image --interactive docker://busybox
image: using image from local store for image name coreos.com/rkt/stage1-coreos:1.2.1
image: remote fetching from URL "docker://busybox"
Downloading sha256:385e281300c: [==============================] 676 KB/676 KB
Downloading sha256:a3ed95caeb0: [==============================] 32 B/32 B
networking: loading networks from /etc/rkt/net.d
networking: loading network default with type ptp
/ #
List pods:
core@core-02 ~ $ rkt list pods
UUID APP IMAGE NAME STATE CREATED
STARTED NETWORK
S
6bafd510 busybox registry-1.docker.io/library/busybox:latest running 6
minutes ago 6 minutes ago default
:ip4=172.16.28.2
List images:
core@core-02 ~ $ rkt image list
ID NAME IMPORT TIME LAST USED
SIZE LATEST
sha512-c0ab44d87499 coreos.com/rkt/stage1-coreos:1.2.1 10
minutes ago 10 minutes ago 152MiB false
sha512-cdb74a334f97 registry-1.docker.io/library/busybox:latest 10
minutes ago 10 minutes ago 2.4MiB true
Rkt Container in Systemd:
[Unit]
Description=nginx
[Service]
# Resource limits
CPUShares=512
MemoryLimit=1G
# Prefetch the image
ExecStartPre=/usr/bin/rkt fetch --insecure-skip-
verify docker://nginx
ExecStart=/usr/bin/rkt run --insecure-skip-verify
--private-net
--port=80-tcp:8080 --volume volume-var-cache-
nginx,kind=host,source=/
home/co
re docker://nginx
KillMode=mixed
Restart=always

22. CoreOS Installation
• CoreOS can be installed in baremetal using ISO image or
PXE boot.
• Works with majority of Cloud providers – AWS, Google
cloud, DigitalOcean, Azure.
• Works with baremetal cloud like Packet.
• CoreOS can be deployed on Openstack cloud.
• Vagrant based installation can be used for development
purposes.
• Base configuration of CoreOS node can be provided using
cloud-config YAML file. Cloud-init program in CoreOS
system takes care of setting up configuration.
• For Cloud providers, System cloud-config is embedded in
base image and user cloud-config is provided by the user.

23. Cloud-config sample
#cloud-config
coreos:
etcd2:
#generate a new token for each unique cluster from https://discovery.etcd.io/new
discovery: https://discovery.etcd.io/99ce5d499df6a4a9c004e8596687ccc7
# multi-region and multi-cloud deployments need to use $public_ipv4
advertise-client-urls: http://$public_ipv4:2379
initial-advertise-peer-urls: http://$private_ipv4:2380
# listen on both the official ports and the legacy ports
# legacy ports can be omitted if your application doesn't depend on them
listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
listen-peer-urls: http://$private_ipv4:2380,http://$private_ipv4:7001
fleet:
public-ip: $public_ipv4
units:
# To use etcd2, comment out the above service and uncomment these
# Note: this requires a release that contains etcd2
- name: etcd2.service
command: start
- name: fleet.service
command: start
Dynamic node discovery
Etcd environment
variables
Fleet environment
variable
Services to be started
automatically

24. CoreOS Ecosystem
• Supports Docker for running Containers.
• Integrates nicely with Kubernetes.
• Mesos can be run on top of CoreOS.
• Kubernetes can be used for orchestrating
Docker and Rkt Containers.
• CoreOS can be used with Openstack.

25. Tectonic – GIFEE(Google Infrastructure for
everyone else)
Tectonic Distributed Trusted computing
• At the firmware level, the customer key can be
embedded, and this allows customers to verify all
the software running in the system.
• Secure keys embedded in the firmware can verify
the bootloader as well as CoreOS.
• Containers such as Rkt can be verified with their
image signature.
• Logs can be made tamper-proof using the TPM
hardware module embedded in the CPU
motherboard.
• Tectonic integrates CoreOS’s open source components and Kubernetes
along with some integration software to provide Enterprise ready
microservices infrastructure platform.
• With Tectonic, CoreOS is integrating their other commercial offerings
such as CoreUpdate, Quay repository, and Enterprise CoreOS into
Tectonic.
• Tectonic will support both Docker and Rkt Container runtime.
Distributed Trusted computing(Image from
https://tectonic.com/)
Tectonic Software Components

26. Newer projects
• Ignition – Better cloud-config.
• Clair – Container scanning service for
vulnerabilities
• DEX – Identity management
• Openstack deployment on
Tectonic(https://tectonic.com/blog/openstac
k-and-kubernetes-come-together.html)

27. References
• CoreOS docs -
https://coreos.com/os/docs/latest/
• DigitalOcean CoreOS tutorials
(https://www.digitalocean.com/
community/tags/coreos?type=tutorials )
• CoreOS Github page(https://github.com/coreos)
• Mastering CoreOS
book(https://www.packtpub.com/networking-
and-servers/mastering-coreos)

28. DEMO!

29. APPC(Rkt) vs OCI(Docker)
Reference:
https://coreos.com/blog/making-sense-of-standards.html