This document provides instructions for setting up a CoreOS cluster with services like MongoDB, SkyDNS, HAProxy, and Datadog monitoring. It describes using systemd unit files to run services like MongoDB and SkyDNS in Docker containers managed by Fleet. Etcd is used for service discovery and configuration management with ConfD. HAProxy provides load balancing of services, and Datadog monitors the cluster. Cloud-init scripts initialize new nodes joining the cluster.

1. HOW TO RIDE A WHALE
Creating a functionnal CoreOS cluster

2. THE SERVICE
Based on systemd
Unit or template files with basic variables

3. A MONGODB DATABASE
units template : mongodb@.service
[Unit]
Description=MongoDB
After=docker.service
[Service]
ExecStartPre=/usr/bin/docker pull mongo:latest
ExecStartPre=-/usr/bin/docker rm -f -v %p.%i
ExecStart=/usr/bin/docker run --rm --name %p.%i -p 27017:27017
--volume=/home/data/mongo/latest:/data/db
--cpu-shares=4 -m="30g"
mongo:latest
ExecStartPost=/bin/etcdctl set /skydns/fr/mycloud/%p/%i '{"host":"%H", "port"
ExecStartPost=/bin/etcdctl set /databases/%p/%i '{"host":"%H", "port":27017
ExecStop=/bin/etcdctl rm /skydns/fr/mycloud/%p/%i
ExecStop=/bin/etcdctl rm /databases/%p.%i
ExecStop=-/usr/bin/docker stop %p.%i
Restart=always

4. FLEET
Remote control
Basic scheduler
Make services survive to machine crash

5. Simple command line
fleetctl submit mongodb@.service
fleetctl start mongodb@x1
fleetctl start mongodb@x2
fleetctl start mongodb@x3
fleetctl list-units
fleetctl journal -f mongodb@x1
coreos:
fleet:
public-ip: $public_ipv4
metadata: disk=ssd,hoster=ovh,location=FR

6. ETCD
Cluster registry
Raft consensus implementation
Use etcd2
http://thesecretlivesofdata.com/raft/

7. Access anywhere
# Use it via command line
> etcdctl set /skydns/fr/mycloud/mongodb/x1 {"host":"core-1", "port":27017
# Or HTTP
> curl -L http://127.0.0.1:4001/v2/keys/skydns/fr/mycloud/mongodb/x1
{"host":"core-1", "port":27017}
coreos:
etcd:
# generate a new token for each unique cluster from https://discovery.etcd.io
discovery: https://discovery.etcd.io/{token}
addr: $public_ipv4:4001
peer-addr: $public_ipv4:7001
Or etcd-browser for human beings

9. SKYDNS
Nameserver (@95%)
Store and read data from etcd (or consul)
Simplest DNS to administrate ever

10. dig *.mongodb.mycloud.fr SRV @localhost
[Unit]
Description=SkyDNS
After=docker.service
[Service]
ExecStartPre=-/usr/bin/docker rm -f skydns
ExecStart=/usr/bin/docker run --rm --name skydns
-e ETCD_MACHINES=http://127.0.0.1:4001
-e SKYDNS_ADDR=0.0.0.0:53
-e SKYDNS_DOMAIN=mycloud.fr
-e SKYDNS_NAMESERVERS=8.8.8.8:53,8.8.4.4:53
--net=host
skynetservices/skydns:latest
ExecStop=-/usr/bin/docker stop skydns
Restart=always

11. CONFD
Read etcd and write config files
Basic go template
check, and reload services

12. [mongodb]
port = 27017{{ range gets "/databases/mongodb/*"}}{{ $server := json .Value
servers[] = {{ $server.host }}{{end}}
[Unit]
Description=Confd
Wants=etcd.service
[Service]
ExecStartPre=-/usr/bin/mkdir -p /opt/bin
ExecStartPre=-/usr/bin/wget -N -P /opt/bin https://github.com/kelseyhightower
ExecStartPre=-/bin/rm /opt/bin/confd
ExecStartPre=-/bin/mv /opt/bin/confd-0.9.0-linux-amd64 /opt/bin/confd
ExecStartPre=-/bin/chmod +x /opt/bin/confd
ExecStart=/opt/bin/confd -backend etcd -node 127.0.0.1:4001 -interval=10
ExecStop=-/bin/kill $(pidof confd)
Restart=always

13. HAPROXY
TCP proxy, load balancing, health check
and mitigation since 1996

15. http://mycloud.fr:1000
etcdctl set /services/mywordpress.fr/scheme http
etcdctl set /services/mywordpress.fr/hosts/1 1.2.3.4:80
etcdctl set /services/mywordpress.fr/hosts/1 1.2.3.5:80
[Unit]
Description=HA proxy load balancer
After=docker.service
[Service]
ExecStartPre=-/usr/bin/docker rm -f balancer
ExecStart=/usr/bin/docker run --rm --name haproxy
-e ETCD_NODE=127.0.0.1:4001
--volume /etc/certs:/etc/certs/
--net host
cstpdk/haproxy-confd
ExecReload=/usr/bin/docker exec balancer service haproxy reload
ExecStop=/usr/bin/docker stop balancer
Restart=always

16. SYNCTHING
Torrent based private cloud
Neat web interface

17. [Unit]
Description=Syncthing
[Service]
ExecStartPre=-/usr/bin/docker rm -f -v syncthing
ExecStartPre=-/usr/bin/docker pull istepanov/syncthing:latest
ExecStart=/usr/bin/docker run --rm --name syncthing
-p 9080:8080 -p 22000:22000 -p 21025:21025/udp
-v /etc/syncthing:/home/syncthing/.config/syncthing
-v /home/data/sync:/home/syncthing/Sync
istepanov/syncthing
ExecStop=-/usr/bin/docker stop syncthing
Restart=always

18. DATADOG
plug everything in easily
StatsD Graphite on vitamins
Monitoring and Alerting
Or cadvisor+heapster+fluentd or prometheus

19. [Unit]
Description=Datadog monitoring
[Service]
ExecStartPre=-/usr/bin/docker rm -f datadog
ExecStartPre=-/usr/bin/docker pull datadog/docker-dd-agent
ExecStart=/usr/bin/docker run --rm --name datadog
-v /etc/datadog/conf.d:/etc/dd-agent/conf.d
-v /var/run/docker.sock:/var/run/docker.sock
-v /proc/mounts:/host/proc/mounts:ro
-v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
-e API_KEY=GET-IT-ON-DATADOG
--net=host
datadog/docker-dd-agent
ExecStop=-/usr/bin/docker stop datadog
Restart=always

20. CLOUD INIT
BareMetal (ovh, online)
Virtual machines (Ganeti, Openstack)
Cloud providers (AWS, DO, GC ...)

21. #cloud-config
coreos:
update:
reboot-strategy: best-effort
group: stable
fleet:
public-ip: $public_ipv4
units:
- name: haproxy.service
command: start
content: |
[Unit]
....
write_files:
-
path: /etc/resolv.conf
owner: root:root
content: |

22. QUESTIONS ?
git clone https://github.com/Vinceveve/achab
cd achab
vagrant up
https://coreos.com/blog/managing-coreos-with-ansible/
http://www.freedesktop.org/software/systemd/man/systemd.service.html
https://github.com/kelseyhightower/confd/blob/master/docs/templates.md
http://cbonte.github.io/haproxy-dconv/configuration-1.5.html
https://www.digitalocean.com/community/tutorials/how-to-set-up-a-coreos-cluster-on-
digitalocean