CoreOS Fleet and Etcd provide a simple and eloquent framework for application clusters to both auto-configure and recover from node failure. Galera Cluster is a multi-master, open solution for clustering MySQL. Mix the two, sprinkle in a bit of “glue” and you have a Docker based MySQL cluster that will react automatically to container failure. This presentation will cover the nuts and bolts of automating a Galera Cluster, built from Docker Images and deployed in a distributed fashion using etcd, confd, and fleet for both initial and failure recovery configuration.

1. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
CoreOSautomatedMySQL
ClusterFailoverusingGaleraCluster
Hewlett-Packard (Enterprise) Advanced Technology Group – Cloud
Yazz Atlas
Patrick Galbraith Spring 2015

2. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Aboutthespeaker
● Yazz Atlas
● HP Advanced Technology Group
● HP Cloud Services PaaS/DBaaS/
LBaaS Environment
● Has worked at Windermere Real
Estate, Keas, Slashdot, Cobalt
Group, University of Iowa, Antioch
College

3. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Aboutthespeaker
● Patrick Galbraith
● HP Advanced Technology Group
● Has worked at Blue Gecko,
MySQL AB, Classmates, Slashdot,
Cobalt Group, US Navy, K-mart
● MySQL projects: memcached
UDFs, DBD::mysql, federated
storage engine
● Family
● Outdoors

4. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4
Purpose of this talk – why are you here?
Basic Pre-req
• Overview of the Demo Setup
Docker
• Containers vs. Virtulization
• Simple Docker usage
• Clustered Docker
CoreOS
• Container optimized, stripped down Linux
distribution
• Overview of core components of CoreOS – fleet,
etcd, systemd
Galera
• Synchronous replication – excellent solution to
clustering MySQL
• HA Proxy for failover
Discussion
• What is next
• What about Kubernetes

5. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5
Laptop Host
- VirtualBox
- OpenStack
- Vmware
- Bare metal
- et al.
Master-01
Basic Pre-reqs Setup
CoreOS + etcd + confd
Some of the times you will need
• CoreOS
• Go
– https://golang.org/doc/install
• Etcdctl
– go get github.com/coreos/etcd/etcdctl
• Fleetctl
– go get github.com/coreos/fleet/fleetctl
Core-01
Core-02
Core-03

6. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6
What are containers?
• Operating-system-level virtualization
• Encapsulated, hermetically sealed applications
• Relatively isolated
• Small footprint
• Fast to launch!
• Portable. And did I mention, portable?!
• Use of host OS and Kernel
• Execution consists of time to startup application in question
• LXC, Docker, Solaris Zones, BSD Jails, Parallels Virtuozzo, OpenVZ, …
Photo credit: expo_2020 - http://goo.gl/OeSQhi License CC v2.0

7. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7
Containers vs. VMs

8. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8
Docker
What is Docker?
• Set of tools for managing containers
• Command line tool that doubles as a daemon
• Uses Linux Kernel Features:
– Kernel namespaces – the core ingredient to containers working: PID, IPC ,uts (what will be
seen by a group of processes), mount, network and user
– Cgroups (control groups) -- limit, account and isolate resource usage (CPU, memory, disk I/O,
etc.) of process groups
• Originally used lxc, now defaults to Libcontainer but meant for any containerization mechanism
• Much more light weight than VMs
• Encapsulated application containers in a relatively isolated but lightweight operating
environment
• Written in Go

9. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.9
Clustered Docker
• CoreOS (https://coreos.com/)
• Kubernetes (http://kubernetes.io)
• Mesos + Marathon (http://mesos.apache.org/) Apache project, Zookeeper, etc
• Project Atomic (http://www.projectatomic.io/) RH/Fedora/Centos designed for running Docker
• Docker Openstack (https://wiki.openstack.org/wiki/Docker) Hypervisor Driver for Openstack Compute
• Swarm/Compose/Machine
• RancherOS (http://rancher.com/rancher-os) Minimalist Linux, Docker daemon runs as PID 1 first process the
kernel starts known as “System Docker,”
• Flocker (https://clusterhq.com)
• Spotify Helios (https://github.com/spotify/helios) Zookeeper
• Flynn (https://flynn.io/)
• Deis (http://deis.io)
• Maestro (https://github.com/toscanini/maestro)
• Shipyard (http://shipyard-project.com)
• … others to come!

10. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10
What is CoreOS?
CoreOS
• Minimalist Linux
• Optimized for containers
• Easy to run containers
• Service discovery, container management,
• Docker -- Container runtime and management, though Rocket long-term
• Etcd – distributed global key value store for config data on each node
• Fleet – Rudimentary Scheduler interacts with systemd and etcd
• Systemd – Newer Linuxi – system and service manager for Linux
• Flannel – Networking across nodes

11. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11
Connecting it all together
CoreOS
• Etcd
• Fleetd
• Flanneld
Photo credit: Stéfan – http://goo.gl/ye8Icd License CC v2.0

12. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.12
Galera secret sauce
• Galera (Percona XtraDB Cluster) container
• FLEETCTL_ENDPOINT set to ETCD host:port
• etcd included
• Entrypoint script
• Generates SQL file to create SST (single state transfer) user and
clustercheck user
• Builds cluster.cnf
• Wsrep_node_address set to its own IP
• Wsrep_cluster_address set using entries from etcd
• Server-id random number
• clustercheck on each galera node container (port 8000) to determine
health and which nodes available using clustercheck user to connect
• Ports 3306, 4444, 4567, 4568, 8000 exposed provides access to Galera
cluster node containers for external and HAProxy container

13. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13
Galera secret sauce – haproxy sprinkling
• HAProxy container
• FLEETCTL_ENDPOINT set to ETCD host:port
• Confd included
• Confd builds haproxy.cfg from template using etcd entries for each Galera
node
• Confd loop entered to start container
• HAProxy uses clustercheck on each node (port 8000) to determine health
and which nodes available
• Port 33006 exposed provides access to Galera cluster node containers

14. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.14
Laptop
Virtualbox
Master-01
Core-01
Core-02
Core-03
etcd
flanneld
Galera-1
Galera-2
Galera-3
haproxy
Sidekick-1
Sidekick-2
Sidekick-2
Flannel network
Galerareplication
HAProxy
etcdcommunication
Connecting it all together.
$>
vagrant
up

15. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.15
Galera on CoreOS

16. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.16
Disaster Strikes!
Don’t put all your containers on
one ship (CoreOS host).
• HA Proxy is the middleman
connecting to your Galera Cluster
– Minimal impact to connecting
agents
• Adding a new CoreOS host and
redeploy the Galera container
Photo credit Paul Townsend – http://goo.gl/ImL8hX License CC v2.0

17. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.17
Demo time…
DEMO
• Vagrant - 1.7.2
• Virtualbox - 4.3.26r98988
– coreos-alpha (virtualbox, 660.0.0)
• Github Repos
– https://github.com/EntropyWorks/coreos-vagrant
– https://github.com/EntropyWorks/fleet-units-galera-cluster
– https://github.com/EntropyWorks/docker_haproxy
– https://github.com/CaptTofu/percona_xtradb_cluster_docker (coreos branch)
• Asciinema recordings
– https://asciinema.org/a/19559

18. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
ThankYou
Hewlett-Packard, Advanced Technology Group
Yazz Atlas yazz.atlas@hp.com
Patrick Galbaith patg@hp.com