Raghavendra Prabhu presented on orchestrating Percona XtraDB Cluster (PXC) with Kubernetes. Some key points: - Kubernetes provides horizontal scaling, self-healing, automated rollouts/rollbacks, service discovery, storage orchestration and more. - In Kubernetes, PXC nodes would be deployed as pods with a replication controller to maintain a set number of pods. Services provide load balancing to the pods. - Demonstrated deploying a basic PXC cluster on Kubernetes, including creating a network, cluster, service, replicating pods from a template, and exposing ports. - Challenges include load balancing for state transfers between nodes and ensuring nodes are

1. Clusternaut:
Orchestrating Percona XtraDB Cluster
with Kubernetes.
Raghavendra Prabhu
Percona Live Data Performance ’16
rprabhu@yelp.com/me@rdprabhu.com/@randomsurfer

2. Yelp’s Mission
Connecting people with great
local businesses.

3. Yelp Stats
As of Q4 2015
86M 3270%95M

4. Me
Raghavendra Prabhu
Software Engineer, Distributed Systems @ Yelp
rprabhu@yelp.com / me@rdprabhu.com

5. Applicability to any datastore
● Derived datastores
○ Elasticsearch
○ Redis
● Relational
○ MySQL
■ Group replication
■ NDB
○ PostgreSQL
● MongoDB
● Cassandra

6. Galera - “The Oar boat”

7. κυβερνήτης “The Helmsman”

8. Warehouse computing
➔ Mesosphere
➔ Paasta
➔ GCE
◆ Reference
➔ ECS
➔ Smartcloud
➔ Tectonic*

9. Rationale
➔ Nodes v/s Hosts
◆ Resource-based
➔ Reusable components
◆ Monitoring and Tracing
◆ DNS and Service Discovery
◆ Logging
◆ Metrics
◆ Scheduler
➔ Agnostic
➔ Roles

10. The Fit
➔ Layered
◆ Client - Server
◆ Multi-layered
➔ Scaling
◆ Horizontal and Vertical
● Preferred?
● Need for vertical

11. The Fit
➔ Statelessness
◆ Planes of logic:
● Control Plane
● Data Plane : Storage
◆ Anti-pattern for containers
➔ Elasticity
◆ Elastic Scalability
● Scaling down

12. Declarative vs Imperative
➔ Configurable mgmt
◆ Puppet, Nix, Terraform
➔ Microservices
◆ What runs on my laptop
● What runs on server
● Reproducibility
➔ 12-factor app
➔ Composability
➔ Immutable deployment artifact

13. Containers
● What is a container and why should I care
○ Operating system virtualization
● Isolation
○ Hierarchies of isolation - application, cgroups,
namespaces, seccomp…
● Unikernels and VMs
○ Role?
○ MirageOS, Rump kernel
● Some - LXC / LXD, Docker*, Rocket*, runc, jails, solaris
zones, lmcty, systemd-nspawn

14. Galera - really short intro!
➔ MySQL and WSREP api
➔ Galera plugin
➔ Group communication
➔ Synchronous replication
◆ ‘Virtually’
➔ EVS
➔ Certification-based
◆ Optimistic Concurrency
➔ Automatic Node Provisioning

15. Galera - really short intro!
➔ CAP theorem and Galera
◆ CP
➔ How does it fit
◆ Others
◆ Idempotency
➔ Stateless?
◆ Symmetric
◆ Replicas - Cassandra et.al.
◆ MySQL Cluster
➔ Maintenance of Quorum

16. Orchestration
● SOA def
○ ‘Stitching’
○ ‘Composing’
● Automation?
● Choreography
● Best of both worlds

17. Kubernetes
● Started as orchestrator
○ Is an ecosystem for containers
● Horizontal Scaling
● Self-healing
○ Chaos-monkey
● Latest issue
○ Rolling update in clusters
○ How K8s solves this

18. Kubernetes
● Bin packing
● Automated rollouts and rollbacks
● Secret management
○ Elegant
● Storage orchestration
● Service discovery and load balancing
○ Underrated

19. Kubernetes: API
● Consistent and Versioned
○ Very important glue
● Composable
● Developed with Swagger
● API Groups
● Supports both declarative and imperative
○ Rolling-update / Daemon Sets

21. Kubernetes
➔ Components:
◆ Kubelet
◆ Pods
● Main service
● Sidekicks
◆ Services
● The gcomm:// URL.
◆ Replication Controller

22. Kubernetes
➔ Components:
◆ Labels and selectors
● Plumbing / addressing mechanism.
● Metadata - docker-machine, MachineMetadata
● Set-based and equality-based

23. Kubernetes
➔ Higher Order
◆ Daemon Sets
● Logging, Monitoring, Tracing
◆ Replica Sets
◆ Deployments
● Rolling updates declarative
● Bouncing
○ PaaSTA

24. Kubernetes
➔ Components:
◆ Volumes
● Persistent Volumes
● External Storage Providers
◆ Secrets / Vault
◆ Horizontal Pod Autoscaler
➔ Scheduler
◆ Pluggable

25. Kubernetes: Providers
● Bespoke
● Google Container Engine (GKE)
● AWS
● Azure
● Determinants:
○ Network - flannel, Weave, calico, GCE.
○ OS
○ Config Mgmt

26. Kubernetes: Ecosystem
● Deis
● Package manager - Helm
● Fabric8
● Spread
○ From compose to kubernetes
● Openshift

27. Kubernetes
➔ Others:
◆ Mesos
● Supports k8s too.
● Aurora, Chronos, Marathon
◆ Docker Swarm
◆ Fleet
➔ Key Differences

28. Pods

29. Pods
● Herd..
● What should they contain - containers!
● How is the grouping done
● Pods and nodes
○ Colocation
● Pod communication
● Labels

30. Services

31. Services
● Don’t commingle with `microservices`
○ Think of endpoints.
● Layering architecture
○ Logical address of subset of pods
● Communication
○ Environment
■ Ordering requirement
■ Discovery
○ DNS
■ Issues with DNS

32. Services: in general
● Potential issues
○ Staleness
○ Live HUP-ing
○ Propagation
● Haproxy
○ Reload configuration.
○ Solved at Yelp with linux qdiscs.
● Flux from Weave

33. Replication Controller
● “Herd Management”
● ASG
● Pod template: Cookie Cutter
○ Pattern
○ Anti-pattern
■ Asymmetric initialisation

34. Replication Controller
● Role
○ Init/Supervisor for cluster
○ Rolling updates
○ Multi-version
● Replica Sets

35. Networking
● Docker-style linking
● Proxy for Pods
● Types
○ Pod to Pod
○ Pod to Service
○ Intra-Pod
○ External to Service
● Providers:
○ OpenVSwitch / Flannel / Calico / Weave / Google

36. External components
● Flannel / Others
● Etcd
● Fluentd
● Skydns
● Container Registry
● REST server
● Proxy
● cAdvisor / Heapster

37. PAAS: PaaSTA
● Docker
● Mesos
○ Chronos
○ Marathon
● Sensu
● Smartstack
○ Zookeeper
● Jenkins
● Splunk / Signalfx
● Why

38. Deployment
● Declare and build individual Galera/PXC nodes.
○ Keep it minimal and simple
○ No assumptions
● Without Kubernetes
○ Docker-compose
■ Possible issues
● Galera node ⇔ Pod
○ Haproxy
○ xinetd

39. Deployment
● Basic Steps:
○ Create a ‘flat’ network - 10.0.0.0/24
○ Create a ‘cluster’ - zone
○ Create a service endpoint.
■ Internal service - 3306/4567/4568.
■ External service - 3306/3306(?).
■ Expose the external.
■ Session affinities.

40. Deployment
● Next:
○ Bootstrap a node Pod from a template.
■ Query existing with selector.
○ Start rest of nodes from template.
■ Point to Service with selector.
■ Replication controller
○ Volumes

41. Deployment - Implications
● Load balancing in state transfers
● Respawning of nodes on timeout
○ May not be same nodes.
● kubectl to manage
○ Puppet etc. also have modules now.
● Separation of client and cluster traffic

42. Case Study: Safe restarts
● Highly available
● Unattended autonomous
○ and Imperative
● Restarts - services and nodes
● More of a orchestration than choreographing
● Randomness?
● Distributed locks
● Low impedance

43. Service Definition

44. Dockerfile

45. ```
... | kubectl create -f -
kubectl expose service eclient --
port=3306 --target-port=3306 --
name=loadbl --type='LoadBalancer’
```

47. ```
.. | kubectl create -f -
kubectl scale --replicas=8
replicationcontrollers
controller
```
```
kubectl get --no-headers
pods -l 'name=pxc' | wc -l
2
kubectl stop …
kubectl get --no-headers
pods -l 'name=pxc' | wc -l
2
```

48. Credits!
● https://www.pinterest.com/duanejohnson851/star-trek-tng/
● https://upload.wikimedia.org/wikipedia/commons/a/a5/CubeSpace.jpg
● https://upload.wikimedia.org/wikipedia/commons/thumb/b/ba/IUB_Arboretum_-_lotus_pond_-
_dry_seed_pod_-_P1100172.JPG/1280px-IUB_Arboretum_-_lotus_pond_-_dry_seed_pod_-_P1100172.JPG
● https://raw.githubusercontent.com/kubernetes/kubernetes/master/docs/design/architecture.png
● https://pbs.twimg.com/profile_images/511909265720614913/21_d3cvM.png
● https://camo.githubusercontent.
com/96468330aba188dbd7d7eeae0caca32d9a6329df/687474703a2f2f656e67696e656572696e67626c6f67
2e79656c702e636f6d2f696d616765732f70726576696577732f7061617374615f707265766965772e706e67
● http://galeracluster.com/documentation-webpages/_images/replicationapi.png
● https://www.linkedin.com/pulse/containerizing-docker-kubernetes-ramit-surana

49. Further reading!
● http://kubernetes.io
● https://github.com/ramitsurana/awesome-kubernetes
● https://open.mesosphere.com/frameworks/
● https://coreos.com/kubernetes/docs/latest/kubernetes-networking.
html
● http://paasta.readthedocs.org/en/latest/about/paasta_principles.html
● http://12factor.net/
● http://kubernetes.io/v1.1/docs/api-reference/v1/definitions.html

50. Contact
Raghavendra Prabhu
rprabhu@yelp.com / me@rdprabhu.com
Twitter: @randomsurfer
Linkedin: rdprabhu
Slideshare: slidunder
Github: ronin13
http://rdprabhu.com
http://about.me/raghavendra.prabhu

51. We are Hiring!
Visit
yelp.com/careers

52. @YelpEngineering
fb.com/YelpEngineers
engineeringblog.yelp.com
github.com/yelp