Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Clusternaut:
Orchestrating Percona XtraDB Cluster
with Kubernetes.
Raghavendra Prabhu
Percona Live Data Performance ’16
rp...
Yelp’s Mission
Connecting people with great
local businesses.
Yelp Stats
As of Q4 2015
86M 3270%95M
Me
Raghavendra Prabhu
Software Engineer, Distributed Systems @ Yelp
rprabhu@yelp.com / me@rdprabhu.com
Applicability to any datastore
● Derived datastores
○ Elasticsearch
○ Redis
● Relational
○ MySQL
■ Group replication
■ NDB...
Galera - “The Oar boat”
κυβερνήτης “The Helmsman”
Warehouse computing
➔ Mesosphere
➔ Paasta
➔ GCE
◆ Reference
➔ ECS
➔ Smartcloud
➔ Tectonic*
Rationale
➔ Nodes v/s Hosts
◆ Resource-based
➔ Reusable components
◆ Monitoring and Tracing
◆ DNS and Service Discovery
◆ ...
The Fit
➔ Layered
◆ Client - Server
◆ Multi-layered
➔ Scaling
◆ Horizontal and Vertical
● Preferred?
● Need for vertical
The Fit
➔ Statelessness
◆ Planes of logic:
● Control Plane
● Data Plane : Storage
◆ Anti-pattern for containers
➔ Elastici...
Declarative vs Imperative
➔ Configurable mgmt
◆ Puppet, Nix, Terraform
➔ Microservices
◆ What runs on my laptop
● What run...
Containers
● What is a container and why should I care
○ Operating system virtualization
● Isolation
○ Hierarchies of isol...
Galera - really short intro!
➔ MySQL and WSREP api
➔ Galera plugin
➔ Group communication
➔ Synchronous replication
◆ ‘Virt...
Galera - really short intro!
➔ CAP theorem and Galera
◆ CP
➔ How does it fit
◆ Others
◆ Idempotency
➔ Stateless?
◆ Symmetr...
Orchestration
● SOA def
○ ‘Stitching’
○ ‘Composing’
● Automation?
● Choreography
● Best of both worlds
Kubernetes
● Started as orchestrator
○ Is an ecosystem for containers
● Horizontal Scaling
● Self-healing
○ Chaos-monkey
●...
Kubernetes
● Bin packing
● Automated rollouts and rollbacks
● Secret management
○ Elegant
● Storage orchestration
● Servic...
Kubernetes: API
● Consistent and Versioned
○ Very important glue
● Composable
● Developed with Swagger
● API Groups
● Supp...
Kubernetes
➔ Components:
◆ Kubelet
◆ Pods
● Main service
● Sidekicks
◆ Services
● The gcomm:// URL.
◆ Replication Controll...
Kubernetes
➔ Components:
◆ Labels and selectors
● Plumbing / addressing mechanism.
● Metadata - docker-machine, MachineMet...
Kubernetes
➔ Higher Order
◆ Daemon Sets
● Logging, Monitoring, Tracing
◆ Replica Sets
◆ Deployments
● Rolling updates decl...
Kubernetes
➔ Components:
◆ Volumes
● Persistent Volumes
● External Storage Providers
◆ Secrets / Vault
◆ Horizontal Pod Au...
Kubernetes: Providers
● Bespoke
● Google Container Engine (GKE)
● AWS
● Azure
● Determinants:
○ Network - flannel, Weave, ...
Kubernetes: Ecosystem
● Deis
● Package manager - Helm
● Fabric8
● Spread
○ From compose to kubernetes
● Openshift
Kubernetes
➔ Others:
◆ Mesos
● Supports k8s too.
● Aurora, Chronos, Marathon
◆ Docker Swarm
◆ Fleet
➔ Key Differences
Pods
Pods
● Herd..
● What should they contain - containers!
● How is the grouping done
● Pods and nodes
○ Colocation
● Pod comm...
Services
Services
● Don’t commingle with `microservices`
○ Think of endpoints.
● Layering architecture
○ Logical address of subset ...
Services: in general
● Potential issues
○ Staleness
○ Live HUP-ing
○ Propagation
● Haproxy
○ Reload configuration.
○ Solve...
Replication Controller
● “Herd Management”
● ASG
● Pod template: Cookie Cutter
○ Pattern
○ Anti-pattern
■ Asymmetric initi...
Replication Controller
● Role
○ Init/Supervisor for cluster
○ Rolling updates
○ Multi-version
● Replica Sets
Networking
● Docker-style linking
● Proxy for Pods
● Types
○ Pod to Pod
○ Pod to Service
○ Intra-Pod
○ External to Service...
External components
● Flannel / Others
● Etcd
● Fluentd
● Skydns
● Container Registry
● REST server
● Proxy
● cAdvisor / H...
PAAS: PaaSTA
● Docker
● Mesos
○ Chronos
○ Marathon
● Sensu
● Smartstack
○ Zookeeper
● Jenkins
● Splunk / Signalfx
● Why
Deployment
● Declare and build individual Galera/PXC nodes.
○ Keep it minimal and simple
○ No assumptions
● Without Kubern...
Deployment
● Basic Steps:
○ Create a ‘flat’ network - 10.0.0.0/24
○ Create a ‘cluster’ - zone
○ Create a service endpoint....
Deployment
● Next:
○ Bootstrap a node Pod from a template.
■ Query existing with selector.
○ Start rest of nodes from temp...
Deployment - Implications
● Load balancing in state transfers
● Respawning of nodes on timeout
○ May not be same nodes.
● ...
Case Study: Safe restarts
● Highly available
● Unattended autonomous
○ and Imperative
● Restarts - services and nodes
● Mo...
Service Definition
Dockerfile
```
... | kubectl create -f -
kubectl expose service eclient --
port=3306 --target-port=3306 --
name=loadbl --type='LoadBa...
```
.. | kubectl create -f -
kubectl scale --replicas=8
replicationcontrollers
controller
```
```
kubectl get --no-headers...
Credits!
● https://www.pinterest.com/duanejohnson851/star-trek-tng/
● https://upload.wikimedia.org/wikipedia/commons/a/a5/...
Further reading!
● http://kubernetes.io
● https://github.com/ramitsurana/awesome-kubernetes
● https://open.mesosphere.com/...
Contact
Raghavendra Prabhu
rprabhu@yelp.com / me@rdprabhu.com
Twitter: @randomsurfer
Linkedin: rdprabhu
Slideshare: slidun...
We are Hiring!
Visit
yelp.com/careers
@YelpEngineering
fb.com/YelpEngineers
engineeringblog.yelp.com
github.com/yelp
Clusternaut:  Orchestrating  Percona XtraDB Cluster with Kubernetes
Clusternaut:  Orchestrating  Percona XtraDB Cluster with Kubernetes
Upcoming SlideShare
Loading in …5
×

Clusternaut: Orchestrating  Percona XtraDB Cluster with Kubernetes

1,046 views

Published on

Talk is about using Kuberentes to orchestrate datastores, with Percona XtraDB Cluster as the reference.

Published in: Engineering
  • Be the first to comment

Clusternaut: Orchestrating  Percona XtraDB Cluster with Kubernetes

  1. 1. Clusternaut: Orchestrating Percona XtraDB Cluster with Kubernetes. Raghavendra Prabhu Percona Live Data Performance ’16 rprabhu@yelp.com/me@rdprabhu.com/@randomsurfer
  2. 2. Yelp’s Mission Connecting people with great local businesses.
  3. 3. Yelp Stats As of Q4 2015 86M 3270%95M
  4. 4. Me Raghavendra Prabhu Software Engineer, Distributed Systems @ Yelp rprabhu@yelp.com / me@rdprabhu.com
  5. 5. Applicability to any datastore ● Derived datastores ○ Elasticsearch ○ Redis ● Relational ○ MySQL ■ Group replication ■ NDB ○ PostgreSQL ● MongoDB ● Cassandra
  6. 6. Galera - “The Oar boat”
  7. 7. κυβερνήτης “The Helmsman”
  8. 8. Warehouse computing ➔ Mesosphere ➔ Paasta ➔ GCE ◆ Reference ➔ ECS ➔ Smartcloud ➔ Tectonic*
  9. 9. Rationale ➔ Nodes v/s Hosts ◆ Resource-based ➔ Reusable components ◆ Monitoring and Tracing ◆ DNS and Service Discovery ◆ Logging ◆ Metrics ◆ Scheduler ➔ Agnostic ➔ Roles
  10. 10. The Fit ➔ Layered ◆ Client - Server ◆ Multi-layered ➔ Scaling ◆ Horizontal and Vertical ● Preferred? ● Need for vertical
  11. 11. The Fit ➔ Statelessness ◆ Planes of logic: ● Control Plane ● Data Plane : Storage ◆ Anti-pattern for containers ➔ Elasticity ◆ Elastic Scalability ● Scaling down
  12. 12. Declarative vs Imperative ➔ Configurable mgmt ◆ Puppet, Nix, Terraform ➔ Microservices ◆ What runs on my laptop ● What runs on server ● Reproducibility ➔ 12-factor app ➔ Composability ➔ Immutable deployment artifact
  13. 13. Containers ● What is a container and why should I care ○ Operating system virtualization ● Isolation ○ Hierarchies of isolation - application, cgroups, namespaces, seccomp… ● Unikernels and VMs ○ Role? ○ MirageOS, Rump kernel ● Some - LXC / LXD, Docker*, Rocket*, runc, jails, solaris zones, lmcty, systemd-nspawn
  14. 14. Galera - really short intro! ➔ MySQL and WSREP api ➔ Galera plugin ➔ Group communication ➔ Synchronous replication ◆ ‘Virtually’ ➔ EVS ➔ Certification-based ◆ Optimistic Concurrency ➔ Automatic Node Provisioning
  15. 15. Galera - really short intro! ➔ CAP theorem and Galera ◆ CP ➔ How does it fit ◆ Others ◆ Idempotency ➔ Stateless? ◆ Symmetric ◆ Replicas - Cassandra et.al. ◆ MySQL Cluster ➔ Maintenance of Quorum
  16. 16. Orchestration ● SOA def ○ ‘Stitching’ ○ ‘Composing’ ● Automation? ● Choreography ● Best of both worlds
  17. 17. Kubernetes ● Started as orchestrator ○ Is an ecosystem for containers ● Horizontal Scaling ● Self-healing ○ Chaos-monkey ● Latest issue ○ Rolling update in clusters ○ How K8s solves this
  18. 18. Kubernetes ● Bin packing ● Automated rollouts and rollbacks ● Secret management ○ Elegant ● Storage orchestration ● Service discovery and load balancing ○ Underrated
  19. 19. Kubernetes: API ● Consistent and Versioned ○ Very important glue ● Composable ● Developed with Swagger ● API Groups ● Supports both declarative and imperative ○ Rolling-update / Daemon Sets
  20. 20. Kubernetes ➔ Components: ◆ Kubelet ◆ Pods ● Main service ● Sidekicks ◆ Services ● The gcomm:// URL. ◆ Replication Controller
  21. 21. Kubernetes ➔ Components: ◆ Labels and selectors ● Plumbing / addressing mechanism. ● Metadata - docker-machine, MachineMetadata ● Set-based and equality-based
  22. 22. Kubernetes ➔ Higher Order ◆ Daemon Sets ● Logging, Monitoring, Tracing ◆ Replica Sets ◆ Deployments ● Rolling updates declarative ● Bouncing ○ PaaSTA
  23. 23. Kubernetes ➔ Components: ◆ Volumes ● Persistent Volumes ● External Storage Providers ◆ Secrets / Vault ◆ Horizontal Pod Autoscaler ➔ Scheduler ◆ Pluggable
  24. 24. Kubernetes: Providers ● Bespoke ● Google Container Engine (GKE) ● AWS ● Azure ● Determinants: ○ Network - flannel, Weave, calico, GCE. ○ OS ○ Config Mgmt
  25. 25. Kubernetes: Ecosystem ● Deis ● Package manager - Helm ● Fabric8 ● Spread ○ From compose to kubernetes ● Openshift
  26. 26. Kubernetes ➔ Others: ◆ Mesos ● Supports k8s too. ● Aurora, Chronos, Marathon ◆ Docker Swarm ◆ Fleet ➔ Key Differences
  27. 27. Pods
  28. 28. Pods ● Herd.. ● What should they contain - containers! ● How is the grouping done ● Pods and nodes ○ Colocation ● Pod communication ● Labels
  29. 29. Services
  30. 30. Services ● Don’t commingle with `microservices` ○ Think of endpoints. ● Layering architecture ○ Logical address of subset of pods ● Communication ○ Environment ■ Ordering requirement ■ Discovery ○ DNS ■ Issues with DNS
  31. 31. Services: in general ● Potential issues ○ Staleness ○ Live HUP-ing ○ Propagation ● Haproxy ○ Reload configuration. ○ Solved at Yelp with linux qdiscs. ● Flux from Weave
  32. 32. Replication Controller ● “Herd Management” ● ASG ● Pod template: Cookie Cutter ○ Pattern ○ Anti-pattern ■ Asymmetric initialisation
  33. 33. Replication Controller ● Role ○ Init/Supervisor for cluster ○ Rolling updates ○ Multi-version ● Replica Sets
  34. 34. Networking ● Docker-style linking ● Proxy for Pods ● Types ○ Pod to Pod ○ Pod to Service ○ Intra-Pod ○ External to Service ● Providers: ○ OpenVSwitch / Flannel / Calico / Weave / Google
  35. 35. External components ● Flannel / Others ● Etcd ● Fluentd ● Skydns ● Container Registry ● REST server ● Proxy ● cAdvisor / Heapster
  36. 36. PAAS: PaaSTA ● Docker ● Mesos ○ Chronos ○ Marathon ● Sensu ● Smartstack ○ Zookeeper ● Jenkins ● Splunk / Signalfx ● Why
  37. 37. Deployment ● Declare and build individual Galera/PXC nodes. ○ Keep it minimal and simple ○ No assumptions ● Without Kubernetes ○ Docker-compose ■ Possible issues ● Galera node ⇔ Pod ○ Haproxy ○ xinetd
  38. 38. Deployment ● Basic Steps: ○ Create a ‘flat’ network - 10.0.0.0/24 ○ Create a ‘cluster’ - zone ○ Create a service endpoint. ■ Internal service - 3306/4567/4568. ■ External service - 3306/3306(?). ■ Expose the external. ■ Session affinities.
  39. 39. Deployment ● Next: ○ Bootstrap a node Pod from a template. ■ Query existing with selector. ○ Start rest of nodes from template. ■ Point to Service with selector. ■ Replication controller ○ Volumes
  40. 40. Deployment - Implications ● Load balancing in state transfers ● Respawning of nodes on timeout ○ May not be same nodes. ● kubectl to manage ○ Puppet etc. also have modules now. ● Separation of client and cluster traffic
  41. 41. Case Study: Safe restarts ● Highly available ● Unattended autonomous ○ and Imperative ● Restarts - services and nodes ● More of a orchestration than choreographing ● Randomness? ● Distributed locks ● Low impedance
  42. 42. Service Definition
  43. 43. Dockerfile
  44. 44. ``` ... | kubectl create -f - kubectl expose service eclient -- port=3306 --target-port=3306 -- name=loadbl --type='LoadBalancer’ ```
  45. 45. ``` .. | kubectl create -f - kubectl scale --replicas=8 replicationcontrollers controller ``` ``` kubectl get --no-headers pods -l 'name=pxc' | wc -l 2 kubectl stop … kubectl get --no-headers pods -l 'name=pxc' | wc -l 2 ```
  46. 46. Credits! ● https://www.pinterest.com/duanejohnson851/star-trek-tng/ ● https://upload.wikimedia.org/wikipedia/commons/a/a5/CubeSpace.jpg ● https://upload.wikimedia.org/wikipedia/commons/thumb/b/ba/IUB_Arboretum_-_lotus_pond_- _dry_seed_pod_-_P1100172.JPG/1280px-IUB_Arboretum_-_lotus_pond_-_dry_seed_pod_-_P1100172.JPG ● https://raw.githubusercontent.com/kubernetes/kubernetes/master/docs/design/architecture.png ● https://pbs.twimg.com/profile_images/511909265720614913/21_d3cvM.png ● https://camo.githubusercontent. com/96468330aba188dbd7d7eeae0caca32d9a6329df/687474703a2f2f656e67696e656572696e67626c6f67 2e79656c702e636f6d2f696d616765732f70726576696577732f7061617374615f707265766965772e706e67 ● http://galeracluster.com/documentation-webpages/_images/replicationapi.png ● https://www.linkedin.com/pulse/containerizing-docker-kubernetes-ramit-surana
  47. 47. Further reading! ● http://kubernetes.io ● https://github.com/ramitsurana/awesome-kubernetes ● https://open.mesosphere.com/frameworks/ ● https://coreos.com/kubernetes/docs/latest/kubernetes-networking. html ● http://paasta.readthedocs.org/en/latest/about/paasta_principles.html ● http://12factor.net/ ● http://kubernetes.io/v1.1/docs/api-reference/v1/definitions.html
  48. 48. Contact Raghavendra Prabhu rprabhu@yelp.com / me@rdprabhu.com Twitter: @randomsurfer Linkedin: rdprabhu Slideshare: slidunder Github: ronin13 http://rdprabhu.com http://about.me/raghavendra.prabhu
  49. 49. We are Hiring! Visit yelp.com/careers
  50. 50. @YelpEngineering fb.com/YelpEngineers engineeringblog.yelp.com github.com/yelp

×