Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Galera on kubernetes_no_video

2,538 views

Published on

My Galera on Kubernetes on CoreOS presentation from Percona Live 2015 in Santa Clara. Please be patient as I need to edit my videos and upload them to youtube in the next few days.

Published in: Software
  • Be the first to comment

Galera on kubernetes_no_video

  1. 1. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Galera on Kubernetes Running a Galera Cluster on a Kubernetes Cluster Patrick Galbraith Advanced Technology Group/ April 14th, 2015
  2. 2. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP ATG HP's Advanced Technology Group for Open Source and Cloud embraces a vision that is two steps ahead of today's solutions. We use this vision to drive product adoption and incubate technologies to advance HP. Through Open Source initiatives we foster collaboration across HP and beyond.
  3. 3. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. About the speaker ● Patrick Galbraith ● HP Advanced Technology Group ● Has worked at Blue Gecko, MySQL AB, Classmates, Slashdot, Cobalt Group, US Navy, K-mart ● MySQL projects: memcached UDFs, DBD::mysql, federated storage engine ● Family ● Outdoors
  4. 4. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4 Purpose of this talk – why are you here? Docker • Containers vs. Virtualization • Simple Docker usage • Clustered Docker CoreOS • Container optimized, stripped-down Linux distribution • Overview of core components of CoreOS – Fleet, etcd. systemd Kubernetes • A more advanced scheduler and how it works • Using Kubernetes to do work, in this case what it means to MySQL users Galera • Synchronous replication – excellent solution to clustering MySQL Vitesss • Introduction and demo
  5. 5. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5 What do you need? Pre-reqs • Vmware – fusion or workstation. ESXi should work too. • https://github.com/CaptTofu/mysql_replication_kubernetes.git • https://github.com/CaptTofu/kubernetes_cluster_vmware.git • Go -- brew install go (or https://golang.org/doc/install) Clients: • etcdctl: go get github.com/coreos/etcd/etcdctl • fleetctl: go get github.com/coreos/fleet/fleetctl • kubectl – – git clone https://github.com/GoogleCloudPlatform/kubernetes – make – sudo cp cmd/kubectl /usr/local/bin • vtctlclient -- go get github.com/youtube/vitess/go/cmd/vtctlclient
  6. 6. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6 What are containers? • Operating-system-level virtualization • Encapsulated, hermetically sealed applications • Relatively isolated • Small footprint • Fast to launch! • Portable. And did I mention, portable?! • Use of host OS and Kernel • Execution consists of time to startup application in question • LXC, Docker, Solaris Zones, BSD Jails, Parallels Virtuozzo, OpenVZ, …
  7. 7. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7 Containers vs. VMs
  8. 8. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8 Docker What is Docker? • Set of tools for managing containers • Command line tool that doubles as a daemon • Uses Linux Kernel Features: – Kernel namespaces – the core ingredient to containers working: PID, IPC ,uts (what will be seen by a group of processes), mount, network and user – Cgroups (control groups) -- limit, account and isolate resource usage (CPU, memory, disk I/O, etc.) of process groups • Originally used lxc, now defaults to Libcontainer but meant for any containerization mechanism • Much more light weight than VMs • Encapsulated application containers in a relatively isolated but lightweight operating environment • Written in Go
  9. 9. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.9 Docker – common terms and usage • Dockerfile • EXPOSE ports • Entrypoints and CMD • docker build • docker push • docker run • docker inspect • docker exec
  10. 10. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10 Dockerfile • Show example of Dockerfile and explain • Show entrypoint scripts • Environment variables passed to container
  11. 11. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11 Simple Docker usage - forked Video will be provided on youtube
  12. 12. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.12 Clustered Docker • CoreOS (https://coreos.com/) • Kubernetes (http://kubernetes.io) • Mesos + Marathon (http://mesos.apache.org/) Apache project, Zookeeper, etc • Project Atomic (http://www.projectatomic.io/) RH/Fedora/Centos designed for running Docker • Docker Openstack (https://wiki.openstack.org/wiki/Docker) Hypervisor Driver for Openstack Compute • Swarm/Compose/Machine • RancherOS (http://rancher.com/rancher-os) Minimalist Linux, Docker daemon runs as PID 1 first process the kernel starts known as “System Docker,” • Flocker (https://clusterhq.com) • Spotify Helios (https://github.com/spotify/helios) Zookeeper • Flynn (https://flynn.io/) • Deis (http://deis.io) • Maestro (https://github.com/toscanini/maestro) • Shipyard (http://shipyard-project.com) • … others to come!
  13. 13. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13 What is CoreOS? CoreOS • Minimalist Linux • Optimized for containers • Easy to run containers • Service discovery, container management, • Docker -- Container runtime and management, though Rocket long-term • Etcd – distributed global key value store for config data on each node • Fleet – Rudimentary Scheduler interacts with systemd and etcd • Systemd – Newer Linuxi – system and service manager for Linux • Flannel – Networking across nodes
  14. 14. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.14 CoreOS Cluster CoreOS CoreOS DNA Node Docker Containers etcd fleetd dockerd systemd CoreOS DNA Node Docker Containers etcd fleetd dockerd systemd CoreOS DNA Node Docker Containers etcd fleetd dockerd systemd… etcd Discovery Server etcd http://discovery.etcd.io Docker Registry Images Discovery Control Node (jump box) etcdctl fleetctl ssh 1 2 3 1. Cluster Start (etcd discovery) 2. Container Start ( fleetctl ) 3. Docker Download
  15. 15. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.15 What is Kubernetes? Kubernetes • “An open source system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications.” • Pre-Production Beta • Lean • Portable – will run cloud, bare metal, hybrid, etc • Extensible – using modular design allowing for plug-ability and hooks • Self-healing – auto-placement, auto-restart, auto-replication • Google engineering bring good work to the Open-source world
  16. 16. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.16 Master Components Kubernetes • kube-apiserver – API Server (RESTful) – primary management for cluster – reconciles etcd entries with deployed containers • kube-controllermanager — Controller Manager Server – Handle replication precesses defined by replication tasks – Writes details to etcd – Monitors changes and implements procedure to reflect the change • kube-scheduler -- Scheduler Server – Assigns workloads to specific minions in cluster taking into account service’s operating requirements and infrastructure environment • kube-register -- Register Server
  17. 17. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.17 Minion Components Kubernetes • Kubelet Kubelet – Communicates with the master, relaying information to/from – Reads and updates etcd – Receives work in a manifest that defines the workload and operating parameters. – Assumes responsibility for the state of work on minion • kube-proxy Kube proxy – Ensures network environment is accessible but isolated. – Makes services available externally by forwarding requests to containers. – Can perform rudimentary load balancing.
  18. 18. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.18 Kubernetes Terminology • Pod • Group of closely-related containers on the same host • Service • Virtual abstraction • Basic load-balancer • Single consistent access point to a pod • Replication controller • Defines pods to be horizontally scaled • Uses a label query for identifying what containers to run • Maintains specified number of replicas of a particular thing to run • Dynamic resizing • Label • Key/value tag to mark work units a part of group • Management and action targeting • Definition file – YAML/json describing a pod, service, or replication controller
  19. 19. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.19 Kubernetes
  20. 20. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.20 Kubernetes usage pattern • Pod configuration file – YAML or JSON • Service configuration file • Replication controller configuration file • export KUBERNETES_API=http://172.16.230.132:8080 • kubectl create –f mysql_master.json • kubectl create –f mysql_master_service.json • …
  21. 21. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.21 MySQL async replication on Kubernetes • Simple proof of concept • Master pod and service • Slave pod and service • Secret sauce? • The master pod configuration file passes environment variables to set root password, replication password • The entrypoint script runs an SQL file to grant permissions to replication user • The master service configuration is loaded and the slave pod container(s) have as an environment variable MYSQL_MASTER_SERVICE_HOST set • Slave pod when launched has environment variables for replication user and password passed and along with MYSQL_MASTER_SERVICE_HOST variable, changes the master host and user to point to that master
  22. 22. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.22 Galera replication on Kubernetes • A bit more tricky • Currently unable to rely on services because only one port/IP (v1beta3 API required in 0.14.x) • A pod per node (pxc_01, pxc_02, and pxc_03) • Single container per pod • Secret sauce? • The pod configuration file passes environment variables to set root password, sst user and password • Kubectl built into container and uses RO kubernetes API to get POD IP addresses of nodes • Entrypoint script: • wsrep cluster address on pxc_01 set to gcomm:// • wsrep_cluster_address on pxc_02 and pxc_03 set to POD IP addresses gcomm://<pxc_01>,<pxc_02>, …
  23. 23. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.23 Galera on Kubernetes
  24. 24. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.24 Galera on Kubernetes Video will be provided on youtube
  25. 25. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.25 Vitess • YouTube – since 2011 • Vitess from french “Vitesse” for fast. Came about because of cat movies (remember Gearman?) • Backed by consistent data store (etcd, Chubby, Zookeeper) • Clients with simple interface to provide a view of a single instance • Lightweight connections (around 32kb) using BSON-based protocol • Row cache • SQL Parser the uses a configurable set of rules to rewrite queries • Sharding and shard management built-in: • range based, supports horizontal and vertical sharding • can accommodate your existing sharding scheme • Supports split replication stream (keyspace ID in statement-based replication stream) • Handles failover and backups • Includes a proxy to route queries to most appropriate MySQL instance • Supports transactions within a shard, plans to support cross-shard transactions using two phase commit • Web and CLI
  26. 26. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.26 Vitess components • Vtgate • Lightweight proxy • Routes traffic to the correct vttablet based on sharding scheme, required latency, and availability • Allows for simple client – client only needs to find the vtgate instance • Vttablet: • Succeeded vtocc. • Fronts actual MySQL database • Provides connection pooling, query rewriting, query de-duping • Performs management tasks initiated by vtctl • Topology server • Contains information about running servers, the sharding scheme, and the replication graph • Vtctld • Web UI • Vtctl • Command line tool for administering the cluster
  27. 27. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.27 Vitess components, topology server (cont)
  28. 28. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.28 Vitess on Kubernetes Video will be provided on youtube
  29. 29. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Questions http://patg.net
  30. 30. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Thank you! Thanks to: Google Kelsey Hightower Tim Hockin Daniel Smith Anthony Yeh #google-containers .* and #coreos .*
  31. 31. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Please attend: SCALING MYSQL IN THE CLOUD WITH VITESS AND KUBERNETES Anthony Yeh 14 April 5:15PM - 5:40PM @ Ballroom A
  32. 32. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.32 Vitess components • Topology server – used to store information about keyspaces, shards, tablets, replication graph, and serving graph. Also supports watch interface for changes on a node • Global instance • Keyspace object • sharding of keyspace • name of sharding column • How to split incoming queries • For use during resharding to change what shard is serving what inside of keyspace • Shard • Subset of data for a keyspace • The shard record contains the master tablet alias for shard • Sharding key range • Tablet types • Vschema Data • Sharding and routing information
  33. 33. © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.33 Vitess components, topology server (cont) • Topology server – used to store information about keyspaces, shards, tablets, replication graph, and serving graph. Also supports watch interface for changes on a node • Local instance (per cell) -- Tablet record contains information about both vtablet process and MySQL process. Contains tablet alias (cell + unique ID), hostname, IP, Port, Tablet type, Which keyspace/shard the tablet is part of, health map, sharding key range. Note that a tablet record is created before it is run • Replication Graph – what clients use to find what endpoints to send queries to. The objects are: • SrvKeySpace – local representation of a Keyspace containing information about what shard to use to access data • SrvShard – local representation of a Shard, details internal only to this shard • EndPoints – for all each possible serving type

×