Downloaded 36 times
Uploaded byPatrick Galbraith
Galera on kubernetes_no_video
The document discusses running a Galera cluster on Kubernetes, focusing on containerization technologies like Docker and CoreOS. It covers the essential components of Kubernetes, including its architecture and key features such as self-healing and scaling applications. Additionally, it introduces Vitess as a solution for managing MySQL in a clustered environment, showcasing its functionalities and components.
More Related Content
Similar to Galera on kubernetes_no_video
![20260201 [FOSDEM] gomodjail - library sandboxing for Go modules.pdf](https://cdn.slidesharecdn.com/ss_thumbnails/20260201fosdemgomodjail-librarysandboxingforgomodules-260201225659-76609ec4-thumbnail.jpg?width=640&height=640&fit=bounds)
Galera on kubernetes_no_video
- 1. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Galera on Kubernetes Running a Galera Cluster on a Kubernetes Cluster Patrick Galbraith Advanced Technology Group/ April 14th, 2015
- 2. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP ATG HP's Advanced Technology Group for Open Source and Cloud embraces a vision that is two steps ahead of today's solutions. We use this vision to drive product adoption and incubate technologies to advance HP. Through Open Source initiatives we foster collaboration across HP and beyond.
- 3. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. About the speaker ● Patrick Galbraith ● HP Advanced Technology Group ● Has worked at Blue Gecko, MySQL AB, Classmates, Slashdot, Cobalt Group, US Navy, K-mart ● MySQL projects: memcached UDFs, DBD::mysql, federated storage engine ● Family ● Outdoors
- 4. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4 Purpose of this talk – why are you here? Docker • Containers vs. Virtualization • Simple Docker usage • Clustered Docker CoreOS • Container optimized, stripped-down Linux distribution • Overview of core components of CoreOS – Fleet, etcd. systemd Kubernetes • A more advanced scheduler and how it works • Using Kubernetes to do work, in this case what it means to MySQL users Galera • Synchronous replication – excellent solution to clustering MySQL Vitesss • Introduction and demo
- 5. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5 What do you need? Pre-reqs • Vmware – fusion or workstation. ESXi should work too. • https://github.com/CaptTofu/mysql_replication_kubernetes.git • https://github.com/CaptTofu/kubernetes_cluster_vmware.git • Go -- brew install go (or https://golang.org/doc/install) Clients: • etcdctl: go get github.com/coreos/etcd/etcdctl • fleetctl: go get github.com/coreos/fleet/fleetctl • kubectl – – git clone https://github.com/GoogleCloudPlatform/kubernetes – make – sudo cp cmd/kubectl /usr/local/bin • vtctlclient -- go get github.com/youtube/vitess/go/cmd/vtctlclient
- 6. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6 What are containers? • Operating-system-level virtualization • Encapsulated, hermetically sealed applications • Relatively isolated • Small footprint • Fast to launch! • Portable. And did I mention, portable?! • Use of host OS and Kernel • Execution consists of time to startup application in question • LXC, Docker, Solaris Zones, BSD Jails, Parallels Virtuozzo, OpenVZ, …
- 7. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7 Containers vs. VMs
- 8. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8 Docker What is Docker? • Set of tools for managing containers • Command line tool that doubles as a daemon • Uses Linux Kernel Features: – Kernel namespaces – the core ingredient to containers working: PID, IPC ,uts (what will be seen by a group of processes), mount, network and user – Cgroups (control groups) -- limit, account and isolate resource usage (CPU, memory, disk I/O, etc.) of process groups • Originally used lxc, now defaults to Libcontainer but meant for any containerization mechanism • Much more light weight than VMs • Encapsulated application containers in a relatively isolated but lightweight operating environment • Written in Go
- 9. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.9 Docker – common terms and usage • Dockerfile • EXPOSE ports • Entrypoints and CMD • docker build • docker push • docker run • docker inspect • docker exec
- 10. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10 Dockerfile • Show example of Dockerfile and explain • Show entrypoint scripts • Environment variables passed to container
- 11. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11 Simple Docker usage - forked Video will be provided on youtube
- 12. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.12 Clustered Docker • CoreOS (https://coreos.com/) • Kubernetes (http://kubernetes.io) • Mesos + Marathon (http://mesos.apache.org/) Apache project, Zookeeper, etc • Project Atomic (http://www.projectatomic.io/) RH/Fedora/Centos designed for running Docker • Docker Openstack (https://wiki.openstack.org/wiki/Docker) Hypervisor Driver for Openstack Compute • Swarm/Compose/Machine • RancherOS (http://rancher.com/rancher-os) Minimalist Linux, Docker daemon runs as PID 1 first process the kernel starts known as “System Docker,” • Flocker (https://clusterhq.com) • Spotify Helios (https://github.com/spotify/helios) Zookeeper • Flynn (https://flynn.io/) • Deis (http://deis.io) • Maestro (https://github.com/toscanini/maestro) • Shipyard (http://shipyard-project.com) • … others to come!
- 13. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13 What is CoreOS? CoreOS • Minimalist Linux • Optimized for containers • Easy to run containers • Service discovery, container management, • Docker -- Container runtime and management, though Rocket long-term • Etcd – distributed global key value store for config data on each node • Fleet – Rudimentary Scheduler interacts with systemd and etcd • Systemd – Newer Linuxi – system and service manager for Linux • Flannel – Networking across nodes
- 14. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.14 CoreOS Cluster CoreOS CoreOS DNA Node Docker Containers etcd fleetd dockerd systemd CoreOS DNA Node Docker Containers etcd fleetd dockerd systemd CoreOS DNA Node Docker Containers etcd fleetd dockerd systemd… etcd Discovery Server etcd http://discovery.etcd.io Docker Registry Images Discovery Control Node (jump box) etcdctl fleetctl ssh 1 2 3 1. Cluster Start (etcd discovery) 2. Container Start ( fleetctl ) 3. Docker Download
- 15. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.15 What is Kubernetes? Kubernetes • “An open source system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications.” • Pre-Production Beta • Lean • Portable – will run cloud, bare metal, hybrid, etc • Extensible – using modular design allowing for plug-ability and hooks • Self-healing – auto-placement, auto-restart, auto-replication • Google engineering bring good work to the Open-source world
- 16. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.16 Master Components Kubernetes • kube-apiserver – API Server (RESTful) – primary management for cluster – reconciles etcd entries with deployed containers • kube-controllermanager — Controller Manager Server – Handle replication precesses defined by replication tasks – Writes details to etcd – Monitors changes and implements procedure to reflect the change • kube-scheduler -- Scheduler Server – Assigns workloads to specific minions in cluster taking into account service’s operating requirements and infrastructure environment • kube-register -- Register Server
- 17. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.17 Minion Components Kubernetes • Kubelet Kubelet – Communicates with the master, relaying information to/from – Reads and updates etcd – Receives work in a manifest that defines the workload and operating parameters. – Assumes responsibility for the state of work on minion • kube-proxy Kube proxy – Ensures network environment is accessible but isolated. – Makes services available externally by forwarding requests to containers. – Can perform rudimentary load balancing.
- 18. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.18 Kubernetes Terminology • Pod • Group of closely-related containers on the same host • Service • Virtual abstraction • Basic load-balancer • Single consistent access point to a pod • Replication controller • Defines pods to be horizontally scaled • Uses a label query for identifying what containers to run • Maintains specified number of replicas of a particular thing to run • Dynamic resizing • Label • Key/value tag to mark work units a part of group • Management and action targeting • Definition file – YAML/json describing a pod, service, or replication controller
- 19. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.19 Kubernetes
- 20. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.20 Kubernetes usage pattern • Pod configuration file – YAML or JSON • Service configuration file • Replication controller configuration file • export KUBERNETES_API=http://172.16.230.132:8080 • kubectl create –f mysql_master.json • kubectl create –f mysql_master_service.json • …
- 21. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.21 MySQL async replication on Kubernetes • Simple proof of concept • Master pod and service • Slave pod and service • Secret sauce? • The master pod configuration file passes environment variables to set root password, replication password • The entrypoint script runs an SQL file to grant permissions to replication user • The master service configuration is loaded and the slave pod container(s) have as an environment variable MYSQL_MASTER_SERVICE_HOST set • Slave pod when launched has environment variables for replication user and password passed and along with MYSQL_MASTER_SERVICE_HOST variable, changes the master host and user to point to that master
- 22. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.22 Galera replication on Kubernetes • A bit more tricky • Currently unable to rely on services because only one port/IP (v1beta3 API required in 0.14.x) • A pod per node (pxc_01, pxc_02, and pxc_03) • Single container per pod • Secret sauce? • The pod configuration file passes environment variables to set root password, sst user and password • Kubectl built into container and uses RO kubernetes API to get POD IP addresses of nodes • Entrypoint script: • wsrep cluster address on pxc_01 set to gcomm:// • wsrep_cluster_address on pxc_02 and pxc_03 set to POD IP addresses gcomm://<pxc_01>,<pxc_02>, …
- 23. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.23 Galera on Kubernetes
- 24. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.24 Galera on Kubernetes Video will be provided on youtube
- 25. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.25 Vitess • YouTube – since 2011 • Vitess from french “Vitesse” for fast. Came about because of cat movies (remember Gearman?) • Backed by consistent data store (etcd, Chubby, Zookeeper) • Clients with simple interface to provide a view of a single instance • Lightweight connections (around 32kb) using BSON-based protocol • Row cache • SQL Parser the uses a configurable set of rules to rewrite queries • Sharding and shard management built-in: • range based, supports horizontal and vertical sharding • can accommodate your existing sharding scheme • Supports split replication stream (keyspace ID in statement-based replication stream) • Handles failover and backups • Includes a proxy to route queries to most appropriate MySQL instance • Supports transactions within a shard, plans to support cross-shard transactions using two phase commit • Web and CLI
- 26. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.26 Vitess components • Vtgate • Lightweight proxy • Routes traffic to the correct vttablet based on sharding scheme, required latency, and availability • Allows for simple client – client only needs to find the vtgate instance • Vttablet: • Succeeded vtocc. • Fronts actual MySQL database • Provides connection pooling, query rewriting, query de-duping • Performs management tasks initiated by vtctl • Topology server • Contains information about running servers, the sharding scheme, and the replication graph • Vtctld • Web UI • Vtctl • Command line tool for administering the cluster
- 27. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.27 Vitess components, topology server (cont)
- 28. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.28 Vitess on Kubernetes Video will be provided on youtube
- 29. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Questions http://patg.net
- 30. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Thank you! Thanks to: Google Kelsey Hightower Tim Hockin Daniel Smith Anthony Yeh #google-containers .* and #coreos .*
- 31. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Please attend: SCALING MYSQL IN THE CLOUD WITH VITESS AND KUBERNETES Anthony Yeh 14 April 5:15PM - 5:40PM @ Ballroom A
- 32. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.32 Vitess components • Topology server – used to store information about keyspaces, shards, tablets, replication graph, and serving graph. Also supports watch interface for changes on a node • Global instance • Keyspace object • sharding of keyspace • name of sharding column • How to split incoming queries • For use during resharding to change what shard is serving what inside of keyspace • Shard • Subset of data for a keyspace • The shard record contains the master tablet alias for shard • Sharding key range • Tablet types • Vschema Data • Sharding and routing information
- 33. © Copyright 2015Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.33 Vitess components, topology server (cont) • Topology server – used to store information about keyspaces, shards, tablets, replication graph, and serving graph. Also supports watch interface for changes on a node • Local instance (per cell) -- Tablet record contains information about both vtablet process and MySQL process. Contains tablet alias (cell + unique ID), hostname, IP, Port, Tablet type, Which keyspace/shard the tablet is part of, health map, sharding key range. Note that a tablet record is created before it is run • Replication Graph – what clients use to find what endpoints to send queries to. The objects are: • SrvKeySpace – local representation of a Keyspace containing information about what shard to use to access data • SrvShard – local representation of a Shard, details internal only to this shard • EndPoints – for all each possible serving type
Editor's Notes
- #14 Using coreos as a container-optimized VM
- #15 Cluster Start (etcd discovery) Container Start (fleetctl) Docker Image download.
- #16 Cluster management Scheduler is cognizant of where to place containers , spreading across multiple machines to avoid SPF, resource isolation (based on cores, free cores…) declarative high-level abstract Everything at Google runs in a container. 2 Billion containers a week! 3300 containers a second,
- #26 Performance Connection pooling - Scale front-end connections while optimizing MySQL performance. Query de-duping – Reuse results of an in-flight query for any identical requests received while the in-flight query was still executing. Transaction manager – Limit number of concurrent transactions and manage deadlines to optimize overall throughput. Rowcache – Maintain a row-based cache (using memcached) to more efficiently field queries that require random access by primary key, very useful for OLTP workloads. (The MySQL buffer cache is optimized for range scans over indices and tables.). This can replace a custom caching layer implementation at the application layer. Protection Query rewriting and sanitation – Add limits and avoid non-deterministic updates. Query blacklisting – Customize rules to prevent potentially problematic queries from hitting your database. Query killer – Terminate queries that take too long to return data. Table ACLs – Specify access control lists (ACLs) for tables based on the connected user. Monitoring Performance analysis: Tools let you monitor, diagnose, and analyze your database performance. Query streaming – Use a list of incoming queries to serve OLAP workloads. Update stream – A server streams the list of rows changing in the database, which can be used as a mechanism to propagate changes to other data stores. Topology Management Tools Master management tools (handles reparenting) Web-based management GUI Designed to work in multiple data centers / regions Sharding Virtually seamless dynamic re-sharding Vertical and Horizontal sharding support Built-in range-based, or application-defined sharding support