Kubecon - Democratizing my sql_ cloud managed to k8s managed (1)
•Download as PPTX, PDF•
1 like•1,423 views
This document summarizes a presentation about Presslabs migrating their MySQL database from cloud-managed to Kubernetes-managed using their own MySQL Operator. Some key points: - Presslabs built a MySQL Operator for Kubernetes to manage MySQL clusters with replication and provide high availability, automated backups, and other features. - Migrating to Kubernetes-managed MySQL addressed needs for ease of operations, elasticity, availability, data safety, and observability. - Challenges in the migration included integrating with their Orchestrator, cleaning up persistent volume claims, upgrading the operator, and handling MySQL upgrades. - Future plans include adding more features to the operator and increasing community contributions and support.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Kubecon - Democratizing my sql_ cloud managed to k8s managed (1)
Similar to Kubecon - Democratizing my sql_ cloud managed to k8s managed (1) (20)
Recently uploaded
Recently uploaded (20)
Kubecon - Democratizing my sql_ cloud managed to k8s managed (1)
- 1. Democratizing MySQL: Cloud Managed to Kubernetes Managed KubeCon 2019
- 2. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs Sachin Manpathak Technical Lead @Platform9 Flavius Mecea Project Lead @Presslabs Case study Migrate from Cloud Managed SQL to K8S managed The story of building Presslabs Operator for MySQL 00. Who we are
- 3. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs Context Needs Solution Challenges Future plans 01. 02. 03. 04. 05.
- 4. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs Context01.
- 5. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs WordPress development agency Managed WordPress hosting Record: 2,2 BN pageviews in a month Top tier in Enterprise hosting Open-Source Stack 2007 2011 2013 2015 2018 Who is Presslabs01.1
- 6. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs Democratizing WordPress hosting infrastructure Presslabs mission01.2
- 7. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs Presslabs objectives01.3 MySQL Operator for WordPress hosting Open infrastructure using Kubernetes to run and operate WordPress
- 8. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs ➔ Runs everywhere ➔ Open-source ➔ We had experience with containers before they were cool ➔ Our core services already run on Kubernetes since version 1.7 ➔ Support for a lot of integrations Why Kubernetes?01.4
- 9. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs Needs02.
- 10. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs Ease of operations Needs02. 1 2 3 4 5 Elasticity Service availability Data safety Observable
- 11. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs Solution03.
- 12. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs Existing solutions03.1 ↑ Increased Costs ✓ Oracle MySQL Operator ✓ Percona XtraDB Operator
- 13. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs ✓ Self-healing clusters ✓ Highly available reads ✓ Virtually highly available writes ✓ Replication lag detection and mitigation ✓ Resource abuse control ✓ Automated backups and restores MySQL Operator03.2 A Kubernetes Operator for managing MySQL Clusters with asynchronous or semi-synchronous replication:
- 14. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs ➔ Control plane ◆ Operator ◆ Orchestrator ➔ Data plane ◆ MySQL deployment ➔ Monitoring ◆ Prometheus Architecture overview03.3
- 15. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs ➔ Statefulset ➔ Persistent volumes ➔ Services ◆ Master ◆ Healthy nodes MySQL Cluster03.4
- 16. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs ➔ Init: ◆ MySQL configuration ➔ Main: ◆ Percona Server for MySQL ➔ Sidecar: ◆ Lag Detection and Monitoring ◆ Resource abuse control ◆ Backups and Initializations MySQL Node03.5
- 17. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs Challenges04. Orchestrator integration 1 2 3 4 PVC clean-up Operator upgrade/ deploy MySQL Upgrade
- 18. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs ➔ Orchestrator is a MySQL high availability and replication management tool ➔ State reconciliation between Orchestrator and Kubernetes Orchestrator integration04.1
- 19. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs ➔ MySQL nodes keeps data into Persistent Volumes ➔ Scale down does not delete PVCs ➔ Scale up may be an issue because of obsolete data ➔ Delete PVC at scale down ➔ Special case for Node 0 PVC clean-up04.2
- 20. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs ➔ Helm the standard for packaging apps in Kubernetes ➔ CRD management is painful ➔ No CRD validation Operator upgrades / deployment04.3 helm.sh/crd-install
- 21. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs ➔ Default policy: MySQL Upgrade04.4 Rolling Updates
- 22. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs MySQL Upgrade04.4 Rolling Updates ➔ Default policy:
- 23. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs MySQL Upgrade04.4 ➔ Default policy: ➔ Not gentle for MySQL ➔ Master should be the last one standing Rolling Updates
- 24. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs MySQL Upgrade ✘ Pod finalizer ✘ Container lifecycle hooks 04.4 ➔ Default policy: ➔ Not gentle for MySQL ➔ Master should be the last one standing ➔ Recommended policy: Rolling Updates On Delete
- 25. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs Current status05.
- 26. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs ➔ Integration with Google Cloud Marketplace, OperatorHub.io, AWS Marketplace ➔ CRD validation and webhooks ➔ Multiple backup policies ➔ Proxy SQL integration Future Plans05.1
- 27. Democratizing MySQL: Cloud Managed to K8S Managed @presslabs Community Contributors Project status05.2 github.com/presslabs/mysql-operator #mysql-operator https://kubernetes.slack.com/ Presslabs HeurekaPlatform 9 KinvolkAgri Terra
- 28. Democratizing MySQL: Cloud Managed to Kubernetes Managed KubeCon 2019
- 29. Democratizing MySQL: Cloud Managed to K8S Managed @Platform9Sys Who we are
- 30. Democratizing MySQL: Cloud Managed to K8S Managed @Platform9Sys ● Customer base growth == Substantial increase in public cloud costs ● At ~300 cloud regions, just RDS bill amounted to 10s of thousands Problem of Scale
- 31. Democratizing MySQL: Cloud Managed to K8S Managed @Platform9Sys Infra evolution 2014-2016 2017-2018 2019-2020 AWS for compute + RDS Private cloud for compute + RDS Private cloud for compute + DB
- 32. Democratizing MySQL: Cloud Managed to K8S Managed @Platform9Sys ● Automation set up to use of public self service API ● Reliance on RDS snapshots, performance charts and alerting ● No MySQL expertise ● Required Comparable performance ● Needed drop-in replacement for MySQL to minimize impact Hurdles to DBaaS
- 33. Democratizing MySQL: Cloud Managed to K8S Managed @Platform9Sys Requirement Mysql-Operator Featureset Simple, self service, open API K8s CRD implementation Drop in replacement: MySQL Percona: 100% compatible Automated backups, API driven recovery Scheduled Backups to S3 High Availability & Failover Replica support with automated failover with Orchestrator Open Source Yes Built-in monitoring Yes: Prometheus metrics Searching for DBaaS
- 34. Democratizing MySQL: Cloud Managed to K8S Managed @Platform9Sys On-prem Architecture Openstack Cloud AZ1 AZ2 AZ3 Storage K8s-Master1 K8s-Master2 K8s-Master3
- 35. Democratizing MySQL: Cloud Managed to K8S Managed @Platform9Sys The Rollout Current State ● ~10 accounts using MySQL managed on K8s by the operator ● 3 managed multi-master K8s clusters: Dev, Stage and Prod ● Automated failover with 3-AZ deployment Plan: ● 100% deployments managed with MySQL operator ● Standardize on Operator Paradigm: Prometheus Monitoring, Log collection, etc.
- 36. Thank You!
Editor's Notes
- Hello everyone, I’m happy to be here with you. Thank you for joining our session on Democratizing MySQL Cloud Managed to K8S Managed.
- My name is Flavius Mecea and I will talk about... This is a joint session with Platform 9’s Sachin Manpathak. Unfortunately he is not present today here, because some visa issues. But he prepared a video for you and his colleague, Daniel, is here to answer your questions. He will talk about a case study on migrating from cloud managed to k8s managed Sa-chin Man-Pa-tak
- First of all, I want to start by presenting the MSQL operator, I will go through this 5 main topics: Context in which the operator was born The needs that we had in mind when we first started building With what needs we started in mind What we have achieved, the operator overview Also, some challenges that we encountered during development And the project status and future plans
- First, let me introduce the company that I work for, Presslabs and the context in which the operator was born.
- We are a managed WordPress hosting company, doing business for more than 10 years We started as a WP dev agency, then we pivoted towards the hosting business. After serving both publishers and Enterprise clients for several years, we came to realize that all the companies (including us) in the global top Enterprise tier were doing the same thing. That’s when we started thinking about the Stack; an open-source infrastructure that could become the standard in WP hosting.
- It’s not just about the stack. It’s our commitment to our mission to democratize WP hosting infrastructure, to share our accumulated knowledge.
- As part of our mission, we have 2 key objectives: We are currently building an open infrastructure using Kubernetes to run and operate WordPress, named: Presslabs Stack The other one is building the MySQL Operator; Because half of WordPress hosting is about MySQL
- Runs everywhere (dev’s laptop, public clouds, private data centers) Open-source We had experience with containers before they were cool Our core services already run on K8s since version 1.7 It already offers support for a lot of integrations: cert manager, nginx(Ingress), Prometheus(monitoring)
- As part of the Stack we needed a way to automate certain operations such as: deploying, scaling, maintaining and backing-up MySQL
- For that: We’ve identified some key requirements (for the infrastructure), to focus on: First we wanted something that is easy to operate, that doesn’t get in our way Second, we needed an elastic service, to help us scale with the demand. You all know that in hosting, service-uptime is paramount, so we had(have) to maximize the availability of our service Also, no one wants to lose data, especially when it comes to someone else’s data And in order to reliably operate the service(system) we need a method to observe what’s happening from top, down to the request level.
- With this in mind we checked some of the available solutions and concluded that they were not suitable for us.
- For example, both Oracle and Percona operators perform group replication — (which implies that ) they required more nodes to operate(at least 3), (which is not suitable for us because) it increases costs. As great engineers do, we’ve ended up building a solution ourselves.
- In the past 10 year, we’ve identified several must-have features, which have been integrated into the Operator,(to fulfill our basic needs) such as: Self-healing clusters - (w/o this feature the operator doesn’t make sense) - the operator has to continuously reconcile and solve replication issues Highly available reads - when more nodes are available Virtually highly available writes - that provides us minimum downtime due to fast failovers Replication lag detection and mitigation - takes lagging nodes out of rotation when lag is above a set threshold or in case of unhealthy nodes Resource abuse control - (which is useful) to limit noisy queries (that may slow down the cluster) Automated backups and restores - this one speaks for itself. All of these features have proven to be very helpful compared with our old setup.
- Now let’s move on to the practical aspects of building the operator. In this figure is presented the entire system overview. The architecture is split into 3 main parts: control plane, data plane and monitoring The control plane consists of the operator and its components, which are deployed using helm, usually in a dedicated namespace. Here we have: The controller itself The Orchestrator, a MySQL high availability and replication management tool (I will come back to it later) The data plane represents a MySQL deployment, made of basic k8s resources (like pods, services, etc) which can be spread across multiple namespaces. And last but not least we have monitoring which is performed by Prometheus, the standard k8s monitoring system.
- Going deeper into the dataplane, we can see that the MySQL cluster has multiple components: Statefulset - that represents the main resource, which provisions the pods and the PVs for each MySQL node. Also there are 2 services for each cluster: Master service - that always points to the master MySQL node Healthy nodes service - that points to all the pods that are considered healthy by the operator The selections are made based on K8S labels, which are set by the operator based on information gathered from Orchestrator Your application will interact with those two services for writes and for reads (and it’s the application’s responsibility to split them, by using app specific logic or by using some dedicated software like proxySQL)
- Internally, a node consists of several components: Init containers: for MySQL initialization and configuration A main container: which is the Percona Server for MySQL. We chose Percona because it’s battle tested in enterprise environments and a MySQL drop-in replacement. Sidecar containers: Some of them are based on Percona toolkit which are responsible for several actions: lag detection, MySQL monitoring and resource limit policy enforcement There is an extra container that provides an endpoint for node initialization or for backups.
- I want to mention some specific challenges we’ve had during the implementation of this operator such as: Orchestrator integration - how do we integrate Orchestrator, a third-party tool, so we don’t have to reinvent the wheel? PV clean-up -we have to manage PVs ourselves because the way K8S manages PVs is not suitable for MySQL (later we’ll see why) Operators upgrades - which is a common problem for operators because helm provides very modest CRDs support MySQL Upgrades - this is a specific problem, because usually it’s done by humans and it’s difficult to automate # is a difficult operation especially when it comes to k8s Let me start by presenting the Orchestrator integration.
- Orchestrator is a subcomponent of the entire operator and it’s the tool that handles MySQL topology and failovers ... but it’s not meant to be stateless, as operators usually are. K8s keeps a state, also Orchestrator keeps a state - the operator doesn’t know which one to listen to. ...I’m talking about an information flow conflict. To fix this, we chose to implement a reconciliation loop between Orchestrator and K8S which, at every few seconds reconciles the state between the 2. On one hand, the Orchestrator is responsible for updating replication topology(in emergency situations) and to observe the current status of the MySQL cluster. On the other hand, the Operator reconciles the desire replication topology into Orchestrator and provides service discovery. Even if the Orchestrator data is lost, the operator is able to restore all the data to Orchestrator. As a conclusion, the operator (has to) take decisions based only on the information found in k8s which is up-to-date, thanks to the reconciliation loop.
- Another challenge was how k8s manages PVs. The MySQL data is being stored in PVs, managed by the statefulset. But this implies that, when a cluster is scaled down, the volume is not deleted, so after a while the data might become obsolete, and when the statefulset is scaled up again the replication can fail. To fix this, we’ve implemented a cleaner that deletes the PVC when the cluster is scaled down, except for node 0 which is special case and the data should be kept as long as the cluster exists, to avoid losing cluster data.
- A common problem in the world of operators is CRD management. Currently the defacto standard for packaging application is Helm. If you are a helm user you probably know that CRD management is still very painful, because Helm does not provides an upgrade path for CRDs. What is more MySQL Operator is still in development and CRDs specifications are still subject to change. This made us to install CRDs without validation, to minimize user intervention at upgrades. However we hope that this is a temporary solution until Helm improves its support for managing CRDs.
- A specific challenge for this operator is how MySQL upgrades are performed. K8S already provides some upgrade policies, like: Rolling updates (update policy)- which is not exactly gentle with MySQL, as it can choose to upgrade the master first, which forces a failover to the replica. Then, when the replica is updated will triggers another failover, which is unnecessary and can be avoided if the master is the last one to be updated. That’s why the master should be the last one standing to avoid failover flip-flop, or downtime A contributor came up with an idea to use: On Delete policy, which fits better our needs because the operator can choose which pod to update. Therefore we can control the order in which the pods are upgraded. We tried to use other techniques, as well, like pod finalizers, to block pod deletion until the failover is done. But we hit a dead end because we misunderstood how k8s finalizers work. Using containers lifecycle hooks to trigger a failover was proven to be too complicated So we chose to implement ‘On Delete’ policy which is still work in progress
- A specific challenge for this operator is how MySQL upgrades are performed. K8S already provides some upgrade policies, like: Rolling updates (update policy)- which is not exactly gentle with MySQL, as it can choose to upgrade the master first, which forces a failover to the replica. Then, when the replica is updated will triggers another failover, which is unnecessary. That’s why the master should be the last one standing to avoid failover flip-flop, or downtime A contributor came up with an idea to use: On Delete policy, which fits better our needs because the operator can choose which pod to update. Therefore we can control the order in which the pods are upgraded. We tried to use other techniques, as well, like pod finalizers, to block pod deletion until the failover is done. But we hit a dead end because we misunderstood how k8s finalizers work. Using containers lifecycle hooks to trigger a failover was proven to be too complicated So we chose to implement ‘On Delete’ policy which is still work in progress
- A specific challenge for this operator is how MySQL upgrades are performed. K8S already provides some upgrade policies, like: Rolling updates (update policy)- which is not exactly gentle with MySQL, as it can choose to upgrade the master first, which forces a failover to the replica. Then, when the replica is updated will triggers another failover, which is unnecessary. ...and can be avoided if the master is the last one that is updated. A contributor came up with an idea to use: On Delete policy, which fits better our needs because the operator can choose which pod to update. Therefore we can control the order in which the pods are upgraded. We tried to use other techniques, as well, like pod finalizers, to block pod deletion until the failover is done. But we hit a dead end because we misunderstood how k8s finalizers work. Using containers lifecycle hooks to trigger a failover was proven to be too complicated So we chose to implement ‘On Delete’ policy which is still work in progress
- A specific challenge for this operator is how MySQL upgrades are performed. K8S already provides some upgrade policies, like: Rolling updates (update policy)- which is not exactly gentle with MySQL, as it can choose to upgrade the master first, which forces a failover to the replica. Then, when the replica is updated will triggers another failover, which is unnecessary. That’s why the master should be the last one standing to avoid failover flip-flop, or downtime A contributor came up with an idea to use: On Delete policy, which fits better our needs because the operator can choose which pod to update. Therefore we can control the order in which the pods are upgraded. We tried to use other techniques, as well, like pod finalizers, to block pod deletion until the failover is done. But we hit a dead end because we misunderstood how k8s finalizers work. Using containers lifecycle hooks to trigger a failover was proven to be too complicated So we chose to implement ‘On Delete’ policy which is still work in progress
- Now, I want to share with you the current status of the project and future plans.
- Integration with Marketplaces - like Google Cloud Marketplace, OperatorHub, AWS Marketplace - to make easier for end users to install it We would like to finish what we’ve started, so we would add CRD validation and webhooks Multiple backup policies - for granular control over backups To make it easy for your application to connect to the cluster we want to integrate ProxySQL, instead of using, that 2 services, the app can connect only to ProxySQL, which will do the routing for you.
- The Operator is still in alpha version and we’re really close to beta. We have a good feedback from the community, and some major platforms actively use and contribute to the Operator. I would like to invite you to visit the project page on Github and for any question to join the #mysql-operator slack channel. Coming up next, my co-presenter Daniel will continue with second part of the presentation. Let’s encourage him with a round of applause, thank you for your time.
- Customer base growth == Substantial increase in public cloud costs At ~300 cloud regions, just RDS bill amounted to 10s of thousands
- Prototype: MySQL as backend on managed on-prem Kubernetes & storage Lessons: Multi-master kubernetes is essential Storage story is still developing (as of v1.10), better (as of v1.13) MySQL backups save lives!