Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Customer Experience: InnoDB Cluster Implementation by PR Karthik

86 views

Published on

PR Karthik sharing challenges faced while migrating data to the Open Source Platform and the solutions his team employed to address the issues.

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Customer Experience: InnoDB Cluster Implementation by PR Karthik

  1. 1. Customer Experience InnoDB Cluster Implementation Presenter Karthik P R Founder Mydbops www.mydbops.com info@mydbops.com
  2. 2. About Mydbops ● Founded in 2015, HQ in Bangalore India with 150+ customer base across the globe. ● Mydbops is on Database Consulting with core specialization on MySQL and MongoDB Administration and Support. ● We have expert team with 20+ certified DBA’s providing full time support and currently managing 300+ servers. ● Mydbops was created with a motto of developing a DevOPS model for Database administration offering 24*7 expert remote DBA support. ● We help organisations to architect and scale systems in MySQL/Mongo by implementing the advanced technologies in industry which are completely open source.
  3. 3. Mydbops is into MySQL/MongoDB Support and Consulting. It is founded by experts who have scaled database at Yahoo! ,Percona and Datavail. We are providing an expert level support and 24*7 monitoring for MySQL databases and its related technologies like MariaDB , Percona ( also clustering ) . We support modern database technologies in MySQL which includes Galera ( Clustering ), Group Replication , SQL aware Load balancers like Maxscale / ProxySQL. About Mydbops
  4. 4. mysqlsupport@mydbops.com Our Clients www.mydbops.com
  5. 5. CEO / DB Architect About Me
  6. 6. 1. Introduction to InnoDB Cluster 2. Project Background 3. Challenges faced 4. Future Enhancements planned Table of Contents
  7. 7. ● Group Replication Library ● Virtually Synchronous replication (MySQL 5.7 ) ● GCS is based on Paxos ( Variant of Mencius ) ● High Availability solution on InnoDB ● Consistent data across nodes ● Shared Nothing Cluster ● Supported on all Platforms ● Inbuilt in MySQL ( No third party plugin ) Introduction : InnoDB Cluster
  8. 8. Introduction : InnoDB Cluster MySQL Router MySQL Shell MySQL Group Replication
  9. 9. Transaction in Group Replication Transaction in Group Replication
  10. 10. 1. Introduction to InnoDB Cluster 2. Project Background 3. Challenges faced 4. Future Enhancements planned Table of Contents
  11. 11. A largest retail store in USA makes a migration to Open source Platform. ● Migration to Cloud ( Google Cloud ) ● Monolithic to Microservice Platform ● Application servers on Containers ( Dockers ) ● MySQL and MongoDB as backend data stores. ● Redis as a caching layer ● Consul for service discovery and Ansible for Config management. Project Background
  12. 12. MySQL Info ● Automated Deployment with MySQL Shell ( In GCC) ● MySQL Enterprise servers. ● Virtual Environment ● Single Primary Mode cluster ● MySQL Version 5.7.17 ( started with ) ● No read / write Split up Project Background
  13. 13. 1. Introduction to InnoDB Cluster 2. Project Background 3. Challenges faced 4. Future Enhancements planned Table of Contents
  14. 14. Case 1) Heavy Data loading. Initial Loading of CSV files of heavy size breaks the cluster and rollback the transaction. Becomes our first blocker as data validating can be made once the data is loaded. Challenges Faced
  15. 15. Solutions Proposed. ● Stop the group replication reset the Master GTID’s and make the sync back in async mode ( normal replication ) ● Use a hot backup solution like MySQL Enterprise backup and rebuilt it. The Solution 1 was comparatively faster in comparing the data set we had. Challenges Faced
  16. 16. Case 2 ) Setting up HA for MySQL Router. MySQL router can become a single point of failure Challenges Faced
  17. 17. Solutions Proposed. Implement Router as a layer with docker containers as a part of application deployment and maintain the config repository based on microservices. Challenges Faced MR MR MR MR MR
  18. 18. Case 3: MySQL bugs affected the Stress tests MySQL Version : 5.7.17 Replication: In previous versions issuing STOP GROUP_REPLICATION stopped the plugin but the server still accepted transactions. This meant the transactions were not transmitted to the group. To make STOP GROUP_REPLICATION safer, now super_read_only is set to ON immediately upon issuingSTOP GROUP_REPLICATION, which ensures no transactions are accepted. (Bug #25495067, Bug #84795) Replication: Interleaved transactions could sometimes deadlock the slave applier when the transaction isolation level was set to REPEATABLE-READ. (Bug #25040331) Challenges Faced
  19. 19. Solutions Proposed. MySQL Version : Those bugs were fixed in MySQL 5.7.20 and MySQL 5.7.19 respectively. We made MySQL 5.7.20 as a standard in our repository for production. Note : Group_replication_transaction_size_limit was added in MySQL 5.7.19 Challenges Faced
  20. 20. Case 4: Isolation Level and Open transaction Repeatable read can be performance killer in a cluster ( optimistic locking ). And open transactions from application or users make it much worse. Solution Proposed : Read Committed was made as a standard isolation level and open transaction alerts based on performance_schema was set to find application code bugs or user level Challenges Faced
  21. 21. Case 5: Schema changes The heavy DDL can block the parallelism in the cluster and can make the secondary to RECOVERY state.Even INNODB Online alter can't help much as it needs a table rebuild. Solution Proposed : Third party Tools can be saviour - Percona toolkit or Gh-OST ( can be ) Challenges Faced
  22. 22. Case 6 : InnoDB cluster Can’t rejoin . When the node is rebuild using MEB , the Group Replication is running fine. But the InnoDB cluster can’t find the right UUID and marks the cluster as failed node. Can be validated via MySQL Shell select * from performance_schema.replication_group_members; +---------------------------+--------------------------------------+-------------+-------------+--------------+ | CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | +---------------------------+--------------------------------------+-------------+-------------+--------------+ | group_replication_applier | b0b1603f-83ef-11e6-85a6-080027de0e0e | box01 | 3306 | ONLINE | | group_replication_applier | bb29750c-83ef-11e6-8b4f-080027de0e0e | box02 | 3306 | ONLINE | | group_replication_applier | bbu3761b-83ef-11e6-894c-080027de0t0e | box03 | 3306 | ONLINE | +---------------------------+--------------------------------------+-------------+-------------+--------------+ Challenges Faced
  23. 23. mysql> show tables; +-----------------------------------------+ | Tables_in_mysql_innodb_cluster_metadata | +-----------------------------------------+ | clusters | | hosts | | instances | | replicasets | | schema_version | +-----------------------------------------+ 5 rows in set (0,00 sec) Solution and fix ( Temporary Patch ). The cluster metadata table has to be updated manually post a rebuild if there is a mismatch in UUID. Challenges Faced
  24. 24. 1. Data loading 2. Setting up HA for MySQL Router. 3. MySQL and Router Version. 4. Isolation level and open transaction 5. Handling the DDL’s 6. DB Recovery and InnoDB Cluster. Challenges Faced
  25. 25. 1. Introduction to InnoDB Cluster 2. Project Background 3. Challenges faced 4. Future Enhancements planned Table of Contents
  26. 26. ● Read Write Split up from App / ProxySQL ● Planning to Upgrade MySQL 5.7.22 or MySQL 8.0 ● Migration to Bare Metal server. ● Single Click Upgrades on Ansible Future Enhancement
  27. 27. Community Sponsorship
  28. 28. Thank You !!!

×