Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

High performance and high availability proxies for MySQL


Published on

This is presented at LSPE meet up on 17-06-2017.
ProxySQL and MaxScale are the next-generation database proxy that manages security, scalability, high availability and data streaming in scale-out deployments

ProxySQL and MaxScale have been most widely adapted in industry these days. They can be used in smaller platforms to bigger platforms. The Objective of this talk is to make awareness about the next-gen database proxies and their value addition.

Published in: Technology

High performance and high availability proxies for MySQL

  1. 1. Presenters Karthik P R ( CEO/ Founder Mydbops ) Kabilesh P R ( Database Consultant ) High Performance and High Availability Proxies for MySQL
  2. 2. Mydbops is into MySQL Support and Consulting. It is founded by experts who have scaled MySQL database at Yahoo! ,Percona and Datavail. We are providing an expert level support and monitoring for MySQL databases and its related technologies like MariaDB , Percona ( also clustering ) . We support modern database technologies in MySQL which includes Galera ( Clustering ), Group Replication , SQL aware Load balancers like Maxscale / ProxySQL, MySQL Engine like InnoDB, TokuDB and MyRocks. We offer seamless end to end DB solutions like migration, installation, up-gradation, design, performance tuning, troubleshooting, monitoring, recovery with 24x7 on call support and Remote DBA support inevitably decreasing the downtime by 99.9%. Our services will improve your DB Performance and helps in saving cost on IT. About Mydbops
  3. 3. Our Clients
  4. 4. Karthik P R Founder of Mydbops. Has an experience of around 7 years in handling MySQL servers . Handled large scale systems at Yahoo! ( 3 years ). Active MySQL speakers and blogger. Currently providing solutions to many established startups and organisation to scale in MySQL. Email : Twitter: @prkart Kabilesh P R Lead Consultant in Mydbops. Has an experience of about 5 years. Worked with large remote database support company Datavail on multi customer platform. Expertise in monitoring of MySQL and actively working with mysql on cloud About US
  5. 5. ● Need for Load Balancer ● SQL Awareness Load Balancers ● Introduction to Maxscale ● Maxscale use cases ● Introduction to ProxySQL ● ProxySQL use cases ● Maxscale Vs ProxySQL Agenda
  6. 6. ● Load balancer distributes the incoming traffic efficiently to underlying backend servers ● Provides efficient use of the underlying hardware ● Avoids single point of failures ie., High Availability ● Provides way for easy switchover and failover ● Additional Features Need for a Load balancer
  7. 7. Types Of Load Balancers ● Reverse Proxies ○ HAProxy ○ Ngnix ● SQL aware Proxies ○ Maxscale ○ MySQL Router ○ ProxySQL Need for a Load balancer
  8. 8. ● SQL aware load balancer, forward the queries/statements by understanding the underlying state of servers. ● Read/Write splitting. ● Support for external tools and scripts. ● Monitoring of backend servers. ● Query Caching. ● Data Streaming SQL Aware Load Balancer
  9. 9. Maxscale
  10. 10. ● Maxscale is an intelligent SQL Aware proxy. ● Built by MariaDB corporation ● Pluggable router ● Easy to configure ● Highly Scalable and flexible ● Provides High Availability Introduction To Maxscale
  11. 11. ● Maxscale comes with two license model they are GPL (Open Source License) and BSL (Business Source License). ● Till version 1.4 Maxscale comes with GPL ● From version 2.0 Maxscale comes with BSL ● Usage Limitation of BSL : The usage of software is free for when you are using total or less than of 3 server for production purposes. ● Change Date of BSL : 2019-01-01 “Business source is simply proprietary software released under a Microsoft-esque shared source license that magically becomes fully open source after a period of time.” License In Maxscale
  12. 12. ● Maxscale Comes with a simple pluggable modular architecture ● Modules are loaded dynamically on demand ● The maxscale core provides the following Network layer interface Buffer management Config Management Synchronization Maxscale Architecture
  13. 13. Maxscale Architecture
  14. 14. Maxscale Internal
  15. 15. ● Maxscale support the below routing algorithms for various use cases ○ ReadConnRoute - Connection based load balancing ○ ReadWriteSplit - Read/Write aware statement based routing ○ SchemaRouter - Simple Sharding based on database level ○ Binlogrouter - As Binlog server Maxscale Routers
  16. 16. ● Maxscale monitor module monitors the underlying cluster/server and provides the status information to the router based on which the query is routed to the server, the various supported monitor modules include ○ MySQL Monitor(mysqlmon) - Monitors M-S replication and assign roles ○ Galera Monitor(galeramon) - Monitors the SYNC status of the each cluster node ○ NDBCluster Monitor(ndbclustermon) - Monitors the NDB cluster ○ Multi-Master Monitor(mmmon) - monitors the multi-master replication cluster. Maxscale Monitor
  17. 17. ● Configuration file to store the maxscale configurations (/etc/maxscale.cnf) default. ● Similar interface like mysql cmd line ● Has query parser similar to mysql server ● Default port for maxscaled 6603 ● Default password is ‘mariadb’ for maxadmin (admin cmd line) Maxscale Configuration
  18. 18. Maxscale Configuration [MaxScale] threads=auto [Service Name] type=service router= servers=server1,server2 user= passwd= routeroptions= [listener] type=listener service=Service Name protocol=MySQLClient port=3306 socket=/tmp/cluster [monitorName] type=momitor module= servers=server1,server2 user= passwd= [server1] type=server address= port=3306 [CLI type=service router=cli [CLI Listener] type=listener service=CLI protocol=maxscaled address=localhost port=6603
  19. 19. Maxscale For Replication Writes Reads
  20. 20. Maxscale For Galera Cluster Writes Reads Reads
  21. 21. Maxscale As Binlog Router
  22. 22. Maxscale As Schema Router
  23. 23. Maxscale As Data Streamer
  24. 24. ● Compression is not supported in mysql server handshake ● With R/W read queries routed to master when they are executed as transaction, in case of prepared statement, procedure call and multiple statement in one query. ● Maxscale cannot manage authentication with % as hostname, the only accepted wildcard will be with IP as ‘192.167.2.%’ ● Has some limitation with client session variable Maxscale Limitations
  25. 25. ● From 2.1 maxscale accepts runtime changes to most of the config ● Support for Aurora Cluster (AWS) ● Selective data Masking ● Secure binlog server using SSL ● Great improvements with query caching. ● Inbuild failover management for promoting slave ( no more external scripts ) Maxscale Advanced Features
  26. 26. ProxySQL
  27. 27. ● Built for DBA’s by a DBA. ● Major Contribution by René Cannaò (Founder) ● Winner MySQL Community Contributor (2017) ● Light weight application. ● Advanced Multi Core Architecture. ● Complete Open Source GPL License (GPL 3.0) ● Current Version 1.3.7 and 1.4 is Beta. ( 17-06-2017 ) ProxySQL Introduction
  28. 28. ● More Granular Configuration ● SQL based Administration. ● Dynamic Configuration ● Advanced Metric Collections. ● Supports More Complex Topologies. ● Seamless Integrations. ● Manage database at large scale ( 1000’s ) ProxySQL Introduction
  29. 29. ProxySQL Architecture
  30. 30. ● SQLite database to store config. ● MySQL Compatible interface. ● There is no real parser in ProxySQL ● Multi Layer Configuration ● Default port 6032 $mysql -u admin -padmin -h -P6032 ProxySQL Configuration
  31. 31. ● SQLite database to store config. ● MySQL Compatible interface. ● There is no real parser in ProxySQL ● Multi Layer Configuration ● Default port 6032 $mysql -u admin -padmin -h -P6032 ProxySQL Configuration
  32. 32. Memory: Represent SQLite3 in memory database Runtime: In memory store of ProxySQL with RUNTIME Config. Disk: On Disk location of SQLite3. Contains Persistent storage of config. Config File: Load the config (proxysql.cnf) when there is no disk file ( proxysql.db) ProxySQL Configuration
  33. 33. Loading configuration three levels: SQL (writes ) are made to table in Memory. Then load to RUNTIME ( Dynamically ) and saved to DISK for persistence. Eg) Load MYSQL USERS TO RUNTIME; SAVE MYSQL USERS TO DISK; ProxySQL Configuration
  34. 34. Scenario Loading a User Host groups 101 - Writer , 102 -Reader insert into mysql_users(username,password,active,default_hostgroup,default_schema) values ('writer_app','test',1,101,'sbtest'); insert into mysql_users(username,password,active,default_hostgroup,default_schema) values ('reader_app','test',1,102,'sbtest'); LOAD MYSQL USERS TO RUNTIME; SAVE MYSQL USERS TO DISK; ProxySQL Configuration
  35. 35. ProxySQL Replication ( case 1 )
  36. 36. Loading Server Info: INSERT INTO mysql_servers ( hostname,hostgroup_id,port,weight) VALUES ('',101,3306,1000); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES ('',102,3306,10000,2); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES ('',102,3306,10000,2); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES ('',102,3306,1,2); INSERT INTO mysql_replication_hostgroups VALUES (101,102); Loading Config LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK; ProxySQL Replication ( case 1 )
  37. 37. ● ProxySQL can be Integrated with application server to reduce latency. ● Read_only flag defines the availability of readers ● Group of ProxySQL can be designed for complex Infra. ● Schedulers can be made for custom solutions. ● HA Solution for ProxySQL can be made with Keepalived. ● Prepared Statements are not supported in Read/write split ProxySQL Replication ( case 1 )
  38. 38. ● ProxySQL has in built Query caching. ● Caching is based on the query Rules. ● Cache invalidation is through cache_ttl ( in mysql_query_rules) ● mysql_query_cache_size_MB variables controls it. ● mysql_threshold_resultset_size defines the Maximum size of can be buffered. ● stats_mysql_global has metric related to Query Cache too. ● More Granular control over caching ( Query , User , Schema , Hint ) Note : Query Cache inbuilt with MySQL is Retiring in MySQL 8.0 ProxySQL Caching ( case 2)
  39. 39. ProxySQL Caching ( case 2) Reader - 102 ProxySQL - With Cache
  40. 40. ● Caching Rules has to be made in Query rules. ( digest,cache_ttl ) ● Rules are made based on digest ( stats_mysql_query_digest ) ● TTL defines the invalidation in ms . ( cache_ttl) ● Top Queries can loaded based on Count of execution, Sum of execution time. ● Cache size helps in tuning it based on global stats. ● Background threads take care of memory cleanup and expired entries. ● It can replace Redis , memcache and other 3rd party caching for DB. ProxySQL Caching ( case 2)
  41. 41. ProxySQL Caching Vs MySQL Query Cache from official proxysql website ProxySQL Caching ( case 2 )
  42. 42. ● Galera is cluster based on InnoDB engine. ● Developed by Codership and it is an open source. ● Synchronous replication ( Zero lag ) ● Multi Threaded replication. ● Automated node Recovery in case of crash. ● Distributions - Percona XtraDB Cluster , MariaDB Cluster and MySQL ( with plugin ). ● Galera with ProxySQL makes a better High Availability. ProxySQL For Galera ( case 3 )
  43. 43. ProxySQL For Galera ( case 3 ) Cluster - 103 ProxySQL ( Cluster )
  44. 44. ● Inbuilt schedulers to supports ● Support third party schedulers ● Awares on Galera States during SST / IST ● Desync states in galera. ● Easy Maintenance with status ( ONLINE , SHUNNED,OFFLINE_SOFT,OFFLINE_HARD ) ProxySQL For Galera ( case 3)
  45. 45. Integrations: ● ProxySQL is bundled in Percona XtraDB Cluster 5.7 ● Proxy-admin tool by Percona to ease configuration. ● Orchestrator has integration for proxysql. ● Ansible modules helps in easy deployment ( 7 modules are present ) ● Prometheus exporters for metrics. ● Integrated with PMM ( Grafana based Dashboards ) ProxySQL Integration & Deployment
  46. 46. ● Query Rewrite ● Sharding ( Schema , User , Query , Hints ) ● Persistent Connections ● NDB Cluster Support ● Mirroring ● Query Filter ● Data Masking and more ProxySQL Other use cases
  47. 47. Maxscale Vs ProxySQL Features Maxscale ProxySQL License BSL/GPL GPL Group Replication Not Supported Supported Galera Support Supported Supported Read/Write Split Query Module ( Classifier ) REGEX Aurora Support Supported(2.1) Not Supported Binlog router Supported Not Supported Kafka Streaming Supported Not Supported
  48. 48. Maxscale Vs ProxySQL Features Maxscale ProxySQL Multiplexing Not Supported Supported Caching Control Not Supported Supported Query Digest Not Supported Supported Ansible No Modules Ansible Modules NagiOS Plugins are present No Official Prometheus No exporters Exporters are present Clustering Not Supported Not Supported
  49. 49. Queries Reach US 080 - 48505683
  50. 50. Thank You