Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ProxySQL for MySQL

1,995 views

Published on

This is the presentation delivered by Karthik.P.R at MySQL User Camp Bangalore on 09th June 2017. ProxySQL is a high performance MySQL Load Balancer Designed to scale database servers.

Published in: Technology
  • Be the first to comment

ProxySQL for MySQL

  1. 1. www.mydbops.com mysqlsupport@mydbops.com Presenter Karthik P R Founder Mydbops ProxySQL for MySQL An Efficient Load Balancer
  2. 2. ● MySQL Consulting ● MySQL Support ● Remote DBA support ( 24*7 ) ● Expert MySQL solutions ● Percona Cluster Support ● MySQL Load balancer Support www.mydbops.com mysqlsupport@mydbops.com About Mydbops
  3. 3. www.mydbops.com mysqlsupport@mydbops.com Our Clients
  4. 4. ● Founder Mydbops & MySQL Solution Architect . ● 7 years experience in handling MySQL and its forks. ● Currently Consulting Organizations to scale in MySQL ● Worked on large scale systems at Yahoo! ● MySQL Speaker and Blogger. Contact : prk@mydbops.com Twitter : @prkart www.mydbops.com mysqlsupport@mydbops.com About Me
  5. 5. ● Need for Load Balancer ● SQL Aware Load Balancers ● ProxySQL Introduction. ● ProxySQL Configuration ● ProxySQL for MySQL Replication. ● ProxySQL Caching ● ProxySQL Mirroring ● ProxySQL Monitoring and deployment ● Other Features www.mydbops.com mysqlsupport@mydbops.com Agenda
  6. 6. ● High Availability of servers ● Performance of servers ● Avoid Complexity at application ● Better Usage of Hardware ● Additional features www.mydbops.com mysqlsupport@mydbops.com Need for a Load balancer
  7. 7. Types Of Load Balancers ● Reverse Proxies ○ HAProxy ○ Ngnix ● SQL aware Proxies ○ Maxscale ○ MySQL Router ○ ProxySQL www.mydbops.com mysqlsupport@mydbops.com Need for a Load balancer
  8. 8. Reverse Proxies ( Eg, HAProxy Scenario ) ● HAProxy will split the connections to the read servers based on the algorithms ( RR/ LC ) ● It serves the connections to slaves ● The slave connection are made on diff port. ● Works on TCP ( Layer 7 ) www.mydbops.com mysqlsupport@mydbops.com Need for a Load balancer
  9. 9. Reverse Proxies ( Eg, HAProxy Scenario ) ● When everything is running fine, load balancer will split connections based on Round Robin or Least recently used algorithms. ● It’s good for application not for database. ● In case if any one of the slave IO thread stopped, it won’t stop sharing queries. ● It lacks SQL Intelligence. www.mydbops.com mysqlsupport@mydbops.com Need for a Load balancer
  10. 10. SQL aware Proxies ( Eg, ProxySQL Scenario ) ● ProxySQL make health checks for each servers in the loop. ● It’s database friendly which removes the failed server from load balancer. ● In case if any one of the slave IO thread stopped, it will take out from load balancer until the issue get resolved. www.mydbops.com mysqlsupport@mydbops.com Need for a Load balancer
  11. 11. ● Designed for DBA’s ● Awareness MySQL states ● Read /Write Split ● Support for external tools and plugins ● Health of Database on query basics ● Routing of Queries ● Query Caching www.mydbops.com mysqlsupport@mydbops.com SQL Aware Load Balancers
  12. 12. ● Built for DBA’s by a DBA. ● Major Contribution by René Cannaò (Founder) ● Winner MySQL Community Contributor (2017) ● Light weight application. ● Advanced Multi Core Architecture. ● Complete Open Source GPL License (GPL 3.0) ● Current Version 1.3.6 and 1.4 is Beta. ( 09-06-2017 ) www.mydbops.com mysqlsupport@mydbops.com ProxySQL Introduction
  13. 13. ● More Granular Configuration ● SQL based Administration. ● Dynamic Configuration ● Advanced Metric Collections. ● Supports More Complex Topologies. ● Seamless Integrations. ● Manage database at large scale ( 1000’s ) www.mydbops.com mysqlsupport@mydbops.com ProxySQL Introduction
  14. 14. ● REGEX Based Routing ● Support for Group Replication ( 1.4) ● Query Rewrite and Mirroring ● Easy Integration with Ansible Module ● PMM Graphs with ProxySQL Exporters www.mydbops.com mysqlsupport@mydbops.com ProxySQL Introduction
  15. 15. www.mydbops.com mysqlsupport@mydbops.com ProxySQL Architecture
  16. 16. Query Cache & Rewrite: ● Caching of queries & result ● Caching or mirroring ● In memory store and pattern based User Authentication ● User Credentials of ProxySQL connectivity ● Max connections limitation & Throttling ● Credentials are encrypted www.mydbops.com mysqlsupport@mydbops.com ProxySQL Architecture
  17. 17. Connection Pool ● Reuses the connection and reduces the overhead ● Automatic retry of queries ● Connection Multiplexing ● Failover management Monitoring : ● Collect metrics and monitor server healths ● Asynchronous replication support ● 1.4 has support for Group replication ● Monitor read_only variables ( Reader host groups) www.mydbops.com mysqlsupport@mydbops.com ProxySQL Architecture
  18. 18. Query Processor: ● Based on Query rules defined ● Timeout and delay ● Queries to be cached ● Hostgroup Targets ● Mirroring and firewall ( query blocking ) www.mydbops.com mysqlsupport@mydbops.com ProxySQL Architecture
  19. 19. Query rules: ● Regex on Query ○ RE2 ( 1.3 ) ○ PCRE ( Default) & RE2 (1.4) ● Fine granular control on incoming traffic based on ○ Username ○ Host group ○ Source IP and Database name ○ Digest ( stats_mysql_query_digest_reset ) ○ Bind Address and port www.mydbops.com mysqlsupport@mydbops.com ProxySQL Architecture
  20. 20. Host Groups Manager ● All Backend or Grouped into hostgroups ● Hostgroups has logical function ● Tracks Server status ● Integrated with Connection pool Eg) HGO : Write Master ( Only one Server ) HG1 : Read Slaves ( One or Many Servers ) HG2 : Reporting Slave ( One or Many Servers ) www.mydbops.com mysqlsupport@mydbops.com ProxySQL Architecture
  21. 21. ● SQLite database to store config. ● MySQL Compatible interface. ● There is no real parser in ProxySQL ● Multi Layer Configuration ● Default port 6032 $mysql -u admin -padmin -h 127.0.0.1 -P6032 www.mydbops.com mysqlsupport@mydbops.com ProxySQL Configuration
  22. 22. Database in ProxySQL. mysql> show databases; +-----+---------+-------------------------------+ | seq | name | file | +-----+---------+-------------------------------+ | 0 | main | | | 2 | disk | /var/lib/proxysql/proxysql.db | | 3 | stats | | | 4 | monitor | | +-----+---------+-------------------------------+ By default main is DB used when connected through Admin interface. www.mydbops.com mysqlsupport@mydbops.com ProxySQL Configuration
  23. 23. Tables in ProxySQL ( Main ) Global_variables Config of proxysql Mysql_query_rules Rules for R/w split,Caching,routing,etc Mysql_servers Info of all MySQL Servers Mysql_replication_hostgroups Reader_hostgroup and writer_hostgroup groups Mysql_users Defines MySQL user to connect DB’s Scheduler Jobs running at regular interval (Custom scripts)www.mydbops.com mysqlsupport@mydbops.com ProxySQL Configuration
  24. 24. Memory: Represent SQLite3 in memory database Runtime: In memory store of ProxySQL with RUNTIME Config. Disk: On Disk location of SQLite3. Contains Persistent storage of config. Config File: Load the config (proxysql.cnf) when there is no disk file ( proxysql.db) www.mydbops.com mysqlsupport@mydbops.com ProxySQL Configuration
  25. 25. Loading configuration three levels: SQL (writes ) are made to table in Memory. Then load to RUNTIME ( Dynamically ) and saved to DISK for persistence. Eg) Load MYSQL USERS TO RUNTIME; SAVE MYSQL USERS TO DISK; www.mydbops.com mysqlsupport@mydbops.com ProxySQL Configuration
  26. 26. Scenario Loading a User Host groups 101 - Writer , 102 -Reader insert into mysql_users(username,password,active,default_hostgroup,default_schema) values ('writer_app','test',1,101,'sbtest'); insert into mysql_users(username,password,active,default_hostgroup,default_schema) values ('reader_app','test',1,102,'sbtest'); LOAD MYSQL USERS TO RUNTIME; SAVE MYSQL USERS TO DISK; www.mydbops.com mysqlsupport@mydbops.com ProxySQL Configuration
  27. 27. Replication ( Common Use case Scenario ) ● Read write Splitting ● Read scaling ● Easy Maintenance with status ( ONLINE , SHUNNED,OFFLINE_SOFT,OFFLINE_HARD ) ● Zero downtime Failover ● Avoid stale data at slaves caused by lag ( max_replication_lag ) ● REGEX Based Routing ● Do not support re-slaving or promotion ( mysqlrpladmin , MHA , Orchestrator ) www.mydbops.com mysqlsupport@mydbops.com ProxySQL Replication
  28. 28. Loading Server Info: INSERT INTO mysql_servers ( hostname,hostgroup_id,port,weight) VALUES ('192.168.1.1',101,3306,1000); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES ('192.168.1.2',102,3306,10000,2); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES ('192.168.1.3',102,3306,10000,2); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES ('192.168.1.4',102,3306,1,2); INSERT INTO mysql_replication_hostgroups VALUES (101,102); Loading Config LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK; www.mydbops.com mysqlsupport@mydbops.com ProxySQL Replication
  29. 29. Loading Query Rules: UPDATE mysql_users SET default_hostgroup=101; --- All Queries Goes to writer group INSERT INTO mysql_query_rules (username,destination_hostgroup,active,retries,match_digest,apply)values(’writer_app’,101,1,3,'^SELECT.*FOR UPDATE',1); INSERT INTO mysql_query_rules (username,destination_hostgroup,active,retries,match_digest,apply)values(‘reader_app’,102,1,3,'^SELECT ',1); INSERT INTO mysql_query_rules (username,destination_hostgroup,active) values(’reader_app’,102,1); Loading Config LOAD MYSQL QUERY RULES TO RUNTIME; SAVE MYSQL QUERY RULES TO DISK; www.mydbops.com mysqlsupport@mydbops.com ProxySQL Replication
  30. 30. www.mydbops.com mysqlsupport@mydbops.com ProxySQL Replication
  31. 31. ● ProxySQL can be Integrated with application server to reduce latency. ● Read_only flag defines the availability of readers ● Group of ProxySQL can be designed for complex Infra. ● Schedulers can be made for custom solutions. ● HA Solution for ProxySQL can be made with Keepalived. ● Prepared Statements are not supported in Read/write split www.mydbops.com mysqlsupport@mydbops.com ProxySQL Replication
  32. 32. ● ProxySQL has in built Query caching. ● Caching is based on the query Rules. ● Cache invalidation is through cache_ttl ( in mysql_query_rules) ● mysql_query_cache_size_MB variables controls it. ● mysql_threshold_resultset_size defines the Maximum size of can be buffered. ● stats_mysql_global has metric related to Query Cache too. Note : Query Cache inbuilt MySQL is Retiring in MySQL 8.0 www.mydbops.com mysqlsupport@mydbops.com ProxySQL Caching
  33. 33. ProxySQL Caching Vs MySQL Query Cache from official proxysql website www.mydbops.com mysqlsupport@mydbops.com ProxySQL Caching
  34. 34. ● Caching Rules has to be made in Query rules. ( digest,cache_ttl ) ● Rules are made based on digest ( stats_mysql_query_digest ) ● TTL defines the invalidation in ms . ( cache_ttl) ● Top Queries can loaded based on Count of execution, Sum of execution time. ● Cache size helps in tuning it based on global stats. ● Background threads take care of memory cleanup and expired entries. www.mydbops.com mysqlsupport@mydbops.com ProxySQL Caching
  35. 35. ● ProxySQL Mirroring Will make exact query mirroring to the testing server. Can be tested without affecting live production database. Extensions to mysql_query_rules Table mysql_query_rules was modified to add 2 more columns: mirror_flagOUT mirror_hostgroup www.mydbops.com mysqlsupport@mydbops.com ProxySQL Mirroring
  36. 36. ● The green arrow represents live production traffic that is sent to the production database server, which sends all responses back to the application as if nothing changed. ● ProxySQL issues the mirrored query, represented by the violet arrow, independently without impact to the application, collecting metrics and errors for both queries. www.mydbops.com mysqlsupport@mydbops.com ProxySQL Mirroring
  37. 37. ● ProxySQL stores metrics and digest info ● STATS and Monitor DB stores the information. ● Response time of queries , Connections usage , stats on traffic ● SHOW MYSQL STATUS ( Query cache, Buffer, Connections ) ● Connection pool usage stats from stats_mysql_connection_pool table. ● ProxySQL exporters for Prometheus helps integration with PMM . www.mydbops.com mysqlsupport@mydbops.com ProxySQL Monitoring
  38. 38. www.mydbops.com mysqlsupport@mydbops.com ProxySQL Monitoring Query Routed
  39. 39. www.mydbops.com mysqlsupport@mydbops.com ProxySQL Monitoring Client Connections
  40. 40. www.mydbops.com mysqlsupport@mydbops.com ProxySQL Monitoring Endpoint Status
  41. 41. ● Alerting can be made on Grafana 4 ( Slack/Email) ● Datadog has monitoring for ProxySQL. ● Monitor DB has info regarding ○ Connect log ○ Ping log ○ Replication lag log ○ Read only log ● ProxySQL Lacks direct NagiOS integration www.mydbops.com mysqlsupport@mydbops.com ProxySQL Monitoring
  42. 42. ● Large scale deployment can be made with Ansible ● Ansible Modules ○ proxysql_backend_server ○ proxysql_global_variables ○ proxysql_manage_config ○ proxysql_mysql_users ○ proxysql_query_rules ○ proxysql_replication_hostgroup ○ proxysql_scheduler www.mydbops.com mysqlsupport@mydbops.com ProxySQL Deployment & Integration
  43. 43. Integrations: ● ProxySQL is bundled in Percona XtraDB Cluster 5.7 ● Proxy-admin tool by Percona to ease configuration. ● Orchestrator has integration for proxysql. www.mydbops.com mysqlsupport@mydbops.com ProxySQL Deployment & Integration
  44. 44. ● Query Rewrite ● Sharding ● Persistent Connections ● NDB Cluster Support ● Galera Support Native With Schedulers . ● Query Filter ● Data Masking and more www.mydbops.com mysqlsupport@mydbops.com ProxySQL Other features
  45. 45. Queries www.mydbops.com mysqlsupport@mydbops.com
  46. 46. Contact US www.mydbops.com mysqlsupport@mydbops.com www.mydbops.com mysqlsupport@mydbops.com 080 - 4850 5683
  47. 47. www.mydbops.com mysqlsupport@mydbops.com Thank You

×