ProxySQL for MySQL

www.mydbops.com mysqlsupport@mydbops.com
Presenter
Karthik P R
Founder Mydbops
ProxySQL for MySQL
An Efficient Load Balancer

● MySQL Consulting
● MySQL Support
● Remote DBA support ( 24*7 )
● Expert MySQL solutions
● Percona Cluster Support
● MySQL Load balancer Support
About Mydbops

Our Clients

● Founder Mydbops & MySQL Solution Architect .
● 7 years experience in handling MySQL and its forks.
● Currently Consulting Organizations to scale in MySQL
● Worked on large scale systems at Yahoo!
● MySQL Speaker and Blogger.
Contact : prk@mydbops.com
Twitter : @prkart
About Me

● Need for Load Balancer
● SQL Aware Load Balancers
● ProxySQL Introduction.
● ProxySQL Configuration
● ProxySQL for MySQL Replication.
● ProxySQL Caching
● ProxySQL Mirroring
● ProxySQL Monitoring and deployment
● Other Features
Agenda

● High Availability of servers
● Performance of servers
● Avoid Complexity at application
● Better Usage of Hardware
● Additional features
Need for a Load balancer

Types Of Load Balancers
● Reverse Proxies
○ HAProxy
○ Ngnix
● SQL aware Proxies
○ Maxscale
○ MySQL Router
○ ProxySQL

Reverse Proxies ( Eg, HAProxy Scenario )
● HAProxy will split the connections to the read
servers based on the algorithms ( RR/ LC )
● It serves the connections to slaves
● The slave connection are made on diff port.
● Works on TCP ( Layer 7 )

Reverse Proxies ( Eg, HAProxy Scenario )
● When everything is running fine, load balancer
will split connections based on Round Robin
or Least recently used algorithms.
● It’s good for application not for database.
● In case if any one of the slave IO thread stopped,
it won’t stop sharing queries.
● It lacks SQL Intelligence.

SQL aware Proxies ( Eg, ProxySQL Scenario )
● ProxySQL make health checks for each servers in
the loop.
● It’s database friendly which removes the failed
server from load balancer.
● In case if any one of the slave IO thread stopped,
it will take out from load balancer until the issue
get resolved.

● Designed for DBA’s
● Awareness MySQL states
● Read /Write Split
● Support for external tools and plugins
● Health of Database on query basics
● Routing of Queries
● Query Caching
SQL Aware Load Balancers

● Built for DBA’s by a DBA.
● Major Contribution by René Cannaò (Founder)
● Winner MySQL Community Contributor (2017)
● Light weight application.
● Advanced Multi Core Architecture.
● Complete Open Source GPL License (GPL 3.0)
● Current Version 1.3.6 and 1.4 is Beta. ( 09-06-2017 )
ProxySQL Introduction

● More Granular Configuration
● SQL based Administration.
● Dynamic Configuration
● Advanced Metric Collections.
● Supports More Complex Topologies.
● Seamless Integrations.
● Manage database at large scale ( 1000’s )

● REGEX Based Routing
● Support for Group Replication ( 1.4)
● Query Rewrite and Mirroring
● Easy Integration with Ansible Module
● PMM Graphs with ProxySQL Exporters

ProxySQL Architecture

Query Cache & Rewrite:
● Caching of queries & result
● Caching or mirroring
● In memory store and pattern based
User Authentication
● User Credentials of ProxySQL connectivity
● Max connections limitation & Throttling
● Credentials are encrypted

Connection Pool
● Reuses the connection and reduces the overhead
● Automatic retry of queries
● Connection Multiplexing
● Failover management
Monitoring :
● Collect metrics and monitor server healths
● Asynchronous replication support
● 1.4 has support for Group replication
● Monitor read_only variables ( Reader host groups)

Query Processor:
● Based on Query rules defined
● Timeout and delay
● Queries to be cached
● Hostgroup Targets
● Mirroring and firewall ( query blocking )

Query rules:
● Regex on Query
○ RE2 ( 1.3 )
○ PCRE ( Default) & RE2 (1.4)
● Fine granular control on incoming traffic based on
○ Username
○ Host group
○ Source IP and Database name
○ Digest ( stats_mysql_query_digest_reset )
○ Bind Address and port

Host Groups Manager
● All Backend or Grouped into hostgroups
● Hostgroups has logical function
● Tracks Server status
● Integrated with Connection pool
Eg)
HGO : Write Master ( Only one Server )
HG1 : Read Slaves ( One or Many Servers )
HG2 : Reporting Slave ( One or Many Servers )

● SQLite database to store config.
● MySQL Compatible interface.
● There is no real parser in ProxySQL
● Multi Layer Configuration
● Default port 6032
$mysql -u admin -padmin -h 127.0.0.1 -P6032
ProxySQL Configuration

Tables in ProxySQL ( Main )
Global_variables Config of proxysql
Mysql_query_rules Rules for R/w split,Caching,routing,etc
Mysql_servers Info of all MySQL Servers
Mysql_replication_hostgroups Reader_hostgroup and writer_hostgroup groups
Mysql_users Defines MySQL user to connect
DB’s
Scheduler Jobs running at regular interval
(Custom scripts)www.mydbops.com mysqlsupport@mydbops.com

Memory: Represent SQLite3 in memory
database
Runtime: In memory store of ProxySQL with
RUNTIME Config.
Disk: On Disk location of SQLite3. Contains
Persistent storage of config.
Config File: Load the config (proxysql.cnf)
when there is no disk file ( proxysql.db)

Loading configuration three levels:
SQL (writes ) are made to table in Memory.
Then load to RUNTIME ( Dynamically ) and
saved to DISK for persistence.
Eg)
Load MYSQL USERS TO RUNTIME;
SAVE MYSQL USERS TO DISK;

Scenario Loading a User
Host groups 101 - Writer , 102 -Reader
insert into mysql_users(username,password,active,default_hostgroup,default_schema) values
('writer_app','test',1,101,'sbtest');
insert into mysql_users(username,password,active,default_hostgroup,default_schema) values
('reader_app','test',1,102,'sbtest');
LOAD MYSQL USERS TO RUNTIME;
SAVE MYSQL USERS TO DISK;

Replication ( Common Use case Scenario )
● Read write Splitting
● Read scaling
● Easy Maintenance with status ( ONLINE , SHUNNED,OFFLINE_SOFT,OFFLINE_HARD )
● Zero downtime Failover
● Avoid stale data at slaves caused by lag ( max_replication_lag )
● REGEX Based Routing
● Do not support re-slaving or promotion ( mysqlrpladmin , MHA , Orchestrator )
ProxySQL Replication

Loading Server Info:
INSERT INTO mysql_servers ( hostname,hostgroup_id,port,weight) VALUES ('192.168.1.1',101,3306,1000);
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES
('192.168.1.2',102,3306,10000,2);
('192.168.1.3',102,3306,10000,2);
('192.168.1.4',102,3306,1,2);
INSERT INTO mysql_replication_hostgroups VALUES (101,102);
Loading Config
LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK;

Loading Query Rules:
UPDATE mysql_users SET default_hostgroup=101; --- All Queries Goes to writer group
INSERT INTO mysql_query_rules
(username,destination_hostgroup,active,retries,match_digest,apply)values(’writer_app’,101,1,3,'^SELECT.*FOR
UPDATE',1);
INSERT INTO mysql_query_rules
(username,destination_hostgroup,active,retries,match_digest,apply)values(‘reader_app’,102,1,3,'^SELECT ',1);
INSERT INTO mysql_query_rules (username,destination_hostgroup,active) values(’reader_app’,102,1);
Loading Config
LOAD MYSQL QUERY RULES TO RUNTIME; SAVE MYSQL QUERY RULES TO DISK;

● ProxySQL can be Integrated with application server to reduce latency.
● Read_only flag defines the availability of readers
● Group of ProxySQL can be designed for complex Infra.
● Schedulers can be made for custom solutions.
● HA Solution for ProxySQL can be made with Keepalived.
● Prepared Statements are not supported in Read/write split

● ProxySQL has in built Query caching.
● Caching is based on the query Rules.
● Cache invalidation is through cache_ttl ( in mysql_query_rules)
● mysql_query_cache_size_MB variables controls it.
● mysql_threshold_resultset_size defines the Maximum size of can be buffered.
● stats_mysql_global has metric related to Query Cache too.
Note : Query Cache inbuilt MySQL is Retiring in MySQL 8.0
ProxySQL Caching

ProxySQL Caching Vs MySQL Query Cache from official proxysql website
ProxySQL Caching

● Caching Rules has to be made in Query rules. ( digest,cache_ttl )
● Rules are made based on digest ( stats_mysql_query_digest )
● TTL defines the invalidation in ms . ( cache_ttl)
● Top Queries can loaded based on Count of execution, Sum of execution time.
● Cache size helps in tuning it based on global stats.
● Background threads take care of memory cleanup and expired entries.
ProxySQL Caching

● ProxySQL Mirroring
Will make exact query mirroring to the testing server.
Can be tested without affecting live production database.
Extensions to mysql_query_rules
Table mysql_query_rules was modified to add 2 more columns:
mirror_flagOUT
mirror_hostgroup
ProxySQL Mirroring

● The green arrow represents live production
traffic that is sent to the production database
server, which sends all responses back to the
application as if nothing changed.
● ProxySQL issues the mirrored query, represented
by the violet arrow, independently without impact
to the application, collecting metrics and errors
for both queries.
ProxySQL Mirroring

● ProxySQL stores metrics and digest info
● STATS and Monitor DB stores the information.
● Response time of queries , Connections usage , stats on traffic
● SHOW MYSQL STATUS ( Query cache, Buffer, Connections )
● Connection pool usage stats from stats_mysql_connection_pool table.
● ProxySQL exporters for Prometheus helps integration with PMM .
ProxySQL Monitoring

ProxySQL Monitoring
Query Routed

ProxySQL Monitoring
Client Connections

ProxySQL Monitoring
Endpoint Status

● Alerting can be made on Grafana 4 ( Slack/Email)
● Datadog has monitoring for ProxySQL.
● Monitor DB has info regarding
○ Connect log
○ Ping log
○ Replication lag log
○ Read only log
● ProxySQL Lacks direct NagiOS integration
ProxySQL Monitoring

● Large scale deployment can be made with Ansible
● Ansible Modules
○ proxysql_backend_server
○ proxysql_global_variables
○ proxysql_manage_config
○ proxysql_mysql_users
○ proxysql_query_rules
○ proxysql_replication_hostgroup
○ proxysql_scheduler
ProxySQL Deployment & Integration

Integrations:
● ProxySQL is bundled in Percona XtraDB Cluster 5.7
● Proxy-admin tool by Percona to ease configuration.
● Orchestrator has integration for proxysql.
ProxySQL Deployment & Integration

● Query Rewrite
● Sharding
● Persistent Connections
● NDB Cluster Support
● Galera Support Native With Schedulers .
● Query Filter
● Data Masking and more
ProxySQL Other features

Queries

Contact US
www.mydbops.com
mysqlsupport@mydbops.com
080 - 4850 5683

Thank You

ProxySQL for MySQL

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to ProxySQL for MySQL

Similar to ProxySQL for MySQL (20)

More from Mydbops

More from Mydbops (20)

Recently uploaded

Recently uploaded (20)

ProxySQL for MySQL