ProxySQL Cluster - Percona Live 2022

ProxySQL Cluster
Challenges and solutions to
synchronize config across multiple
decentralized cluster nodes

Who am I?
René Cannaò
• Founder of ProxySQL LLC
• Author of ProxySQL
• MySQL DBA

ProxySQL LLC
We provide services to help build, support as well as improve the
performance & reliability of your Cloud-Based or On-Premise MySQL
infrastructure:
● ProxySQL Development
● Remote DBRE Consulting
● ProxySQL Support Services

We are hiring!
● Experience coding in C/C++?
● MySQL DBA / Development?
● DevOps / Automation?
● Working remotely?

Application and Database layers
APPLICATIONS
DATABASES

Database as a Service (layered)
APPLICATIONS
DATABASES + MANAGER(s)
DAAS – REVERSE PROXY

What is ProxySQL?
MySQL protocol aware data gateway
– Clients connect to ProxySQL
– Requests are evaluated
– Various actions are performed
Visit https://proxysql.com for more information

Main features
● High Availability and Scalability
● seamless failover
● firewall
● query throttling
● query timeout
● query mirroring
● runtime reconfiguration
● Scheduler
● Support for Async Replication, Galera/PXC, Group Replication, Aurora

Main features
● on-the-fly rewrite of queries
● caching reads outside the database
● connection pooling and multiplexing
● complex query routing and r/w split
● load balancing
● real time statistics
● monitoring
● Data masking
● Management of hundreds of backend servers
● Native Clustering

App Server Deployment
ProxySQL is deployed locally
• No network overhead
• No single point of failure
• Isolated configuration
• Rolling upgrades
2 • DB monitoring overhead
• More backend connections
• Configuration effort
• Query cache isolated
Configuration management is essential

ProxySQL Layer Deployment
ProxySQL is deployed on a standalone server
• DB monitoring overhead
• Less backend connections
• Configuration effort
• Shared Query Cache
• Additional Round Trip Time
• Single point of failure
• Shared configuration
• No rolling upgrades
2
Configuration management is optional

ProxySQL Cascading
• ProxySQL is deployed on
each application server
and in a ProxySQL layer
• Applications connect to
the local ProxySQL server
• Provides load balancing as
well as HA
ProxySQL

Distributed system
Multiple proxysql instances
Communicate to, and coordinate with
each other
Advantages:
● Scalability
● redundancy

Configuration Management
what is?
Maintaining systems in a state that is:
● Desired
● Consistent
● According to requirements

Consul
https://learn.hashicorp.com/tutorials/consul/get-started

ZooKeeper
https://zookeeper.apache.org/doc/r3.5.1-alpha/zookeeperOver.html

ProxySQL Cluster
ProxySQL native implementation of Configuration
management:
● Doesn’t rely on external tool
● Always support new features or extensions
● Other enhancements (details later)

ProxySQL Cluster
What is it able to synchronize:
MySQL Users
MySQL Servers and MySQL Cluster configurations
MySQL Query Rules
ProxySQL Nodes
Admin Variables
LDAP Variables
MySQL Variables

ProxySQL Cluster Topologies
A ProxySQL Cluster can be configured to pull configuration
data from a single specific node

A ProxySQL Cluster can also be
configured to pull configuration data
from any node
● Currently no leader election
● Time based synchronization
● Configuration checked at regular
intervals
● Pulling mechanism synchronization

Cluster core nodes
Admin> SELECT * FROM proxysql_servers;
+-----------+-------+--------+------------+
| hostname | port | weight | comment |
+-----------+-------+--------+------------+
| 127.0.0.1 | 6032 | 0 | proxysql |
| 127.0.0.1 | 26001 | 0 | proxysql01 |
| 127.0.0.1 | 26002 | 0 | proxysql02 |
| 127.0.0.1 | 26003 | 0 | proxysql03 |
+-----------+-------+--------+------------+

What to synchronize?
Controlled by several admin variables:
Admin> SELECT variable_name, variable_value val FROM global_variables WHERE variable_name
LIKE 'admin-cluster%sync' OR variable_name IN
('admin-cluster_check_interval_ms','admin-cluster_username','admin-cluster_password');
+---------------------------------------------------+-------------+
| variable_name | val |
+---------------------------------------------------+-------------+
| admin-cluster_username | cluster1 |
| admin-cluster_password | secret1pass |
| admin-cluster_check_interval_ms | 1000 |
| admin-cluster_mysql_query_rules_diffs_before_sync | 3 |
| admin-cluster_mysql_servers_diffs_before_sync | 3 |
| admin-cluster_mysql_users_diffs_before_sync | 3 |
| admin-cluster_proxysql_servers_diffs_before_sync | 3 |
| admin-cluster_mysql_variables_diffs_before_sync | 3 |
| admin-cluster_admin_variables_diffs_before_sync | 3 |
| admin-cluster_ldap_variables_diffs_before_sync | 3 |
+---------------------------------------------------+-------------+

Admin> SELECT * FROM stats_proxysql_servers_checksums ORDER BY hostname, port, name;
+-----------+-------+-------------------+---------+------------+--------------------+------------+------------+------------+
| hostname | port | name | version | epoch | checksum | changed_at | updated_at | diff_check |
+-----------+-------+-------------------+---------+------------+--------------------+------------+------------+------------+
| 127.0.0.1 | 6032 | admin_variables | 2 | 1643239330 | 0x48F46583E487DA09 | 1643239664 | 1643313066 | 0 |
| 127.0.0.1 | 6032 | mysql_query_rules | 4 | 1643239330 | 0x2607031643FD24F1 | 1643239664 | 1643313066 | 0 |
| 127.0.0.1 | 6032 | mysql_servers | 4 | 1643239240 | 0xE7A444A2B6CEEC30 | 1643239664 | 1643313066 | 0 |
| 127.0.0.1 | 6032 | mysql_users | 3 | 1643239330 | 0x5278D9F01DAB674E | 1643239664 | 1643313066 | 0 |
| 127.0.0.1 | 6032 | mysql_variables | 2 | 1643239330 | 0x4812D46ED6B25CC6 | 1643239664 | 1643313066 | 0 |
| 127.0.0.1 | 6032 | proxysql_servers | 3 | 1643239661 | 0xB80BD23DB0FE25F0 | 1643239664 | 1643313066 | 0 |
+-----------+-------+-------------------+---------+------------+--------------------+------------+------------+------------+

Admin> SELECT hostname, port, name, version ver, epoch, checksum, changed_at, updated_at, diff_check diff
FROM stats_proxysql_servers_checksums WHERE name = 'mysql_servers' ORDER BY hostname, port, name;
+-----------+-------+---------------+-----+------------+--------------------+------------+------------+------+
| hostname | port | name | ver | epoch | checksum | changed_at | updated_at | diff |
+-----------+-------+---------------+-----+------------+--------------------+------------+------------+------+
+-----------+-------+---------------+-----+------------+--------------------+------------+------------+------+
Why version is important?
Timestamps are in second.
Tune properly variables:
● Admin-cluster_check_interval_ms
● admin-cluster_*_diffs_before_sync accordingly.

Poor design patterns
Nasty stories:
● Too fast resynchronization
● Conflicting configuration tools
● Use of DNS
● Use of LB

Variables Synchronization
Challenges and solutions:
● variables must match
○ ProxySQL versions must match during handshake
● Running on different listeners
○ Variable cluster_sync_interfaces (boolean) in proxysql.cnf

ProxySQL Cluster can also be configured to pull configuration from a
set of nodes

Core/Satellites Layers
ProxySQL Cluster configured to pull configuration from a set of
nodes

Core/Satellites Layers
ProxySQL Cluster configured to pull configuration from a set of
nodes
Core
Layer
Satellites
Layer

Satellite nodes registration
Admin> SELECT * FROM proxysql_servers;
+-----------+-------+--------+------------+
| hostname | port | weight | comment |
+-----------+-------+--------+------------+
| 127.0.0.1 | 6032 | 0 | proxysql |
| 127.0.0.1 | 26001 | 0 | proxysql01 |
| 127.0.0.1 | 26002 | 0 | proxysql02 |
| 127.0.0.1 | 26003 | 0 | proxysql03 |
+-----------+-------+--------+------------+
Admin> SELECT * FROM disk.global_settings;
+---------------+--------------------------------------+
| variable_name | variable_value |
+---------------+--------------------------------------+
| uuid | c96f918a-933a-48b9-b037-b9d874133a2f |
+---------------+--------------------------------------+

Satellite nodes registration
Admin> SELECT * FROM stats_proxysql_servers_clients_status;
+--------------------------------------+-----------+-------+--------------------+--------------+
| uuid | hostname | port | admin_mysql_ifaces | last_seen_at |
+--------------------------------------+-----------+-------+--------------------+--------------+
| b94a322f-510a-4297-86c6-aaa98a82a741 | 127.0.0.1 | 54024 | 0.0.0.0:26005 | 1643495163 |
| c96f918a-933a-48b9-b037-b9d874133a2f | 127.0.0.1 | 54040 | 0.0.0.0:6032 | 1643495163 |
| abbfd59a-559a-45db-aeb7-61ef9e6e5d55 | 127.0.0.1 | 54052 | 0.0.0.0:26007 | 1643495163 |
| 1c337b69-63d1-45f3-85e7-2c0f1cf5016b | 127.0.0.1 | 53994 | 0.0.0.0:26004 | 1643495163 |
| 9d5b904b-25b8-4cce-b35a-8887b418fb41 | 127.0.0.1 | 54044 | 0.0.0.0:26006 | 1643495163 |
| 44736f56-686c-45f1-a622-c015db76a79c | 127.0.0.1 | 53996 | 0.0.0.0:26008 | 1643495163 |
| 7d0f2ff5-9c12-4885-aaff-a7c0242fc1f7 | 127.0.0.1 | 54020 | 0.0.0.0:26009 | 1643495163 |
| 12041e13-08aa-4034-be77-0efc1bba34f3 | 127.0.0.1 | 54048 | 0.0.0.0:26003 | 1643495163 |
| 287fdb40-8d74-4a6b-8d14-0ab05a08d70c | 127.0.0.1 | 53952 | 0.0.0.0:26001 | 1643495164 |
| 2601e4ef-c5ff-4d3d-8c5d-eb09db67ab0c | 127.0.0.1 | 54038 | 0.0.0.0:26002 | 1643495164 |
+--------------------------------------+-----------+-------+--------------------+--------------+
10 rows in set (0.00 sec)

Why nodes registration?
• Autoscale and management
• Monitoring tools
• Aggregate metrics
• Log collections
• Troubleshooting
• Remote analysis
• Web UI to remote nodes

Todo: Leader election
Admin in RW or RO mode
Monitor read_only: built-in failure
detection
Use of replication hostgroups

Synchronization slowdown
Causes:
• Network delay
• Complex configuration. Example:
– hundreds of thousands of users
– very complex query rules
• Various internal locking
Side effect:
● Configuration rollback

Avoid config rollback
A 1
B 1
C 1
D 1
All ProxySQL nodes start with the same
configuration

A 2
B 2
C 1
D 1
Node A get a new configuration.
Node B gets it immediately.
Node C and D are slow slow at synchronizing.

A 3
B 3
C 2
D 2
When node C finally synchronizes with node
A, node A get a new configuration.
Node B synchronizes with node A.
while node D synchronizes with node C.

A 3
B 3
C 2
D 2
Because nodes C and node D got their
configuration after node A and node B got
theirs, they were considered to have a newer
configuration.

A 2
B 2
C 2
D 2
Old configuration (2) was applied to node A
and node B.

In ProxySQL 2.4, to avoid
configuration rollback, each
configuration carries with it both
version and timestamp.
A 1
A 1
A 1
A 1

Prevent incorrect sync
Before ProxySQL 2.4 :
1. Detection of a new configuration
2. Wait admin-cluster_*_diffs_before_sync
3. Pull configuration
4. Load new configuration
What happens if configuration changes right before point 3?
● An incorrect configuration is pulled, possible incomplete
● The incorrect configuration was then considered the newer configuration and
distributed in all the cluster

Prevent incorrect sync
Since ProxySQL 2.4 :
1. Detection of a new configuration
2. Wait admin-cluster_*_diffs_before_sync
3. Pull configuration
4. Verify configuration checksum. Either:
○ Apply it if it matches
○ Discard it if doesn’t match (configuration has changed during pull)

Feedback & Thank you!
Visit: https://proxysql.com
Github: https://github.com/sysown/proxysql/
Mailing list: https://groups.google.com/g/proxysql
Twitter: @proxysql

ProxySQL Cluster - Percona Live 2022

More Related Content

What's hot

Similar to ProxySQL Cluster - Percona Live 2022

Recently uploaded

ProxySQL Cluster - Percona Live 2022