Marco Tusa, profile picture
Uploaded byMarco Tusa
2,174 views

Proxysql sharding

The document outlines the implementation and configuration of ProxySQL for sharding MySQL databases. It discusses various sharding techniques, query rules, and user management in a ProxySQL environment using real-life case scenarios and examples. Additionally, it covers query analysis, rewriting, and how to set up necessary components for effective operation.

Embed presentation

Downloaded 53 times
ProxySQL Sharding Marco (the Grinch) Tusa September 2017 Dublin
2 ● Marco “The Grinch” • Open source enthusiast • In love with ProxySQL • Consulting team leader • Consultant myself About me
3 Example of ProxySQL/MySQL setup Understand Query Rules Examples: IP/port Schema using field=value Comment -> schema or host Use combination of host/schema Real life case of need to reduce sharding A lot of code so you can reuse Agenda
4 Examples will use world DB. http://downloads.mysql.com/docs/world-db.tar.gz Sharding …
5 Way of Sharding
5 Way of Sharding 6034 HG 10 6035 HG 20 6036 HG 30 Basic By IP/Port
5 Way of Sharding 6034 HG 10 6035 HG 20 6036 HG 30 Basic By IP/Port By parameter capture or comment injection Performing query rewrite HG 10
5 Way of Sharding 6034 HG 10 6035 HG 20 6036 HG 30 Basic By IP/Port By parameter capture or comment injection Performing query rewrite Comment Injection/ HG Direct. Query analysis and rewrite HG 10
6 You will need: 3 hosts: 192.168.0.[5-21-231] 4 schemas: Continent X + world schema 1 user : user_shardRW 3 hostgroups: 10, 20, 30 Prepare MySQL
7 HostGroup (HG) & Replication HG Logical group of server Unless specify HG receives reads and writes Properties: id; Server list Special Replicaton Host Group (RHG) Manage the separation between Write and Read HG. Properties: writer_hostgroup ID reader_hostgroup ID Understand the basics
8 HostGroup (HG) & Replication HG Understand the basics All writers PXC
8 HostGroup (HG) & Replication HG Understand the basics All writers PXC 1 writer more readers PXC with weight
8 HostGroup (HG) & Replication HG Understand the basics All writers PXC 1 writer more readers PXC with weight Master - Slave
8 HostGroup (HG) & Replication HG Understand the basics All writers PXC 1 writer more readers PXC with weight Master - Slave
9 Is the logic in ProxySQL that link Users/Schema/IP/Port/SQL to (Replication) Host Group. OR That allow SQL rewrite. And more What is a Query rule?
9 Is the logic in ProxySQL that link Users/Schema/IP/Port/SQL to (Replication) Host Group. OR That allow SQL rewrite. And more What is a Query rule?
9 Is the logic in ProxySQL that link Users/Schema/IP/Port/SQL to (Replication) Host Group. OR That allow SQL rewrite. And more What is a Query rule?
9 Is the logic in ProxySQL that link Users/Schema/IP/Port/SQL to (Replication) Host Group. OR That allow SQL rewrite. And more What is a Query rule?
9 Is the logic in ProxySQL that link Users/Schema/IP/Port/SQL to (Replication) Host Group. OR That allow SQL rewrite. And more What is a Query rule?
9 Is the logic in ProxySQL that link Users/Schema/IP/Port/SQL to (Replication) Host Group. OR That allow SQL rewrite. And more What is a Query rule?
9 Is the logic in ProxySQL that link Users/Schema/IP/Port/SQL to (Replication) Host Group. OR That allow SQL rewrite. And more What is a Query rule?
9 Is the logic in ProxySQL that link Users/Schema/IP/Port/SQL to (Replication) Host Group. OR That allow SQL rewrite. And more What is a Query rule?
9 Is the logic in ProxySQL that link Users/Schema/IP/Port/SQL to (Replication) Host Group. OR That allow SQL rewrite. And more What is a Query rule?
9 Is the logic in ProxySQL that link Users/Schema/IP/Port/SQL to (Replication) Host Group. OR That allow SQL rewrite. And more What is a Query rule?
10 Load DB test World Create schemas Asia|Europe|North_America|Africa For each continent: create table {continent}.City as select a.* from world.City a join Country on a.CountryCode = Country.code where Continent='{continent}’; create table {continent}.Country as select * from world.Country where Continent='{continent}' ; End for grant all on *.* to user_shardRW@'%' identified by 'test'; Prepare MySQL
11 What you need to do is: Set the user insert into mysql_users (username,password,active,default_hostgroup,default_schema,transaction_persistent) values ('user_shardRW','test',1,10,'test',1); LOAD MYSQL USERS TO RUNTIME;SAVE MYSQL USERS TO DISK; Set variable: update global_variables set Variable_Value=0 where Variable_name='mysql- query_cache_size_MB'; update global_variables set Variable_Value=0/20 where Variable_name='mysql- query_processor_iterations'; load mysql variables to run;save mysql variables to disk; Prepare Proxysql
12 Create basic set of server to play with like: INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.0.5',10,3306,100); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.0.21',20,3306,100); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.0.231',30,3306,100); LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK; Prepare Proxysql
13 Add Ports to play later with: update global_variables set Variable_Value='0.0.0.0:6033;0.0.0.0:6034;0.0.0.0:6035;0.0.0.0:60 36;’ where Variable_name='mysql-interfaces'; Prepare Proxysql
13 Add Ports to play later with: update global_variables set Variable_Value='0.0.0.0:6033;0.0.0.0:6034;0.0.0.0:6035;0.0.0.0:60 36;’ where Variable_name='mysql-interfaces'; Prepare Proxysql
13 Add Ports to play later with: update global_variables set Variable_Value='0.0.0.0:6033;0.0.0.0:6034;0.0.0.0:6035;0.0.0.0:60 36;’ where Variable_name='mysql-interfaces'; save mysql variables to disk; Prepare Proxysql
13 Add Ports to play later with: update global_variables set Variable_Value='0.0.0.0:6033;0.0.0.0:6034;0.0.0.0:6035;0.0.0.0:60 36;’ where Variable_name='mysql-interfaces'; save mysql variables to disk; Exit and restart proxysql Prepare Proxysql
14 MySQL Query Rules Table Sharding by: ● username ● schemaname ● client_addr ● proxy_addr ● proxy_port ● match_digest ● match_pattern Sharding all the logic is in:
15 Destination_hostgroup Pointing to different HG Replace_pattern Rewriting the Query Or if firewalling blocking the query Sharding Targets
16 Understand FlagIN FlagOUT
17 IP/port Schema using field=value Comment -> schema or host Use combination of host/schema Sharding Examples
18 IP/port Sharding Examples 6034 HG 10 6035 HG 20 6036 HG 30 Basic By IP/Port
19 For $port in 6034 6035 6036 do INSERT INTO mysql_query_rules (rule_id,active,username,proxy_port,destination_hostgroup,appl y) VALUES (10/11/12,1,'user_shardRW’, $port,10/20/30,1); Done LOAD MYSQL QUERY RULES TO RUNTIME;SAVE MYSQL QUERY RULES TO DISK; Sharding Examples – IP or Port settings
19 For $port in 6034 6035 6036 do INSERT INTO mysql_query_rules (rule_id,active,username,proxy_port,destination_hostgroup,appl y) VALUES (10/11/12,1,'user_shardRW’, $port,10/20/30,1); Done LOAD MYSQL QUERY RULES TO RUNTIME;SAVE MYSQL QUERY RULES TO DISK; Sharding Examples – IP or Port settings
20 rule_id |hg| username |schemaname |flagIN |flagOUT |apply|proxy_addr|proxy_port 41 |10| user_shardRW | NULL | 0 | NULL | 1 | NULL | 6034 42 |20| user_shardRW | NULL | 0 | NULL | 1 | NULL | 6035 43 |30| user_shardRW | NULL | 0 | NULL | 1 | NULL | 6036 Sharding Examples – IP or Port Result
20 rule_id |hg| username |schemaname |flagIN |flagOUT |apply|proxy_addr|proxy_port 41 |10| user_shardRW | NULL | 0 | NULL | 1 | NULL | 6034 42 |20| user_shardRW | NULL | 0 | NULL | 1 | NULL | 6035 43 |30| user_shardRW | NULL | 0 | NULL | 1 | NULL | 6036 Sharding Examples – IP or Port Result 6035 : HG 20 6036 : HG 30 6034 : HG 10
20 rule_id |hg| username |schemaname |flagIN |flagOUT |apply|proxy_addr|proxy_port 41 |10| user_shardRW | NULL | 0 | NULL | 1 | NULL | 6034 42 |20| user_shardRW | NULL | 0 | NULL | 1 | NULL | 6035 43 |30| user_shardRW | NULL | 0 | NULL | 1 | NULL | 6036 Sharding Examples – IP or Port Result 6035 : HG 20 6036 : HG 30 for port in 6034 6035 6036;do echo "---- $port ---"; mysql - uuser_shardRW -ptest -h 192.168.0.11 -P $port -e "show global variables like 'wsrep_node_name';";done ---- 6034/5/6 --- +-----------------+-------+ | Variable_name | Value | +-----------------+-------+ | wsrep_node_name | node1/2/3 +-----------------+-------+ 6034 : HG 10
21 Schema using field=value Query ReWrite Sharding Examples By parameter analysis and rewrite HG 10
22 Define a simple query rule using regular expression Query: SELECT name,population from world.City WHERE Continent=’Asia' and name like 'B%‘ order by population desc limit 1; match_pattern: ”^SELECTs*(.*)s*froms*world.(S*) s(.*).*Continent='(S*)'s*(s*.*)$” Replace_pattern: SELECT 1 from 4.2 WHERE 1=1 5 Sharding Examples Schema – Value - How
22 Define a simple query rule using regular expression Query: SELECT name,population from world.City WHERE Continent=’Asia' and name like 'B%‘ order by population desc limit 1; match_pattern: ”^SELECTs*(.*)s*froms*world.(S*) s(.*).*Continent='(S*)'s*(s*.*)$” Replace_pattern: SELECT 1 from 4.2 WHERE 1=1 5 Sharding Examples Schema – Value - How Comment HG 22 1 user_shardRW NULL 20 CASELESS,GLOBAL /*s*shard_host_HG=.*Africas**. NULL 0 NULL 0 Value Schema 31 1 user_shardRW NULL NULL CASELESS ^SELECTs*(.*)s*froms*world.(S*)s(.*).*Continent='(S*)'s*(s*.*)$ SELECT 1 from 4.2 WHERE 1=1 5 1 NULL 0
23 ProxySQL can read your query and identify a value SELECT name,population from world.City WHERE Continent=’Asia' and CountryCode=‘IND' order by population desc limit 1; Become SELECT name,population from Asia.City WHERE ?=? and CountryCode=? order by population desc limit ? Sharding Examples Schema - Value
24 Test for $continent in Asia Europe Africa North_America ;do echo "---- $continent ---"; mysql -uuser_shardRW -ptest -h 192.168.0.11 -P6033 -e ” SELECT name, population from world.City WHERE Continent=’$continent' and name like 'B%' order by population desc limit 1;” ;done Sharding Examples Schema – Value
25 To check what ProxySQL processed: hits: 4 !Because I run the script for 4 continents rule_id: 31 match_pattern: ^SELECTs*(.*)s*froms*world.(S*) s(.*).*Continent='(S*)'s*(s*.*)$ replace_pattern: SELECT 1 from 4.2 WHERE 1=1 5 cache_ttl: NULL apply: 1 select active,hits, mysql_query_rules.rule_id, match_digest, match_pattern, replace_pattern, apply FROM mysql_query_rules NATURAL JOIN stats.stats_mysql_query_rules ORDER BY mysql_query_rules.rule_idG Sharding Examples Schema: Value - Check
26 Comment Injection Causing: ●Query ReWrite Or ●HG redirection Or ●Both Sharding Examples
27 Comments are the easiest thing to implement in the application QueryRules chains (FlagIN – FlagOUT) What is a comment in MYSQL: Select name /* I am a comment*/ from world.City; Sharding Examples - Comment
27 Comments are the easiest thing to implement in the application QueryRules chains (FlagIN – FlagOUT) What is a comment in MYSQL: Select name /* I am a comment*/ from world.City; If testing from mysql console do not forget the flag “-c” Sharding Examples - Comment
28 Use comments to hit Different schemas Sharding Examples - Comment /* Asia */ /* Europe */ /* Africa */ Asia Europe Africa
29 Given a query: Original : Select /* continent=Asia */ * from world.Country join world.City on world.City.CountryCode=world.Country.Code where Country.code='IND’ Transformed: Select * from Asia.Country join Asia.City on Asia.City.CountryCode=Asia.Country.Code where Country.code=? Sharding Examples – Comment - How
30 2 Rules: INSERT INTO mysql_query_rules (rule_id,active,username,match_pattern,replace_pattern,apply,FlagOUT,Fl agIN) VALUES (32,1,'user_shardRW',"S*s*/ *s*continent=.*Asias**.*",null,0,23,0); INSERT INTO mysql_query_rules (rule_id,active,username,match_pattern,replace_pattern,apply,FlagIN,Fla gOUT) VALUES (33,1,'user_shardRW','world.','Asia.',0,23,23); Sharding Examples - Comment
30 2 Rules: INSERT INTO mysql_query_rules (rule_id,active,username,match_pattern,replace_pattern,apply,FlagOUT,Fl agIN) VALUES (32,1,'user_shardRW',"S*s*/ *s*continent=.*Asias**.*",null,0,23,0); INSERT INTO mysql_query_rules (rule_id,active,username,match_pattern,replace_pattern,apply,FlagIN,Fla gOUT) VALUES (33,1,'user_shardRW','world.','Asia.',0,23,23); Sharding Examples - Comment
31 Sharding Examples – Comment - How hits: 1 rule_id: 32 match_pattern: /*s*continent=.*Asias* replace_pattern: NULL FlagIN: 0 FlagOUT: 23 hits: 1 (old versions (1.3.x: 4) rule_id: 33 match_pattern: world. replace_pattern: Asia. FlagIN: 23 FlagOUT: 23 Select hits, mysql_query_rules.rule_id, match_digest, match_pattern, replace_pattern FROM mysql_query_rules NATURAL JOIN stats.stats_mysql_query_rules ORDER BY mysql_query_rules.rule_idG
32 Point to a HG directly is even easier Sharding Examples – Comment – To HG /* Asia */ /* Europe */ /* Africa */
33 Point to a HG directly is even easier The scope is that whatever match the rule, should go to a defined HG. INSERT INTO mysql_query_rules (rule_id, active, username, match_pattern, destination_hostgroup,apply) VALUES (10,1,'user_shardRW',"/*s*shard_host_HG=.*Europes**.", 10,0); Sharding Examples – Comment – To HG
33 Point to a HG directly is even easier The scope is that whatever match the rule, should go to a defined HG. INSERT INTO mysql_query_rules (rule_id, active, username, match_pattern, destination_hostgroup,apply) VALUES (10,1,'user_shardRW',"/*s*shard_host_HG=.*Europes**.", 10,0); Sharding Examples – Comment – To HG
34 Sharding Examples – Comment - How hits: 1 rule_id: 20 match_pattern: /*s*continent=.*Asias* replace_pattern: NULL FlagIN: 0 FlagOUT: NULL destination_hostgroup: 10 apply: 0 Select hits, mysql_query_rules.rule_id, match_digest, match_pattern, replace_pattern FROM mysql_query_rules NATURAL JOIN stats.stats_mysql_query_rules ORDER BY mysql_query_rules.rule_idG
35 Use both, by HG and Query Rewrite Sharding Examples – Comment: HG + Schema
36 Finally is possible to mix the two things. I want to use: Asia on Server 192.168.1.5 -> HG 10, points to a COUNTRY schema Europe on Server 192.168.1.6 -> HG 20, points to a COUNTRY schema Africa on Server 192.168.1.7 -> HG 30, points to a COUNTRY schema Select /* shard_host_HG=Asia */ /* country=India */ City.Name, City.Population from world.Country join world.City on world.City.CountryCode=world.Country.Code limit 5; Sharding Examples – Comment: HG + Schema
37 Sharding Examples - Comment 2 Rules for country : INSERT INTO mysql_query_rules (rule_id,active,username,match_pattern,replace_pattern,apply,FlagOUT,Fl agIN) VALUES (32,1,'user_shardRW',"S*s*/ *s*country=.*Indias**.*",null,0,23,0); INSERT INTO mysql_query_rules (rule_id,active,username,match_pattern,replace_pattern,apply,FlagIN,Fla gOUT) VALUES (33,1,'user_shardRW','world.',’India.',0,23,23);
38 Sharding Examples – Comment - How Three different rules to achieve the result. Order is important QR ID: 10 match_pattern: /*s*continent=.*Asias* Target: HG Apply: 0 QR ID: 32 match_pattern: /*s*country=.*Indias* Target: Query Rule 33 Apply: 0 QR ID: 33 match_pattern: world. Target: Query Rewrite to Country schema Apply: 0
39 Problem Real case
39 Problem ●Customer must reduce the number of shards Real case
39 Problem ●Customer must reduce the number of shards ●Aggregate several schemas in multitenant DB Real case
39 Problem ●Customer must reduce the number of shards ●Aggregate several schemas in multitenant DB ●Collapse several servers in few Real case
39 Problem ●Customer must reduce the number of shards ●Aggregate several schemas in multitenant DB ●Collapse several servers in few ●No code change (as usual) or minimal Real case
39 Problem ●Customer must reduce the number of shards ●Aggregate several schemas in multitenant DB ●Collapse several servers in few ●No code change (as usual) or minimal ●Java with Hibernate Real case
40 Solution Real case
40 Solution ●Inject limited comment in the query ● /*shard_ip=x.x.x.x,db=my_db,grp_id=123 */ Real case
40 Solution ●Inject limited comment in the query ● /*shard_ip=x.x.x.x,db=my_db,grp_id=123 */ ●Create Query rules to manage ●4 rules to manage all + 1 for each additional server (HG) Real case
41 The query: /* shard_ip=10.11.4.80,db=grp_aaaa,grp_id=23456 */ SELECT * FROM grp_1211.users where pippo like tt; Real case
42 The QRs: INSERT INTO mysql_query_rules (rule_id,active,username,match_pattern,apply,FlagIN,FlagOUT) VALUES (10,1,'apptest1','where',0,0,20); 
 INSERT INTO mysql_query_rules (rule_id,active,username,match_pattern,replace_pattern,apply,FlagIN,FlagOUT) VALUES (15,1,'apptest1',"(/*.*,)grp_id=(d*)s**/s*(.*).*$","1 */ 3 WHERE grp_id=2;",0,0,25);
 
 INSERT INTO mysql_query_rules (rule_id,active,username,match_pattern,replace_pattern,apply,FlagIN,FlagOUT) VALUES (18,1,'apptest1',"(/*.*,)grp_id=(d*)s**/(.*)s*(where)s*(.*).*$","1 */ 3 WHERE grp_id=2 AND 5;",0,20,25);
 
 INSERT INTO mysql_query_rules (rule_id,active,username,match_pattern,replace_pattern,apply,FlagIN,FlagOUT) VALUES (22,1,'apptest1',"(^/*.*,)db=(S*),s**/s*(.*(FROM|UPDATE|INSERT INTO))s*(.*).(.*)","1 */ 3 2.6",0,25,25); INSERT INTO mysql_query_rules (rule_id,active,username,match_pattern,destination_hostgroup,apply,FlagIN,FlagOUT) VALUES (30,1,'apptest1',"shard_ip=10.11.4.79",20,1,25,NULL); Real case
43 Starting query: /* shard_ip=10.11.4.80,db=grp_aaaa,grp_id=23456 */ SELECT * FROM grp_1211.users where pippo like tt; Real case
43 Starting query: /* shard_ip=10.11.4.80,db=grp_aaaa,grp_id=23456 */ SELECT * FROM grp_1211.users where pippo like tt; Become: /* shard_ip=10.11.4.80,db=grp_aaaa, */ SELECT * FROM grp_1211.users where grp_id=23456 AND pippo like tt; Real case
43 Starting query: /* shard_ip=10.11.4.80,db=grp_aaaa,grp_id=23456 */ SELECT * FROM grp_1211.users where pippo like tt; Become: /* shard_ip=10.11.4.80,db=grp_aaaa, */ SELECT * FROM grp_1211.users where grp_id=23456 AND pippo like tt; /* shard_ip=10.11.4.80,*/ SELECT * FROM grp_aaaa.users WHERE grp_id=23456 AND pippo like tt; Ready to be dispatch to final HG with rules above id 30 Real case
44 Real case
44 Obviously if a customer can screw your perfect plan …. Real case
44 Obviously if a customer can screw your perfect plan …. … he will. Real case
44 Obviously if a customer can screw your perfect plan …. … he will. ●Try to use NON standard SQL like: INSERT… SET (MySQL extension) Real case
44 Obviously if a customer can screw your perfect plan …. … he will. ●Try to use NON standard SQL like: INSERT… SET (MySQL extension) ●Omit the schema name in the SQL Real case
45 There is no magic solution for sharding.
 ProxySQL is a tool to help not a sharding mechanism. ● Per port/IP (require to be define at infrastructure design time) ● Query filtering/rewrite (may impact performance if many QR, and require application to be aware of the shards) ● Cannot pass variables cross QR ● Queries are NOT spanning on multiple HG. ● Cannot aggregate resultsets from different shards It is NOT a kind of magic
46 Cost in Execution (trx in 5 minutes) Review the blogs: https://www.percona.com/blog/2017/04/12/proxysql-applying-and- chaining-the-rules/ https://www.percona.com/blog/2017/04/10/proxysql-rules-do-i-have-too-many/
47 Multiple element to use to shard:
 User/default schema/IP-Port/value/Hint Shards can be all pointing to same HG different schemas; pointing to different HG or a Mix Plan Your QR and chains to avoid performance degradation Brief summary
48 http://www.proxysql.com/ ProxySQL http://www.pcre.org/ Perl Compatible Regular Expressions https://goo.gl/cMkngr Sharding with ProxySQL https://goo.gl/vovszl stop an offending query with ProxySQL https://goo.gl/OC8RbN ProxySQL Rules: Do I Have Too Many? https://goo.gl/rzYDr7 Applying and Chaining the Rules https://www.percona.com/blog 
 To Know more
49 René Cannaò for ProxySQL Tibor Köröcz for performance Tests Some personal thanks to:
50
51
52 Contacts To contact Me Marco.tusa@percona.com marcotusa@tusacentral.net To follow me http://www.tusacentral.net/ http://www.percona.com/blog/ https://www.facebook.com/marco.tusa.94 @marcotusa http://it.linkedin.com/in/marcotusa/ “Consulting = No mission refused!”

More Related Content

PDF
MySQL/MariaDB Proxy Software Test
byI Goo Lee
 
PDF
ProxySQL High Avalability and Configuration Management Overview
byRené Cannaò
 
PDF
ProxySQL - High Performance and HA Proxy for MySQL
byRené Cannaò
 
PPTX
MySQL8.0_performance_schema.pptx
byNeoClova
 
PDF
Redo log improvements MYSQL 8.0
byMydbops
 
DOCX
Keepalived+MaxScale+MariaDB_운영매뉴얼_1.0.docx
byNeoClova
 
PDF
Maxscale switchover, failover, and auto rejoin
byWagner Bianchi
 
PDF
MariaDB Performance Tuning and Optimization
byMariaDB plc
 
MySQL/MariaDB Proxy Software Test
byI Goo Lee
 
ProxySQL High Avalability and Configuration Management Overview
byRené Cannaò
 
ProxySQL - High Performance and HA Proxy for MySQL
byRené Cannaò
 
MySQL8.0_performance_schema.pptx
byNeoClova
 
Redo log improvements MYSQL 8.0
byMydbops
 
Keepalived+MaxScale+MariaDB_운영매뉴얼_1.0.docx
byNeoClova
 
Maxscale switchover, failover, and auto rejoin
byWagner Bianchi
 
MariaDB Performance Tuning and Optimization
byMariaDB plc
 

What's hot

PDF
MariaDB 마이그레이션 - 네오클로바
byNeoClova
 
PDF
MySQL Database Architectures - InnoDB ReplicaSet & Cluster
byKenny Gryp
 
PDF
MariaDB MaxScale monitor 매뉴얼
byNeoClova
 
PDF
[2018] MySQL 이중화 진화기
byNHN FORWARD
 
PDF
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
byJesmar Cannao'
 
PDF
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
byAltinity Ltd
 
PPTX
Maria db 이중화구성_고민하기
byNeoClova
 
PPTX
ProxySQL for MySQL
byMydbops
 
PDF
MySQL GTID 시작하기
byI Goo Lee
 
PPTX
Query logging with proxysql
byYoungHeon (Roy) Kim
 
PDF
Transparent sharding with Spider: what's new and getting started
byMariaDB plc
 
PDF
ProxySQL Tutorial - PLAM 2016
byDerek Downey
 
PDF
[오픈소스컨설팅]Day #1 MySQL 엔진소개, 튜닝, 백업 및 복구, 업그레이드방법
byJi-Woong Choi
 
PDF
What is new in PostgreSQL 14?
byMydbops
 
PPTX
ProxySQL & PXC(Query routing and Failover Test)
byYoungHeon (Roy) Kim
 
PDF
All about Zookeeper and ClickHouse Keeper.pdf
byAltinity Ltd
 
PDF
Galera cluster for high availability
byMydbops
 
PDF
ClickHouse Monitoring 101: What to monitor and how
byAltinity Ltd
 
PDF
MariaDB 10.11 key features overview for DBAs
byFederico Razzoli
 
PDF
Automated master failover
byYoshinori Matsunobu
 
MariaDB 마이그레이션 - 네오클로바
byNeoClova
 
MySQL Database Architectures - InnoDB ReplicaSet & Cluster
byKenny Gryp
 
MariaDB MaxScale monitor 매뉴얼
byNeoClova
 
[2018] MySQL 이중화 진화기
byNHN FORWARD
 
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
byJesmar Cannao'
 
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
byAltinity Ltd
 
Maria db 이중화구성_고민하기
byNeoClova
 
ProxySQL for MySQL
byMydbops
 
MySQL GTID 시작하기
byI Goo Lee
 
Query logging with proxysql
byYoungHeon (Roy) Kim
 
Transparent sharding with Spider: what's new and getting started
byMariaDB plc
 
ProxySQL Tutorial - PLAM 2016
byDerek Downey
 
[오픈소스컨설팅]Day #1 MySQL 엔진소개, 튜닝, 백업 및 복구, 업그레이드방법
byJi-Woong Choi
 
What is new in PostgreSQL 14?
byMydbops
 
ProxySQL & PXC(Query routing and Failover Test)
byYoungHeon (Roy) Kim
 
All about Zookeeper and ClickHouse Keeper.pdf
byAltinity Ltd
 
Galera cluster for high availability
byMydbops
 
ClickHouse Monitoring 101: What to monitor and how
byAltinity Ltd
 
MariaDB 10.11 key features overview for DBAs
byFederico Razzoli
 
Automated master failover
byYoshinori Matsunobu
 

Similar to Proxysql sharding

PDF
HandsOn ProxySQL Tutorial - PLSC18
byDerek Downey
 
PDF
ProxySQL Cluster - Percona Live 2022
byRené Cannaò
 
PDF
ProxySQL High Availability (Clustering)
byMydbops
 
PPTX
Proxysql use case scenarios fosdem17
byAlkin Tezuysal
 
PDF
ProxySQL in the Cloud
byRené Cannaò
 
PDF
MySQL High Availability Solutions
byLenz Grimmer
 
PDF
Proxysql ha plam_2016_2_keynote
byMarco Tusa
 
PDF
Webinar slides: MySQL & MariaDB load balancing with ProxySQL & ClusterControl...
bySeveralnines
 
PDF
Massively Distributed Backups at Facebook Scale - Shlomo Priymak, Facebook - ...
byDevOpsDays Tel Aviv
 
PDF
Shard-Query, an MPP database for the cloud using the LAMP stack
byJustin Swanhart
 
PDF
Fortify aws aurora_proxy
byMarco Tusa
 
PDF
Fortify aws aurora_proxy_2019_pleu
byMarco Tusa
 
PDF
Deploying Percona XtraDB Cluster in Openshift
byAlexander Rubin
 
PPTX
DB Floripa - ProxySQL para MySQL
byMarcelo Altmann
 
PDF
Percona Live 4/15/15: Transparent sharding database virtualization engine (DVE)
byTesora
 
PDF
4 docker small_web_micro_services
byFEG
 
PDF
Robust ha solutions with proxysql
byMarco Tusa
 
PDF
Mysqlhacodebits20091203 1260184765-phpapp02
byLouis liu
 
PDF
MySQL High Availability Solutions
byLenz Grimmer
 
PPTX
Executing Queries on a Sharded Database
byNeha Narula
 
HandsOn ProxySQL Tutorial - PLSC18
byDerek Downey
 
ProxySQL Cluster - Percona Live 2022
byRené Cannaò
 
ProxySQL High Availability (Clustering)
byMydbops
 
Proxysql use case scenarios fosdem17
byAlkin Tezuysal
 
ProxySQL in the Cloud
byRené Cannaò
 
MySQL High Availability Solutions
byLenz Grimmer
 
Proxysql ha plam_2016_2_keynote
byMarco Tusa
 
Webinar slides: MySQL & MariaDB load balancing with ProxySQL & ClusterControl...
bySeveralnines
 
Massively Distributed Backups at Facebook Scale - Shlomo Priymak, Facebook - ...
byDevOpsDays Tel Aviv
 
Shard-Query, an MPP database for the cloud using the LAMP stack
byJustin Swanhart
 
Fortify aws aurora_proxy
byMarco Tusa
 
Fortify aws aurora_proxy_2019_pleu
byMarco Tusa
 
Deploying Percona XtraDB Cluster in Openshift
byAlexander Rubin
 
DB Floripa - ProxySQL para MySQL
byMarcelo Altmann
 
Percona Live 4/15/15: Transparent sharding database virtualization engine (DVE)
byTesora
 
4 docker small_web_micro_services
byFEG
 
Robust ha solutions with proxysql
byMarco Tusa
 
Mysqlhacodebits20091203 1260184765-phpapp02
byLouis liu
 
MySQL High Availability Solutions
byLenz Grimmer
 
Executing Queries on a Sharded Database
byNeha Narula
 

More from Marco Tusa

PDF
Galera explained 3
byMarco Tusa
 
PDF
Comparing high availability solutions with percona xtradb cluster and percona...
byMarco Tusa
 
PDF
Percona xtra db cluster(pxc) non blocking operations, what you need to know t...
byMarco Tusa
 
PDF
MySQL innoDB split and merge pages
byMarco Tusa
 
PDF
MySQL developing Store Procedure
byMarco Tusa
 
PPT
Empower my sql server administration with 5.7 instruments
byMarco Tusa
 
PDF
Geographically dispersed perconaxtra db cluster deployment
byMarco Tusa
 
PDF
Accessing Data Through Hibernate; What DBAs Should Tell Developers and Vice V...
byMarco Tusa
 
PDF
Accessing data through hibernate: what DBAs should tell to developers and vic...
byMarco Tusa
 
PDF
Mysql8 advance tuning with resource group
byMarco Tusa
 
PDF
My sql on kubernetes demystified
byMarco Tusa
 
PPTX
Improve aws withproxysql
byMarco Tusa
 
PDF
Discard inport exchange table & tablespace
byMarco Tusa
 
PDF
Scaling with sync_replication using Galera and EC2
byMarco Tusa
 
PDF
MySQL cluster 72 in the Cloud
byMarco Tusa
 
PDF
Are we there Yet?? (The long journey of Migrating from close source to opens...
byMarco Tusa
 
PDF
Sync rep aurora_2016
byMarco Tusa
 
PDF
Plmce 14 be a_hero_16x9_final
byMarco Tusa
 
PDF
MySQL overview
byMarco Tusa
 
PDF
Best practice-high availability-solution-geo-distributed-final
byMarco Tusa
 
Galera explained 3
byMarco Tusa
 
Comparing high availability solutions with percona xtradb cluster and percona...
byMarco Tusa
 
Percona xtra db cluster(pxc) non blocking operations, what you need to know t...
byMarco Tusa
 
MySQL innoDB split and merge pages
byMarco Tusa
 
MySQL developing Store Procedure
byMarco Tusa
 
Empower my sql server administration with 5.7 instruments
byMarco Tusa
 
Geographically dispersed perconaxtra db cluster deployment
byMarco Tusa
 
Accessing Data Through Hibernate; What DBAs Should Tell Developers and Vice V...
byMarco Tusa
 
Accessing data through hibernate: what DBAs should tell to developers and vic...
byMarco Tusa
 
Mysql8 advance tuning with resource group
byMarco Tusa
 
My sql on kubernetes demystified
byMarco Tusa
 
Improve aws withproxysql
byMarco Tusa
 
Discard inport exchange table & tablespace
byMarco Tusa
 
Scaling with sync_replication using Galera and EC2
byMarco Tusa
 
MySQL cluster 72 in the Cloud
byMarco Tusa
 
Are we there Yet?? (The long journey of Migrating from close source to opens...
byMarco Tusa
 
Sync rep aurora_2016
byMarco Tusa
 
Plmce 14 be a_hero_16x9_final
byMarco Tusa
 
MySQL overview
byMarco Tusa
 
Best practice-high availability-solution-geo-distributed-final
byMarco Tusa
 

Recently uploaded

PDF
Introducing Cohesity NetBackup (2).pdf
byNestorNavarro20
 
PDF
10 PyData Piraeus Meetup: Retrieval and Matching at Scale: from embeddings to...
byPyData Piraeus
 
PPTX
IT231_Presenation-Template oftentimes-1.pptx
bygoneweldon026
 
PPTX
essential in computer science structure and other
byhncseven7
 
PDF
[DSC Europe 25] Nikola Rajovic - Hardware Technologies Under the Hood: RISC-V...
byDataScienceConferenc1
 
PDF
DiaryofaWebSDKImplemenation-LaunchDebugging.pdf
bygmuir1066
 
PPTX
Data Science in IOT which is utilized by .pptx
byyanKhan
 
PPTX
[DSC Europe 25] Vladimir Jelic - The AI-Driven Security Shift From Reactive D...
byDataScienceConferenc1
 
PPTX
491456461_laudon_mis16_ppt_ch11_KL_CE_Updated_Content_for_2021_Managing.pptx
byFantaneshTegegn1
 
PDF
Data+Security+Masterclass Presentation.pdf
byckurt9676
 
PPTX
Safe_Spaces_Act_Modern_Presentation.pptx
byairesmijares2003
 
PPTX
[DSC Europe 25] Marko Krstic - Understanding the AI Threat Landscape - Risks,...
byDataScienceConferenc1
 
PPSX
[DSC Europe 25] Milan Sekuloski - Data, Defence, and Development: Cybersecuri...
byDataScienceConferenc1
 
PPTX
Orientation-on-ARAL-Check-in-Assessment.pptx
byJenTeruel1
 
PPTX
[DSC Europe 25] Goran Obradovic - The Rise of Sovereign AI: Building the Regi...
byDataScienceConferenc1
 
PPTX
[DSC Europe 25] Andy Cotgreave - Nothing is new in analytics.pptx
byDataScienceConferenc1
 
PDF
OPPOTUS - Malaysians on Malaysia (MOM) Q3 2025
byOppotus
 
PDF
[DSC Europe 25] Sara Polak - The Archaeology of Innovation: AI as the Next Cr...
byDataScienceConferenc1
 
PPTX
GRADE 8 MATH - Q3 - LESSON 1 -LINEAR EQUATION IN ONE VARIABLE.pptx
byEdlynMalangPanti
 
PPTX
[DSC Europe 25] Dragan Vucic - Building the Learning Organization - How AI Tr...
byDataScienceConferenc1
 
Introducing Cohesity NetBackup (2).pdf
byNestorNavarro20
 
10 PyData Piraeus Meetup: Retrieval and Matching at Scale: from embeddings to...
byPyData Piraeus
 
IT231_Presenation-Template oftentimes-1.pptx
bygoneweldon026
 
essential in computer science structure and other
byhncseven7
 
[DSC Europe 25] Nikola Rajovic - Hardware Technologies Under the Hood: RISC-V...
byDataScienceConferenc1
 
DiaryofaWebSDKImplemenation-LaunchDebugging.pdf
bygmuir1066
 
Data Science in IOT which is utilized by .pptx
byyanKhan
 
[DSC Europe 25] Vladimir Jelic - The AI-Driven Security Shift From Reactive D...
byDataScienceConferenc1
 
491456461_laudon_mis16_ppt_ch11_KL_CE_Updated_Content_for_2021_Managing.pptx
byFantaneshTegegn1
 
Data+Security+Masterclass Presentation.pdf
byckurt9676
 
Safe_Spaces_Act_Modern_Presentation.pptx
byairesmijares2003
 
[DSC Europe 25] Marko Krstic - Understanding the AI Threat Landscape - Risks,...
byDataScienceConferenc1
 
[DSC Europe 25] Milan Sekuloski - Data, Defence, and Development: Cybersecuri...
byDataScienceConferenc1
 
Orientation-on-ARAL-Check-in-Assessment.pptx
byJenTeruel1
 
[DSC Europe 25] Goran Obradovic - The Rise of Sovereign AI: Building the Regi...
byDataScienceConferenc1
 
[DSC Europe 25] Andy Cotgreave - Nothing is new in analytics.pptx
byDataScienceConferenc1
 
OPPOTUS - Malaysians on Malaysia (MOM) Q3 2025
byOppotus
 
[DSC Europe 25] Sara Polak - The Archaeology of Innovation: AI as the Next Cr...
byDataScienceConferenc1
 
GRADE 8 MATH - Q3 - LESSON 1 -LINEAR EQUATION IN ONE VARIABLE.pptx
byEdlynMalangPanti
 
[DSC Europe 25] Dragan Vucic - Building the Learning Organization - How AI Tr...
byDataScienceConferenc1
 
In this document
Powered by AI

Introduces ProxySQL, its creator Marco Tusa, and provides an agenda for sharding examples using the world database.

Discusses fundamental concepts of sharding, including basic strategies by IP/Port distribution and preparing MySQL for sharding.Details on HostGroup (HG), replication settings, and the structure necessary to understand sharding logic and configurations.

Explains how ProxySQL links users, schemas, IP, and SQL to replication Host Groups through query rules.

Instructions on setting up schemas and users for sharding, adjusting MySQL and ProxySQL for optimal configuration.

Shows how to create query rules for sharding, explanation of sharding examples using IP, schema values, and comment injections.

Discusses using comments in queries for sharding, rules for comment injections, and how to manage sharding with comments.

Describes a real-life issue faced by a customer regarding sharding, and potential solutions leveraging ProxySQL.

Concludes with caution against expecting magic solutions for sharding, emphasizing ProxySQL as a supporting tool.

Provides resources for further learning about ProxySQL, sharding, and ways to connect with the speaker and contributors.

Proxysql sharding

  • 1.
    ProxySQL Sharding Marco (theGrinch) Tusa September 2017 Dublin
  • 2.
    2 ● Marco “TheGrinch” • Open source enthusiast • In love with ProxySQL • Consulting team leader • Consultant myself About me
  • 3.
    3 Example of ProxySQL/MySQLsetup Understand Query Rules Examples: IP/port Schema using field=value Comment -> schema or host Use combination of host/schema Real life case of need to reduce sharding A lot of code so you can reuse Agenda
  • 4.
    4 Examples will useworld DB. http://downloads.mysql.com/docs/world-db.tar.gz Sharding …
  • 5.
  • 6.
    5 Way of Sharding 6034 HG10 6035 HG 20 6036 HG 30 Basic By IP/Port
  • 7.
    5 Way of Sharding 6034 HG10 6035 HG 20 6036 HG 30 Basic By IP/Port By parameter capture or comment injection Performing query rewrite HG 10
  • 8.
    5 Way of Sharding 6034 HG10 6035 HG 20 6036 HG 30 Basic By IP/Port By parameter capture or comment injection Performing query rewrite Comment Injection/ HG Direct. Query analysis and rewrite HG 10
  • 9.
    6 You will need: 3hosts: 192.168.0.[5-21-231] 4 schemas: Continent X + world schema 1 user : user_shardRW 3 hostgroups: 10, 20, 30 Prepare MySQL
  • 10.
    7 HostGroup (HG) &Replication HG Logical group of server Unless specify HG receives reads and writes Properties: id; Server list Special Replicaton Host Group (RHG) Manage the separation between Write and Read HG. Properties: writer_hostgroup ID reader_hostgroup ID Understand the basics
  • 11.
    8 HostGroup (HG) &Replication HG Understand the basics All writers PXC
  • 12.
    8 HostGroup (HG) &Replication HG Understand the basics All writers PXC 1 writer more readers PXC with weight
  • 13.
    8 HostGroup (HG) &Replication HG Understand the basics All writers PXC 1 writer more readers PXC with weight Master - Slave
  • 14.
    8 HostGroup (HG) &Replication HG Understand the basics All writers PXC 1 writer more readers PXC with weight Master - Slave
  • 15.
    9 Is the logicin ProxySQL that link Users/Schema/IP/Port/SQL to (Replication) Host Group. OR That allow SQL rewrite. And more What is a Query rule?
  • 16.
    9 Is the logicin ProxySQL that link Users/Schema/IP/Port/SQL to (Replication) Host Group. OR That allow SQL rewrite. And more What is a Query rule?
  • 17.
    9 Is the logicin ProxySQL that link Users/Schema/IP/Port/SQL to (Replication) Host Group. OR That allow SQL rewrite. And more What is a Query rule?
  • 18.
    9 Is the logicin ProxySQL that link Users/Schema/IP/Port/SQL to (Replication) Host Group. OR That allow SQL rewrite. And more What is a Query rule?
  • 19.
    9 Is the logicin ProxySQL that link Users/Schema/IP/Port/SQL to (Replication) Host Group. OR That allow SQL rewrite. And more What is a Query rule?
  • 20.
    9 Is the logicin ProxySQL that link Users/Schema/IP/Port/SQL to (Replication) Host Group. OR That allow SQL rewrite. And more What is a Query rule?
  • 21.
    9 Is the logicin ProxySQL that link Users/Schema/IP/Port/SQL to (Replication) Host Group. OR That allow SQL rewrite. And more What is a Query rule?
  • 22.
    9 Is the logicin ProxySQL that link Users/Schema/IP/Port/SQL to (Replication) Host Group. OR That allow SQL rewrite. And more What is a Query rule?
  • 23.
    9 Is the logicin ProxySQL that link Users/Schema/IP/Port/SQL to (Replication) Host Group. OR That allow SQL rewrite. And more What is a Query rule?
  • 24.
    9 Is the logicin ProxySQL that link Users/Schema/IP/Port/SQL to (Replication) Host Group. OR That allow SQL rewrite. And more What is a Query rule?
  • 25.
    10 Load DB testWorld Create schemas Asia|Europe|North_America|Africa For each continent: create table {continent}.City as select a.* from world.City a join Country on a.CountryCode = Country.code where Continent='{continent}’; create table {continent}.Country as select * from world.Country where Continent='{continent}' ; End for grant all on *.* to user_shardRW@'%' identified by 'test'; Prepare MySQL
  • 26.
    11 What you needto do is: Set the user insert into mysql_users (username,password,active,default_hostgroup,default_schema,transaction_persistent) values ('user_shardRW','test',1,10,'test',1); LOAD MYSQL USERS TO RUNTIME;SAVE MYSQL USERS TO DISK; Set variable: update global_variables set Variable_Value=0 where Variable_name='mysql- query_cache_size_MB'; update global_variables set Variable_Value=0/20 where Variable_name='mysql- query_processor_iterations'; load mysql variables to run;save mysql variables to disk; Prepare Proxysql
  • 27.
    12 Create basic setof server to play with like: INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.0.5',10,3306,100); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.0.21',20,3306,100); INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.0.231',30,3306,100); LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK; Prepare Proxysql
  • 28.
    13 Add Ports toplay later with: update global_variables set Variable_Value='0.0.0.0:6033;0.0.0.0:6034;0.0.0.0:6035;0.0.0.0:60 36;’ where Variable_name='mysql-interfaces'; Prepare Proxysql
  • 29.
    13 Add Ports toplay later with: update global_variables set Variable_Value='0.0.0.0:6033;0.0.0.0:6034;0.0.0.0:6035;0.0.0.0:60 36;’ where Variable_name='mysql-interfaces'; Prepare Proxysql
  • 30.
    13 Add Ports toplay later with: update global_variables set Variable_Value='0.0.0.0:6033;0.0.0.0:6034;0.0.0.0:6035;0.0.0.0:60 36;’ where Variable_name='mysql-interfaces'; save mysql variables to disk; Prepare Proxysql
  • 31.
    13 Add Ports toplay later with: update global_variables set Variable_Value='0.0.0.0:6033;0.0.0.0:6034;0.0.0.0:6035;0.0.0.0:60 36;’ where Variable_name='mysql-interfaces'; save mysql variables to disk; Exit and restart proxysql Prepare Proxysql
  • 32.
    14 MySQL Query RulesTable Sharding by: ● username ● schemaname ● client_addr ● proxy_addr ● proxy_port ● match_digest ● match_pattern Sharding all the logic is in:
  • 33.
    15 Destination_hostgroup Pointing to differentHG Replace_pattern Rewriting the Query Or if firewalling blocking the query Sharding Targets
  • 34.
  • 35.
    17 IP/port Schema using field=value Comment-> schema or host Use combination of host/schema Sharding Examples
  • 36.
  • 37.
    19 For $port in6034 6035 6036 do INSERT INTO mysql_query_rules (rule_id,active,username,proxy_port,destination_hostgroup,appl y) VALUES (10/11/12,1,'user_shardRW’, $port,10/20/30,1); Done LOAD MYSQL QUERY RULES TO RUNTIME;SAVE MYSQL QUERY RULES TO DISK; Sharding Examples – IP or Port settings
  • 38.
    19 For $port in6034 6035 6036 do INSERT INTO mysql_query_rules (rule_id,active,username,proxy_port,destination_hostgroup,appl y) VALUES (10/11/12,1,'user_shardRW’, $port,10/20/30,1); Done LOAD MYSQL QUERY RULES TO RUNTIME;SAVE MYSQL QUERY RULES TO DISK; Sharding Examples – IP or Port settings
  • 39.
    20 rule_id |hg| username|schemaname |flagIN |flagOUT |apply|proxy_addr|proxy_port 41 |10| user_shardRW | NULL | 0 | NULL | 1 | NULL | 6034 42 |20| user_shardRW | NULL | 0 | NULL | 1 | NULL | 6035 43 |30| user_shardRW | NULL | 0 | NULL | 1 | NULL | 6036 Sharding Examples – IP or Port Result
  • 40.
    20 rule_id |hg| username|schemaname |flagIN |flagOUT |apply|proxy_addr|proxy_port 41 |10| user_shardRW | NULL | 0 | NULL | 1 | NULL | 6034 42 |20| user_shardRW | NULL | 0 | NULL | 1 | NULL | 6035 43 |30| user_shardRW | NULL | 0 | NULL | 1 | NULL | 6036 Sharding Examples – IP or Port Result 6035 : HG 20 6036 : HG 30 6034 : HG 10
  • 41.
    20 rule_id |hg| username|schemaname |flagIN |flagOUT |apply|proxy_addr|proxy_port 41 |10| user_shardRW | NULL | 0 | NULL | 1 | NULL | 6034 42 |20| user_shardRW | NULL | 0 | NULL | 1 | NULL | 6035 43 |30| user_shardRW | NULL | 0 | NULL | 1 | NULL | 6036 Sharding Examples – IP or Port Result 6035 : HG 20 6036 : HG 30 for port in 6034 6035 6036;do echo "---- $port ---"; mysql - uuser_shardRW -ptest -h 192.168.0.11 -P $port -e "show global variables like 'wsrep_node_name';";done ---- 6034/5/6 --- +-----------------+-------+ | Variable_name | Value | +-----------------+-------+ | wsrep_node_name | node1/2/3 +-----------------+-------+ 6034 : HG 10
  • 42.
    21 Schema using field=value Query ReWrite ShardingExamples By parameter analysis and rewrite HG 10
  • 43.
    22 Define a simplequery rule using regular expression Query: SELECT name,population from world.City WHERE Continent=’Asia' and name like 'B%‘ order by population desc limit 1; match_pattern: ”^SELECTs*(.*)s*froms*world.(S*) s(.*).*Continent='(S*)'s*(s*.*)$” Replace_pattern: SELECT 1 from 4.2 WHERE 1=1 5 Sharding Examples Schema – Value - How
  • 44.
    22 Define a simplequery rule using regular expression Query: SELECT name,population from world.City WHERE Continent=’Asia' and name like 'B%‘ order by population desc limit 1; match_pattern: ”^SELECTs*(.*)s*froms*world.(S*) s(.*).*Continent='(S*)'s*(s*.*)$” Replace_pattern: SELECT 1 from 4.2 WHERE 1=1 5 Sharding Examples Schema – Value - How Comment HG 22 1 user_shardRW NULL 20 CASELESS,GLOBAL /*s*shard_host_HG=.*Africas**. NULL 0 NULL 0 Value Schema 31 1 user_shardRW NULL NULL CASELESS ^SELECTs*(.*)s*froms*world.(S*)s(.*).*Continent='(S*)'s*(s*.*)$ SELECT 1 from 4.2 WHERE 1=1 5 1 NULL 0
  • 45.
    23 ProxySQL can readyour query and identify a value SELECT name,population from world.City WHERE Continent=’Asia' and CountryCode=‘IND' order by population desc limit 1; Become SELECT name,population from Asia.City WHERE ?=? and CountryCode=? order by population desc limit ? Sharding Examples Schema - Value
  • 46.
    24 Test for $continent inAsia Europe Africa North_America ;do echo "---- $continent ---"; mysql -uuser_shardRW -ptest -h 192.168.0.11 -P6033 -e ” SELECT name, population from world.City WHERE Continent=’$continent' and name like 'B%' order by population desc limit 1;” ;done Sharding Examples Schema – Value
  • 47.
    25 To check whatProxySQL processed: hits: 4 !Because I run the script for 4 continents rule_id: 31 match_pattern: ^SELECTs*(.*)s*froms*world.(S*) s(.*).*Continent='(S*)'s*(s*.*)$ replace_pattern: SELECT 1 from 4.2 WHERE 1=1 5 cache_ttl: NULL apply: 1 select active,hits, mysql_query_rules.rule_id, match_digest, match_pattern, replace_pattern, apply FROM mysql_query_rules NATURAL JOIN stats.stats_mysql_query_rules ORDER BY mysql_query_rules.rule_idG Sharding Examples Schema: Value - Check
  • 48.
    26 Comment Injection Causing: ●Query ReWrite Or ●HGredirection Or ●Both Sharding Examples
  • 49.
    27 Comments are theeasiest thing to implement in the application QueryRules chains (FlagIN – FlagOUT) What is a comment in MYSQL: Select name /* I am a comment*/ from world.City; Sharding Examples - Comment
  • 50.
    27 Comments are theeasiest thing to implement in the application QueryRules chains (FlagIN – FlagOUT) What is a comment in MYSQL: Select name /* I am a comment*/ from world.City; If testing from mysql console do not forget the flag “-c” Sharding Examples - Comment
  • 51.
    28 Use comments tohit Different schemas Sharding Examples - Comment /* Asia */ /* Europe */ /* Africa */ Asia Europe Africa
  • 52.
    29 Given a query: Original: Select /* continent=Asia */ * from world.Country join world.City on world.City.CountryCode=world.Country.Code where Country.code='IND’ Transformed: Select * from Asia.Country join Asia.City on Asia.City.CountryCode=Asia.Country.Code where Country.code=? Sharding Examples – Comment - How
  • 53.
    30 2 Rules: INSERT INTOmysql_query_rules (rule_id,active,username,match_pattern,replace_pattern,apply,FlagOUT,Fl agIN) VALUES (32,1,'user_shardRW',"S*s*/ *s*continent=.*Asias**.*",null,0,23,0); INSERT INTO mysql_query_rules (rule_id,active,username,match_pattern,replace_pattern,apply,FlagIN,Fla gOUT) VALUES (33,1,'user_shardRW','world.','Asia.',0,23,23); Sharding Examples - Comment
  • 54.
    30 2 Rules: INSERT INTOmysql_query_rules (rule_id,active,username,match_pattern,replace_pattern,apply,FlagOUT,Fl agIN) VALUES (32,1,'user_shardRW',"S*s*/ *s*continent=.*Asias**.*",null,0,23,0); INSERT INTO mysql_query_rules (rule_id,active,username,match_pattern,replace_pattern,apply,FlagIN,Fla gOUT) VALUES (33,1,'user_shardRW','world.','Asia.',0,23,23); Sharding Examples - Comment
  • 55.
    31 Sharding Examples –Comment - How hits: 1 rule_id: 32 match_pattern: /*s*continent=.*Asias* replace_pattern: NULL FlagIN: 0 FlagOUT: 23 hits: 1 (old versions (1.3.x: 4) rule_id: 33 match_pattern: world. replace_pattern: Asia. FlagIN: 23 FlagOUT: 23 Select hits, mysql_query_rules.rule_id, match_digest, match_pattern, replace_pattern FROM mysql_query_rules NATURAL JOIN stats.stats_mysql_query_rules ORDER BY mysql_query_rules.rule_idG
  • 56.
    32 Point to aHG directly is even easier Sharding Examples – Comment – To HG /* Asia */ /* Europe */ /* Africa */
  • 57.
    33 Point to aHG directly is even easier The scope is that whatever match the rule, should go to a defined HG. INSERT INTO mysql_query_rules (rule_id, active, username, match_pattern, destination_hostgroup,apply) VALUES (10,1,'user_shardRW',"/*s*shard_host_HG=.*Europes**.", 10,0); Sharding Examples – Comment – To HG
  • 58.
    33 Point to aHG directly is even easier The scope is that whatever match the rule, should go to a defined HG. INSERT INTO mysql_query_rules (rule_id, active, username, match_pattern, destination_hostgroup,apply) VALUES (10,1,'user_shardRW',"/*s*shard_host_HG=.*Europes**.", 10,0); Sharding Examples – Comment – To HG
  • 59.
    34 Sharding Examples –Comment - How hits: 1 rule_id: 20 match_pattern: /*s*continent=.*Asias* replace_pattern: NULL FlagIN: 0 FlagOUT: NULL destination_hostgroup: 10 apply: 0 Select hits, mysql_query_rules.rule_id, match_digest, match_pattern, replace_pattern FROM mysql_query_rules NATURAL JOIN stats.stats_mysql_query_rules ORDER BY mysql_query_rules.rule_idG
  • 60.
    35 Use both, byHG and Query Rewrite Sharding Examples – Comment: HG + Schema
  • 61.
    36 Finally is possibleto mix the two things. I want to use: Asia on Server 192.168.1.5 -> HG 10, points to a COUNTRY schema Europe on Server 192.168.1.6 -> HG 20, points to a COUNTRY schema Africa on Server 192.168.1.7 -> HG 30, points to a COUNTRY schema Select /* shard_host_HG=Asia */ /* country=India */ City.Name, City.Population from world.Country join world.City on world.City.CountryCode=world.Country.Code limit 5; Sharding Examples – Comment: HG + Schema
  • 62.
    37 Sharding Examples -Comment 2 Rules for country : INSERT INTO mysql_query_rules (rule_id,active,username,match_pattern,replace_pattern,apply,FlagOUT,Fl agIN) VALUES (32,1,'user_shardRW',"S*s*/ *s*country=.*Indias**.*",null,0,23,0); INSERT INTO mysql_query_rules (rule_id,active,username,match_pattern,replace_pattern,apply,FlagIN,Fla gOUT) VALUES (33,1,'user_shardRW','world.',’India.',0,23,23);
  • 63.
    38 Sharding Examples –Comment - How Three different rules to achieve the result. Order is important QR ID: 10 match_pattern: /*s*continent=.*Asias* Target: HG Apply: 0 QR ID: 32 match_pattern: /*s*country=.*Indias* Target: Query Rule 33 Apply: 0 QR ID: 33 match_pattern: world. Target: Query Rewrite to Country schema Apply: 0
  • 64.
  • 65.
    39 Problem ●Customer must reducethe number of shards Real case
  • 66.
    39 Problem ●Customer must reducethe number of shards ●Aggregate several schemas in multitenant DB Real case
  • 67.
    39 Problem ●Customer must reducethe number of shards ●Aggregate several schemas in multitenant DB ●Collapse several servers in few Real case
  • 68.
    39 Problem ●Customer must reducethe number of shards ●Aggregate several schemas in multitenant DB ●Collapse several servers in few ●No code change (as usual) or minimal Real case
  • 69.
    39 Problem ●Customer must reducethe number of shards ●Aggregate several schemas in multitenant DB ●Collapse several servers in few ●No code change (as usual) or minimal ●Java with Hibernate Real case
  • 70.
  • 71.
    40 Solution ●Inject limited commentin the query ● /*shard_ip=x.x.x.x,db=my_db,grp_id=123 */ Real case
  • 72.
    40 Solution ●Inject limited commentin the query ● /*shard_ip=x.x.x.x,db=my_db,grp_id=123 */ ●Create Query rules to manage ●4 rules to manage all + 1 for each additional server (HG) Real case
  • 73.
    41 The query: /* shard_ip=10.11.4.80,db=grp_aaaa,grp_id=23456*/ SELECT * FROM grp_1211.users where pippo like tt; Real case
  • 74.
    42 The QRs: INSERT INTOmysql_query_rules (rule_id,active,username,match_pattern,apply,FlagIN,FlagOUT) VALUES (10,1,'apptest1','where',0,0,20); 
 INSERT INTO mysql_query_rules (rule_id,active,username,match_pattern,replace_pattern,apply,FlagIN,FlagOUT) VALUES (15,1,'apptest1',"(/*.*,)grp_id=(d*)s**/s*(.*).*$","1 */ 3 WHERE grp_id=2;",0,0,25);
 
 INSERT INTO mysql_query_rules (rule_id,active,username,match_pattern,replace_pattern,apply,FlagIN,FlagOUT) VALUES (18,1,'apptest1',"(/*.*,)grp_id=(d*)s**/(.*)s*(where)s*(.*).*$","1 */ 3 WHERE grp_id=2 AND 5;",0,20,25);
 
 INSERT INTO mysql_query_rules (rule_id,active,username,match_pattern,replace_pattern,apply,FlagIN,FlagOUT) VALUES (22,1,'apptest1',"(^/*.*,)db=(S*),s**/s*(.*(FROM|UPDATE|INSERT INTO))s*(.*).(.*)","1 */ 3 2.6",0,25,25); INSERT INTO mysql_query_rules (rule_id,active,username,match_pattern,destination_hostgroup,apply,FlagIN,FlagOUT) VALUES (30,1,'apptest1',"shard_ip=10.11.4.79",20,1,25,NULL); Real case
  • 75.
    43 Starting query: /* shard_ip=10.11.4.80,db=grp_aaaa,grp_id=23456*/ SELECT * FROM grp_1211.users where pippo like tt; Real case
  • 76.
    43 Starting query: /* shard_ip=10.11.4.80,db=grp_aaaa,grp_id=23456*/ SELECT * FROM grp_1211.users where pippo like tt; Become: /* shard_ip=10.11.4.80,db=grp_aaaa, */ SELECT * FROM grp_1211.users where grp_id=23456 AND pippo like tt; Real case
  • 77.
    43 Starting query: /* shard_ip=10.11.4.80,db=grp_aaaa,grp_id=23456*/ SELECT * FROM grp_1211.users where pippo like tt; Become: /* shard_ip=10.11.4.80,db=grp_aaaa, */ SELECT * FROM grp_1211.users where grp_id=23456 AND pippo like tt; /* shard_ip=10.11.4.80,*/ SELECT * FROM grp_aaaa.users WHERE grp_id=23456 AND pippo like tt; Ready to be dispatch to final HG with rules above id 30 Real case
  • 78.
  • 79.
    44 Obviously if acustomer can screw your perfect plan …. Real case
  • 80.
    44 Obviously if acustomer can screw your perfect plan …. … he will. Real case
  • 81.
    44 Obviously if acustomer can screw your perfect plan …. … he will. ●Try to use NON standard SQL like: INSERT… SET (MySQL extension) Real case
  • 82.
    44 Obviously if acustomer can screw your perfect plan …. … he will. ●Try to use NON standard SQL like: INSERT… SET (MySQL extension) ●Omit the schema name in the SQL Real case
  • 83.
    45 There is nomagic solution for sharding.
 ProxySQL is a tool to help not a sharding mechanism. ● Per port/IP (require to be define at infrastructure design time) ● Query filtering/rewrite (may impact performance if many QR, and require application to be aware of the shards) ● Cannot pass variables cross QR ● Queries are NOT spanning on multiple HG. ● Cannot aggregate resultsets from different shards It is NOT a kind of magic
  • 84.
    46 Cost in Execution(trx in 5 minutes) Review the blogs: https://www.percona.com/blog/2017/04/12/proxysql-applying-and- chaining-the-rules/ https://www.percona.com/blog/2017/04/10/proxysql-rules-do-i-have-too-many/
  • 85.
    47 Multiple element touse to shard:
 User/default schema/IP-Port/value/Hint Shards can be all pointing to same HG different schemas; pointing to different HG or a Mix Plan Your QR and chains to avoid performance degradation Brief summary
  • 86.
    48 http://www.proxysql.com/ ProxySQL http://www.pcre.org/ PerlCompatible Regular Expressions https://goo.gl/cMkngr Sharding with ProxySQL https://goo.gl/vovszl stop an offending query with ProxySQL https://goo.gl/OC8RbN ProxySQL Rules: Do I Have Too Many? https://goo.gl/rzYDr7 Applying and Chaining the Rules https://www.percona.com/blog 
 To Know more
  • 87.
    49 René Cannaò forProxySQL Tibor Köröcz for performance Tests Some personal thanks to:
  • 88.
  • 89.
  • 90.
    52 Contacts To contact Me Marco.tusa@percona.com marcotusa@tusacentral.net Tofollow me http://www.tusacentral.net/ http://www.percona.com/blog/ https://www.facebook.com/marco.tusa.94 @marcotusa http://it.linkedin.com/in/marcotusa/ “Consulting = No mission refused!”