Presentation shows how ProxySQL can improve the HA in solution like MySQL async and sync replication without the need to increase the platform complexity.

1. Percona Live Europe 2016
Use ProxySQL to Improve Your
MySQL High Availability Solution
Marco Tusa Manager Consulting
Amsterdam, Netherlands | October 3 – 5, 2016

2. 2
● Marco “The Grinch”
• Open source enthusiast
About me

3. 3
What is HA

4. 3
What is HA

5. 3
What is HA
9 0. 0 0 0 % (36 days) MySQL Replication
9 9. 9 0 0 % (8 hours) Linux Heartbeat with DRBD
9 9. 9 0 0 % (8 hours) RHCS with Shared Storage (Active/Passive)
9 9. 9 9 0 % (52 minutes) MHA with at least 3 nodes
9 9. 9 9 0 % (52 minutes) Linux Heartbeat with DRBD and Replication
9 9. 9 9 0 % (52 minutes) Linux Heartbeat with Shared Storage and
Replication
9 9 .9 9 5 % (26 minutes) Multi Master (galera - percona cluster) 3 node
minimum
9 9. 9 9 9 % (5 minutes) MySQL Cluster

6. 4
What is HA

7. 5
What is HA

8. 6
Objectively What we have is:

9. 6
Objectively What we have is:

10. 6
Objectively What we have is:

11. 6
Objectively What we have is:

12. 6
Objectively What we have is:
MySQL Group
Replication

13. 7
The telephone game

14. 8
All talking

15. 9
Black box:

16. 10
Use the Right Tool for the Job…

17. 11
What we need

18. 11
What we need

19. 11
What we need

20. 11
What we need

21. 12
ProxySQL

22. 13
ProxySQL

23. 14
Understand the basics

24. 14
Understand the basics
Write / Read (Master/Slave)

25. 14
Understand the basics
Write / Read (Master/Slave)
Multiple Writers

26. 14
Understand the basics
Write / Read (Master/Slave)
Multiple Writers
Sharding &
Geographic Distribution

27. 14
Understand the basics
Write / Read (Master/Slave)
Multiple Writers
Sharding &
Geographic Distribution
Mirroring

28. 14
Understand the basics
Write / Read (Master/Slave)
Multiple Writers
Sharding &
Geographic Distribution
Mirroring
Query re-write

29. 15
Understand the basics – how this is done?

30. 15
Servers group by function
Understand the basics – how this is done?

31. 15
Servers group by function
Host Groups
Understand the basics – how this is done?

32. 15
Servers group by function
Host Groups
Rules define what, where and how
Understand the basics – how this is done?

33. 15
Servers group by function
Host Groups
Rules define what, where and how
Query Rules
Understand the basics – how this is done?

34. 15
Servers group by function
Host Groups
Rules define what, where and how
Query Rules
Special Grouping for Replication
Understand the basics – how this is done?

35. 15
Servers group by function
Host Groups
Rules define what, where and how
Query Rules
Special Grouping for Replication
Replication Host Group
Understand the basics – how this is done?

36. 16
HostGroup (HG) & Replication HG
Understand the basics

37. 16
HostGroup (HG) & Replication HG
Understand the basics

38. 17
Query rules
Understand the basics

39. 18
Scheduler
Understand the basics

40. 19
Basic scenario

41. 19
Basic scenario
HG: 10
U: app_RW
HG: 11
U: app_R
HG: 12
U: ETL_R

42. 20
insert into mysql_users
(username,password,active,default_hostgroup,default_schema)
values ('app_RW','test',1,10,'test_schema');
insert into mysql_users
(username,password,active,default_hostgroup,default_schema)
values ('app_R','test',1,11,'test_schema');
insert into mysql_users
(username,password,active,default_hostgroup,default_schema)
values ('etl_R','test',1,12,’etl_schema');
LOAD MYSQL USERS TO RUNTIME;SAVE MYSQL USERS TO DISK;
Basic scenario - Users

43. 21
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES
('192.168.1.5',10,3306,1000);
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight,
max_replication_lag) VALUES ('192.168.1.6',11,3306,1000000,2);
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight,
max_replication_lag) VALUES ('192.168.1.7',11,3306,1000000,2);
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight,
max_replication_lag) VALUES ('192.168.1.5',11,3306,1,2);
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES
('192.168.1.8',12,3306,10);
INSERT INTO mysql_replication_hostgroups VALUES (10,11);
LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK;
Basic scenario – Servers / HG

44. 22
insert into mysql_query_rules
(username,destination_hostgroup,active,retries,match_digest,apply)
values(’app_RW',10,1,3,'^SELECT.*FOR UPDATE',1);
insert into mysql_query_rules
(username,destination_hostgroup,active,retries,match_digest,apply)
values(’app_RW',11,1,3,'^SELECT ',1);
insert into mysql_query_rules (username,destination_hostgroup,active)
values(’app_R',11,1);
insert into mysql_query_rules (username,destination_hostgroup,active)
values('etl_R',12,1);
LOAD MYSQL QUERY RULES TO RUNTIME;SAVE MYSQL QUERY RULES TO DISK;
Basic scenario – Query rules

45. 23
Basic scenario – with ProxySQL

46. 23
Basic scenario – with ProxySQL
Writes

47. 23
Basic scenario – with ProxySQL
Writes
Reads

48. 23
Basic scenario – with ProxySQL
Writes
Reads
Reads

49. 23
Basic scenario – with ProxySQL
Reads
Reads

50. 24
READ_ONLY = 0 WRITE HG
READ_ONLY = 1 READ HG
ProxySQL – Replication HG
READ_ONLY=0 READ_ONLY=1 READ_ONLY=1
HG 11 HG 11HG 10 & HG 11

51. 24
READ_ONLY = 0 WRITE HG
READ_ONLY = 1 READ HG
ProxySQL – Replication HG
READ_ONLY=0 READ_ONLY=1
HG 10 & HG:11 HG 11
READ_ONLY=0
HG 10 & HG 11

52. 24
READ_ONLY = 0 WRITE HG
READ_ONLY = 1 READ HG
ProxySQL – Replication HG
READ_ONLY=0 READ_ONLY=1
HG 10 & HG:11 HG 11
READ_ONLY=0
HG 10 & HG 11

53. 25
ProxySQL + MHA (or any controller)
Writes
Reads
Reads

54. 25
ProxySQL + MHA (or any controller)
Writes
Reads
Reads

55. 25
ProxySQL + MHA (or any controller)
Writes
Reads
Reads

56. 26
No movable part like:
ProxySQL + MHA (or any controller)

57. 26
No movable part like:
VIP
ProxySQL + MHA (or any controller)

58. 26
No movable part like:
VIP
DNS (Route 53 or similar)
ProxySQL + MHA (or any controller)

59. 26
No movable part like:
VIP
DNS (Route 53 or similar)
No Crazy scripts to manage other proxy (ie HAProxy)
ProxySQL + MHA (or any controller)

60. 26
No movable part like:
VIP
DNS (Route 53 or similar)
No Crazy scripts to manage other proxy (ie HAProxy)
Better distributed HA
ProxySQL + MHA (or any controller)

61. 27
What timing with MHA?

62. 27
MHA discovery time (ping x 3 times) -> ~9 sec
What timing with MHA?

63. 27
MHA discovery time (ping x 3 times) -> ~9 sec
MHA recovery (binlog) variable time
What timing with MHA?

64. 27
MHA discovery time (ping x 3 times) -> ~9 sec
MHA recovery (binlog) variable time
MHA Slave shift + Master READ-ONLY < 2 sec
What timing with MHA?

65. 27
MHA discovery time (ping x 3 times) -> ~9 sec
MHA recovery (binlog) variable time
MHA Slave shift + Master READ-ONLY < 2 sec
ProxySQL will react as soon as READ_ONLY=0
What timing with MHA?

66. 28
Nothing all defaults
master_ip_failover_script=master_ip_failover
master_ip_online_change_script=master_ip_online_change
Test it:
masterha_master_switch --master_state=alive
--conf=/etc/mha.cnf --
orig_master_is_new_slave --interactive=0 --
running_updates_limit=0
What is needed with MHA?

67. 29
Given no VIP or centralize element.
Why better distributed HA?

68. 30
Integration with PXCGalera

69. 31
Integration with PXCGalera

70. 31
Integration with PXCGalera

71. 31
Integration with PXCGalera

72. 32
Integration with PXCGalera

73. 32
Integration with PXCGalera
read_only
wsrep_status
wsrep_rejectqueries
wsrep_donorrejectqueries
wsrep_connected
wsrep_desinccount
wsrep_ready
wsrep_provider
wsrep_segment
Number of nodes in by segment
Retry loop

74. 32
Integration with PXCGalera
read_only
wsrep_status
wsrep_rejectqueries
wsrep_donorrejectqueries
wsrep_connected
wsrep_desinccount
wsrep_ready
wsrep_provider
wsrep_segment
Number of nodes in by segment
Retry loop
Online
Offline_soft
Offline_hard
Shunned

75. 32
Integration with PXCGalera
read_only
wsrep_status
wsrep_rejectqueries
wsrep_donorrejectqueries
wsrep_connected
wsrep_desinccount
wsrep_ready
wsrep_provider
wsrep_segment
Number of nodes in by segment
Retry loop
Online
Offline_soft
Offline_hard
Shunned
But we also have
Hostgroups
Replication HG

76. 33
Integration with PXCGalera

77. 33
Integration with PXCGalera

78. 33
Integration with PXCGalera

79. 34
Integration with PXCGalera
HG: 10
HG: 10
HG: 12
HG: 11
HG: 11
HG: 12

80. 35
Multiple writer vs Single writer
Integration with PXCGalera

81. 35
Multiple writer vs Single writer
Multi writer based on weight and Hostgroups
Integration with PXCGalera

82. 35
Multiple writer vs Single writer
Multi writer based on weight and Hostgroups
Need Replication HG and READ_ONLY
Integration with PXCGalera

83. 35
Multiple writer vs Single writer
Multi writer based on weight and Hostgroups
Need Replication HG and READ_ONLY
Failover is done without the need of performing actions on
the nodes
Integration with PXCGalera

84. 35
Multiple writer vs Single writer
Multi writer based on weight and Hostgroups
Need Replication HG and READ_ONLY
Failover is done without the need of performing actions on
the nodes
Failover is done modifying the READ_ONLY value
Integration with PXCGalera

85. 35
Multiple writer vs Single writer
Multi writer based on weight and Hostgroups
Need Replication HG and READ_ONLY
Failover is done without the need of performing actions on
the nodes
Failover is done modifying the READ_ONLY value
Single WRITER node isolation not guarantee
Integration with PXCGalera

86. 35
Multiple writer vs Single writer
Multi writer based on weight and Hostgroups
Need Replication HG and READ_ONLY
Failover is done without the need of performing actions on
the nodes
Failover is done modifying the READ_ONLY value
Single WRITER node isolation not guarantee
Single WRITER node isolation is guarantee
Integration with PXCGalera

87. 36
insert into mysql_users
(username,password,active,default_hostgroup,default_schema)
values ('app_RW','test',1,10,'test_schema');
insert into mysql_users
(username,password,active,default_hostgroup,default_schema)
values ('app_R','test',1,11,'test_schema');
insert into mysql_users
(username,password,active,default_hostgroup,default_schema)
values (‘appDR_R','test',1,12,’etl_schema');
LOAD MYSQL USERS TO RUNTIME;SAVE MYSQL USERS TO DISK;
Integration with PXCGalera - Users

88. 37
insert into mysql_query_rules
(username,destination_hostgroup,active,retries,match_digest,apply)
values(’app_RW',10,1,3,'^SELECT.*FOR UPDATE',1);
insert into mysql_query_rules
(username,destination_hostgroup,active,retries,match_digest,apply)
values(’app_RW',11,1,3,'^SELECT ',1);
insert into mysql_query_rules (username,destination_hostgroup,active)
values(’app_R',11,1);
insert into mysql_query_rules (username,destination_hostgroup,active)
values(‘appDR_R',12,1);
LOAD MYSQL QUERY RULES TO RUNTIME;SAVE MYSQL QUERY RULES TO DISK;
Integration with PXCGalera – QR

89. 38
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight)
VALUES ('192.168.1.5',10,3306,1000000000);
192.168.1.6 … 10000; 192.168.1.7 … 10
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight)
VALUES ('192.168.1.5',11,3306,1);
192.168.1.6 … 100000; 192.168.1.7 … 100000
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight)
VALUES ('192.168.1.8',12,3306,100);
192.168.1.9 … 100;
LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK;
Integration with PXCGalera – HG MW

90. 39
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES
('192.168.1.5',10,3306,100);
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES
('192.168.1.5',11,3306,1);
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES
('192.168.1.6',11,3306,100);
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES
('192.168.1.7',11,3306,100);
INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES
('192.168.1.8',12,3306,100);
192.168.1.9 … 100;
INSERT INTO mysql_replication_hostgroups VALUES (10,11);
LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK;
Integration with PXCGalera HG SW

91. 40
ProxySQL is configured Now what?
Integration with PXC

92. 40
ProxySQL is configured Now what?
Integration with PXC

93. 40
ProxySQL is configured Now what?
Example script
https://github.com/Tusamarco/proxy_sql_tools
Full guide: https://goo.gl/vcDFZx
Integration with PXC

94. 41
Set the script:
INSERT INTO scheduler
(id,active,interval_ms,filename,arg1) values
(10,0,2000,"/var/lib/proxysql/galera_check.pl","-
u=admin -p=admin -h=192.168.1.50 -H=500:W,501:R -
P=3310 --execution_time=1 --retry_down=2 --retry_up=1
--main_segment=1 --debug=0 --log=/var/lib/
proxysql/galeraLog");
LOAD SCHEDULER TO RUNTIME;SAVE SCHEDULER TO DISK;
update scheduler set active=1 where id=10;
LOAD SCHEDULER TO RUNTIME;
Integration with PXC

95. 42
+--------------+-------------+--------+------------+---------+
| hostgroup_id | hostname | status | weight | comment |
+--------------+-------------+--------+------------+---------+
| 10 | 192.168.1.5 | ONLINE | 1000000000 | |
| 10 | 192.168.1.6 | ONLINE | 10000 | |
| 10 | 192.168.1.7 | ONLINE | 10 | |
| 11 | 192.168.1.5 | ONLINE | 1 | |
| 11 | 192.168.1.6 | ONLINE | 100000 | |
| 11 | 192.168.1.7 | ONLINE | 100000 | |
| 12 | 192.168.1.8 | ONLINE | 100 | |
| 12 | 192.168.1.9 | ONLINE | 100 | |
+--------------+-------------+--------+------------+---------+
Integration with PXC MW

96. 43
+--------------+-------------+--------+------------+---------+
| hostgroup_id | hostname | status | weight | comment |
+--------------+-------------+--------+------------+---------+
| 10 | 192.168.1.5 |OFFLINE_SOFT | 1000000000 |
10_W_11_R_retry_up=0;10_W_11_R_retry_down=0;
| 10 | 192.168.1.6 | ONLINE | 10000 | |
| 10 | 192.168.1.7 | ONLINE | 10 | |
| 11 | 192.168.1.5 | OFFLINE_SOFT | 1 | ... |
| 11 | 192.168.1.6 | ONLINE | 100000 | |
| 11 | 192.168.1.7 | ONLINE | 100000 | |
| 12 | 192.168.1.8 | ONLINE | 100 | |
| 12 | 192.168.1.9 | ONLINE | 100 | |
+--------------+-------------+--------+------------+---------+
Integration with PXC MW

97. 44
+---------------------------------+-------+
| Variable_name | Value |
+---------------------------------+-------+
| wsrep_sst_donor_rejects_queries | ON |
+---------------------------------+-------+
Integration with PXC MW

98. 44
+---------------------------------+-------+
| Variable_name | Value |
+---------------------------------+-------+
| wsrep_sst_donor_rejects_queries | ON |
+---------------------------------+-------+
ProxySQL Think all is ok
Integration with PXC MW

99. 44
+---------------------------------+-------+
| Variable_name | Value |
+---------------------------------+-------+
| wsrep_sst_donor_rejects_queries | ON |
+---------------------------------+-------+
ProxySQL Think all is ok
Application writes will fail
Integration with PXC MW

100. 45
+--------------+-------------+--------+------------+---------+
| hostgroup_id | hostname | status | weight | comment |
+--------------+-------------+--------+------------+---------+
| 10 | 192.168.1.6 | ONLINE | 10000 | |
| 10 | 192.168.1.7 | ONLINE | 10 | |
| 11 | 192.168.1.6 | ONLINE | 100000 | |
| 11 | 192.168.1.7 | ONLINE | 100000 | |
| 12 | 192.168.1.8 | ONLINE | 100 | |
| 12 | 192.168.1.9 | ONLINE | 100 | |
| 9010 | 192.168.1.5 | ONLINE | 1000000000 | |
| 9011 | 192.168.1.5 | ONLINE | 1 | |
+--------------+-------------+--------+------------+---------+
Integration with PXC MW

101. 46
WRITE Fail-over
is automatically perform by ProxySQL
following the weight
Until an active node exists the HG
Integration with PXC MW

102. 47
WRITE Fail-over
will not happen
Until READ_ONLY variable
is modified on a reader HG node.
WRITER if DONOR
will never be OFFLINE_SOFT
Integration with PXC SW

103. 48
Integration with PXCGalera SW manual

104. 48
Integration with PXCGalera SW manual

105. 48
Integration with PXCGalera SW manual

106. 49
Integration with PXCGalera SW Auto

107. 49
Integration with PXCGalera SW Auto

108. 49
Integration with PXCGalera SW Auto

109. 50
Change state to:
OFFLINE_SOFT = ~2 sec
HostGroup 9000 = ~2 sec
ProxySQL Shunned = ~1.2 sec (5 fails in 1sec)
Failover (auto) = ~3 sec
Integration with PXC - Timing

110. 51
Avoid SPOF

111. 51
Avoid SPOF

112. 51
Avoid SPOF

113. 52
Cost in time

114. 52
Cost in time

115. 53
Cost in time

116. 53
Cost in time

117. 54
So what..
9 0. 0 0 0 % (36 days) MySQL Replication
9 9. 9 0 0 % (8 hours) Linux Heartbeat with DRBD
9 9. 9 0 0 % (8 hours) RHCS with Shared Storage (Active/Passive)
9 9. 9 9 0 % (52 minutes) MHA with at least 3 nodes
9 9. 9 9 0 % (52 minutes) Linux Heartbeat with DRBD and Replication
9 9. 9 9 0 % (52 minutes) Linux Heartbeat with Shared Storage and
Replication
9 9 .9 9 5 % (26 minutes) Multi Master (galera - percona
cluster) 3 node minimum
9 9. 9 9 9 % (5 minutes) MySQL Cluster

118. 55
Brief summary

119. 55
Improve service flexibility with r/w split;
Brief summary

120. 55
Improve service flexibility with r/w split;
Allow simple way to manage access to specific servers by
usage (like DW-ETL);
Brief summary

121. 55
Improve service flexibility with r/w split;
Allow simple way to manage access to specific servers by
usage (like DW-ETL);
Implement sharding;
Brief summary

122. 55
Improve service flexibility with r/w split;
Allow simple way to manage access to specific servers by
usage (like DW-ETL);
Implement sharding;
Allow Query revrite/query fix/blocking without need to modify
code;
Brief summary

123. 55
Improve service flexibility with r/w split;
Allow simple way to manage access to specific servers by
usage (like DW-ETL);
Implement sharding;
Allow Query revrite/query fix/blocking without need to modify
code;
Improve failover time;
Brief summary

124. 55
Improve service flexibility with r/w split;
Allow simple way to manage access to specific servers by
usage (like DW-ETL);
Implement sharding;
Allow Query revrite/query fix/blocking without need to modify
code;
Improve failover time;
Reduce master downtime impact on reads;
Brief summary

125. 55
Improve service flexibility with r/w split;
Allow simple way to manage access to specific servers by
usage (like DW-ETL);
Implement sharding;
Allow Query revrite/query fix/blocking without need to modify
code;
Improve failover time;
Reduce master downtime impact on reads;
Not SPOF (as vip)
Brief summary

126. 56

127. 57

128. 58
Contacts
To contact Me
Marco.tusa@percona.com
marcotusa@tusacentral.net
To follow me
http://www.tusacentral.net/
http://www.percona.com/blog/
https://www.facebook.com/marco.tusa.94
@marcotusa
http://it.linkedin.com/in/marcotusa/
“Consulting = No mission refused!”