COSCUP 2017 presentation: MySQL High Availability made easy

1. 1 / 84

2. 2 / 84

3. MySQL InnoDB Cluster
MySQL High Availability made easy
August 2017
Frédéric Descamps - MySQL Community Manager - Oracle
3 / 84

4.
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purpose only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied up in
making purchasing decisions. The development, release and timing of any features or
functionality described for Oracle´s product remains at the sole discretion of Oracle.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
4 / 84

5. about.me/lefred
Who am I ?
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
5 / 84

6. Frédéric Descamps
@lefred
MySQL Evangelist
Hacking MySQL since 3.23
devops believer
MySQL Community Manager since May
2016
living in Belgium 🇧🇪
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
6 / 84

7. 100%
virtually all organizations require their most
critical systems to be highly available !
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
7 / 84

8.
"HighAvailabilitybecomesacorefirst
classfeatureofMySQL!"
MySQL InnoDB Cluster
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
8 / 84

9. MySQL InnoDB Cluster: vision
"Asingleproduct-MySQL-withhighavailabilityandscalingfeaturesbakedin;providing
anintegratedend-to-endsolutionthatiseasytouse."
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
9 / 84

10. MySQL InnoDB Cluster: vision
"Asingleproduct-MySQL-withhighavailabilityandscalingfeaturesbakedin;providing
anintegratedend-to-endsolutionthatiseasytouse."
InnoDB
cluster
Ease-of-Use
Extreme Scale-Out
Out-of-Box Solution
Built-in HA
High Performance
Everything Integrated
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
10 / 84

11. One Product: MySQL
All components created together
Tested together
Packaged together
Easy to Use
One client: MySQL Shell
Easy packaging
Integrated orchestration
Homogenous servers
Flexible and Moden
SQL and NoSQL together
Protocol Buffers
Asynchronous API
Developer friendly
Support Read/Write Scale Out
Shared clusters
Federated systemof N replica sets
supporting cross shard
operations
Each replica set manages a shard
MySQL InnoDB Cluster: goals
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
11 / 84

12. Our vision in 4 steps
MySQLDocumentStore
Relational&DocumentModels
MySQLHA
Out-Of-BoxHA
ReadScale-Out
AsyncReplication+AutoFailover
WriteScale-Out
Sharding
E1 E3
E2 E4
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
12 / 84

13. Step 2´s Architecture
Application
MySQL Connector
MySQL Router
MySQL Shell
InnoDB
cluster
Application
MySQL Connector
MySQL Router
Mp
M
M
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
13 / 84

14. Step 3´s Architecture
Application
MySQL Connector
MySQL Router
MySQL Shell
InnoDB
cluster
Application
MySQL Connector
MySQL Router
Mp
M
M
S1 S2 S3 S4 S...
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
14 / 84

15. Step 4´s Architecture
Application
MySQL Connector
MySQL Router
MySQL Shell
Application
MySQL Connector
MySQL Router
InnoDB
cluster
Mp
M
M
S2 S3 S4 S...S1
InnoDB
cluster
Mp
M
M
S1 S2 S3 S4 S...
InnoDB
cluster
Mp
M
M
S1 S2 S3 S4 S...
Application
MySQL Connector
MySQL Router
Application
MySQL Connector
MySQL Router
Application
MySQL Connector
MySQL Router
replicaset1
replicaset2
replicaset3
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
15 / 84

16. Group Replication: heart of MySQL InnoDB
Cluster
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
16 / 84

17. Group Replication: heart of MySQL InnoDB
Cluster
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
17 / 84

18. MySQL Group Replication
but what is it ?!?
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
18 / 84

19. MySQL Group Replication
but what is it ?!?
GR is a plugin for MySQL, made by MySQL and packaged with MySQL
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
19 / 84

20. MySQL Group Replication
but what is it ?!?
GR is a plugin for MySQL, made by MySQL and packaged with MySQL
GR is an implementation of Replicated Database State Machine theory
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
20 / 84

21. MySQL Group Replication
but what is it ?!?
GR is a plugin for MySQL, made by MySQL and packaged with MySQL
GR is an implementation of Replicated Database State Machine theory
GR uses a Paxos based protocol
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
21 / 84

22. MySQL Group Replication
but what is it ?!?
GR is a plugin for MySQL, made by MySQL and packaged with MySQL
GR is an implementation of Replicated Database State Machine theory
GR uses a Paxos based protocol
GR allows to write on all Group Members (cluster nodes) simultaneously while
retaining consistency
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
22 / 84

23. MySQL Group Replication
but what is it ?!?
GR is a plugin for MySQL, made by MySQL and packaged with MySQL
GR is an implementation of Replicated Database State Machine theory
GR uses a Paxos based protocol
GR allows to write on all Group Members (cluster nodes) simultaneously while
retaining consistency
GR implements conflict detection and resolution
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
23 / 84

24. MySQL Group Replication
but what is it ?!?
GR is a plugin for MySQL, made by MySQL and packaged with MySQL
GR is an implementation of Replicated Database State Machine theory
GR uses a Paxos based protocol
GR allows to write on all Group Members (cluster nodes) simultaneously while
retaining consistency
GR implements conflict detection and resolution
GR allows automatic distributed recovery
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
24 / 84

25. MySQL Group Replication
but what is it ?!?
GR is a plugin for MySQL, made by MySQL and packaged with MySQL
GR is an implementation of Replicated Database State Machine theory
GR uses a Paxos based protocol
GR allows to write on all Group Members (cluster nodes) simultaneously while
retaining consistency
GR implements conflict detection and resolution
GR allows automatic distributed recovery
Supported on all MySQL platforms !!
Linux, Windows, Solaris, OSX, FreeBSD
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
25 / 84

26. And for users ?
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
26 / 84

27. And for users ?
no longer necessary to handle server fail-over manually or with a complicated script
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
27 / 84

28. And for users ?
no longer necessary to handle server fail-over manually or with a complicated script
GR provides fault tolerance
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
28 / 84

29. And for users ?
no longer necessary to handle server fail-over manually or with a complicated script
GR provides fault tolerance
GR enables update-everywhere setups
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
29 / 84

30. And for users ?
no longer necessary to handle server fail-over manually or with a complicated script
GR provides fault tolerance
GR enables update-everywhere setups
GR handles crashes, failures, re-connetcs automatically
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
30 / 84

31. And for users ?
no longer necessary to handle server fail-over manually or with a complicated script
GR provides fault tolerance
GR enables update-everywhere setups
GR handles crashes, failures, re-connetcs automatically
Allows an easy setup of a MySQL service high available !
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
31 / 84

32. OK, but how does it work ?
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
32 / 84

33. OK, but how does it work ?
it´s just ...
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
33 / 84

34. OK, but how does it work ?
it´s just ...
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
34 / 84

35. OK, but how does it work ?
it´s just ...
... no, in fact the writesets replication is synchronous and then certification and apply of
the changes are local to each nodes and asynchronous.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
35 / 84

36. OK, but how does it work ?
it´s just ...
... no, in fact the writesets replication is synchronous and then certification and apply of
the changes are local to each nodes and asynchronous.
not that easy to understand... right ? As a picture is worth a 1000 words, let´s illustrate
this...
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
36 / 84

37. MySQL Group Replication
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
37 / 84

38. MySQL Group Replication
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
38 / 84

39. MySQL Group Replication
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
39 / 84

40. MySQL Group Replication
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
40 / 84

41. MySQL Group Replication
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
41 / 84

42. MySQL Group Replication
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
42 / 84

43. MySQL Group Replication
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
43 / 84

44. MySQL Group Replication
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
44 / 84

45. MySQL Group Replication
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
45 / 84

46. MySQL Group Replication
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
46 / 84

47. MySQL Group Replication
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
47 / 84

48. MySQL Group Replication
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
48 / 84

49. Certification
Certification is the process that only needs to answer the following unique question:
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
49 / 84

50. Certification
Certification is the process that only needs to answer the following unique question:
canthewrite(transaction)beapplied?
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
50 / 84

51. Certification
Certification is the process that only needs to answer the following unique question:
canthewrite(transaction)beapplied?
based on unapplied earlier transactions
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
51 / 84

52. Certification
Certification is the process that only needs to answer the following unique question:
canthewrite(transaction)beapplied?
based on unapplied earlier transactions
such conflicts must come for other members/nodes
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
52 / 84

53. Certification
Certification is the process that only needs to answer the following unique question:
canthewrite(transaction)beapplied?
based on unapplied earlier transactions
such conflicts must come for other members/nodes
happens on every member/node
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
53 / 84

54. Certification
Certification is the process that only needs to answer the following unique question:
canthewrite(transaction)beapplied?
based on unapplied earlier transactions
such conflicts must come for other members/nodes
happens on every member/node
should be deterministic on every node
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
54 / 84

55. Certification
Certification is the process that only needs to answer the following unique question:
canthewrite(transaction)beapplied?
based on unapplied earlier transactions
such conflicts must come for other members/nodes
happens on every member/node
should be deterministic on every node
results are not reported to the group
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
55 / 84

56. Certification
Certification is the process that only needs to answer the following unique question:
canthewrite(transaction)beapplied?
based on unapplied earlier transactions
such conflicts must come for other members/nodes
happens on every member/node
should be deterministic on every node
results are not reported to the group
pass: enter in the apply queue
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
56 / 84

57. Certification
Certification is the process that only needs to answer the following unique question:
canthewrite(transaction)beapplied?
based on unapplied earlier transactions
such conflicts must come for other members/nodes
happens on every member/node
should be deterministic on every node
results are not reported to the group
pass: enter in the apply queue
fail: drop the transaction
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
57 / 84

58. Certification
Certification is the process that only needs to answer the following unique question:
canthewrite(transaction)beapplied?
based on unapplied earlier transactions
such conflicts must come for other members/nodes
happens on every member/node
should be deterministic on every node
results are not reported to the group
pass: enter in the apply queue
fail: drop the transaction
serialized by the total order in GCS/XCOM+ GTID
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
58 / 84

59. Certification
Certification is the process that only needs to answer the following unique question:
canthewrite(transaction)beapplied?
based on unapplied earlier transactions
such conflicts must come for other members/nodes
happens on every member/node
should be deterministic on every node
results are not reported to the group
pass: enter in the apply queue
fail: drop the transaction
serialized by the total order in GCS/XCOM+ GTID
cost is based on trx size (# rows &# keys)
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
59 / 84

60. Default = Single Primary Mode
By default, MySQL InnoDB Cluster runs in Single Primary Mode.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
60 / 84

61. Default = Single Primary Mode
By default, MySQL InnoDB Cluster runs in Single Primary Mode.
mysql> show global variables like 'group_replication_single_primary_mode';
+---------------------------------------+-------+
| Variable_name | Value |
+---------------------------------------+-------+
| group_replication_single_primary_mode | ON |
+---------------------------------------+-------+
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
61 / 84

62. Default = Single Primary Mode
By default, MySQL InnoDB Cluster runs in Single Primary Mode.
mysql> show global variables like 'group_replication_single_primary_mode';
+---------------------------------------+-------+
| Variable_name | Value |
+---------------------------------------+-------+
| group_replication_single_primary_mode | ON |
+---------------------------------------+-------+
In Single Primary Mode, a single member acts as the writable master (PRIMARY) and the
rest of the members act as hot-standbys (SECONDARY).
The group itself coordinates and configures itself automatically to determine which
member will act as the PRIMARY, through a leader election mechanism.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
62 / 84

63. Single Primary Mode - new in 8.0.2
group_replication_member_weight: influence the primary member election in Single-
Primary mode (from0to100).
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
63 / 84

64. Single Primary Mode - new in 8.0.2
group_replication_member_weight: influence the primary member election in Single-
Primary mode (from0to100).
more visibility
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
64 / 84

65. Single Primary Mode - new in 8.0.2
group_replication_member_weight: influence the primary member election in Single-
Primary mode (from0to100).
more visibility
mysql-sql> select * from performance_schema.replication_group_members;
+--------------+--------------+--------+------+---------+-------------+-------+
| CHANNEL_NAME | MEMBER_ID | M_HOST | PORT | M_STATE | MEMBER_ROLE | M_VER |
+--------------+--------------+--------+------+---------+-------------+-------+
| gr_applier | ...-8d90-... | mysql1 | 3306 | ONLINE | SECONDARY | 8.0.2 |
| gr_applier | ...-be23-... | mysql2 | 3306 | ONLINE | SECONDARY | 8.0.2 |
| gr_applier | ...-a747-... | mysql3 | 3306 | ONLINE | PRIMARY | 8.0.2 |
+--------------+--------------+--------+------+---------+-------------+-------+
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
65 / 84

66. Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
66 / 84

67. MySQL Router (GA!)
MySQL Router is lightweight middleware that provides transparent routing between your
application and backend MySQL Servers. It can be used for a wide variety of use cases,
such as providing high availability and scalability by effectively routing database traffic to
appropriate backend MySQL Servers.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
67 / 84

68. MySQL Router (GA!)
MySQL Router is lightweight middleware that provides transparent routing between your
application and backend MySQL Servers. It can be used for a wide variety of use cases,
such as providing high availability and scalability by effectively routing database traffic to
appropriate backend MySQL Servers.
MySQL Router doesn´t require any specific configuration. It configures itself automatically
(bootstrap) using MySQL InnoDB Cluster´s metadata.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
68 / 84

69. MySQL Router (GA!)
MySQL Router is lightweight middleware that provides transparent routing between your
application and backend MySQL Servers. It can be used for a wide variety of use cases,
such as providing high availability and scalability by effectively routing database traffic to
appropriate backend MySQL Servers.
MySQL Router doesn´t require any specific configuration. It configures itself automatically
(bootstrap) using MySQL InnoDB Cluster´s metadata.
The MySQL Router development will be focusing on sharding.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
69 / 84

70. ProxySQL has native support for Group
Replication which makes it a good choice for
advanced users.
ProxySQL
If you need some specific features that are not yet available in MySQL Router, like
transparent R/Wsplitting, then you can use your software of choice.
We are also collaborating with ProxySQL.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
70 / 84

71. Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
71 / 84

72. MySQL Shell (GA!)
The MySQL Shell is an interactive Javascript, Python, or SQL interface supporting
development and administration for the MySQL Server and is a component of the MySQL
Server. You can use the MySQL Shell to performdata queries and updates as well as
various administration operations.
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
72 / 84

73. MySQL Shell (2)
The MySQL Shell provides:
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
73 / 84

74. MySQL Shell (2)
The MySQL Shell provides:
Both Interactive and Batch operations
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
74 / 84

75. MySQL Shell (2)
The MySQL Shell provides:
Both Interactive and Batch operations
Document and Relational Models
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
75 / 84

76. MySQL Shell (2)
The MySQL Shell provides:
Both Interactive and Batch operations
Document and Relational Models
CRUD Document and Relational APIs via scripting
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
76 / 84

77. MySQL Shell (2)
The MySQL Shell provides:
Both Interactive and Batch operations
Document and Relational Models
CRUD Document and Relational APIs via scripting
Traditional Table, JSON, Tab Separated output results formats
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
77 / 84

78. MySQL Shell (2)
The MySQL Shell provides:
Both Interactive and Batch operations
Document and Relational Models
CRUD Document and Relational APIs via scripting
Traditional Table, JSON, Tab Separated output results formats
MySQL Standard and X Protocols
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
78 / 84

79. MySQL Shell (2)
The MySQL Shell provides:
Both Interactive and Batch operations
Document and Relational Models
CRUD Document and Relational APIs via scripting
Traditional Table, JSON, Tab Separated output results formats
MySQL Standard and X Protocols
and more...
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
79 / 84

80. MySQL Shell (3)
PREVIEW: the newShell´s prompt is nice and clear !
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
80 / 84

81. don´t forget, this is a HA solution !
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
81 / 84

82. Deploying a MySQL InnoDB Cluster using MySQL
Shell´s adminAPI:
mysql-js> var i1 = 'root@instance01:3306';
mysql-js> var i2 = 'root@instance02:3306';
mysql-js> var i3 = 'root@instance03:3306';
mysql-js> dba.checkInstanceCon guration(i1);
mysql-js> dba.checkInstanceCon guration(i2);
mysql-js> dba.checkInstanceCon guration(i3);
mysql-js> shell.connect(i1);
mysql-js> var cluster = dba.createCluster('ChinaCluster');
mysql-js> cluster.checkInstanceState(i2);
mysql-js> cluster.addInstance(i2);
mysql-js> cluster.checkInstanceState(i3);
mysql-js> cluster.addInstance(i3);
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
82 / 84

83. See this in action and do it yourself during the
next session !
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
83 / 84

84. Thank you !
Any Questions ?
more at http://lefred.be/content/category/mysql/group-replication/
http://mysqlhighavailability.com/
Copyright @ 2017 Oracle and/or its affiliates. All rights reserved.
84 / 84