Topics covered in these slides: How to perform Zero Downtime Schema Changes 2 main methods: TOI and RSU Total Order Isolation: predictability and consistency Rolling Schema Upgrades pt-online-schema-change Schema synchronization with re-joining nodes Recommended procedures Common pitfalls/user errors

1. Galera
Cluster
Schema Upgrades
Seppo Jaakola
Codership

2. Agenda
●
Schema Upgrades in Synchronous Cluster
●
DDL Classification
●
Schema Backwards Compatibility
●
Alternatives for Schema Upgrades
●
●
●
●
●
●
TOI
RSU / desync
wsrep_desync / wsrep_on method
Dropping Node for DDL
pt-online-schema-change
Summary
www.codership.com
2

3. Schema Upgrades
Applications have evolution, and there will be needs to
change database schema in new application revisions
With 24/7 online requirements, the schema upgrade must
happen while the system is online
Synchronous database cluster, requires that all active
nodes must have same data
–
Note that schema may differ, as long as data content is logically same
We need to figure out way(s) to roll schema upgrades in
online cluster with minimal disruption for the service
www.codership.com
3

4. What is DDL?
SQL statements can be classified in several classes:
DML Data Manipulation Language
–
Transactional data manipulations
–
SELECT, INSERT, UPDATE...
DDL Data Definition Language
–
CREATE..., DROP..., ALTER...
DCL Data Control Language
–
GRANT, REVOKE
TCL Transaction Control Language
…
http://en.wikibooks.org/wiki/MySQL/Language/Definitions:_what_are_DDL,_DML_and_DQL%3F
www.codership.com
4

5. DML vs DDL
But the bottom line is division between transactional and nontransactional statements:
Transactional SQL
–
All DML on InnoDB
–
NON Transactional SQL
Can be rolled back
●
Not possible to rollback
–
●
DDL, DCL..., DML on non
transactional SE
–
●
–
Requires up-front locking
Galera uses optimistic concurrency control, and depends on possibility to rollback a
conflicting operation
Only transactional SQL can be replicated through Galera transactional replication
For others (DDL, DCL...), we either have to skip replication or use up-front locking
www.codership.com
5

6. DML vs DDL
Notes on some DDL:
TRUNCATE
–
is DDL!
–
Is fast to execute, but nevertheless has some impact
OPTIMIZE, REPAIR, ANALYZE
–
Table admin commands are now replicated
CREATE / DROP INDEX
–
Hold MDL on affected table, and can stall the
replication
www.codership.com
6

7. Schema Backward Compatibility

8. Backward Compatibility
App v1
MySQL
Schema v1
App v1
Schema
Upgrade
MySQL
Schema v2
App
Upgrade
App v2
MySQL
Schema v2
Old application version
must be able to use the
new schema
www.codership.com
9

9. Backward Compatibility
App v1
MySQL
Schema v1
App
Upgrade
App v2
MySQL
Schema v1
App v2
Schema
Upgrade
MySQL
Schema v2
New application version
must be able to use the
old schema
www.codership.com
10

10. Backwards Compatibility
New/old application should be able to use both old and
new schema
Application should be backwards compatible
ROW replication between old and new schema should be
possible
Schema change should be backwards compatible
www.codership.com
11

11. App Backwards Compatibility
New/old application should be able to use both old and
new schema
–
New application can have compatibility mode to detect the version of
underlying database
–
If old app cannot use new schema, the old app must connect to one
node only, which will be the last to upgrade
Dropping tables or columns can be a problem
–
But drops can be done delayed: e.g. in v2 -> v3 upgrade, obsolete v1
elements can be dropped as neither v2 or v3 app will use them any
more
www.codership.com
12

12. ROW Replication Compatibility
MySQL guarantees ROW replication event compatibility
with some limitations
Newer MySQL versions tolerate more variation between
source and target tables, check out this page for latest
status:
http://dev.mysql.com/doc/refman/5.6/en/replication-features-differing-tables.html
●
●
Source and target can have different number of columns
But columns must be in same order
●
New columns in the end, and must have default values
●
Some data type conversions are also supported
www.codership.com
13

13. ROW Replication Compatibility
Insert into table-A(col-a,col-b) values (5,7)
col-a
col-b
col-a
Table A
col-b
col-c
Table A
www.codership.com
14

14. ROW Replication Compatibility
Insert into table-A(col-a,col-b) values (5,7)
col-a
col-b
col-a
col-b
col-c
5
7
5
7
def
Replication
Table A
Table A
www.codership.com
15

15. STATEMENT Replication
In STATEMENT format, schema compatibility is not an issue
Galera does not currently support STATEMENT replication,
but:
–
Enabling STATEMENT replication is minor task
Consistency is at risk
● Parallel applying must be limited (OFF, Database or Table level)
● STATEMENT replication, in general, is phasing out
Supporting STATEMENT replication for schema upgrades, is one
potential extension we are looking for
●
–
www.codership.com
16

16. Schema Upgrades in Galera Cluster

17. Schema Upgrades in Galera
Galera Cluster has two inbuilt methods for DDL replication:
–
TOI – Total Order Isolation
–
RSU – Rolling Schema Upgrade
The method of choice is declared by variable:
wsrep_osu_method = TOI | RSU
Pt-online-schema-change is valid tool for upgrades, these
and other DDL replication alternatives are discussed in
following chapters.
www.codership.com
18

18. Total Order Isolation

19. TOI
Total oder Isolation (TOI) is the default DDL replication
method
●
●
●
●
wsrep_osu_method = TOI
“master node” detects DDL at parsing time and sends out replication event
for the SQL statement before even starting the DDL processing
DDL replication happens in STATEMENT format
Every node in the cluster will process the replicated DDL at the same “slot”
in the cluster transaction stream (Total Order)
www.codership.com
20

20. TOI Replication
ALTER TABLE t1...
Parser
Replication
MySQL
MySQL
Execution
a
Galera
Replication
WS
Seqno
STATEMENT event
G a l e r a R e p l i c a t io n
www.codership.com
21

21. TOI Replication
ALTER TABLE t1...
Parser
apply
continue
Parser
MySQL
MySQL
Execution
Execution
a
Galera
Replication
WS
Seqno slot
reached
www.codership.com
22

22. TOI Replication
Pros
–
Strict consistency, all nodes will get same change
–
No worries about schema backwards compatibility
Cons
–
Strict commit order will make every transaction to wait until DDL is
over
Usable, when:
–
DDL is short term
–
Schema change will not be backwards compatible
–
And/or there is maintenance window
Some future work in road map:
–
TOI replication commit order can be relaxed
–
We are working on optimization based on this
www.codership.com
23

23. Rolling Schema Upgrade

24. RSU
➢
Rolling Schema Upgrade
wsrep_osu_method=RSU
➢
Will desynchronize the node from replication for the
duration of following DDL command
➢
Incoming replication is buffered
➢
Nothing will be replicated out of the node
➢
When DDL is over, the node will automatically join back
in cluster, and catch up missed transactions from the
buffer
www.codership.com
25

25. RSU
SET GLOBAL
wsrep_osu_method=RSU
ALTER TABLE t1...
MySQL
MySQL
MySQL
G a l e r a R e p l i c a t io n
a
Galera Replication
www.codership.com
26

26. RSU
ALTER TABLE
MySQL
MySQL
WS
a
MySQL
WS
Slave queue
Galera Replication
www.codership.com
27

27. RSU
ALTER TABLE t1...
MySQL
WS
a
MySQL
MySQL
WS
Slave queue
Galera Replication
www.codership.com
28

28. RSU
Pros
–
DDL will not slow down cluster
–
Automatic re-sync after DDL is over
Cons
–
–
Schema change has to be backwards compatible
–
a
Has global effect, all sessions will be RSU'ed
Only one RSU operation at a time
–
Rolling over cluster is manual operation
www.codership.com
29

29. wsrep_desync

30. wsrep_desync
Node can be set to omit flow control by:
mysql> SET GLOBAL wsrep_desync=ON;
A session can be declared to not replicate anything by:
mysql> SET wsrep_on=OFF;
●
●
Running DDL in such a session, will result in local
schema change, and processing of the DDL will not
hold back cluster.
However, all cluster transactions will be replicated to
the node, and if there are conflicts, the DDL will be
aborted.
wsrep_desync+wsrep_on method is good only for
non-confliction operations
www.codership.com
31

31. wsrep_desync+wsrep_on
SET GLOBAL wsrep_desync=ON;
SET wsrep_on=OFF;
ALTER TABLE t1...
t1
t2
tn
MySQL
a
WS
WS
Slave queue
Galera Replication
www.codership.com
32

32. wsrep_desync
We are currently working on making better use of
desync mode. The goal is to protect local desynced
transactions from replication aborts.
This way, the DDL will succeed even if there are
conflicts with the cluster. However, cluster replication
will pause at first such conflict and wait until DDL is
complete.
But, this will be future extension, and available in some
of following 3.* release.
www.codership.com
33

33. Drop a Node for DDL

34. Dropping Node for DDL
One way to do “manual RSU”, is to drop a node from
cluster and run DDL on the stand-alone node.
Joining the node back must happen through IST, as we
don't want SST to bring back the old schema.
Make sure to protect the node from any production
connections! Load balancers should be configured first to
isolate the node from unwanted connections.
Adjust your gcache size big enough to allow IST after the
DDL is over.
www.codership.com
35

35. Dropping Node for DDL
Load Balancer
1. configure LB
2. Drop node, e.g.
set global wsrep_cluster_address=gcomm://
MySQL
MySQL
Galera Replication
www.codership.com
36

36. Dropping Node for DDL
Load Balancer
3. ALTER TABLE t1...
MySQL
MySQL
Galera Replication
www.codership.com
37

37. Dropping Node for DDL
Load Balancer
4. Join back
set wsrep_cluster_address...
WS
MySQL
WS
IST
MySQL
Galera Replication
www.codership.com
38

38. Dropping Node for DDL
Load Balancer
5. configure LB
MySQL
MySQL
Galera Replication
www.codership.com
39

39. pt-online-schema-change

40. pt-online-schema-change
Tool in Percona Toolkit to run non blocking schema
changes
http://www.percona.com/doc/percona-toolkit/2.2/pt-online-schema-change.html
1. Creates a shadow copy of target table
2. Installs triggers in source table to forward updates to target table
3. Copies source table data in small chunks to target table
4. Renames target table to replace the source table
Pt-osc does not replicate schema changes, but makes it
possible to change schema without interfering with
replication
However, pt-osc requires TOI to be enabled, and TOI
replication will propagate the changes to whole cluster
www.codership.com
41

41. pt-online-schema-change
updates
t1-new
t1
CREATE similar table
and ALTER
a
www.codership.com
42

42. pt-online-schema-change
updates
t1-new
t1
a
Install trigger to forward updates
to new table
www.codership.com
43

43. pt-online-schema-change
updates
t1-new
t1
Copy data in
chunks
a
www.codership.com
44

44. pt-online-schema-change
updates
t1
t1-new
Copy data in
chunks
a
www.codership.com
45

45. pt-online-schema-change
updates
t1
t1-old
Rename
tables
a
www.codership.com
46

46. pt-online-schema-change
updates
t1
Drop old table
a
www.codership.com
47

47. pt-online-schema-change
Some Caveats:
●
TOI requirement
–
–
●
Pt-osc changes will be run against whole cluster with one go
Could be relaxed, imo
Triggers not supported
–
●
Pt-osc installs new triggers in source table and does not allow any
other triggers to exists in the table
Foreign key support
–
a
Two methods for dealing with FKs
–
Rebuilding child table FK constraint may be needed
–
FK constraint name will be different
www.codership.com
48

48. Summary

49. Codership
Schema upgrades require careful planning
➢
Find out backwards compatibility both from application and from ROW
replication perspective
➢
Plan your upgrade process
➢
Rehearse with test cluster
Instant methods:
➢
TOI replication, pt-osc
➢
ROW replication backwards compatibility is not an issue
Rolling methods
➢
RSU, wsrep_desync/wsrep_on, node dropping
➢
Schema backwards compatibility required
www.codership.com
50

50. Questions?
Thank you for listening!
Happy Clustering :-)