Galera

Cluster

Schema Upgrades
Seppo Jaakola
Codership
Agenda
●

Schema Upgrades in Synchronous Cluster

●

DDL Classification

●

Schema Backwards Compatibility

●

Alternative...
Schema Upgrades
Applications have evolution, and there will be needs to
change database schema in new application revision...
What is DDL?
SQL statements can be classified in several classes:
DML Data Manipulation Language
–

Transactional data man...
DML vs DDL
But the bottom line is division between transactional and nontransactional statements:

Transactional SQL
–

Al...
DML vs DDL
Notes on some DDL:
TRUNCATE
–

is DDL!

–

Is fast to execute, but nevertheless has some impact

OPTIMIZE, REPA...
Schema Backward Compatibility
Backward Compatibility

App v1

MySQL
Schema v1

App v1

Schema
Upgrade

MySQL
Schema v2

App
Upgrade

App v2

MySQL
Schem...
Backward Compatibility

App v1

MySQL
Schema v1

App
Upgrade

App v2

MySQL
Schema v1

App v2

Schema
Upgrade

MySQL
Schem...
Backwards Compatibility
New/old application should be able to use both old and
new schema
Application should be backwards ...
App Backwards Compatibility
New/old application should be able to use both old and
new schema
–

New application can have ...
ROW Replication Compatibility
MySQL guarantees ROW replication event compatibility
with some limitations
Newer MySQL versi...
ROW Replication Compatibility

Insert into table-A(col-a,col-b) values (5,7)

col-a

col-b

col-a

Table A

col-b

col-c

...
ROW Replication Compatibility

Insert into table-A(col-a,col-b) values (5,7)

col-a

col-b

col-a

col-b

col-c

5

7

5

...
STATEMENT Replication
In STATEMENT format, schema compatibility is not an issue
Galera does not currently support STATEMEN...
Schema Upgrades in Galera Cluster
Schema Upgrades in Galera
Galera Cluster has two inbuilt methods for DDL replication:
–

TOI – Total Order Isolation

–

R...
Total Order Isolation
TOI
Total oder Isolation (TOI) is the default DDL replication
method
●

●

●

●

wsrep_osu_method = TOI
“master node” dete...
TOI Replication

ALTER TABLE t1...

Parser
Replication
MySQL

MySQL

Execution

a

Galera
Replication

WS

Seqno
STATEMENT...
TOI Replication

ALTER TABLE t1...

Parser

apply

continue

Parser

MySQL

MySQL

Execution

Execution

a

Galera
Replica...
TOI Replication
Pros
–

Strict consistency, all nodes will get same change

–

No worries about schema backwards compatibi...
Rolling Schema Upgrade
RSU

➢

Rolling Schema Upgrade
wsrep_osu_method=RSU

➢

Will desynchronize the node from replication for the
duration of f...
RSU
SET GLOBAL
wsrep_osu_method=RSU
ALTER TABLE t1...

MySQL

MySQL

MySQL

G a l e r a R e p l i c a t io n

a

Galera Re...
RSU

ALTER TABLE

MySQL
MySQL

WS

a

MySQL

WS

Slave queue

Galera Replication
www.codership.com
27
RSU

ALTER TABLE t1...

MySQL

WS

a

MySQL

MySQL

WS

Slave queue

Galera Replication
www.codership.com
28
RSU
Pros
–

DDL will not slow down cluster

–

Automatic re-sync after DDL is over

Cons
–
–

Schema change has to be back...
wsrep_desync
wsrep_desync
Node can be set to omit flow control by:
mysql> SET GLOBAL wsrep_desync=ON;

A session can be declared to not...
wsrep_desync+wsrep_on
SET GLOBAL wsrep_desync=ON;
SET wsrep_on=OFF;
ALTER TABLE t1...

t1

t2

tn

MySQL

a

WS

WS

Slave...
wsrep_desync
We are currently working on making better use of
desync mode. The goal is to protect local desynced
transacti...
Drop a Node for DDL
Dropping Node for DDL
One way to do “manual RSU”, is to drop a node from
cluster and run DDL on the stand-alone node.
Join...
Dropping Node for DDL

Load Balancer

1. configure LB
2. Drop node, e.g.
set global wsrep_cluster_address=gcomm://

MySQL
...
Dropping Node for DDL

Load Balancer

3. ALTER TABLE t1...

MySQL

MySQL

Galera Replication
www.codership.com
37
Dropping Node for DDL

Load Balancer

4. Join back
set wsrep_cluster_address...

WS

MySQL

WS

IST

MySQL

Galera Replica...
Dropping Node for DDL

Load Balancer

5. configure LB

MySQL

MySQL

Galera Replication
www.codership.com
39
pt-online-schema-change
pt-online-schema-change
Tool in Percona Toolkit to run non blocking schema
changes
http://www.percona.com/doc/percona-tool...
pt-online-schema-change

updates

t1-new
t1

CREATE similar table
and ALTER

a
www.codership.com
42
pt-online-schema-change

updates

t1-new
t1

a

Install trigger to forward updates
to new table
www.codership.com
43
pt-online-schema-change

updates

t1-new
t1

Copy data in
chunks

a
www.codership.com
44
pt-online-schema-change

updates

t1

t1-new

Copy data in
chunks

a
www.codership.com
45
pt-online-schema-change

updates

t1

t1-old

Rename
tables

a
www.codership.com
46
pt-online-schema-change

updates

t1

Drop old table

a
www.codership.com
47
pt-online-schema-change
Some Caveats:
●

TOI requirement
–
–

●

Pt-osc changes will be run against whole cluster with one...
Summary
Codership
Schema upgrades require careful planning
➢

Find out backwards compatibility both from application and from ROW
...
Questions?

Thank you for listening!
Happy Clustering :-)
Upcoming SlideShare
Loading in...5
×

Zero Downtime Schema Changes - Galera Cluster - Best Practices

6,446

Published on

Database schema changes are usually not popular among DBAs or sysadmins, not when you are operating a cluster and cannot afford to switch off the service during a maintenance window. There are different ways to perform schema changes, some procedures being more complicated than others.

Galera Cluster is great at making your MySQL database highly available, but are you concerned about schema changes? Is an ALTER TABLE statement something that requires a lot of advance scheduling? What is the impact on your database uptime?

This is a common question, since ALTER operations in MySQL usually cause the table to be locked and rebuilt – which can potentially be disruptive to your live applications. Fortunately, Galera Cluster has mechanisms to replicate DDL across its nodes.

In these slides, you will learn about the following:

How to perform Zero Downtime Schema Changes
2 main methods: TOI and RSU
Total Order Isolation: predictability and consistency
Rolling Schema Upgrades
pt-online-schema-change
Schema synchronization with re-joining nodes
Recommended procedures
Common pitfalls/user errors

The slides are courtesy of Seppo Jaakola, CEO, Codership - creators of Galera Cluster

Published in: Technology
0 Comments
10 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
6,446
On Slideshare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
103
Comments
0
Likes
10
Embeds 0
No embeds

No notes for slide

Zero Downtime Schema Changes - Galera Cluster - Best Practices

  1. 1. Galera Cluster Schema Upgrades Seppo Jaakola Codership
  2. 2. Agenda ● Schema Upgrades in Synchronous Cluster ● DDL Classification ● Schema Backwards Compatibility ● Alternatives for Schema Upgrades ● ● ● ● ● ● TOI RSU / desync wsrep_desync / wsrep_on method Dropping Node for DDL pt-online-schema-change Summary www.codership.com 2
  3. 3. Schema Upgrades Applications have evolution, and there will be needs to change database schema in new application revisions With 24/7 online requirements, the schema upgrade must happen while the system is online Synchronous database cluster, requires that all active nodes must have same data – Note that schema may differ, as long as data content is logically same We need to figure out way(s) to roll schema upgrades in online cluster with minimal disruption for the service www.codership.com 3
  4. 4. What is DDL? SQL statements can be classified in several classes: DML Data Manipulation Language – Transactional data manipulations – SELECT, INSERT, UPDATE... DDL Data Definition Language – CREATE..., DROP..., ALTER... DCL Data Control Language – GRANT, REVOKE TCL Transaction Control Language … http://en.wikibooks.org/wiki/MySQL/Language/Definitions:_what_are_DDL,_DML_and_DQL%3F www.codership.com 4
  5. 5. DML vs DDL But the bottom line is division between transactional and nontransactional statements: Transactional SQL – All DML on InnoDB – NON Transactional SQL Can be rolled back ● Not possible to rollback – ● DDL, DCL..., DML on non transactional SE – ● – Requires up-front locking Galera uses optimistic concurrency control, and depends on possibility to rollback a conflicting operation Only transactional SQL can be replicated through Galera transactional replication For others (DDL, DCL...), we either have to skip replication or use up-front locking www.codership.com 5
  6. 6. DML vs DDL Notes on some DDL: TRUNCATE – is DDL! – Is fast to execute, but nevertheless has some impact OPTIMIZE, REPAIR, ANALYZE – Table admin commands are now replicated CREATE / DROP INDEX – Hold MDL on affected table, and can stall the replication www.codership.com 6
  7. 7. Schema Backward Compatibility
  8. 8. Backward Compatibility App v1 MySQL Schema v1 App v1 Schema Upgrade MySQL Schema v2 App Upgrade App v2 MySQL Schema v2 Old application version must be able to use the new schema www.codership.com 9
  9. 9. Backward Compatibility App v1 MySQL Schema v1 App Upgrade App v2 MySQL Schema v1 App v2 Schema Upgrade MySQL Schema v2 New application version must be able to use the old schema www.codership.com 10
  10. 10. Backwards Compatibility New/old application should be able to use both old and new schema Application should be backwards compatible ROW replication between old and new schema should be possible Schema change should be backwards compatible www.codership.com 11
  11. 11. App Backwards Compatibility New/old application should be able to use both old and new schema – New application can have compatibility mode to detect the version of underlying database – If old app cannot use new schema, the old app must connect to one node only, which will be the last to upgrade Dropping tables or columns can be a problem – But drops can be done delayed: e.g. in v2 -> v3 upgrade, obsolete v1 elements can be dropped as neither v2 or v3 app will use them any more www.codership.com 12
  12. 12. ROW Replication Compatibility MySQL guarantees ROW replication event compatibility with some limitations Newer MySQL versions tolerate more variation between source and target tables, check out this page for latest status: http://dev.mysql.com/doc/refman/5.6/en/replication-features-differing-tables.html ● ● Source and target can have different number of columns But columns must be in same order ● New columns in the end, and must have default values ● Some data type conversions are also supported www.codership.com 13
  13. 13. ROW Replication Compatibility Insert into table-A(col-a,col-b) values (5,7) col-a col-b col-a Table A col-b col-c Table A www.codership.com 14
  14. 14. ROW Replication Compatibility Insert into table-A(col-a,col-b) values (5,7) col-a col-b col-a col-b col-c 5 7 5 7 def Replication Table A Table A www.codership.com 15
  15. 15. STATEMENT Replication In STATEMENT format, schema compatibility is not an issue Galera does not currently support STATEMENT replication, but: – Enabling STATEMENT replication is minor task Consistency is at risk ● Parallel applying must be limited (OFF, Database or Table level) ● STATEMENT replication, in general, is phasing out Supporting STATEMENT replication for schema upgrades, is one potential extension we are looking for ● – www.codership.com 16
  16. 16. Schema Upgrades in Galera Cluster
  17. 17. Schema Upgrades in Galera Galera Cluster has two inbuilt methods for DDL replication: – TOI – Total Order Isolation – RSU – Rolling Schema Upgrade The method of choice is declared by variable: wsrep_osu_method = TOI | RSU Pt-online-schema-change is valid tool for upgrades, these and other DDL replication alternatives are discussed in following chapters. www.codership.com 18
  18. 18. Total Order Isolation
  19. 19. TOI Total oder Isolation (TOI) is the default DDL replication method ● ● ● ● wsrep_osu_method = TOI “master node” detects DDL at parsing time and sends out replication event for the SQL statement before even starting the DDL processing DDL replication happens in STATEMENT format Every node in the cluster will process the replicated DDL at the same “slot” in the cluster transaction stream (Total Order) www.codership.com 20
  20. 20. TOI Replication ALTER TABLE t1... Parser Replication MySQL MySQL Execution a Galera Replication WS Seqno STATEMENT event G a l e r a R e p l i c a t io n www.codership.com 21
  21. 21. TOI Replication ALTER TABLE t1... Parser apply continue Parser MySQL MySQL Execution Execution a Galera Replication WS Seqno slot reached www.codership.com 22
  22. 22. TOI Replication Pros – Strict consistency, all nodes will get same change – No worries about schema backwards compatibility Cons – Strict commit order will make every transaction to wait until DDL is over Usable, when: – DDL is short term – Schema change will not be backwards compatible – And/or there is maintenance window Some future work in road map: – TOI replication commit order can be relaxed – We are working on optimization based on this www.codership.com 23
  23. 23. Rolling Schema Upgrade
  24. 24. RSU ➢ Rolling Schema Upgrade wsrep_osu_method=RSU ➢ Will desynchronize the node from replication for the duration of following DDL command ➢ Incoming replication is buffered ➢ Nothing will be replicated out of the node ➢ When DDL is over, the node will automatically join back in cluster, and catch up missed transactions from the buffer www.codership.com 25
  25. 25. RSU SET GLOBAL wsrep_osu_method=RSU ALTER TABLE t1... MySQL MySQL MySQL G a l e r a R e p l i c a t io n a Galera Replication www.codership.com 26
  26. 26. RSU ALTER TABLE MySQL MySQL WS a MySQL WS Slave queue Galera Replication www.codership.com 27
  27. 27. RSU ALTER TABLE t1... MySQL WS a MySQL MySQL WS Slave queue Galera Replication www.codership.com 28
  28. 28. RSU Pros – DDL will not slow down cluster – Automatic re-sync after DDL is over Cons – – Schema change has to be backwards compatible – a Has global effect, all sessions will be RSU'ed Only one RSU operation at a time – Rolling over cluster is manual operation www.codership.com 29
  29. 29. wsrep_desync
  30. 30. wsrep_desync Node can be set to omit flow control by: mysql> SET GLOBAL wsrep_desync=ON; A session can be declared to not replicate anything by: mysql> SET wsrep_on=OFF; ● ● Running DDL in such a session, will result in local schema change, and processing of the DDL will not hold back cluster. However, all cluster transactions will be replicated to the node, and if there are conflicts, the DDL will be aborted. wsrep_desync+wsrep_on method is good only for non-confliction operations www.codership.com 31
  31. 31. wsrep_desync+wsrep_on SET GLOBAL wsrep_desync=ON; SET wsrep_on=OFF; ALTER TABLE t1... t1 t2 tn MySQL a WS WS Slave queue Galera Replication www.codership.com 32
  32. 32. wsrep_desync We are currently working on making better use of desync mode. The goal is to protect local desynced transactions from replication aborts. This way, the DDL will succeed even if there are conflicts with the cluster. However, cluster replication will pause at first such conflict and wait until DDL is complete. But, this will be future extension, and available in some of following 3.* release. www.codership.com 33
  33. 33. Drop a Node for DDL
  34. 34. Dropping Node for DDL One way to do “manual RSU”, is to drop a node from cluster and run DDL on the stand-alone node. Joining the node back must happen through IST, as we don't want SST to bring back the old schema. Make sure to protect the node from any production connections! Load balancers should be configured first to isolate the node from unwanted connections. Adjust your gcache size big enough to allow IST after the DDL is over. www.codership.com 35
  35. 35. Dropping Node for DDL Load Balancer 1. configure LB 2. Drop node, e.g. set global wsrep_cluster_address=gcomm:// MySQL MySQL Galera Replication www.codership.com 36
  36. 36. Dropping Node for DDL Load Balancer 3. ALTER TABLE t1... MySQL MySQL Galera Replication www.codership.com 37
  37. 37. Dropping Node for DDL Load Balancer 4. Join back set wsrep_cluster_address... WS MySQL WS IST MySQL Galera Replication www.codership.com 38
  38. 38. Dropping Node for DDL Load Balancer 5. configure LB MySQL MySQL Galera Replication www.codership.com 39
  39. 39. pt-online-schema-change
  40. 40. pt-online-schema-change Tool in Percona Toolkit to run non blocking schema changes http://www.percona.com/doc/percona-toolkit/2.2/pt-online-schema-change.html 1. Creates a shadow copy of target table 2. Installs triggers in source table to forward updates to target table 3. Copies source table data in small chunks to target table 4. Renames target table to replace the source table Pt-osc does not replicate schema changes, but makes it possible to change schema without interfering with replication However, pt-osc requires TOI to be enabled, and TOI replication will propagate the changes to whole cluster www.codership.com 41
  41. 41. pt-online-schema-change updates t1-new t1 CREATE similar table and ALTER a www.codership.com 42
  42. 42. pt-online-schema-change updates t1-new t1 a Install trigger to forward updates to new table www.codership.com 43
  43. 43. pt-online-schema-change updates t1-new t1 Copy data in chunks a www.codership.com 44
  44. 44. pt-online-schema-change updates t1 t1-new Copy data in chunks a www.codership.com 45
  45. 45. pt-online-schema-change updates t1 t1-old Rename tables a www.codership.com 46
  46. 46. pt-online-schema-change updates t1 Drop old table a www.codership.com 47
  47. 47. pt-online-schema-change Some Caveats: ● TOI requirement – – ● Pt-osc changes will be run against whole cluster with one go Could be relaxed, imo Triggers not supported – ● Pt-osc installs new triggers in source table and does not allow any other triggers to exists in the table Foreign key support – a Two methods for dealing with FKs – Rebuilding child table FK constraint may be needed – FK constraint name will be different www.codership.com 48
  48. 48. Summary
  49. 49. Codership Schema upgrades require careful planning ➢ Find out backwards compatibility both from application and from ROW replication perspective ➢ Plan your upgrade process ➢ Rehearse with test cluster Instant methods: ➢ TOI replication, pt-osc ➢ ROW replication backwards compatibility is not an issue Rolling methods ➢ RSU, wsrep_desync/wsrep_on, node dropping ➢ Schema backwards compatibility required www.codership.com 50
  50. 50. Questions? Thank you for listening! Happy Clustering :-)
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×