MySQL User Camp: GTIDs
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

MySQL User Camp: GTIDs

on

  • 407 views

The slde contains an introduction to the global transaction identifiers(GTIDs) in MySQL Replication. The new protocol at re-connect, skipping transactions with GTIDS, replication filters, purging ...

The slde contains an introduction to the global transaction identifiers(GTIDs) in MySQL Replication. The new protocol at re-connect, skipping transactions with GTIDS, replication filters, purging logs, backup/restore ets are covered here.

Statistics

Views

Total Views
407
Views on SlideShare
312
Embed Views
95

Actions

Likes
1
Downloads
14
Comments
0

4 Embeds 95

http://localhost 48
http://shivjijha.com 44
http://markable.in 2
https://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

MySQL User Camp: GTIDs Presentation Transcript

  • 1. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. MySQL Replication: Global Transaction Identifiers(GTIDs) Shivji Kumar Jha Software Developer, MySQL Replication Team
  • 2. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Safe Harbour Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
  • 3. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Program Agenda  Introduction to MySQL Replication  Fail-over Base – Global Transaction Identifiers – Automatic Positioning – Hands-On  Under the Hood: Become a GTID Expert – How Slave Preserves GTIDs – GTIDs Must Be Unique  Integration With Other Features – Seeking & Skipping – Purging Binary Logs – Restoring from Backup
  • 4. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features What is Replication?
  • 5. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Replication: Copy Changes Master → Slave  MySQL Master Server – Changes data – Sends changes to slave  MySQL Slave Server – Receives changes from master – Applies received changes to database Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features M S
  • 6. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Replication: Copy Changes Master → Slave Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features M M/S S S S S M Server can be master, slave or both Master can have multiple slaves
  • 7. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Replication: Copy Changes Master → Slave Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features S M M ● Slave can only have one master S M M Slave can have multiple masters! labs.mysql.com labs Yippee!
  • 8. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Replication: Copy Changes Master → Slave Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features M/S ● Circular replication is also possible M/S M/S M/S M/S M/S
  • 9. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features Why Use Replication?
  • 10. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Why Replication? – Performance Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Read scale-out M S write clients read clients
  • 11. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Why Replication? – Performance Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Read scale-out M S write clients read clients More reads? More slaves!
  • 12. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Why Replication? – Performance Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Read scale-out M S write clients read clients More reads? More slaves! S S S M read clients write clients
  • 13. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Why Replication? – Redundancy Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  If master crashes, promote slave to master C B A
  • 14. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Why Replication? – Redundancy Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  If master crashes, promote slave to master C B ACrash
  • 15. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Why Replication? – Redundancy Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  If master crashes, promote slave to master C B A B is the new master
  • 16. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Why Replication? – Online backup/reporting Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Expensive queries on slave(s) M S Regular clients  Reports  Big queries  Business intelligence
  • 17. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. CB BAAC Image from www.ginkgomaps.com Why Replication? – Long-distance Data Distribution Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features
  • 18. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features How Does Replication Work?
  • 19. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. All Changes Written to Binary Log Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features A binary log Client
  • 20. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. All Changes Written to Binary Log Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features A binary log Client create table t (a int);
  • 21. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. All Changes Written to Binary Log Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features create... A binary log Client Table t create table t (a int);
  • 22. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. All Changes Written to Binary Log Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features create... A binary log Client Table t create table t (a int); insert into t values (1);
  • 23. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. All Changes Written to Binary Log Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features create... insert...A binary log Client Table t 1 create table t (a int); insert into t values (1);
  • 24. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Slave Initiates Replication Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features B binary log A binary log Client 1. Slave sends request to start replication to master 2. Master sends stream of replication data to slave
  • 25. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Binary Log Sent to Slave, Re-executed Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features B binary log A binary log Client
  • 26. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Binary Log Sent to Slave, Re-executed Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features create... B binary log A binary log Client
  • 27. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Binary Log Sent to Slave, Re-executed Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features create... Table t B binary log create... A binary log Client
  • 28. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Binary Log Sent to Slave, Re-executed Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features create... Table t Table t create... B binary log create... A binary log Client
  • 29. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Binary Log Sent to Slave, Re-executed Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features create... insert... Table t Table t create... B binary log create... A binary log Client
  • 30. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Binary Log Sent to Slave, Re-executed Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features create... insert... Table t 1 Table t create... B binary log create... insert...A binary log Client
  • 31. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Binary Log Sent to Slave, Re-executed Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features create... insert... Table t 1 Table t 1 create... insert...B binary log create... insert...A binary log Client
  • 32. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Makes sense?
  • 33. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Program Agenda  Introduction to MySQL Replication  Fail-over Base – Global Transaction Identifiers – Automatic Positioning – Hands-On  Under the Hood: Become a GTID Expert – How Slave Preserves GTIDs – GTIDs Must Be Unique  Integration With Other Features – Seeking & Skipping – Purging Binary Logs – Restoring from Backup
  • 34. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Fail-over  Crash can happen to anyone: – Hardware failure – Human mistake – Bug – Natural disaster  Automatic fail-over with Global Transaction Identifiers: – Reduces admin overhead – Prevents planned downtime – Prevents unplanned downtime Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features
  • 35. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Fail-over  Handle server crash with minimal disruption  Example 1: tree Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features C B A
  • 36. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Fail-over  Handle server crash with minimal disruption  Example 1: tree Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features C B ACrash
  • 37. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Fail-over  Handle server crash with minimal disruption  Example 1: tree Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features C B A
  • 38. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Fail-over  Handle server crash with minimal disruption  Example 1: tree Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features C B A Make B the new master
  • 39. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Fail-over  Handle server crash with minimal disruption  Example 1: tree  Example 2: line Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features
  • 40. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Fail-over  Handle server crash with minimal disruption  Example 1: tree  Example 2: line Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features CA B
  • 41. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Fail-over  Handle server crash with minimal disruption  Example 1: tree  Example 2: line Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features CA BCrash
  • 42. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Fail-over  Handle server crash with minimal disruption  Example 1: tree  Example 2: line Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features CA B
  • 43. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Fail-over  Handle server crash with minimal disruption  Example 1: tree  Example 2: line Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features CA B Make A direct master of C
  • 44. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Fail-over Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Handle server crash with minimal disruption  Example 1: tree  Example 2: line  Example 3: circle C A B D
  • 45. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Fail-over Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Handle server crash with minimal disruption  Example 1: tree  Example 2: line  Example 3: circle C A B DCrash
  • 46. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Fail-over Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Handle server crash with minimal disruption  Example 1: tree  Example 2: line  Example 3: circle C A B D
  • 47. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Fail-over Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Handle server crash with minimal disruption  Example 1: tree  Example 2: line  Example 3: circle C A B D Make a shortcut in the circle
  • 48. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Fail-over Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Handle server crash with minimal disruption  Example 1: tree  Example 2: line  Example 3: circle  Example 4: scheduled maintenance C A B D Take out D!
  • 49. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Fail-over Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Handle server crash with minimal disruption  Example 1: tree  Example 2: line  Example 3: circle  Example 4: scheduled maintenance C A B D Maintain D offline, without disrupting service
  • 50. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Fail-over Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Handle server crash with minimal disruption  Example 1: tree  Example 2: line  Example 3: circle  Example 4: scheduled maintenance C A B D
  • 51. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Fail-over Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Handle server crash with minimal disruption  Example 1: tree  Example 2: line  Example 3: circle  Example 4: scheduled maintenance  Example 5: arbitrary topology A B C D E F
  • 52. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Wow! Lets get started with fail-over.
  • 53. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Program Agenda  Introduction to MySQL Replication  Fail-over Base – Global Transaction Identifiers – Automatic Positioning – Hands-On  Under the Hood: Become a GTID Expert – How Slave Preserves GTIDs – GTIDs Must Be Unique  Integration With Other Features – Seeking & Skipping – Purging Binary Logs – Restoring from Backup
  • 54. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Global Transaction Identifiers Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Server assigns Global Transaction Identifier (GTID) to every transaction: – server_uuid:number a61678ba-4889-4279-9e58-45ba840af334:1 – server_uuid identifies the server; globally unique – number is incremented by 1 for each transaction on this server  Writes GTID to binary log  Slave ensures transaction gets the same GTID when re-executed
  • 55. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Global Transaction Identifiers Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features master> CREATE TABLE t1 (a INT); master> SELECT @@global.gtid_executed; a61678ba­4889­4279­9e58­45ba840af334:1 master> INSERT INTO t1 VALUES (1); master> INSERT INTO t1 VALUES (2); master> SELECT @@global.gtid_executed; a61678ba­4889­4279­9e58­45ba840af334:1­3 Note: interval New variable: gtid_executed
  • 56. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Global Transaction Identifiers Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features master> SELECT @@global.gtid_executed; a61678ba­4889­4279­9e58­45ba840af334:1­10000  slave> SELECT @@global.gtid_executed; a61678ba­4889­4279­9e58­45ba840af334:1­9999 Slave is missing one transaction Slave is missing one transaction
  • 57. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. But how are GTIDs used in fail-over?
  • 58. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Program Agenda  Introduction to MySQL Replication  Fail-over Base – Global Transaction Identifiers – Automatic Positioning – Hands-On  Under the Hood: Become a GTID Expert – How Slave Preserves GTIDs – GTIDs Must Be Unique  Integration With Other Features – Seeking & Skipping – Purging Binary Logs – Restoring from Backup
  • 59. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Automatic Positioning Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Master–slave protocol: – Slave sends @@gtid_executed to master – Master sends all other transactions to slave id1,trx1, id2,trx2 id1,trx1, id2,trx2, id3,trx3 A 2. id3, trx3, … 1. id1…id2 B
  • 60. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Automatic Positioning Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Example 1: tree (master) A id1,trx1, id2,trx2, id3,trx3 (slave) C id1,trx1 (slave) id1,trx1, id2,trx2B
  • 61. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Automatic Positioning Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Example 1: tree (master) A id1,trx1, id2,trx2, id3,trx3 (slave) C id1,trx1 (slave) id1,trx1, id2,trx2B Crash!
  • 62. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Automatic Positioning Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Example 1: tree (master) A id1,trx1, id2,trx2, id3,trx3 (slave) C id1,trx1 (slave) id1,trx1, id2,trx2B
  • 63. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Automatic Positioning Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Example 1: tree (master) A id1,trx1, id2,trx2, id3,trx3 (slave) C id1,trx1 (slave) id1,trx1, id2,trx2B id1 id2, trx2,... What the slave has What the slave does not have
  • 64. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Automatic Positioning Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Example 1: tree  Example 2: circle BA C client client
  • 65. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Automatic Positioning Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Example 1: tree  Example 2: circle id1,trx1 BA C client client trx1
  • 66. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Automatic Positioning Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Example 1: tree  Example 2: circle id1,trx1 B id2,trx2 A C client clienttrx2 trx1
  • 67. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Automatic Positioning Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Example 1: tree  Example 2: circle id1,trx1, id2,trx2B id2,trx2 A C client clienttrx2 trx1
  • 68. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Automatic Positioning Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Example 1: tree  Example 2: circle id1,trx1, id2,trx2, id3,trx3 B id2,trx2 A C client clienttrx2 trx1, trx3
  • 69. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Automatic Positioning Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Example 1: tree  Example 2: circle id1,trx1, id2,trx2, id3,trx3 B id2,trx2 A C client clienttrx2 trx1, trx3 Crash!
  • 70. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Automatic Positioning Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Example 1: tree  Example 2: circle id1,trx1, id2,trx2, id3,trx3 B id2,trx2 A C client clienttrx2 trx1, trx3
  • 71. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Automatic Positioning Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Example 1: tree  Example 2: circle id1,trx1, id2,trx2, id3,trx3 B id2,trx2 A C client clienttrx2 trx1, trx3 id2 id1,trx1,id3,trx3,... What the slave has What the slave does not have
  • 72. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Awesome! How do I set it up?
  • 73. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Program Agenda  Introduction to MySQL Replication  Fail-over Base – Global Transaction Identifiers – Automatic Positioning – Hands-On  Under the Hood: Become a GTID Expert – How Slave Preserves GTIDs – GTIDs Must Be Unique  Integration With Other Features – Seeking & Skipping – Purging Binary Logs – Restoring from Backup
  • 74. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Hands-On Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Prerequisites: – Use transactional storage engine for all tables (InnoDB) – Don't use CREATE TABLE … SELECT – Don't execute CREATE TEMPORARY TABLE or – DROP TEMPORARY TABLE inside a transaction
  • 75. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Hands-On Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Prepare the server for fail-over – Sync and stop all servers – Add to every my.cnf: gtid­mode=on enforce­gtid­consistency=on log­bin log­slave­updates – Start all servers – Execute: > CHANGE MASTER TO MASTER_AUTO_POSITION = 1
  • 76. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Hands-On Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Perform fail-over On the slave, simply point to the new master: > CHANGE MASTER TO MASTER_HOST = '<host>',                    MASTER_PORT = <port number>,                    MASTER_USER = '<user name>'                    MASTER_PASSWORD = 'secret';  No positions needed!
  • 77. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Hands-On Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features  Perform fail-over C B A CHANGE MASTER TO MASTER_HOST = 'B', MASTER_PORT = <B's port>, MASTER_USER = '<user name>' MASTER_PASSWORD = 'secret';
  • 78. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Interesting! I want to learn more.
  • 79. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Program Agenda  Introduction to MySQL Replication  Fail-over Base – Global Transaction Identifiers – Automatic Positioning – Hands-On  Under the Hood: Become a GTID Expert – How Slave Preserves GTIDs – GTIDs Must Be Unique  Integration With Other Features – Seeking & Skipping – Purging Binary Logs – Restoring from Backup
  • 80. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Under the Hood: How Slave Preserves GTIDs Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features B id1,trx1, id2,trx2A Slave = client
  • 81. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Under the Hood: How Slave Preserves GTIDs Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features B id1,trx1, id2,trx2A Slave = client SET GTID_NEXT = 'id1'; trx1
  • 82. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Under the Hood: How Slave Preserves GTIDs Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features id1,trx1 B id1,trx1, id2,trx2A Slave = client SET GTID_NEXT = 'id1'; trx1
  • 83. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Under the Hood: How Slave Preserves GTIDs Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features id1,trx1 B id1,trx1, id2,trx2A Slave = client SET GTID_NEXT = 'id1'; trx1 SET GTID_NEXT = 'id2'; trx2
  • 84. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Under the Hood: How Slave Preserves GTIDs Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features id1,trx1, id2,trx2B id1,trx1, id2,trx2A Slave = client SET GTID_NEXT = 'id1'; trx1 SET GTID_NEXT = 'id2'; trx2
  • 85. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Under the Hood: GTID_NEXT  GTID_NEXT – session system variable  Default: “AUTOMATIC” – → Server generates new GTID for next transaction  Slave thread sets to “UUID:NUMBER” – → Server uses specified GTID for next transaction Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features
  • 86. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Under the Hood: GTID_NEXT  Clients can set GTID_NEXT too:  Mysqlbinlog outputs SET GTID_NEXT statements: id1,trx1 id2,trx2 set gtid_next = “id1”; trx1 set gtid_next = “id2”; trx2 mysqlbinlog binary log id,insert A Client set gtid_next=”id”; insert Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features
  • 87. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Program Agenda  Introduction to MySQL Replication  Fail-over Base – Global Transaction Identifiers – Automatic Positioning – Hands-On  Under the Hood: Become a GTID Expert – How Slave Preserves GTIDs – GTIDs Must Be Unique  Integration With Other Features – Seeking & Skipping – Purging Binary Logs – Restoring from Backup
  • 88. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Under the Hood: GTIDs Must Be Unique  Impossible to execute a GTID twice: Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features id1,trx1 id1,trx2A
  • 89. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Under the Hood: GTIDs Must Be Unique  Impossible to execute a GTID twice:  Try to execute a GTID second time → transaction skipped: Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features id1,trx1 id1,trx2A id1,trx1 A Client
  • 90. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Under the Hood: GTIDs Must Be Unique  Impossible to execute a GTID twice:  Try to execute a GTID second time → transaction skipped: Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features id1,trx1 id1,trx2A id1,trx1 A Client set gtid_next=”id1”; trx2
  • 91. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Under the Hood: GTIDs Must Be Unique  Impossible to execute a GTID twice:  Try to execute a GTID second time → transaction skipped: Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features id1,trx1 id1,trx2A id1,trx1 A Client set gtid_next=”id1”; trx2 trx2 not executed!
  • 92. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. I love GTIDs! Is there anything that I should pay attention to?
  • 93. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Program Agenda  Introduction to MySQL Replication  Fail-over Base – Global Transaction Identifiers – Automatic Positioning – Hands-On  Under the Hood: Become a GTID Expert – How Slave Preserves GTIDs – GTIDs Must Be Unique  Integration With Other Features – Seeking & Skipping – Purging Binary Logs – Restoring from Backup
  • 94. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Skipping a Transaction  Common use case: skip a bad transaction  Transaction should be skipped forever  Not enough to move replication position Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features A id1,trx1, id2,trx2, id3,trx3 id1,trx1 B Bad transaction
  • 95. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Skipping a Transaction  Common use case: skip a bad transaction  Transaction should be skipped forever  Not enough to move replication position Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features A id1,trx1, id2,trx2, id3,trx3 id1,trx1, id3,trx3B Bad transaction
  • 96. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Skipping a Transaction  Common use case: skip a bad transaction  Transaction should be skipped forever  Not enough to move replication position Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features A id1,trx1, id2,trx2, id3,trx3 id1,trx1, id3,trx3B Bad transaction trx2 comes back after reconnect id2, trx2, … id1,id3
  • 97. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Skipping a Transaction  Common use case: skip a bad transaction  Transaction should be skipped forever  Not enough to move replication position Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features A id1,trx1, id2,trx2, id3,trx3 id1,trx1, id3,trx3B Bad transaction trx2 comes back after reconnect So we don't allow seeking! id2, trx2, … id1,id3
  • 98. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Skipping a Transaction  Recall: Try to execute a GTID second time → transaction skipped  Interesting feature: Client executes GTID before slave → slave skips transaction  To force slave thread to skip a transaction: slave> SET GTID_NEXT=“GTID of transaction to skip”; slave> COMMIT;  Transaction is skipped forever Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features
  • 99. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Skipping a Transaction  How to skip a transaction forever Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features A id1,trx1, id2,trx2, id3,trx3 id1,trx1 B Bad transaction set gtid_next=”id2”; commit;Client
  • 100. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Skipping a Transaction  How to skip a transaction forever Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features A id1,trx1, id2,trx2, id3,trx3 id1,trx1, id2, –B Bad transaction set gtid_next=”id2”; commit;Client
  • 101. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Skipping a Transaction  How to skip a transaction forever Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features A id1,trx1, id2,trx2, id3,trx3 id1,trx1, id2, – , id3,trx3 B Bad transaction set gtid_next=”id2”; commit;Client
  • 102. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Skipping a Transaction  How to skip a transaction forever Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features A id1,trx1, id2,trx2, id3,trx3 id1,trx1, id2, – , id3,trx3 B Bad transaction set gtid_next=”id2”; commit;Client id4, trx4, … id1,id2,id3
  • 103. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Filters  --replicate-ignore-table etc  Transactions should be filtered-out forever  Slave commits empty transaction Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features A (crashed)(master) A id1,trx1, id2,trx2, id3,trx3 (slave) id1,trx1, id2,- id3,trx3 B Filters out trx2 Logs id2 with an empty transaction
  • 104. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Skipping Stuff: Summary & Recipe  To seek forward from position A to B: – Commit empty transaction for each GTID between A and B  To seek backward: – Not possible, does not make sense  To skip N transactions: – Get the GTIDs, commit empty transaction for each GTID  Replication filters automatically commit empty transactions Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features
  • 105. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. So the rule is easy: skip = empty transaction
  • 106. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Program Agenda  Introduction to MySQL Replication  Fail-over Base – Global Transaction Identifiers – Automatic Positioning – Hands-On  Under the Hood: Become a GTID Expert – How Slave Preserves GTIDs – GTIDs Must Be Unique  Integration With Other Features – Seeking & Skipping – Purging Binary Logs – Restoring from Backup
  • 107. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Purging Binary Logs  Binary logs are rotated and can be purged Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features A id4,trx4, id5,trx5, id6,trx6 master-bin.02 master-bin.03 id7,trx7, id8,trx8 master-bin.01 id1,trx1, id2,trx2, id3,trx3
  • 108. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Purging Binary Logs  Binary logs are rotated and can be purged Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features A id4,trx4, id5,trx5, id6,trx6 master-bin.02 master-bin.03 id7,trx7, id8,trx8 Client PURGE BINARY LOGS TO 'master­bin.02' master-bin.01 id1,trx1, id2,trx2, id3,trx3
  • 109. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Purging Binary Logs  Binary logs are rotated and can be purged Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features A id4,trx4, id5,trx5, id6,trx6 master-bin.02 master-bin.03 id7,trx7, id8,trx8 XPurged! Client PURGE BINARY LOGS TO 'master­bin.02' master-bin.01
  • 110. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Purging Binary Logs  On Master: cannot send purged GTIDs to slave – @@gtid_purged = all purged GTIDs Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features
  • 111. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Purging Binary Logs  On Master: cannot send purged GTIDs to slave – @@gtid_purged = all purged GTIDs Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features B Purged!A id4,trx4, id5,trx5, id6,trx6 master-bin.02 XPurged! id1,trx1, id2,trx2 slave-bin.01 gtid_purged=id1,id2,id3 gtid_executed=id1,id2 master-bin.01
  • 112. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Purging Binary Logs  On Master: cannot send purged GTIDs to slave – @@gtid_purged = all purged GTIDs Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features B Purged!A id4,trx4, id5,trx5, id6,trx6 master-bin.02 XPurged! id1,trx1, id2,trx2 slave-bin.01 Error! id1,id2 Nothing sensible to do: B needs trx3 which is purged gtid_purged=id1,id2,id3 gtid_executed=id1,id2 master-bin.01
  • 113. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Purging Binary Logs  On Slave: must send all GTIDs (purged or not) to master – @@gtid_executed = all GTIDs (purged or not) Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features A slave-bin.02 XPurged!A master-bin.02 id1,trx1, id2,trx2, id3,trx3 id4,trx4, id5,trx5 id4,trx4 slave-bin.01master-bin.01 id5,trx5 id1…id4 Note: slave must send id1...id3 despite purged or else master would send trx1…trx3
  • 114. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Purging Binary Logs  On Master: cannot send purged GTIDs to slave – @@gtid_purged = all purged GTIDs  On Slave: must send all GTIDs (purged or not) to master – @@gtid_executed = all GTIDs (purged or not) Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features A id4,trx4, id5,trx5, id6,trx6 master-bin.02 master-bin.03 id7,trx7, id8,trx8 XPurged! @@gtid_purged = id1 … id3 @@gtid_executed = id1 … id8 master-bin.01
  • 115. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Purged and Executed are Stored in Binary Log  Binary logs begin with: [ list of all GTIDs in all previous binary logs ] Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features A [id1…id3] id4,trx4, id5,trx5, id6,trx6 master-bin.02 master-bin.03 [id1…id6] id7,trx7, id8,trx8 XPurged! master-bin.01 GTIDs in master-bin.01 + master-bin.02 GTIDs in master-bin.01
  • 116. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Purged and Executed are Stored in Binary Log  Binary logs begin with: [ list of all GTIDs in all previous binary logs ] Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features A [id1…id3] id4,trx4, id5,trx5, id6,trx6 master-bin.02 master-bin.03 [id1…id6] id7,trx7, id8,trx8 XPurged! @@gtid_executed = head of newest log + GTIDs in newest logid1…id3 id1…id6 + id7 + id8 master-bin.01 @@gtid_purged = head of oldest log
  • 117. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Purging Binary Logs: Summary  Works automatically, behind the scenes  Purged GTIDs are stored in @@gtid_purged  @@gtid_purged is a subset of @@gtid_executed Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features
  • 118. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Automatic is the word! Does purging have anything to do with backups?
  • 119. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Program Agenda  Introduction to MySQL Replication  Fail-over Base – Global Transaction Identifiers – Automatic Positioning – Hands-On  Under the Hood: Become a GTID Expert – How Slave Preserves GTIDs – GTIDs Must Be Unique  Integration With Other Features – Seeking & Skipping – Purging Binary Logs – Restoring from Backup
  • 120. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Provision a Slave From a Backup Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features id1,create, id2,insertA Table t 1 B
  • 121. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Provision a Slave From a Backup Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features id1,create, id2,insertA Table t 1 B Backup Table t 1 Backup
  • 122. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Provision a Slave From a Backup Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features id1,create, id2,insertA Table t 1 B Table t 1Backup Table t 1 Backup Restore
  • 123. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Provision a Slave From a Backup Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features id1,create, id2,insertA Table t 1 B Table t 1Backup Table t 1 Backup Restore Binary log not restored!
  • 124. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Provision a Slave From a Backup Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features id1,create, id2,insertA Table t 1 B Table t 1Backup Table t 1 Backup Restore Binary log not restored! Set gtid_executed and gtid_purged to id1,id2 (or else A would send id1,id2 when B connects)
  • 125. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Provision a Slave From a Backup Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features id1,create, id2,insertA Table t 1 B Table t 1Backup Table t 1 Backup Restore Binary log not restored! Set gtid_executed and gtid_purged to id1,id2 (or else A would send id1,id2 when B connects) XPurged! set @@gtid_purged = “id1,id2”
  • 126. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Provision a Slave From a Backup  @@gtid_executed must agree with database state – @@gtid_executed after restore = @@gtid_executed at backup  @@gtid_purged = GTIDs in @@gtid_executed that are not in log – Binary logs not (normally) restored – @@gtid_purged after restore = @@gtid_executed after restore  @@gtid_purged is settable: – SET @@GLOBAL.GTID_PURGED = "GTID_EXECUTED at backup" – Also sets @@GTID_EXECUTED to the same value Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features
  • 127. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Restore a Backup  Backup tools must store @@gtid_executed  Restore tools must execute: – SET @@GLOBAL.GTID_PURGED =     "GTID_EXECUTED at backup" Introduction to MySQL Replication  Fail-over Base  Under the Hood  Integration With Other Features
  • 128. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. You had me at “GTID” What shall I do now?
  • 129. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. Next Steps  More on GTIDs – svenmysql.blogspot.co.uk  More on MySQL 5.6 Replication – http://dev.mysql.com/tech-resources/articles/ mysql-5.6-replication.html  Try Out MySQL 5.6 – http://dev.mysql.com/downloads/mysql/