Un-split brain MySQL
•
2 likes•731 views
Is there a way to amend a MySQL split brain scenario? When two servers are diverged from each other, is it possible to identify and undo the conflicting changes? We introduce gh-mysql-rewind, which combines multiple technologies to achieve auto-resolution of data divergence. This presentation explains how gh-mysql-rewind works, and how it is being tested in production to validate its operation.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Un-split brain MySQL
Similar to Un-split brain MySQL (20)
More from Shlomi Noach
Recently uploaded
Recently uploaded (20)
Un-split brain MySQL
- 1. MySQL Un-split brain aka Move Back in Time viagh-mysql-rewind ShlomiNoach GitHub FOSDEM2019
- 5. ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! Failover & split brain
- 8. gh-mysql-rewind Rewindadirtyserverbackintimeandconnectitasahealthy replicainareplicationtopology.
- 9. Iteratively restore hosts into replication topology ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !
- 12. GTID Eachserverkeepstrackof: • gtid_executed: alltransactionseverexecuted: 00020192-1111-1111-1111-111111111111:1-130541 • gtid_purged: ofwhich,somemayhavebeenpurged: 00020192-1111-1111-1111-111111111111:1-120008
- 13. ! ! ! Binarylogs ! ! ! Binarylogs Split brain: GTID contents ! ! ! ! ! ! Binarylogs
- 14. 00020192-1111-1111-1111-111111111111:1-5042 Split brain: gtid_executed gtid_executedondemotedmaster:
- 16. 00020192-1111-1111-1111-111111111111:1-5042 - 00020192-1111-1111-1111-111111111111:5001-5042 00020192-1111-1111-1111-111111111111:1-5000, 00020193-2222-2222-2222-222222222222:1-200 Identifying bad GTID transactions gtid_executed(demoted)-gtid_executed (promoted)
- 17. Row based replication Withbinlog_row_image=FULL: EachINSERT,DELETE,UPDATE isexpressedinthebinarylog asthecompleterowimagebefore/afterchange.
- 18. Row based replication BINLOG ' L9MtXBMBAAAANgAAAJAWAAAAAG4AAAAAAAEABG1ldGEACWhlYXJ0YmVhdAACAxEBBgCk9AE9 L9MtXB8BAAAAPAAAAMwWAAAAAG4AAAAAAAEAAgAC///8AQAAAFwt0y4BQYr8AQAAAFwt0y8BWKiT IZN+ '/*!*/; ### UPDATE `meta`.`heartbeat` ### WHERE ### @1=1 /* INT meta=0 nullable=0 is_null=0 */ ### @2=1546507054.082314 /* TIMESTAMP(6) meta=6 nullable=0 is_null=0 */ ### SET ### @1=1 /* INT meta=0 nullable=0 is_null=0 */ ### @2=1546507055.088232 /* TIMESTAMP(6) meta=6 nullable=0 is_null=0 */ # at 5836 #190103 11:17:35 server id 1 end_log_pos 5867 CRC32 0x2cf60376 Xid = 114 COMMIT/*!*/;
- 19. MariaDB, flashback DevelopedbyAlibaba ContributedtoMySQLandMariaDB ImplementedinMariaDB’smysqlbinlog: • mysqlbinlog --flashback
- 20. flashback example, pseudo code insert(1, 'a') insert(2, 'b') insert(3, 'c') update(2, 'b')->(2, 'second') update(3, 'c')->(3, 'third') insert(4, 'd') delete(1, 'a') insert(1, 'a') delete(4, 'd') update(3, 'third')->(3, 'c') update(2, 'second')->(2, 'b') delete(3, 'c') delete(2, 'b') delete(1, 'a')
- 22. Time axis ! !
- 23. Rewind to split point ! !
- 24. Rewind beyond split point ! !
- 25. ! ! ! Binarylogs Finding bad transactions in binlogs mysql-bin.0000620 mysql-bin.0000621 mysql-bin.0000622 mysql-bin.0000623
- 26. Finding bad transactions in binlogs # at 4 #190103 11:17:14 server id 1 end_log_pos 123 CRC32 0x60730f6b Start: binlog v 4, server v 5.7.17-log created 190103 11:17:14 at startup ROLLBACK/*!*/; BINLOG ' GtMtXA8BAAAAdwAAAHsAAAAAAAQANS43LjE3LWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAa0y1cEzgNAAgAEgAEBAQEEgAAXwAEGggAAAAICAgCAAAACgoKKioAEjQA AWsPc2A= '/*!*/; # at 123 #190103 11:17:14 server id 1 end_log_pos 194 CRC32 0x48f18c88 Previous-GTIDs # 00020192-1111-1111-1111-111111111111:1-59023 # at 194
- 27. gh-mysql-rewind: preparation • IdentifiesthebadGTIDtransactions. • Identifieswhichbinarylogscontainthosetransactions. • Generatesflashbackper-binlog
- 28. gh-mysql-rewind: inject GTID MariaDB/flashbackdonotspeakMySQL-GTID. WeusesomegoodoldawkhackingtoinjectdummyGTID entries. mysqlbinlog —flashback … | awk 'BEGIN {"uuidgen -r" |& getline u} /^BEGIN/ {c += 1 ; print "SET @@SESSION.GTID_NEXT= x27" u ":" c "x27/*!*/;"} {print}' | sed -e s/', @@session.check_constraint_checks=1//g’
- 30. gh-mysql-rewind: back in time Havewejustmadethingsevenworse? Wherearewe? gtid_executedisamess! Whathappensifwereconnectthe serverintothetopology? ! ! ! Binarylogs
- 31. 00020192-1111-1111-1111-111111111111:1-5042 - 00020192-1111-1111-1111-111111111111:1-4805 00020192-1111-1111-1111-111111111111:4806-5042 Computing target GTID RESET MASTER SET GLOBAL gtid_purged := <computation>
- 32. Join topology as healthy replica CHANGE MASTER TO MASTER_HOST=<healthy host>, MASTER_AUTO_POSITION=1 ⚡ ! ! ! ! ! !
- 37. ! ! ! ! ! ! STOP SLAVE CHANGE MASTER TO MASTER_AUTO_POSITION=1 Rotatesrelaylogs FLUSH BINARY LOGS Opensanewbinarylog Wepredictthatgh-mysql-rewindwilldropusbackinto currentposition. Testing, a dedicated replica
- 38. START SLAVE IO_THREAD Sleep30sec STOP SLAVE Weaggregate30secofproductiontrafficinarelaylog. Testing: grab production data ! Relaylog
- 39. Weparsetherelaylogandlistaffectedtables. Weuseall“small”tables, Andasingle“large”table. Wechecksumallthesetables. Testing: analyze production data ! Relaylog
- 40. ! START SLAVE SQL_THREAD; SELECT MASTER_POS_WAIT(<relaylog end coordinates>) Weapplytheentirecontentoftherelaylog Testing: apply production dataRelaylog Binarylog
- 41. ! Foreachtable,weissue: DELETE FROM <t> ORDER BY id DESC LIMIT 10 DELETE FROM <t> LIMIT 10 Andtomakethingsworse: START SLAVE Sleep30sec STOP SLAVE Testing: contaminateRelaylog Binarylog
- 48. In development at MySQL Query&manipulateGTIDovertheclientprotocol: