Demystifying MySQL Replication Crash Safety

2
Introducing MessageBird
MessageBird is a cloud communications platform
founded in Amsterdam 2011.
Examples of our messaging and voice SaaS:
SMS in and out, call in (IVR) and out (alert), SIP,
WhatsApp, Facebook, Telegram, Twitter, WeChat, …
Omni-Channel Conversation
Details at www.messagebird.com
225+ Direct-to-Carrier Agreements
With operators from around the world
15,000+ Customers
In over 60+ countries
180+ Employees
Engineering office in Amsterdam
Sales and support offices worldwide
We are expanding :
{Software, Front-End, Infrastructure, Data, Security, Telecom, QA} Engineers
{Team, Tech, Product} Leads, Product Owners, Customer Support
{Commercial, Connectivity, Partnership} Managers
www.messagebird.com/careers

3
Summary
(Demystifying MySQL Replication Crash Safety – HLMoscow2018)
• Helicopter view of – and then Zoom in – Replication and Crash Safety
• MySQL 5.6 solution (and its problems)
• Complexifying things with GTIDs and Multi-Threaded Slave (MTS)
• Impacts of reducing / compromising durability
(sync_binlog != 1 and trx_commit != 1)
• Overview of related subjects: Semi-Sync, MariaDB & Pseudo-GTIDs
• Closing, links, bugs and questions

Overview of MySQL Replication
One master with one or more slaves:
• The master records transactions in a journal (binary logs); each slave:
• Downloads the journal and saves it locally in the relay logs (IO thread)
• Executes the relay logs on its local database (SQL thread)
• Could also produce binary logs to be a master (log-slave-updates – lsu)

Replication Crash Safety
What do I mean by Replication Crash Safety ?
• When a slave crashes, it is able to resume replication after recovery
(OK if rewinds its state after recovery, as long as it is eventually consistent)
• When a master crashes, slaves are able to resume replicating from it
• All above without sacrificing data consistency
• In other words: ACID is not compromised by a slave or a master crash
(Discussion limited to transactional SE: InnoDB, TokuDB, MyRocks; obviously not MyISAM)
Intermediate masters (IM) qualify both as master and slave
Slaves are potential master (and IM) in some failover strategy
(Proving replication crash un-safety is easy, proving safety is hard) 5

6
State of the Dolphin and of the Sea Lion
State of the Dolphin in Replication Crash Safety:
• MySQL 5.5 is not crash safe
• MySQL 5.6 can be made crash safe (it is not by default)
• MySQL 5.7 is mostly the same as 5.6
(with complexity added by Logical Lock parallel replication)
• MySQL 8.0 is crash safe by default
(but it can be made unsafe by “tuning” the configuration)
Quick state of the Sea Lion:
• MariaDB 5.5 is not replication crash safe
• MariaDB 10.x can be made crash safe

7
Zoom in the details [1 of 3]
More details about replication:
• The IO Thread stores its state in master info (also configuration stored there)
• The SQL Thread in relay log info
slave1 [localhost] {msandbox} ((none)) > show slave statusG
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
[...]
Master_Log_File: mysql-bin.000001 <-------+-- master info (persisted state)
Read_Master_Log_Pos: 25489 <-------+
Relay_Log_File: mysql-relay.000002 <--+
Relay_Log_Pos: 10788 <--+
Relay_Master_Log_File: mysql-bin.000001 <--+-- relay log info (persisted state)
[...] |
Exec_Master_Log_Pos: 10575 <--+
[...]
1 row in set (0.00 sec)

More parameters: sync_master_info, sync_relay_log and sync_relay_log_info
In MySQL 5.5, master info and relay log info are files:
• No atomicity of “making progress” and “state tracking” for IO & SQL Threads
• Consistency of actual vs registered state is compromised after a crash
Ø This is why replication is not crash-safe in MySQL 5.5 8

9
Even more parameters:
• sync_binlog (and innodb_flush_log_at_trx_commit – trx_commit):
• Binlogs are synchronised to disk after every N writes/transactions
(default 0 in My|SQL 5.5 and 5.6; and in 5.7 and 8.0 it is 1 which is full ACID)
• trx_commit = 1: logs written and flushed each trx (full ACID and default)
= 0: written and flushed once per second (not crash safe)
= 2: written after each trx and flushed once per second
(mysqld crash safe, not OS crash safe)

MySQL 5.6 solution [1 of 4]
Reminder: problems making MySQL 5.5 Replication Crash Un-Safe:
• The position of the SQL Thread cannot be trusted
• The position of the IO Thread cannot be trusted
• The content of the Relay Logs cannot be trusted
10

The MySQL 5.6 solution
• Atomicity for SQL Thread: relay-log-info-repository = TABLE (default = FILE)
• Useless for crash safety: a parameter to store master info in a table:
• master-info-repository = TABLE (default = FILE)
• Providing a way to “fix” the relay logs: relay-log-recovery = 1 (default = 0)

12
More details about Relay Log Recovery:
• relay-log-recovery is only used on mysqld startup (dynamic would be useless)
• If relay-log-recovery = 0, nothing special done (and a new relay log is created)
• If relay-log-recovery = 1:
• The position of the IO Thread is set to the position of the SQL Thread
• The position of the SQL Thread is set to the newly created relay log
• If relay-log-purge = 1: the old relay logs will be deleted on SQL Thread startup
(relay-log-recovery does not delete anything: easy to test with skip-slave-start)
Ø Said otherwise, the previous relay logs are skipped !
(those relay logs are considered improper for SQL Thread consumption)
• This will happen even if MySQL (or the IO Thread) did not crash
OK for 1st implementation but a waste of perfectly good relay logs

13
In MySQL 5.7:
• No change of defaults (for replication crash safety)
• Relay log recovery still simplistic K
In MySQL 8.0:
• Still simplistic relay log recovery L
• New defaults:
• relay-log-info-repository = TABLE J
• relay-log-recovery = 1 J
• master-info-repository = TABLE (not sure this is very useful)
Bug#74323: Avoid overloading the master NIC on relay-log-recovery of a lagging slave
Bug#74321: Execute relay-log-recovery only when needed

Adding complexity with GTIDs [1 of 2]
Not only MySQL 5.6 introduces replication crash safety,
it also introduced Global Transaction IDs (GTIDs)
• This tags every transaction with an ID when writing to the binlogs
• The GTID state of the master and slaves are tracked in the binlogs
Ø IO and SQL Thread states are now partially in the binlogs (and relay logs)
• Optionally, slaves can use GTID to replicate (instead of file+position)
• This allows easier repointing of slaves to a new master (including fail over)
• This heavily relies on precise tracking of GTID states on master and slaves
Ø As this tracking is in the binlogs, this is impeded when sync_binlog != 1
Bug#70659: Make crash safe slave with gtid + less durable settings
14

Adding complexity with GTIDs [2 of 2]
To make replication crash safe with GTIDs in MySQL 5.6:
• relay-log-info-repository = TABLE (default = FILE)
• relay-log-recovery = 1 (default = 0) – (Bug#92093)
• sync_binlog = 1 (default = 0)
• In 5.7, the default is sync_binlog = 1 J (two other unchanged K)
• In 8.0, all the defaults are good for crash safe replication with GTID J J
• MySQL 5.7 adds a table for storing the GTID state of slaves:
• Allows GTIDS slaves without log-slave-updates (lsu)
• With lsu, this table (mysql.gtid_executed) is not updated after each trx
Ø Missed opportunity for OS crash safety with sync_binlog != 1 L L L
Bug#92109: Make GTID replication crash safe with less durable setting 15

16
Master Replication Crash Safety [1 of 5]
Relaxing durability of the binlogs implies losing GTID state (after an OS crash)
• What about the consequence on the master ? With and without GTID ?
• If sync_binlog != 1 on the master, an OS crash will lose binlogs
• With sync_binlog != 1, usually trx_commit != 1 (normally 2, but can be 0)
• trx_commit = 2 preserves data on mysqld crashes, 0 does not (à 2 is better)
Ø InnoDB will also lose transactions on an OS crash
Ø After an OS crash, InnoDB will be out-of-sync with the binlogs
Ø And we cannot trust the binlogs on such master (trx gap or ghost trx)
The failure mode will be different depending on the configuration

17
With file+position
• IO Thread in vanished binlogs
• So slaves executed phantom trx
(ghost in binlogs, maybe not in InnoDB)
• When the master is restarted:
• It records trx in new binlog file
• Most slaves are broken, and they might be out-of-sync with each-others
• Some lagging slave might skip vanished binlogs

18
With file+position
• IO Thread in vanished binlogs
• So slaves executed phantom trx
(ghost in binlogs, maybe not in InnoDB)
• When the master is restarted:
• It records trx in new binlog file
• Most slaves are broken, and they might be out-of-sync with each-others
• Some lagging slave might skip vanished binlogs
Ø Broken slaves have more data than the master (à data drift)
Ø And different data drift on “lucky” lagging slaves that might not break

19
With GTID enabled
• Slave also executed ghost
trx vanished from binlogs
• But those are in their GTID state
• A recovered master reuses
GTIDs of the vanished trx
• Slaves magically reconnect to the master (MASTER_AUTO_POSITION = 1)
1. If master has not reused all ghost GTIDs, then the slave breaks
2. If it has, then the slave skips the new transactions à more data drift
(in illustration, the slave will skip new 50 to 58 as it has the old one)

20
With GTID enabled but MASTER_AUTO_POSITION = 0
• Left as an exercise to the audience…
On the consequences of sync_binlog != 1 (part #1)
https://jfg-mysql.blogspot.com/2018/10/consequences-sync-binlog-neq-1-part-1.html
(more posts to be published in the series)

Summary of running with sync_binlog != 1:
• The binlogs – of the master or slave – cannot be trusted after an OS crash
• On a master, having mysqld normally restarts after such a crash leads to data drift
Ø After an OS crash, make sure no slaves reconnect to the recovered master
(OFFLINE_MODE = ON in config file – failing-over to a slave is the way forward)
• On slaves, having mysqld restarts after such a crash leads to truncated binlogs
Ø After an OS crash, consider purging all binlogs on the recovered slave
• Intermediate Masters (IM) are both master and slaves
Ø After an OS crash make sure no slaves reconnect to the recovered IM
Ø And consider purging all binary logs on it
• Remember: GTID state corrupted on slaves after OS crash (Bug#92109) 21

22
Adding complexity with MTS [1 of 4]
Multi-Threaded Slave (MTS) in MySQL 5.6 is doing out-of-order committing
• Same for MySQL 5.7 with DATABASE and LOGICAL_CLOCK types
• LOGICAL_CLOCK also has the slave_preserve_commit_order option
(OFF by default in 5.7 and 8.0 K, with ON requiring log-slave-updates L)
(Bug#75396: Allow slave_preserve_commit_order without log-slave-updates)
Example: transactions A, B, C, D, E on the master
• On a slave, SHOW SLAVE STATUS points to B, so A is committed
• C and E are also committed, B is running and D is pending scheduling
(maybe B and D are in the same schema with DATABASE type)
With out-of-order commit, a file+position in relay log info is not enough
• GTID allows tracking complex position (generating temporary holes on slaves)
• And there is the mysql.slave_worker_info table
(https://dev.mysql.com/worklog/task/?id=5599: for more details)

23
Without GTID, resuming replication after a crash needs filling the gap in trx
• Manual, error-prone, and not always possible before 5.6.31 and 5.7.13 (Bug#77496)
• Now, automated by doing START SLAVE UNTIL SQL_AFTER_MTS_GAPS
• But this needs relay logs, which might have vanished after an OS crash (Bug#81840)

24
• Bug#81840 makes MTS with File+Position OS crash unsafe (safe for mysqld crash)
• Hard to accept workaround: sync_relay_log = 1 (performance killer)
• Full state in mysql.slave_worker_info à recovery possible with a lot of effort
• The good solution would be a better relay log recovery (Bug#93081)

25
With GTID, MTS in MySQL 5.6, 5.7 & 8.0 is replication crash safe:
• But it needs MASTER_AUTO_POSITION = 1 (and relay log recovery Bug#92093)
• And it comes with all the GTID “goodies” (rogue transactions, lsu for 5.6, …)
• Also needs sync_binlog = 1 (if 5.7+, also works without binlogs or lsu off)
• And care with sync_binlog != 1 on the master (need to fail over if OS crash)
(sync_binlog != 1 should not be needed in 95% of cases)
(Group Commit and MTS make this optimisation almost obsolete)
Example: A, B, C, D, E on the master with GTID 10, 11, 12, 13, 14:
• GTID executed on the slave is 1-10:12:14 before a crash
• Replication resumes by fetching 11:13:15… (after relay log recovery)

26
With GTID, MTS in MySQL 5.6, 5.7 & 8.0 is should be crash safe :
• But it needs MASTER_AUTO_POSITION = 1 (and relay log recovery Bug#92093)
• And it comes with all the GTID “goodies” (rogue transactions, lsu for 5.6, …)
• Also needs sync_binlog = 1 (if 5.7+, also works without binlogs or lsu off)
• And care with sync_binlog != 1 on the master (need to fail over if OS crash)
Bug#92882: MTS not replication crash-safe with GTID and all the right parameters
(Only applies to Operating System crashes)
Example: A, B, C, D, E on the master with GTID 10, 11, 12, 13, 14:
• GTID executed on the slave is 1-10:12:14 before Operating System crash
• Relay log recovery tries to “fill the gaps” but fails because relay logs are gone
(This might be a regression from the fix of Bug#77496)
(Easy workaround: stop slave; reset slave; start slave;)

27
Related subjects – Semi-Sync
In this talk, we did not cover master failover explicitly,
when a master crashes in an unrecoverable way, failover needs to happen
When failing-over to a slave, committed transactions can be lost
(Some transactions on the crashed master might not have reached slaves)
à violation of durability (ACID) in the replication topology (distributed system)
Except if lossless semi-sync is used, more details in:
Question about Semi-Synchronous Replication: the Answer with All the Details
https://www.percona.com/community-blog/2018/08/23/question-about-semi-synchronous-
replication-answer-with-all-the-details/

28
Related subjects – MariaDB
MariaDB still stores its master info and relay log info in files
• But it stores GTID state of slaves in the mysql.gtid_slave_pos table
Ø MariaDB is replication crash safe when using GTID slave positioning
Also, it has an interesting feature:
• If using more than one storage engine, a single state table is not optimal
• Having one such table per storage engine could be better
Improving replication with multiple storage engines (MariaDB 10.3)
https://kristiannielsen.livejournal.com/19223.html

29
Related subjects – Pseudo GTIDs
Pseudo-GTIDs:
• A way to get GTID-like features without GTIDs
• They work with any version of MySQL/MariaDB (even 5.5)
• But they assume in-order-commit à does not work with MTS
They can provide slave replication crash safety:
• With log-slave-updates and sync_binlog = 1
• Even on MySQL 5.5 or MariaDB 5.5
https://github.com/github/orchestrator/blob/master/docs/pseudo-gtid.md

Conclusion
• It is complicated and it depends…
• It has many edge cases
• It might still change as bugs are fixed
• And hopefully improvements will be made
• So sorry: there is no short version

Conclusion [2 of 5]
Some parameters never impact/improve Replication Crash Safety:
• master-info-repository, sync_master_info, sync_relay_log_info
Some parameters are always needed for Replication Crash Safety:
• relay-log-info-repository = TABLE
• relay-log-recovery = 1

32
Conclusion [3 of 5]
MySQL 5.6 with GTID (with and without MTS) à crash safe slave if:
• All above with sync_binlog = 1 (not default) and MASTER_AUTO_POSITION = 1
and maybe a “stop slave; reset slave; start slave;” (Bug#92882)
MySQL 5.6 without GTID and with MTS à not always crash safe slaves:
• OK for MySQL crashes as relay logs are not lost
• For OS crashes, losing the relay logs leads to replication breakage (Bug#81840)
• Possible to recover with some voodoo and dark magic (Bug#93081)
For master and slaves, binlogs cannot be trusted if sync_binlog != 1

33
Conclusion [4 of 5]
MySQL 5.7 is mostly the same as 5.6:
• sync_binlog = 1 is the default J
• Will be crash safe with GTID and sync_binlog != 1 when Bug#92109 fixed
• LOGICAL_CLOCK with slave_preserve_commit_order like single-threaded
• Without slave_preserve_commit_order, same as MTS in 5.6
MySQL 8.0 is mostly the same as 5.7 with safer defaults:
• relay-log-info-repository = TABLE J
• relay-log-recovery = 1 J
• But default for slave_preserve_commit_order is still 0 K

34
Conclusion [5 of 5]
Care with MTS as it has many traps
And in all cases:
• Relay log recovery needs to re-download relay logs from the master
• High load in case of lagging (or delayed) slaves L
• Will fail if the binary logs were purged from the master L
• Relay log recovery also fails for MTS and OS crashes (vanished relay logs)
L L L
We need a better Relay Log Recovery !
Bug#74321, Bug#74323, Bug#74324, Bug#81840
Bug#92882, Bug#93081

35
Links [1 of 3]
Crash-Safe MySQL Replication - A Visual Guide
https://hackmongo.com/post/crash-safe-mysql-replication-a-visual-guide/
(diagrams in this talk are inspired by this post)
Jean-François’s blog posts about Replication Crash Safety:
• Better Crash-safe replication for MySQL
https://medium.com/booking-com-infrastructure/better-crash-safe-replication-for-mysql-a336a69b317f
• Replication crash safety with MTS in MySQL 5.6 and 5.7: reality or illusion?
https://jfg-mysql.blogspot.com/2016/01/replication-crash-safety-with-mts.html
• A discussion about sync-master-info and other replication parameters
https://jfg-mysql.blogspot.com/2016/08/discussion-about-sync-master-info-and-replication-parameters.html
• On the consequences of sync_binlog != 1 (part #1)
https://jfg-mysql.blogspot.com/2018/10/consequences-sync-binlog-neq-1-part-1.html

36
Links [2 of 3]
Directly related bugs:
• Bug#70669: Slave can't continue repl. after master's recovery (old – 5.6.14, and fixed – 5.6.17)
• Bug#70659: Make crash safe slave work with gtid + less durable settings
• Bug#74321: Execute relay-log-recovery only when needed
• Bug#74323: Avoid overloading the master NIC on relay-log-recovery of a lagging slave
• Bug#74324: Make keeping relay logs (relay_log_purge = 0) crash safe
• Bug#77496: Replication position lost after crash on MTS configured slave (really fixed ?)
• Bug#81840: Automatic Replication Recovery Does Not Handle Lost Relay Log Events
• Bug#92093: Replication crash safety needs relay_log_recovery even with GTID
• Bug#92109: Please make replication crash safe with GITD and less durable setting (bis)
• Bug#92882: MTS not replication crash-safe with GTID and all the right parameters
• Bug#93081: Please implement a better relay log recovery
Somehow related bugs:
• Bug#75396: Allow slave_preserve_commit_order without log-slave-updates
• Bug#92891: Please make relay_log_space_limit dynamic

Links [3 of 3]
• Question about Semi-Synchronous Replication: the Answer with All the Details
https://www.percona.com/community-blog/2018/08/23/question-about-semi-synchronous-replication-answer-with-all-the-details/
• Improving replication with multiple storage engines (in MariaDB 10.3)
https://kristiannielsen.livejournal.com/19223.html
• Pseudo-GTID and Orchestrator:
https://github.com/github/orchestrator/blob/master/docs/pseudo-gtid.md
https://speakerdeck.com/shlominoach/pseudo-gtid-and-easy-mysql-replication-topology-management
• The Full MySQL and MariaDB Parallel Replication Tutorial
https://www.slideshare.net/JeanFranoisGagn/the-full-mysql-and-mariadb-parallel-replication-tutorial
• Arg: relay_log_space_limit is (still) not dynamic !
https://jfg-mysql.blogspot.com/2018/10/arg-relay-log-space-limit-is-still-not-dynamic.html
• Evaluating MySQL Parallel Replication Part 2: Slave Group Commit
https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-2-slave-group-commit-459026a141d2
• Evaluating MySQL Parallel Replication Part 4: More Benchmarks in Production
https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-4-more-benchmarks-in-production-49ee255043ab

Thanks !
Presented at HighLoad++ 2018 in Moscow
by Jean-François Gagné
Senior Infrastructure Engineer / System and MySQL Expert
jeanfrancois AT messagebird DOT com

Demystifying MySQL Replication Crash Safety

More Related Content

What's hot

Similar to Demystifying MySQL Replication Crash Safety

Recently uploaded

Demystifying MySQL Replication Crash Safety