SlideShare a Scribd company logo
1 of 32
Download to read offline
The consequences
of sync_binlog != 1
by Jean-François Gagné
Senior Infrastructure Engineer / System and MySQL Expert
jeanfrancois AT messagebird DOT com / @jfg956 #FOSDEM #MySQLDevRoom
The full title of the talk should be
The consequences of sync_binlog != 1
and of
innodb_flush_log_at_trx_commit = 2
as one without the other does not make much sense
(I will use trx_commit for short in this talk)
3
Summary
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
• Faster but by how much ?
• Overview of replication and zoom in sync_binlog & trx_commit
• Avoiding sync_binlog != 1
• The consequences of sync_binlog != 1
• Mitigating sync_binlog != 1
• Closing, links and questions
This talk applies mostly to MySQL 5.6 and 5.7 (unless explicitly mentioned)
some content will apply to MariaDB 10.0+ (and will be explicitly mentioned)
Faster but by how much ?
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
Sysbench, single-thread insert benchmark, without secondary index,
vm on Google Cloud Platform (GCP), SSD persistent-disks, MySQL 5.7.26
With sync_binlog = 1 & trx_commit = 1:
• ~200 Transactions Per Second (TPS) on the master
• ~230 TPS on a slave
With sync_binlog = 0 & trx_commit = 2:
• ~3770 TPS on the master (~18x faster)
• ~7050 TPS on a slave (~30x faster)
https://jfg-mysql.blogspot.com/2019/07/master-replication-crash-safety-part-4-benchmarks-of-high-n-low-durability.html
4
Overview of MySQL Replication
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
One master node with one or more slave nodes:
• The master records transactions in a journal (binary logs); each slave:
• Downloads the journal and saves it locally in the relay logs (IO thread)
• Executes the relay logs on its local database (SQL thread)
• Could also produce binary logs to be a master (log-slave-updates)
Zoom in sync_binlog & trx_commit [1 of 6]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
• sync_binlog = N: the binlogs are flushed to disk every N trx group
• trx_commit = {1,2,0}
• 1: the InnoDB Redo Log (IRL) is flushed to disk after each transaction
• 2: the IRL is written to after each trx (OS RAM buffer) but flushed every second
• 0: the IRL is written to and flushed every second (not covered in this talk)
Zoom in sync_binlog & trx_commit [2 of 6]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
Flushing to disk is not fast !
• Local disks [1]:
• Desktop and Enterprise grade magnetic SATA: between 18 and 25 ms
• Consumer grade SSD SATA and NVMe: between 0.5 ms and 10 ms
• High-end enterprise-grade NVMe: ~0.15 ms
• Dell Perc with BBU: ~0.04 ms
• Network disks (FC or iSCSI à network round-trip): between 0.5 ms and 1 ms
• Cloud environments:
• GCP Persistent Disks (network): ~0.6 ms [2]
• Amazon Web Services Local SSD: ~0.05 ms [3]
(careful about local SSDs in the cloud as they are not always persistent between reboots)
[1]: https://www.percona.com/blog/2018/02/08/fsync-performance-storage-devices/
[2]: https://jfg-mysql.blogspot.com/2019/07/master-replication-crash-safety-part-4-benchmarks-under-the-hood.html
[3]: https://jfg-mysql.blogspot.com/2019/07/master-replication-crash-safety-part-5a-faster-wo-reducing-durability-hardware.html
Zoom in sync_binlog & trx_commit [3 of 6]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
What does this mean ?
• sync_binlog = 1 & trx_commit = 1: trx are on disk after COMMIT
• Everything is fully durable (ACID), nothing lost in the case of an OS crash
• = 0 & = 2: trx are in an OS RAM buffer after COMMIT, but not on disk
• No data lost after a mysqld crash (data from OS RAM buffer is not lost)
• But things are lost in the case of an OS crash
• And after an OS crash, InnoDB and the binary logs are probably out-of-sync
If those transactions are run on the master: …, D, E, F, G, H, I, J, K, L, …
• On an OS crash, binlog could be flushed up to E, and InnoDB up to K
• So after recovery, InnoDB will have data up to K (L and after is lost)
and the transactions F and after are lost from the binary logs
(Note: the scenario where InnoDB looses less than the binlog is more likely
as the Redo Logs are flushed every second, but the opposite might also happen)
8
Zoom in sync_binlog & trx_commit [4 of 6]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
Another thing about sync_binlog != 1:
• Binary logs are flushed to disk at each binlog rotation
and putting up to 1 GB on disk this might take time…
• The associated stalls have been described by Vadim Tkachenko in [1]
[1]: https://www.percona.com/blog/2018/05/04/how-binary-logs-and-filesystems-affect-mysql-performance/
And with MySQL 5.6+ and MariaDB 5.3+, the Binary Log Group Commit
optimization allows to persist many transactions to disk in a single flush
(InnoDB already had Group Commit for some time):
• Running trx in parallel on the master increases TPS with sync_binlog = 1
• MariaDB 10.0+ Slave Group Commit [2] allows the same on slaves
• And so does Parallel Replication in MySQL 5.6+ and MariaDB 10.0+
[2]: https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-2-slave-group-commit-459026a141d2
9
Zoom in sync_binlog & trx_commit [5 of 6]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
Defaults in different versions:
(The binlog_order_commits and innodb_support_xa parameters are also this discussion, but
their defaults are decent, so they are only briefly mention here.) 10
MySQL 5.5 5.6 5.7 8.0 MariaDB 10.0 10.1 10.2 10.3 10.4
sync_binlog
0
K1
0
K1
1
J K2
1
J K2
0
K3
0
K3
0
L3
0
L3
0
L3
trx_commit 1 1 1 1 1 1 1 1 1
binlog_order_commits N/A4 ON ON ON N/A4 N/A4 N/A4 N/A4 N/A4
innodb_support_xa ON ON ON N/A5 ON ON ON N/A5 N/A5
Zoom in sync_binlog & trx_commit [6 of 6]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
Notes from previous page:
1) MySQL introduced binary log group commit (BLGC) in 5.6, so up to, and arguably including 5.6,
sync_binlog needed to be to 0 for performance reasons, but this was unsafe.
2) As MySQL had GLGC since 5.6, it was possible to make sync_binlog = 1 the default without
penalizing performance on the master (J), but this change made replication the bottleneck and
MySQL does not have slave group commit (SGC) nor does it enable parallel replication (// rpl) by
default (K). Still good to be safe by default (J).
3) MariaDB introduced BLGC in 5.3, so arguably, they should have made sync_binlog = 1 the
default in 10.0, but as this might have impacted replication performance (// rpl just had been
introduced in 10.0 and SGC explicitly only in 10.1) this is only a K up to 10.1. But from 10.2, this
is a L as databases should be safe by default (sync_binlog = 1) and fast (// rpl or SGC enabled).
4) The binlog_order_commits was introduced in MySQL 5.6, probably as part of the binary group
commit implementation, and it never was introduced in MariaDB.
5) The innodb_support_xa parameter was deprecated in MySQL 5.7 and MariaDB 10.2 and
removed in 8.0 and 10.3.
11
Avoiding sync_binlog != 1 [1 of 5]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
Ideally, we would always run with sync_binlog = 1 (and trx_commit = 1)
When reaching the transaction throughput limit of sync_binlog = 1,
because of disk flush latencies becoming the bottleneck,
and before setting sync_binlog to 0, we can:
1. Get faster disks [1] (reducing the latency of a flush)
2. Run transactions in parallel on the master
(persisting many trx with a single flush thanks to Binary Log Group Commit)
3. Use parallel replication (hoping to persist many trx with a single flush)
(including if running MariaDB 10.0+, use Slave Group Commit)
We will quickly explore #2 and 3 in the next slides (#1 is described in [1])
[1]: https://jfg-mysql.blogspot.com/2019/07/master-replication-crash-safety-part-5a-faster-wo-reducing-durability-hardware.html
12
Avoiding sync_binlog != 1 [2 of 5]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
Running transactions in parallel on the master (Binary Log Group Commit):
• Very nice increase in TPS with sync_binlog = 1, but needs a lot of threads
• And sync_binlog = 0 can do much more TPS with less threads
Note: benchmarks done in GCP, so not as reliable as dedicated server.
(sysbench insert bench. with SSD persistent disks without secondary index) 13
14
Avoiding sync_binlog != 1 [3 of 5]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
MariaDB 10.0+ Slave Group Commit [1] (named minimal in 10.1+):
• Nice increase in TPS (without allocating too much extra CPU resources)
• Almost as fast as single-threaded sync_binlog = 0 !
• But not as good as multi-threaded on the master
(obviously as transactions are serialized and not run in parallel)
[1]: https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-2-slave-group-commit-459026a141d2
Avoiding sync_binlog != 1 [4 of 5]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
Parallel Replication (this is a complex subject, not covered in detail here, see [1]):
• Nice increase in TPS with sync_binlog = 1, but needs a lot of threads
• And sync_binlog = 0 can do more TPS with less threads
• Sadly, not as good as the master in this case
(Note that the insert benchmark is probably the worse for Parallel Repl.)
[1]: https://www.slideshare.net/JeanFranoisGagn/the-full-mysql-and-mariadb-parallel-replication-tutorial
15
Avoiding sync_binlog != 1 [5 of 5]
17
Consequences of sync_binlog != 1 [1 of 6]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
Trx are in an OS RAM buffer after COMMIT, but they are not on disk:
• No data lost after a mysqld crash (data from OS RAM buffer is not lost)
• But loosing data in case of an OS crash
• And after an OS crash, InnoDB and the binary logs are not in sync
If those transactions are run on the master: …, D, E, F, G, H, I, J, K, L, …
• On an OS crash, binlog could be flushed up to E, and InnoDB up to K
• So after recovery, InnoDB will have data up to K (L and after is lost)
and the transactions F and after are lost from the binary logs
(Note that the scenario where InnoDB looses less than the binlog is more likely as
the Redo Logs are flushed every seconds, but the opposite of above might also happen.)
Consequences of sync_binlog != 1 [2 of 6]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
Master OS crash
and rpl. with file+position:
• IO Thread in vanished binlogs
• So slaves executed phantom trx
(ghost in binlogs, maybe not in InnoDB)
• When the master is restarted:
• It records trx in new binlog file
• Most slaves are broken (pointing in vanished binlog),
and they might be out-of-sync with each-others
• Some lagging slave might skip vanished binlogs
Consequences of sync_binlog != 1 [2 of 6]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
Master OS crash
and rpl. with file+position:
• IO Thread in vanished binlogs
• So slaves executed phantom trx
(ghost in binlogs, maybe not in InnoDB)
• When the master is restarted:
• It records trx in new binlog file
• Most slaves are broken (pointing in vanished binlog),
and they might be out-of-sync with each-others
• Some lagging slave might skip vanished binlogs
Ø Broken slaves have more data than the master (à data drift)
Ø With different data drift on “lucky” lagging slaves that might not break
20
Consequences of sync_binlog != 1 [3 of 6]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
Master OS crash
and replication with GTID:
• Slave also executed ghost
trx vanished from binlogs
• But those are in their GTID state
• A recovered master reuses
GTIDs of the vanished trx
• Slaves magically reconnect to the master (MASTER_AUTO_POSITION = 1)
1. If master has not reused all ghost GTIDs, then the slave breaks
2. If it has, then the slave skips the new transactions à more data drift
(in illustration, the slave will skip new 50 to 58 as it has the old one)
Consequences of sync_binlog != 1 [4 of 6]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
Slave OS crash and replication with file+position:
• If using replication crash-safe configuration [1] à no problem !
(well, there is still the corrupted binlog problem, but this is covered in master crashes)
• With crash-safe replication, the position is stored in an InnoDB table
• So replication will resume at the exact place where InnoDB rewinded
• Obviously, if this slave has binlog enabled, they will be corrupted
(same problems as for a master crash)
[1]: https://medium.com/booking-com-infrastructure/better-crash-safe-replication-for-mysql-a336a69b317f
Slave OS crash and replication with GTID:
• If binlogs disabled on the slave (only possible with 5.7+) à no problem !
• With binlogs disabled, the GTID state is stored in an InnoDB table
• So replication will resume at the exact place where InnoDB rewinded 21
22
Consequences of sync_binlog != 1 [5 of 6]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
Slave OS crash and replication with GTID:
• If binary logs are enabled à problems as the GTID state is lost !
• With binlogs enabled, the GTID state is stored in the binary logs
• As the binlogs rewinded, replication will resume at this past position
• And as InnoDB is not in sync with the binlogs, replication will probably break
• Disappointing: the GTID table is not updated after each trx (only on binlog rotation)
Bug#92109: Please make replication crash-safe with GITD and less durable setting (bis)
(Note: this is not a problem with MariaDB as its GTID state is stored in a table.)
23
Consequences of sync_binlog != 1 [6 of 6]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
It looks like MySQL 8.0.17 has the GTID position in InnoDB Logs [1]
• But unclear if this solves the problem
(no communication from Oracle on this subject including no bugs closed)
• And GTID position in InnoDB does not work for MyRocks or other Storage Engines
Ø Updating the GTID table after each trx would be better IMHO
[1]: WL#9211: InnoDB: Clone Replication Coordinates: https://dev.mysql.com/worklog/task/?id=9211
Mitigating sync_binlog != 1 [1 of 2]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
When running with sync_binlog != 1:
• The binlogs – of the master or slave – cannot be trusted after an OS crash
• On a master, mysqld restarting after OS crash leads to broken slaves and data drift
Ø After an OS crash, make sure no slaves reconnect to the recovered master
(OFFLINE_MODE = ON in conf file - failing-over to a slave is the way forward)
• On slaves, having mysqld restarts after such a crash leads to truncated binlogs
Ø After an OS crash, purge all binlogs on the recovered slave (RESET MASTER)
(A little more complicated with GTID, see next slide)
• Intermediate Masters (IM) are both master and slaves
Ø After an OS crash of an intermediate master, make sure no slaves reconnect to it
Ø And purge all its binary logs 24
Mitigating sync_binlog != 1 [2 of 2]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
When running a GTID slave with sync_binlog != 1:
• The GTID state is corrupted after an OS crash à you should restore a backup
• But file+pos. in a table can be trusted, so there is hope to avoid restoring a backup
if single-threaded replication or no gap in relay log execution
(always the case if slave_preserve_commit_order = ON)
This is how to salvage a GTID slave after an OS crash (this is voodoo):
• Make sure mysql starts with replication disabled (skip-slave-start)
• Note the GTID position (gtid_executed) and wipe the binlogs (RESET MASTER)
• Start replication with file+position
(CHANGE MASTER TO AUTO_POSITION=0; START SLAVE)
• After running a few transactions, stop replication and restore the GTID state
(this is the voodoo part: you will have to figure it out on your own
and this uses the noted GTID position above) 25
Conclusion [1 of 2]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
• We saw why someone would run with sync_binlog != 1
• We understood the consequences, and saw how to avoid and mitigates them
• Using sync_binlog != 1 will be very common in the Cloud
(because of the high latency of network disks)
• To be fully Cloud Ready,
MySQL should make running with sync_binlog != 1 easier:
• Auto OFFLINE_MODE after an OS crash when sync_binlog != 1 ?
• Until MySQL is replication crash-safe with GTID and sync_binlog != 1:
Auto skip-slave-start after an OS crash when sync_binlog != 1 ?
• Bug#98448: Please make running MySQL with sync_binlog != 1 safer
• Rant: MySQL implementation of GTID makes things complicated !
• Rant2: GTID state in a table, in the binlogs and in the InnoDB Logs
à solving problems by patching things, no consistent vision, cleanup needed !
Conclusion [2 of 2]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
• More GTID rant: gtid_executed shared by a DBA in a MySQL Slack channel:
(how to clean this up is left as an exercise for the audience)
(this was reported to Oracle years ago, no good way of dealing with this provided yet)
000d4a8b-ee59-11e9-ab79-008cfa5440e4:1-18448,0048b71c-d850-11e9-bc30-008cfa542442:1-18563,005f87ce-ade3-11e9-9c87-008cfa542c9e:1-82355,008367c9-971f-
11e9-95d9-008cfa544064:1-80594,00f2a2a9-ab92-11e9-ac53-008cfa544064:1-81871,0100879f-e8fc-11e9-b340-008cfa5440e4:1-18531,0103609f-d53a-11e9-828e-
008cfa542442:1-18206,011f5cb1-a92c-11e9-9fdb-008cfa542c9e:1-83848,018c0de3-ba7e-11e9-a890-008cfa542c9e:1-83934,02a5faa1-98b6-11e9-97cd-008cfa542c9e:1-
85046136,037f8134-c89f-11e9-8e3a-008cfa542c9e:1-18332,03ba38a5-a6db-11e9-8af2-008cfa5440e4:1-74794,03de556d-eb30-11e9-84df-008cfa542c9e:1-
18466,03eb1177-cd50-11e9-9949-008cfa542442:1-1818897,04d72700-b5c7-11e9-a0a9-008cfa542c9e:1-85470,05a10dbe-d5ff-11e9-8e87-008cfa544064:1-18386,068e6393-
c323-11e9-ac17-008cfa542442:1-103482,069bb1f8-caf4-11e9-919f-008cfa542c9e:1-18420,069eb739-e1c0-11e9-b03d-008cfa5440e4:1-18496,06cde7cb-989c-11e9-bd1e-
008cfa5440e4:1-935909,07035800-e036-11e9-9775-008cfa5440e4:1-18497,071c672a-be70-11e9-a302-008cfa542442:1-138571,075883a2-d211-11e9-a3a2-008cfa542c9e:1-
18225,07d8cfd0-a79e-11e9-9fbb-008cfa544064:1-84475,08241557-db7f-11e9-9801-008cfa542442:1-18287,08eee297-e4ed-11e9-8eb3-008cfa542c9e:1-18230,090590ae-
e5ac-11e9-8b4f-008cfa544064:1-18527,0939cfc6-d462-11e9-b8f1-008cfa542c9e:1-18275,09d48f25-c971-11e9-99ec-008cfa542c9e:1-18493,0a121d94-cee6-11e9-b5ab-
008cfa544064:1-18595,0a1ae309-d6c4-11e9-b732-008cfa542442:1-18168,0aa0e300-ca2d-11e9-a81b-008cfa542c9e:1-18347,0b3d7eed-b29a-11e9-87d6-008cfa5440e4:1-
77023,0b7c58fd-cb01-11e9-bfcd-008cfa544064:1-18508,0bb517b6-cd52-11e9-8903-008cfa542c9e:1-18173,0bcfd1ac-b488-11e9-89b8-e4434b27ec50:1-39442,0c34eca3-
ad1e-11e9-bf76-008cfa542c9e:1-82933,0c46cff1-b823-11e9-bbb4-008cfa5440e4:1-7001796,0c72d5c7-ee57-11e9-9cde-a0369f9432f4:1-50694858,0d36c194-a7a2-11e9-
965a-008cfa542c9e:1-7572323,0d534018-ade1-11e9-934e-008cfa544064:1-83481,0e751980-cc8f-11e9-92ef-008cfa5440e4:1-17609,0ea04e41-b757-11e9-a049-
008cfa542442:1-80864,0f0e919e-b1d5-11e9-b22a-008cfa5440e4:1-61341,0f7c6956-bc0a-11e9-8628-008cfa542442:1-130513,10528142-95c0-11e9-8218-008cfa5440e4:1-
38346,10636758-bccf-11e9-bdf4-008cfa542c9e:1-133216,1076c777-b110-11e9-8dfa-008cfa542c9e:1-85695,110d67aa-d603-11e9-b24d-008cfa542442:1-18271,11d66846-
d601-11e9-b3da-008cfa542c9e:1-1671806,11fb613f-aeaa-11e9-87ad-008cfa542c9e:1-81172,1202852e-d854-11e9-bb27-008cfa542c9e:1-18331,12259ccd-a867-11e9-868f-
008cfa5440e4:1-7005845,128e5b58-ac59-11e9-a987-008cfa542c9e:1-84165,128ff97f-d2d8-11e9-85ff-008cfa542442:1-1668330,1395d36e-d39b-11e9-9b67-
008cfa542c9e:1-18186,1398ce80-e73e-11e9-b6d6-008cfa544064:1-18295,13aa8365-ca2f-11e9-9b9c-008cfa544064:1-18393,13b2a7f8-fefc-11e9-8d3c-b4969136e8e0:1-
3,13ca371a-d919-11e9-9e7b-008cfa5440e4:1-18470,13d4cd9c-c0c1-11e9-8a19-008cfa5440e4:1-121684,13fa0ac6-b04b-11e9-ab02-008cfa544064:1-83899,14c9fee1-bb45-
11e9-bc9b-008cfa542c9e:1-136184,1509d718-9a40-11e9-bca9-008cfa542442:1-961253,15a5c07d-ddd0-11e9-b050-008cfa542442:1-18432,15c06864-b361-11e9-8e4c-
008cfa542c9e:1-81510,15da65c1-a6dd-11e9-b919-008cfa542c9e:1-83624,1695a57f-a9f3-11e9-b8d9-008cfa542c9e:1-81772,1730b2c1-a63f-11e9-a9a7-008cfa542442:1-
4112383,176edb77-c96a-11e9-a4ee-008cfa5440e4:1-18533,178da034-9b07-11e9-a1bc-008cfa542442:1-962266,17c75d9f-ab94-11e9-8d5e-008cfa542c9e:1-
84025,18bcec4e-b818-11e9-ac66-008cfa5440e4:1-78637,1936ceb2-c327-11e9-aead-008cfa542442:1-104809,196ee579-e1c2-11e9-8f96-008cfa542c9e:1-18151,19e72316-
ea67-11e9-b4cf-a0369f9432f4:1-18255,1a094a28-ea6b-11e9-a501-008cfa542c9e:1-18296,1a5f9739-ba80-11e9-9bbf-008cfa542442:1-138749,1a88b452-d6ca-11e9-ab67-
008cfa5440e4:1-18288,1ac1bdef-b753-11e9-84b5-008cfa544064:1-35746,1b425cb0-d78f-11e9-8ed5-008cfa542442:1-18153,1b730452-d852-11e9-8190-008cfa5440e4:1-
18538,1b7c329b-df6f-11e9-9906-008cfa544064:1-1663592,1bb9e21f-e679-11e9-ab63-008cfa5440e4:1-18516,1bea1785-c8a5-11e9-a768-008cfa542c9e:1-1671715
28
Links [1 of 3]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
• On the consequences of sync_binlog != 1 (part #1)
https://jfg-mysql.blogspot.com/2018/10/consequences-sync-binlog-neq-1-part-1.html
• Evaluating MySQL Parallel Replication Part 2: Slave Group Commit
https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-2-slave-group-commit-459026a141d2
• Better Crash-safe replication for MySQL
https://medium.com/booking-com-infrastructure/better-crash-safe-replication-for-mysql-a336a69b317f
• Bug#70659: Make crash safe slave work with gtid + less durable settings
• Bug#92109: Please make repl. crash safe with GITD and less durable setting (bis)
• Bug#98448: Please make running MySQL with sync_binlog != 1 safer
• WL#9211: InnoDB Clone Replication Coordinates
Links [2 of 3]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
• MySQL Master High Availability and Failover: more thoughts
https://jfg-mysql.blogspot.com/2019/02/mysql-master-high-availability-and-failover-more-thoughts.html
• MySQL Scalability and Reliability for Replicated Environment
https://www.slideshare.net/JeanFranoisGagn/mysql-scalability-and-reliability-for-replicated-environment-150876075
• The Full MySQL and MariaDB Parallel Replication Tutorial
https://www.slideshare.net/JeanFranoisGagn/the-full-mysql-and-mariadb-parallel-replication-tutorial
• Evaluating MySQL Parallel Replication Part 4: More Benchmarks in Production
https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-4-more-benchmarks-in-production-49ee255043ab
29
Links [3 of 3]
(The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020)
• Pseudo-GTID and Orchestrator:
https://github.com/github/orchestrator/blob/master/docs/pseudo-gtid.md
https://speakerdeck.com/shlominoach/pseudo-gtid-and-easy-mysql-replication-topology-management
• How Binary Logs (and Filesystems) Affect MySQL Performance
https://www.percona.com/blog/2018/05/04/how-binary-logs-and-filesystems-affect-mysql-performance/
• Fsync Performance on Storage Devices
https://www.percona.com/blog/2018/02/08/fsync-performance-storage-devices/
30
MessageBird is hiring
messagebird.com/en/careers/
Thanks !
by Jean-François Gagné
Senior Infrastructure Engineer / System and MySQL Expert
jeanfrancois AT messagebird DOT com / @jfg956 #FOSDEM #MySQLDevRoom

More Related Content

What's hot

ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdfProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdfJesmar Cannao'
 
MySQL GTID 시작하기
MySQL GTID 시작하기MySQL GTID 시작하기
MySQL GTID 시작하기I Goo Lee
 
Optimizing MariaDB for maximum performance
Optimizing MariaDB for maximum performanceOptimizing MariaDB for maximum performance
Optimizing MariaDB for maximum performanceMariaDB plc
 
Replication Troubleshooting in Classic VS GTID
Replication Troubleshooting in Classic VS GTIDReplication Troubleshooting in Classic VS GTID
Replication Troubleshooting in Classic VS GTIDMydbops
 
Redo log improvements MYSQL 8.0
Redo log improvements MYSQL 8.0Redo log improvements MYSQL 8.0
Redo log improvements MYSQL 8.0Mydbops
 
MySQL GTID Concepts, Implementation and troubleshooting
MySQL GTID Concepts, Implementation and troubleshooting MySQL GTID Concepts, Implementation and troubleshooting
MySQL GTID Concepts, Implementation and troubleshooting Mydbops
 
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...Jean-François Gagné
 
Galera cluster for high availability
Galera cluster for high availability Galera cluster for high availability
Galera cluster for high availability Mydbops
 
Parallel Replication in MySQL and MariaDB
Parallel Replication in MySQL and MariaDBParallel Replication in MySQL and MariaDB
Parallel Replication in MySQL and MariaDBMydbops
 
Wars of MySQL Cluster ( InnoDB Cluster VS Galera )
Wars of MySQL Cluster ( InnoDB Cluster VS Galera ) Wars of MySQL Cluster ( InnoDB Cluster VS Galera )
Wars of MySQL Cluster ( InnoDB Cluster VS Galera ) Mydbops
 
MariaDB Performance Tuning Crash Course
MariaDB Performance Tuning Crash CourseMariaDB Performance Tuning Crash Course
MariaDB Performance Tuning Crash CourseSeveralnines
 
Handling Schema Changes Using pt-online-schema change.
Handling Schema Changes Using pt-online-schema change.Handling Schema Changes Using pt-online-schema change.
Handling Schema Changes Using pt-online-schema change.Mydbops
 
AWS 환경에서 MySQL BMT
AWS 환경에서 MySQL BMTAWS 환경에서 MySQL BMT
AWS 환경에서 MySQL BMTI Goo Lee
 
How to Manage Scale-Out Environments with MariaDB MaxScale
How to Manage Scale-Out Environments with MariaDB MaxScaleHow to Manage Scale-Out Environments with MariaDB MaxScale
How to Manage Scale-Out Environments with MariaDB MaxScaleMariaDB plc
 
MySQL Data Encryption at Rest
MySQL Data Encryption at RestMySQL Data Encryption at Rest
MySQL Data Encryption at RestMydbops
 
MySQL InnoDB Cluster - A complete High Availability solution for MySQL
MySQL InnoDB Cluster - A complete High Availability solution for MySQLMySQL InnoDB Cluster - A complete High Availability solution for MySQL
MySQL InnoDB Cluster - A complete High Availability solution for MySQLOlivier DASINI
 
MariaDB AX: Analytics with MariaDB ColumnStore
MariaDB AX: Analytics with MariaDB ColumnStoreMariaDB AX: Analytics with MariaDB ColumnStore
MariaDB AX: Analytics with MariaDB ColumnStoreMariaDB plc
 
MySQL Performance Schema in 20 Minutes
 MySQL Performance Schema in 20 Minutes MySQL Performance Schema in 20 Minutes
MySQL Performance Schema in 20 MinutesSveta Smirnova
 
MariaDB MaxScale
MariaDB MaxScaleMariaDB MaxScale
MariaDB MaxScaleMariaDB plc
 

What's hot (20)

ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdfProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdf
 
MySQL GTID 시작하기
MySQL GTID 시작하기MySQL GTID 시작하기
MySQL GTID 시작하기
 
Optimizing MariaDB for maximum performance
Optimizing MariaDB for maximum performanceOptimizing MariaDB for maximum performance
Optimizing MariaDB for maximum performance
 
Replication Troubleshooting in Classic VS GTID
Replication Troubleshooting in Classic VS GTIDReplication Troubleshooting in Classic VS GTID
Replication Troubleshooting in Classic VS GTID
 
Redo log improvements MYSQL 8.0
Redo log improvements MYSQL 8.0Redo log improvements MYSQL 8.0
Redo log improvements MYSQL 8.0
 
MySQL GTID Concepts, Implementation and troubleshooting
MySQL GTID Concepts, Implementation and troubleshooting MySQL GTID Concepts, Implementation and troubleshooting
MySQL GTID Concepts, Implementation and troubleshooting
 
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
 
Galera cluster for high availability
Galera cluster for high availability Galera cluster for high availability
Galera cluster for high availability
 
Parallel Replication in MySQL and MariaDB
Parallel Replication in MySQL and MariaDBParallel Replication in MySQL and MariaDB
Parallel Replication in MySQL and MariaDB
 
Wars of MySQL Cluster ( InnoDB Cluster VS Galera )
Wars of MySQL Cluster ( InnoDB Cluster VS Galera ) Wars of MySQL Cluster ( InnoDB Cluster VS Galera )
Wars of MySQL Cluster ( InnoDB Cluster VS Galera )
 
MariaDB Performance Tuning Crash Course
MariaDB Performance Tuning Crash CourseMariaDB Performance Tuning Crash Course
MariaDB Performance Tuning Crash Course
 
Handling Schema Changes Using pt-online-schema change.
Handling Schema Changes Using pt-online-schema change.Handling Schema Changes Using pt-online-schema change.
Handling Schema Changes Using pt-online-schema change.
 
AWS 환경에서 MySQL BMT
AWS 환경에서 MySQL BMTAWS 환경에서 MySQL BMT
AWS 환경에서 MySQL BMT
 
How to Manage Scale-Out Environments with MariaDB MaxScale
How to Manage Scale-Out Environments with MariaDB MaxScaleHow to Manage Scale-Out Environments with MariaDB MaxScale
How to Manage Scale-Out Environments with MariaDB MaxScale
 
MySQL Data Encryption at Rest
MySQL Data Encryption at RestMySQL Data Encryption at Rest
MySQL Data Encryption at Rest
 
MySQL InnoDB Cluster - A complete High Availability solution for MySQL
MySQL InnoDB Cluster - A complete High Availability solution for MySQLMySQL InnoDB Cluster - A complete High Availability solution for MySQL
MySQL InnoDB Cluster - A complete High Availability solution for MySQL
 
Query logging with proxysql
Query logging with proxysqlQuery logging with proxysql
Query logging with proxysql
 
MariaDB AX: Analytics with MariaDB ColumnStore
MariaDB AX: Analytics with MariaDB ColumnStoreMariaDB AX: Analytics with MariaDB ColumnStore
MariaDB AX: Analytics with MariaDB ColumnStore
 
MySQL Performance Schema in 20 Minutes
 MySQL Performance Schema in 20 Minutes MySQL Performance Schema in 20 Minutes
MySQL Performance Schema in 20 Minutes
 
MariaDB MaxScale
MariaDB MaxScaleMariaDB MaxScale
MariaDB MaxScale
 

Similar to The consequences of sync_binlog != 1

Percona 服务器与 XtraDB 存储引擎
Percona 服务器与 XtraDB 存储引擎Percona 服务器与 XtraDB 存储引擎
Percona 服务器与 XtraDB 存储引擎YUCHENG HU
 
Demystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash SafetyDemystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash SafetyJean-François Gagné
 
Demystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash SafetyDemystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash SafetyJean-François Gagné
 
Best practices for MySQL/MariaDB Server/Percona Server High Availability
Best practices for MySQL/MariaDB Server/Percona Server High AvailabilityBest practices for MySQL/MariaDB Server/Percona Server High Availability
Best practices for MySQL/MariaDB Server/Percona Server High AvailabilityColin Charles
 
Best practices for MySQL High Availability
Best practices for MySQL High AvailabilityBest practices for MySQL High Availability
Best practices for MySQL High AvailabilityColin Charles
 
MySQL Parallel Replication: inventory, use-case and limitations
MySQL Parallel Replication: inventory, use-case and limitationsMySQL Parallel Replication: inventory, use-case and limitations
MySQL Parallel Replication: inventory, use-case and limitationsJean-François Gagné
 
The InnoDB Storage Engine for MySQL
The InnoDB Storage Engine for MySQLThe InnoDB Storage Engine for MySQL
The InnoDB Storage Engine for MySQLMorgan Tocker
 
About MySQL sync_binlog
About MySQL sync_binlog About MySQL sync_binlog
About MySQL sync_binlog Dehua Yang
 
Best practices for MySQL High Availability Tutorial
Best practices for MySQL High Availability TutorialBest practices for MySQL High Availability Tutorial
Best practices for MySQL High Availability TutorialColin Charles
 
介绍 Percona 服务器 XtraDB 和 Xtrabackup
介绍 Percona 服务器 XtraDB 和 Xtrabackup介绍 Percona 服务器 XtraDB 和 Xtrabackup
介绍 Percona 服务器 XtraDB 和 XtrabackupYUCHENG HU
 
Loadays MySQL
Loadays MySQLLoadays MySQL
Loadays MySQLlefredbe
 
MySQL Parallel Replication by Booking.com
MySQL Parallel Replication by Booking.comMySQL Parallel Replication by Booking.com
MySQL Parallel Replication by Booking.comJean-François Gagné
 
High Performance Scaling Techniques in Golang Using Go Assembly
High Performance Scaling Techniques in Golang Using Go AssemblyHigh Performance Scaling Techniques in Golang Using Go Assembly
High Performance Scaling Techniques in Golang Using Go AssemblyMinio
 
20140128 webinar-get-more-out-of-mysql-with-tokudb-140319063324-phpapp02
20140128 webinar-get-more-out-of-mysql-with-tokudb-140319063324-phpapp0220140128 webinar-get-more-out-of-mysql-with-tokudb-140319063324-phpapp02
20140128 webinar-get-more-out-of-mysql-with-tokudb-140319063324-phpapp02Francisco Gonçalves
 
Consistency between Engine and Binlog under Reduced Durability
Consistency between Engine and Binlog under Reduced DurabilityConsistency between Engine and Binlog under Reduced Durability
Consistency between Engine and Binlog under Reduced DurabilityYoshinori Matsunobu
 
Kettunen, miaubiz fuzzing at scale and in style
Kettunen, miaubiz   fuzzing at scale and in styleKettunen, miaubiz   fuzzing at scale and in style
Kettunen, miaubiz fuzzing at scale and in styleDefconRussia
 
Deployment Strategies (Mongo Austin)
Deployment Strategies (Mongo Austin)Deployment Strategies (Mongo Austin)
Deployment Strategies (Mongo Austin)MongoDB
 
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale by ...
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale  by ...[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale  by ...
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale by ...Insight Technology, Inc.
 
Faster, better, stronger: The new InnoDB
Faster, better, stronger: The new InnoDBFaster, better, stronger: The new InnoDB
Faster, better, stronger: The new InnoDBMariaDB plc
 
Parallel Graphics in Frostbite - Current & Future (Siggraph 2009)
Parallel Graphics in Frostbite - Current & Future (Siggraph 2009)Parallel Graphics in Frostbite - Current & Future (Siggraph 2009)
Parallel Graphics in Frostbite - Current & Future (Siggraph 2009)Johan Andersson
 

Similar to The consequences of sync_binlog != 1 (20)

Percona 服务器与 XtraDB 存储引擎
Percona 服务器与 XtraDB 存储引擎Percona 服务器与 XtraDB 存储引擎
Percona 服务器与 XtraDB 存储引擎
 
Demystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash SafetyDemystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash Safety
 
Demystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash SafetyDemystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash Safety
 
Best practices for MySQL/MariaDB Server/Percona Server High Availability
Best practices for MySQL/MariaDB Server/Percona Server High AvailabilityBest practices for MySQL/MariaDB Server/Percona Server High Availability
Best practices for MySQL/MariaDB Server/Percona Server High Availability
 
Best practices for MySQL High Availability
Best practices for MySQL High AvailabilityBest practices for MySQL High Availability
Best practices for MySQL High Availability
 
MySQL Parallel Replication: inventory, use-case and limitations
MySQL Parallel Replication: inventory, use-case and limitationsMySQL Parallel Replication: inventory, use-case and limitations
MySQL Parallel Replication: inventory, use-case and limitations
 
The InnoDB Storage Engine for MySQL
The InnoDB Storage Engine for MySQLThe InnoDB Storage Engine for MySQL
The InnoDB Storage Engine for MySQL
 
About MySQL sync_binlog
About MySQL sync_binlog About MySQL sync_binlog
About MySQL sync_binlog
 
Best practices for MySQL High Availability Tutorial
Best practices for MySQL High Availability TutorialBest practices for MySQL High Availability Tutorial
Best practices for MySQL High Availability Tutorial
 
介绍 Percona 服务器 XtraDB 和 Xtrabackup
介绍 Percona 服务器 XtraDB 和 Xtrabackup介绍 Percona 服务器 XtraDB 和 Xtrabackup
介绍 Percona 服务器 XtraDB 和 Xtrabackup
 
Loadays MySQL
Loadays MySQLLoadays MySQL
Loadays MySQL
 
MySQL Parallel Replication by Booking.com
MySQL Parallel Replication by Booking.comMySQL Parallel Replication by Booking.com
MySQL Parallel Replication by Booking.com
 
High Performance Scaling Techniques in Golang Using Go Assembly
High Performance Scaling Techniques in Golang Using Go AssemblyHigh Performance Scaling Techniques in Golang Using Go Assembly
High Performance Scaling Techniques in Golang Using Go Assembly
 
20140128 webinar-get-more-out-of-mysql-with-tokudb-140319063324-phpapp02
20140128 webinar-get-more-out-of-mysql-with-tokudb-140319063324-phpapp0220140128 webinar-get-more-out-of-mysql-with-tokudb-140319063324-phpapp02
20140128 webinar-get-more-out-of-mysql-with-tokudb-140319063324-phpapp02
 
Consistency between Engine and Binlog under Reduced Durability
Consistency between Engine and Binlog under Reduced DurabilityConsistency between Engine and Binlog under Reduced Durability
Consistency between Engine and Binlog under Reduced Durability
 
Kettunen, miaubiz fuzzing at scale and in style
Kettunen, miaubiz   fuzzing at scale and in styleKettunen, miaubiz   fuzzing at scale and in style
Kettunen, miaubiz fuzzing at scale and in style
 
Deployment Strategies (Mongo Austin)
Deployment Strategies (Mongo Austin)Deployment Strategies (Mongo Austin)
Deployment Strategies (Mongo Austin)
 
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale by ...
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale  by ...[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale  by ...
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale by ...
 
Faster, better, stronger: The new InnoDB
Faster, better, stronger: The new InnoDBFaster, better, stronger: The new InnoDB
Faster, better, stronger: The new InnoDB
 
Parallel Graphics in Frostbite - Current & Future (Siggraph 2009)
Parallel Graphics in Frostbite - Current & Future (Siggraph 2009)Parallel Graphics in Frostbite - Current & Future (Siggraph 2009)
Parallel Graphics in Frostbite - Current & Future (Siggraph 2009)
 

More from Jean-François Gagné

Almost Perfect Service Discovery and Failover with ProxySQL and Orchestrator
Almost Perfect Service Discovery and Failover with ProxySQL and OrchestratorAlmost Perfect Service Discovery and Failover with ProxySQL and Orchestrator
Almost Perfect Service Discovery and Failover with ProxySQL and OrchestratorJean-François Gagné
 
Autopsy of a MySQL Automation Disaster
Autopsy of a MySQL Automation DisasterAutopsy of a MySQL Automation Disaster
Autopsy of a MySQL Automation DisasterJean-François Gagné
 
MySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated EnvironmentMySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated EnvironmentJean-François Gagné
 
MySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated EnvironmentMySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated EnvironmentJean-François Gagné
 
MySQL/MariaDB Parallel Replication: inventory, use-case and limitations
MySQL/MariaDB Parallel Replication: inventory, use-case and limitationsMySQL/MariaDB Parallel Replication: inventory, use-case and limitations
MySQL/MariaDB Parallel Replication: inventory, use-case and limitationsJean-François Gagné
 
The two little bugs that almost brought down Booking.com
The two little bugs that almost brought down Booking.comThe two little bugs that almost brought down Booking.com
The two little bugs that almost brought down Booking.comJean-François Gagné
 
How Booking.com avoids and deals with replication lag
How Booking.com avoids and deals with replication lagHow Booking.com avoids and deals with replication lag
How Booking.com avoids and deals with replication lagJean-François Gagné
 
MySQL Parallel Replication: inventory, use-cases and limitations
MySQL Parallel Replication: inventory, use-cases and limitationsMySQL Parallel Replication: inventory, use-cases and limitations
MySQL Parallel Replication: inventory, use-cases and limitationsJean-François Gagné
 
Riding the Binlog: an in Deep Dissection of the Replication Stream
Riding the Binlog: an in Deep Dissection of the Replication StreamRiding the Binlog: an in Deep Dissection of the Replication Stream
Riding the Binlog: an in Deep Dissection of the Replication StreamJean-François Gagné
 

More from Jean-François Gagné (10)

Almost Perfect Service Discovery and Failover with ProxySQL and Orchestrator
Almost Perfect Service Discovery and Failover with ProxySQL and OrchestratorAlmost Perfect Service Discovery and Failover with ProxySQL and Orchestrator
Almost Perfect Service Discovery and Failover with ProxySQL and Orchestrator
 
Autopsy of a MySQL Automation Disaster
Autopsy of a MySQL Automation DisasterAutopsy of a MySQL Automation Disaster
Autopsy of a MySQL Automation Disaster
 
MySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated EnvironmentMySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated Environment
 
MySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated EnvironmentMySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated Environment
 
MySQL/MariaDB Parallel Replication: inventory, use-case and limitations
MySQL/MariaDB Parallel Replication: inventory, use-case and limitationsMySQL/MariaDB Parallel Replication: inventory, use-case and limitations
MySQL/MariaDB Parallel Replication: inventory, use-case and limitations
 
The two little bugs that almost brought down Booking.com
The two little bugs that almost brought down Booking.comThe two little bugs that almost brought down Booking.com
The two little bugs that almost brought down Booking.com
 
Autopsy of an automation disaster
Autopsy of an automation disasterAutopsy of an automation disaster
Autopsy of an automation disaster
 
How Booking.com avoids and deals with replication lag
How Booking.com avoids and deals with replication lagHow Booking.com avoids and deals with replication lag
How Booking.com avoids and deals with replication lag
 
MySQL Parallel Replication: inventory, use-cases and limitations
MySQL Parallel Replication: inventory, use-cases and limitationsMySQL Parallel Replication: inventory, use-cases and limitations
MySQL Parallel Replication: inventory, use-cases and limitations
 
Riding the Binlog: an in Deep Dissection of the Replication Stream
Riding the Binlog: an in Deep Dissection of the Replication StreamRiding the Binlog: an in Deep Dissection of the Replication Stream
Riding the Binlog: an in Deep Dissection of the Replication Stream
 

Recently uploaded

Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubKalema Edgar
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Wonjun Hwang
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machinePadma Pradeep
 
Vector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector DatabasesVector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector DatabasesZilliz
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst
 
The Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdfThe Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdfSeasiaInfotech2
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLScyllaDB
 
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsVertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsMiki Katsuragi
 
Search Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdfSearch Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdfRankYa
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Enterprise Knowledge
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsMemoori
 
My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024The Digital Insurer
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piececharlottematthew16
 
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr LapshynFwdays
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxArtificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxhariprasad279825
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii SoldatenkoFwdays
 

Recently uploaded (20)

Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding Club
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machine
 
Vector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector DatabasesVector Databases 101 - An introduction to the world of Vector Databases
Vector Databases 101 - An introduction to the world of Vector Databases
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR Systems
 
The Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdfThe Future of Software Development - Devin AI Innovative Approach.pdf
The Future of Software Development - Devin AI Innovative Approach.pdf
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQL
 
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsVertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering Tips
 
Search Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdfSearch Engine Optimization SEO PDF for 2024.pdf
Search Engine Optimization SEO PDF for 2024.pdf
 
Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024Designing IA for AI - Information Architecture Conference 2024
Designing IA for AI - Information Architecture Conference 2024
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial Buildings
 
My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piece
 
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxArtificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptx
 
DMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special EditionDMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special Edition
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
 

The consequences of sync_binlog != 1

  • 1. The consequences of sync_binlog != 1 by Jean-François Gagné Senior Infrastructure Engineer / System and MySQL Expert jeanfrancois AT messagebird DOT com / @jfg956 #FOSDEM #MySQLDevRoom
  • 2. The full title of the talk should be The consequences of sync_binlog != 1 and of innodb_flush_log_at_trx_commit = 2 as one without the other does not make much sense (I will use trx_commit for short in this talk)
  • 3. 3 Summary (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) • Faster but by how much ? • Overview of replication and zoom in sync_binlog & trx_commit • Avoiding sync_binlog != 1 • The consequences of sync_binlog != 1 • Mitigating sync_binlog != 1 • Closing, links and questions This talk applies mostly to MySQL 5.6 and 5.7 (unless explicitly mentioned) some content will apply to MariaDB 10.0+ (and will be explicitly mentioned)
  • 4. Faster but by how much ? (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) Sysbench, single-thread insert benchmark, without secondary index, vm on Google Cloud Platform (GCP), SSD persistent-disks, MySQL 5.7.26 With sync_binlog = 1 & trx_commit = 1: • ~200 Transactions Per Second (TPS) on the master • ~230 TPS on a slave With sync_binlog = 0 & trx_commit = 2: • ~3770 TPS on the master (~18x faster) • ~7050 TPS on a slave (~30x faster) https://jfg-mysql.blogspot.com/2019/07/master-replication-crash-safety-part-4-benchmarks-of-high-n-low-durability.html 4
  • 5. Overview of MySQL Replication (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) One master node with one or more slave nodes: • The master records transactions in a journal (binary logs); each slave: • Downloads the journal and saves it locally in the relay logs (IO thread) • Executes the relay logs on its local database (SQL thread) • Could also produce binary logs to be a master (log-slave-updates)
  • 6. Zoom in sync_binlog & trx_commit [1 of 6] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) • sync_binlog = N: the binlogs are flushed to disk every N trx group • trx_commit = {1,2,0} • 1: the InnoDB Redo Log (IRL) is flushed to disk after each transaction • 2: the IRL is written to after each trx (OS RAM buffer) but flushed every second • 0: the IRL is written to and flushed every second (not covered in this talk)
  • 7. Zoom in sync_binlog & trx_commit [2 of 6] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) Flushing to disk is not fast ! • Local disks [1]: • Desktop and Enterprise grade magnetic SATA: between 18 and 25 ms • Consumer grade SSD SATA and NVMe: between 0.5 ms and 10 ms • High-end enterprise-grade NVMe: ~0.15 ms • Dell Perc with BBU: ~0.04 ms • Network disks (FC or iSCSI à network round-trip): between 0.5 ms and 1 ms • Cloud environments: • GCP Persistent Disks (network): ~0.6 ms [2] • Amazon Web Services Local SSD: ~0.05 ms [3] (careful about local SSDs in the cloud as they are not always persistent between reboots) [1]: https://www.percona.com/blog/2018/02/08/fsync-performance-storage-devices/ [2]: https://jfg-mysql.blogspot.com/2019/07/master-replication-crash-safety-part-4-benchmarks-under-the-hood.html [3]: https://jfg-mysql.blogspot.com/2019/07/master-replication-crash-safety-part-5a-faster-wo-reducing-durability-hardware.html
  • 8. Zoom in sync_binlog & trx_commit [3 of 6] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) What does this mean ? • sync_binlog = 1 & trx_commit = 1: trx are on disk after COMMIT • Everything is fully durable (ACID), nothing lost in the case of an OS crash • = 0 & = 2: trx are in an OS RAM buffer after COMMIT, but not on disk • No data lost after a mysqld crash (data from OS RAM buffer is not lost) • But things are lost in the case of an OS crash • And after an OS crash, InnoDB and the binary logs are probably out-of-sync If those transactions are run on the master: …, D, E, F, G, H, I, J, K, L, … • On an OS crash, binlog could be flushed up to E, and InnoDB up to K • So after recovery, InnoDB will have data up to K (L and after is lost) and the transactions F and after are lost from the binary logs (Note: the scenario where InnoDB looses less than the binlog is more likely as the Redo Logs are flushed every second, but the opposite might also happen) 8
  • 9. Zoom in sync_binlog & trx_commit [4 of 6] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) Another thing about sync_binlog != 1: • Binary logs are flushed to disk at each binlog rotation and putting up to 1 GB on disk this might take time… • The associated stalls have been described by Vadim Tkachenko in [1] [1]: https://www.percona.com/blog/2018/05/04/how-binary-logs-and-filesystems-affect-mysql-performance/ And with MySQL 5.6+ and MariaDB 5.3+, the Binary Log Group Commit optimization allows to persist many transactions to disk in a single flush (InnoDB already had Group Commit for some time): • Running trx in parallel on the master increases TPS with sync_binlog = 1 • MariaDB 10.0+ Slave Group Commit [2] allows the same on slaves • And so does Parallel Replication in MySQL 5.6+ and MariaDB 10.0+ [2]: https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-2-slave-group-commit-459026a141d2 9
  • 10. Zoom in sync_binlog & trx_commit [5 of 6] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) Defaults in different versions: (The binlog_order_commits and innodb_support_xa parameters are also this discussion, but their defaults are decent, so they are only briefly mention here.) 10 MySQL 5.5 5.6 5.7 8.0 MariaDB 10.0 10.1 10.2 10.3 10.4 sync_binlog 0 K1 0 K1 1 J K2 1 J K2 0 K3 0 K3 0 L3 0 L3 0 L3 trx_commit 1 1 1 1 1 1 1 1 1 binlog_order_commits N/A4 ON ON ON N/A4 N/A4 N/A4 N/A4 N/A4 innodb_support_xa ON ON ON N/A5 ON ON ON N/A5 N/A5
  • 11. Zoom in sync_binlog & trx_commit [6 of 6] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) Notes from previous page: 1) MySQL introduced binary log group commit (BLGC) in 5.6, so up to, and arguably including 5.6, sync_binlog needed to be to 0 for performance reasons, but this was unsafe. 2) As MySQL had GLGC since 5.6, it was possible to make sync_binlog = 1 the default without penalizing performance on the master (J), but this change made replication the bottleneck and MySQL does not have slave group commit (SGC) nor does it enable parallel replication (// rpl) by default (K). Still good to be safe by default (J). 3) MariaDB introduced BLGC in 5.3, so arguably, they should have made sync_binlog = 1 the default in 10.0, but as this might have impacted replication performance (// rpl just had been introduced in 10.0 and SGC explicitly only in 10.1) this is only a K up to 10.1. But from 10.2, this is a L as databases should be safe by default (sync_binlog = 1) and fast (// rpl or SGC enabled). 4) The binlog_order_commits was introduced in MySQL 5.6, probably as part of the binary group commit implementation, and it never was introduced in MariaDB. 5) The innodb_support_xa parameter was deprecated in MySQL 5.7 and MariaDB 10.2 and removed in 8.0 and 10.3. 11
  • 12. Avoiding sync_binlog != 1 [1 of 5] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) Ideally, we would always run with sync_binlog = 1 (and trx_commit = 1) When reaching the transaction throughput limit of sync_binlog = 1, because of disk flush latencies becoming the bottleneck, and before setting sync_binlog to 0, we can: 1. Get faster disks [1] (reducing the latency of a flush) 2. Run transactions in parallel on the master (persisting many trx with a single flush thanks to Binary Log Group Commit) 3. Use parallel replication (hoping to persist many trx with a single flush) (including if running MariaDB 10.0+, use Slave Group Commit) We will quickly explore #2 and 3 in the next slides (#1 is described in [1]) [1]: https://jfg-mysql.blogspot.com/2019/07/master-replication-crash-safety-part-5a-faster-wo-reducing-durability-hardware.html 12
  • 13. Avoiding sync_binlog != 1 [2 of 5] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) Running transactions in parallel on the master (Binary Log Group Commit): • Very nice increase in TPS with sync_binlog = 1, but needs a lot of threads • And sync_binlog = 0 can do much more TPS with less threads Note: benchmarks done in GCP, so not as reliable as dedicated server. (sysbench insert bench. with SSD persistent disks without secondary index) 13
  • 14. 14 Avoiding sync_binlog != 1 [3 of 5] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) MariaDB 10.0+ Slave Group Commit [1] (named minimal in 10.1+): • Nice increase in TPS (without allocating too much extra CPU resources) • Almost as fast as single-threaded sync_binlog = 0 ! • But not as good as multi-threaded on the master (obviously as transactions are serialized and not run in parallel) [1]: https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-2-slave-group-commit-459026a141d2
  • 15. Avoiding sync_binlog != 1 [4 of 5] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) Parallel Replication (this is a complex subject, not covered in detail here, see [1]): • Nice increase in TPS with sync_binlog = 1, but needs a lot of threads • And sync_binlog = 0 can do more TPS with less threads • Sadly, not as good as the master in this case (Note that the insert benchmark is probably the worse for Parallel Repl.) [1]: https://www.slideshare.net/JeanFranoisGagn/the-full-mysql-and-mariadb-parallel-replication-tutorial 15
  • 17. 17 Consequences of sync_binlog != 1 [1 of 6] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) Trx are in an OS RAM buffer after COMMIT, but they are not on disk: • No data lost after a mysqld crash (data from OS RAM buffer is not lost) • But loosing data in case of an OS crash • And after an OS crash, InnoDB and the binary logs are not in sync If those transactions are run on the master: …, D, E, F, G, H, I, J, K, L, … • On an OS crash, binlog could be flushed up to E, and InnoDB up to K • So after recovery, InnoDB will have data up to K (L and after is lost) and the transactions F and after are lost from the binary logs (Note that the scenario where InnoDB looses less than the binlog is more likely as the Redo Logs are flushed every seconds, but the opposite of above might also happen.)
  • 18. Consequences of sync_binlog != 1 [2 of 6] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) Master OS crash and rpl. with file+position: • IO Thread in vanished binlogs • So slaves executed phantom trx (ghost in binlogs, maybe not in InnoDB) • When the master is restarted: • It records trx in new binlog file • Most slaves are broken (pointing in vanished binlog), and they might be out-of-sync with each-others • Some lagging slave might skip vanished binlogs
  • 19. Consequences of sync_binlog != 1 [2 of 6] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) Master OS crash and rpl. with file+position: • IO Thread in vanished binlogs • So slaves executed phantom trx (ghost in binlogs, maybe not in InnoDB) • When the master is restarted: • It records trx in new binlog file • Most slaves are broken (pointing in vanished binlog), and they might be out-of-sync with each-others • Some lagging slave might skip vanished binlogs Ø Broken slaves have more data than the master (à data drift) Ø With different data drift on “lucky” lagging slaves that might not break
  • 20. 20 Consequences of sync_binlog != 1 [3 of 6] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) Master OS crash and replication with GTID: • Slave also executed ghost trx vanished from binlogs • But those are in their GTID state • A recovered master reuses GTIDs of the vanished trx • Slaves magically reconnect to the master (MASTER_AUTO_POSITION = 1) 1. If master has not reused all ghost GTIDs, then the slave breaks 2. If it has, then the slave skips the new transactions à more data drift (in illustration, the slave will skip new 50 to 58 as it has the old one)
  • 21. Consequences of sync_binlog != 1 [4 of 6] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) Slave OS crash and replication with file+position: • If using replication crash-safe configuration [1] à no problem ! (well, there is still the corrupted binlog problem, but this is covered in master crashes) • With crash-safe replication, the position is stored in an InnoDB table • So replication will resume at the exact place where InnoDB rewinded • Obviously, if this slave has binlog enabled, they will be corrupted (same problems as for a master crash) [1]: https://medium.com/booking-com-infrastructure/better-crash-safe-replication-for-mysql-a336a69b317f Slave OS crash and replication with GTID: • If binlogs disabled on the slave (only possible with 5.7+) à no problem ! • With binlogs disabled, the GTID state is stored in an InnoDB table • So replication will resume at the exact place where InnoDB rewinded 21
  • 22. 22 Consequences of sync_binlog != 1 [5 of 6] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) Slave OS crash and replication with GTID: • If binary logs are enabled à problems as the GTID state is lost ! • With binlogs enabled, the GTID state is stored in the binary logs • As the binlogs rewinded, replication will resume at this past position • And as InnoDB is not in sync with the binlogs, replication will probably break • Disappointing: the GTID table is not updated after each trx (only on binlog rotation) Bug#92109: Please make replication crash-safe with GITD and less durable setting (bis) (Note: this is not a problem with MariaDB as its GTID state is stored in a table.)
  • 23. 23 Consequences of sync_binlog != 1 [6 of 6] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) It looks like MySQL 8.0.17 has the GTID position in InnoDB Logs [1] • But unclear if this solves the problem (no communication from Oracle on this subject including no bugs closed) • And GTID position in InnoDB does not work for MyRocks or other Storage Engines Ø Updating the GTID table after each trx would be better IMHO [1]: WL#9211: InnoDB: Clone Replication Coordinates: https://dev.mysql.com/worklog/task/?id=9211
  • 24. Mitigating sync_binlog != 1 [1 of 2] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) When running with sync_binlog != 1: • The binlogs – of the master or slave – cannot be trusted after an OS crash • On a master, mysqld restarting after OS crash leads to broken slaves and data drift Ø After an OS crash, make sure no slaves reconnect to the recovered master (OFFLINE_MODE = ON in conf file - failing-over to a slave is the way forward) • On slaves, having mysqld restarts after such a crash leads to truncated binlogs Ø After an OS crash, purge all binlogs on the recovered slave (RESET MASTER) (A little more complicated with GTID, see next slide) • Intermediate Masters (IM) are both master and slaves Ø After an OS crash of an intermediate master, make sure no slaves reconnect to it Ø And purge all its binary logs 24
  • 25. Mitigating sync_binlog != 1 [2 of 2] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) When running a GTID slave with sync_binlog != 1: • The GTID state is corrupted after an OS crash à you should restore a backup • But file+pos. in a table can be trusted, so there is hope to avoid restoring a backup if single-threaded replication or no gap in relay log execution (always the case if slave_preserve_commit_order = ON) This is how to salvage a GTID slave after an OS crash (this is voodoo): • Make sure mysql starts with replication disabled (skip-slave-start) • Note the GTID position (gtid_executed) and wipe the binlogs (RESET MASTER) • Start replication with file+position (CHANGE MASTER TO AUTO_POSITION=0; START SLAVE) • After running a few transactions, stop replication and restore the GTID state (this is the voodoo part: you will have to figure it out on your own and this uses the noted GTID position above) 25
  • 26. Conclusion [1 of 2] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) • We saw why someone would run with sync_binlog != 1 • We understood the consequences, and saw how to avoid and mitigates them • Using sync_binlog != 1 will be very common in the Cloud (because of the high latency of network disks) • To be fully Cloud Ready, MySQL should make running with sync_binlog != 1 easier: • Auto OFFLINE_MODE after an OS crash when sync_binlog != 1 ? • Until MySQL is replication crash-safe with GTID and sync_binlog != 1: Auto skip-slave-start after an OS crash when sync_binlog != 1 ? • Bug#98448: Please make running MySQL with sync_binlog != 1 safer • Rant: MySQL implementation of GTID makes things complicated ! • Rant2: GTID state in a table, in the binlogs and in the InnoDB Logs à solving problems by patching things, no consistent vision, cleanup needed !
  • 27. Conclusion [2 of 2] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) • More GTID rant: gtid_executed shared by a DBA in a MySQL Slack channel: (how to clean this up is left as an exercise for the audience) (this was reported to Oracle years ago, no good way of dealing with this provided yet) 000d4a8b-ee59-11e9-ab79-008cfa5440e4:1-18448,0048b71c-d850-11e9-bc30-008cfa542442:1-18563,005f87ce-ade3-11e9-9c87-008cfa542c9e:1-82355,008367c9-971f- 11e9-95d9-008cfa544064:1-80594,00f2a2a9-ab92-11e9-ac53-008cfa544064:1-81871,0100879f-e8fc-11e9-b340-008cfa5440e4:1-18531,0103609f-d53a-11e9-828e- 008cfa542442:1-18206,011f5cb1-a92c-11e9-9fdb-008cfa542c9e:1-83848,018c0de3-ba7e-11e9-a890-008cfa542c9e:1-83934,02a5faa1-98b6-11e9-97cd-008cfa542c9e:1- 85046136,037f8134-c89f-11e9-8e3a-008cfa542c9e:1-18332,03ba38a5-a6db-11e9-8af2-008cfa5440e4:1-74794,03de556d-eb30-11e9-84df-008cfa542c9e:1- 18466,03eb1177-cd50-11e9-9949-008cfa542442:1-1818897,04d72700-b5c7-11e9-a0a9-008cfa542c9e:1-85470,05a10dbe-d5ff-11e9-8e87-008cfa544064:1-18386,068e6393- c323-11e9-ac17-008cfa542442:1-103482,069bb1f8-caf4-11e9-919f-008cfa542c9e:1-18420,069eb739-e1c0-11e9-b03d-008cfa5440e4:1-18496,06cde7cb-989c-11e9-bd1e- 008cfa5440e4:1-935909,07035800-e036-11e9-9775-008cfa5440e4:1-18497,071c672a-be70-11e9-a302-008cfa542442:1-138571,075883a2-d211-11e9-a3a2-008cfa542c9e:1- 18225,07d8cfd0-a79e-11e9-9fbb-008cfa544064:1-84475,08241557-db7f-11e9-9801-008cfa542442:1-18287,08eee297-e4ed-11e9-8eb3-008cfa542c9e:1-18230,090590ae- e5ac-11e9-8b4f-008cfa544064:1-18527,0939cfc6-d462-11e9-b8f1-008cfa542c9e:1-18275,09d48f25-c971-11e9-99ec-008cfa542c9e:1-18493,0a121d94-cee6-11e9-b5ab- 008cfa544064:1-18595,0a1ae309-d6c4-11e9-b732-008cfa542442:1-18168,0aa0e300-ca2d-11e9-a81b-008cfa542c9e:1-18347,0b3d7eed-b29a-11e9-87d6-008cfa5440e4:1- 77023,0b7c58fd-cb01-11e9-bfcd-008cfa544064:1-18508,0bb517b6-cd52-11e9-8903-008cfa542c9e:1-18173,0bcfd1ac-b488-11e9-89b8-e4434b27ec50:1-39442,0c34eca3- ad1e-11e9-bf76-008cfa542c9e:1-82933,0c46cff1-b823-11e9-bbb4-008cfa5440e4:1-7001796,0c72d5c7-ee57-11e9-9cde-a0369f9432f4:1-50694858,0d36c194-a7a2-11e9- 965a-008cfa542c9e:1-7572323,0d534018-ade1-11e9-934e-008cfa544064:1-83481,0e751980-cc8f-11e9-92ef-008cfa5440e4:1-17609,0ea04e41-b757-11e9-a049- 008cfa542442:1-80864,0f0e919e-b1d5-11e9-b22a-008cfa5440e4:1-61341,0f7c6956-bc0a-11e9-8628-008cfa542442:1-130513,10528142-95c0-11e9-8218-008cfa5440e4:1- 38346,10636758-bccf-11e9-bdf4-008cfa542c9e:1-133216,1076c777-b110-11e9-8dfa-008cfa542c9e:1-85695,110d67aa-d603-11e9-b24d-008cfa542442:1-18271,11d66846- d601-11e9-b3da-008cfa542c9e:1-1671806,11fb613f-aeaa-11e9-87ad-008cfa542c9e:1-81172,1202852e-d854-11e9-bb27-008cfa542c9e:1-18331,12259ccd-a867-11e9-868f- 008cfa5440e4:1-7005845,128e5b58-ac59-11e9-a987-008cfa542c9e:1-84165,128ff97f-d2d8-11e9-85ff-008cfa542442:1-1668330,1395d36e-d39b-11e9-9b67- 008cfa542c9e:1-18186,1398ce80-e73e-11e9-b6d6-008cfa544064:1-18295,13aa8365-ca2f-11e9-9b9c-008cfa544064:1-18393,13b2a7f8-fefc-11e9-8d3c-b4969136e8e0:1- 3,13ca371a-d919-11e9-9e7b-008cfa5440e4:1-18470,13d4cd9c-c0c1-11e9-8a19-008cfa5440e4:1-121684,13fa0ac6-b04b-11e9-ab02-008cfa544064:1-83899,14c9fee1-bb45- 11e9-bc9b-008cfa542c9e:1-136184,1509d718-9a40-11e9-bca9-008cfa542442:1-961253,15a5c07d-ddd0-11e9-b050-008cfa542442:1-18432,15c06864-b361-11e9-8e4c- 008cfa542c9e:1-81510,15da65c1-a6dd-11e9-b919-008cfa542c9e:1-83624,1695a57f-a9f3-11e9-b8d9-008cfa542c9e:1-81772,1730b2c1-a63f-11e9-a9a7-008cfa542442:1- 4112383,176edb77-c96a-11e9-a4ee-008cfa5440e4:1-18533,178da034-9b07-11e9-a1bc-008cfa542442:1-962266,17c75d9f-ab94-11e9-8d5e-008cfa542c9e:1- 84025,18bcec4e-b818-11e9-ac66-008cfa5440e4:1-78637,1936ceb2-c327-11e9-aead-008cfa542442:1-104809,196ee579-e1c2-11e9-8f96-008cfa542c9e:1-18151,19e72316- ea67-11e9-b4cf-a0369f9432f4:1-18255,1a094a28-ea6b-11e9-a501-008cfa542c9e:1-18296,1a5f9739-ba80-11e9-9bbf-008cfa542442:1-138749,1a88b452-d6ca-11e9-ab67- 008cfa5440e4:1-18288,1ac1bdef-b753-11e9-84b5-008cfa544064:1-35746,1b425cb0-d78f-11e9-8ed5-008cfa542442:1-18153,1b730452-d852-11e9-8190-008cfa5440e4:1- 18538,1b7c329b-df6f-11e9-9906-008cfa544064:1-1663592,1bb9e21f-e679-11e9-ab63-008cfa5440e4:1-18516,1bea1785-c8a5-11e9-a768-008cfa542c9e:1-1671715
  • 28. 28 Links [1 of 3] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) • On the consequences of sync_binlog != 1 (part #1) https://jfg-mysql.blogspot.com/2018/10/consequences-sync-binlog-neq-1-part-1.html • Evaluating MySQL Parallel Replication Part 2: Slave Group Commit https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-2-slave-group-commit-459026a141d2 • Better Crash-safe replication for MySQL https://medium.com/booking-com-infrastructure/better-crash-safe-replication-for-mysql-a336a69b317f • Bug#70659: Make crash safe slave work with gtid + less durable settings • Bug#92109: Please make repl. crash safe with GITD and less durable setting (bis) • Bug#98448: Please make running MySQL with sync_binlog != 1 safer • WL#9211: InnoDB Clone Replication Coordinates
  • 29. Links [2 of 3] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) • MySQL Master High Availability and Failover: more thoughts https://jfg-mysql.blogspot.com/2019/02/mysql-master-high-availability-and-failover-more-thoughts.html • MySQL Scalability and Reliability for Replicated Environment https://www.slideshare.net/JeanFranoisGagn/mysql-scalability-and-reliability-for-replicated-environment-150876075 • The Full MySQL and MariaDB Parallel Replication Tutorial https://www.slideshare.net/JeanFranoisGagn/the-full-mysql-and-mariadb-parallel-replication-tutorial • Evaluating MySQL Parallel Replication Part 4: More Benchmarks in Production https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-4-more-benchmarks-in-production-49ee255043ab 29
  • 30. Links [3 of 3] (The consequences of sync_binlog != 1 – MySQL.FOSDEM.2020) • Pseudo-GTID and Orchestrator: https://github.com/github/orchestrator/blob/master/docs/pseudo-gtid.md https://speakerdeck.com/shlominoach/pseudo-gtid-and-easy-mysql-replication-topology-management • How Binary Logs (and Filesystems) Affect MySQL Performance https://www.percona.com/blog/2018/05/04/how-binary-logs-and-filesystems-affect-mysql-performance/ • Fsync Performance on Storage Devices https://www.percona.com/blog/2018/02/08/fsync-performance-storage-devices/ 30
  • 32. Thanks ! by Jean-François Gagné Senior Infrastructure Engineer / System and MySQL Expert jeanfrancois AT messagebird DOT com / @jfg956 #FOSDEM #MySQLDevRoom