SlideShare a Scribd company logo
MySQL Parallel Replication:
all the 5.7 and 8.0 details
(LOGICAL_CLOCK)
by Jean-François Gagné
System and MySQL Expert at HubSpot
Presented at Percona Live Austin 2022
jfg.mysql AT gmail DOT com
Twitter: @jfg956
Benchmark Preview
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
2
Replication throughput (TPS) without and with parallel replication
Terminology [1 of 2]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
MySQL Terminology Updates in 2020:
https://dev.mysql.com/blog-archive/mysql-terminology-updates/
• master / slave à primary / replica
• the master of a slave à the source of a replica
This presentation uses above terminology, with below additions:
• Multi-Threaded Slave (MTS) à Multi-Threaded Replication (MTR)
• intermediary master à intermediary source or non-primary source
(Exceptions for citing content from before the terminology change)
3
Terminology [2 of 2]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
From MySQL 8.0.22 (not in 5.7), new commands were introduced:
• START/STOP SLAVE à START/STOP REPLICA
• And more…
• Some missing: START REPLICA UNTIL SQL_AFTER_MTS_GAPS
From 8.0.26 (not in 5.7 either), new variables were introduced:
• log_slave_updates à log_replica_updates
• slave_parallel_workers à replica_parallel_workers
• And more…
Because this talk applies to 5.7, and not everyone is on 8.0-latest,
this presentation uses old command and variable names
Summary
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
• Replication reminders
• Why parallel replication is important (and complicated)
• How to enable parallel replication and what is LOGICAL_CLOCK
• How parallel replication works on replicas
• How dependency tracking works on primaries
• Benchmarks
• Closing thoughts
5
MySQL Replication
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
• The primary records transactions in a journal (binary logs), and replicas
• downloads the journal and saves them locally in the relay logs (IO thread)
• executes the relay logs on their local database (SQL thread)
• could also write binlogs to be themselves a source (log-slave-updates)
Other Replication Concepts: Durability
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Persistence / Durability (D of ACID):
• Is there data lost after a crash ?
(different types of crash: MySQL process (mysqld) or OS (Linux))
• Parameters controlling durability: sync_binlog and trx_commit
(trx_commit is short for innodb_flush_logs_at_trx_commit)
• High Durability (HD) or no data lost after crashes
(sync_binlog = 1 and trx_commit = 1)
(most expensive / slower because flush to disk after each trx)
• No Durability (ND sometimes called low durability)
(sync_binlog = 0 and trx_commit = 2)
(faster, with data lost on OS crash, but no loss on mysqld crash)
• (Data lost for mysqld crash in case trx_commit = 0 but not covered here) 7
Other Replication Concepts: Group Commit
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Group Commit is an optimization making high durability faster
• Before Group Commit (GC), each trx needed its disk flush
• With GC, many trx can be persisted in the same flush
à better write throughput on primaries with concurency
• (GC was introduced in MariaDB xx and then in MySQL 5.6)
• (Single threaded replication cannot Group Commit)
• (MariaDB 10.0 introduced Replica Group Commit)
8
From Single to Multi-Threaded Replication
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Without MTR, a single CPU is used for applying writes on replicas:
• on the primary, many CPUs are used for writes (one per session)
WO MTR, a single read IO can be “in flight” for writes on replicas:
• this is a bottleneck, especially for high latency disks (cloud)
• on primary, many IOs can be in flights for writes (one per session)
WO MTR, no group commit on replicas:
• on the p., a single flush can persist many trx in binlogs and redo logs
• on replicas, flushing each trx is a bottleneck
Multi-Threaded Replication is a solution to all above problems 9
Challenges of Multi-Threaded Replication
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
But running transactions in parallel on replica is not trivial:
• Updating a row cannot be run in parallel with the insert of this row
• MTR implementation needs to keep data consistency
Enabling MTR is also not straightforward:
• Many parameters and complex tunning
MTR also still has some rough edges:
• In 5.7, START SLAVE UNTIL not supported with MTR (ok from 8.0.0)
• Latest -1 Percona Server 5.7 crashes on STOP SLAVE with MTR
(PS-8030: 5.7.27 to .36 affected and 8.0.17 to .21, Oracle MySQL not affected)
• I have also seen deadlock with PS 5.7.36, and rumor of another crash
(to be investigated, unknown if both in Percona Server and Oracle MySQL)
Solutions for MTR Data Consistency
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
• MySQL 5.6: schema based parallel replication
• MariaDB 10.0: group commit based parallel replication
• MariaDB 10.1: optimistic parallel replication
• Improvement on 10.0, probably the fastest solution, but high CPU cost
• MySQL 5.7: same schema parallel replication (Logical Clock)
• Not as fast as optimistic, but not wasteful in CPU
• MySQL 8.0: Write-Set improvement (still Logical Clock)
• Lot to say, including need for RBR and back-ported in 5.7.22
This talk is about Logical Clock in 5.7 and 8.0 11
Solutions for MTR Data Consistency
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
• MySQL 5.6: schema based parallel replication
• MariaDB 10.0: group commit based parallel replication
• MariaDB 10.1: optimistic parallel replication
• Improvement on 10.0, probably the fastest solution, but high CPU cost
• MySQL 5.7: same schema parallel replication (Logical Clock)
• Not as fast as optimistic, but not wasteful in CPU
• MySQL 8.0: Write-Set improvement (still Logical Clock)
• Lot to say, including need for RBR and back-ported in 5.7.22
This talk is about Logical Clock in 5.7 and 8.0 12
Enabling Parallel Replication
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Set slave_parallel_type to LOGICAL_CLOCK
• This is the new default in 8.0.27
(was DATABASE before (schema-based solution of 5.6), including in 5.7)
(in 8.0.26, this variable is deprecated for dropping DATABASE support)
And set slave_preserve_commit_order to ON
• This is also the new default in 8.0.27(was OFF before, including in 5.7)
(more about this parameter later in the talk)
And set slave_parallel_workers to 2 or more
• New default to 4 in 8.0.27 (was 0 before including in 5.7)
These enable MTR, but are not sufficient for best throughput 13
What is LOGICAL_CLOCK ?
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
From the MySQL Reference Manual for LOGICAL_CLOCK:
• Transactions that are part of the same binary log group commit on a
source are applied in parallel on a replica.
• But this is not 100% exact (Bug#85977)
When using LOGICAL_CLOCK, MTR is scheduling trx on replica
using dependency tracking information from the binary logs:
• We will name these information Intervals
• By default, trx intervals are generated with Commit Timestamps
(which is close, but not strictly the same as group commit)
• Much better intervals can be generated with Write Set
(and Write Set is completely different to group commit) 14
Intervals
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Each trx is tagged with two numbers in the binlogs (starting with 5.7):
• sequence_number: increasing id for each trx (not to confuse with GTID)
• last_committed: id of the latest trx on which this trx depends
The last_committed / sequence_number is the trx interval
Here an example of intervals from a real system:
15
mysqlbinlog $f | awk '$11 ~ "last_committed"{print $1,$2,$11,$12}'
[...]
#170206 20:08:33 last_committed=6201 sequence_number=6203
#170206 20:08:33 last_committed=6203 sequence_number=6204
#170206 20:08:33 last_committed=6203 sequence_number=6205
#170206 20:08:33 last_committed=6203 sequence_number=6206
#170206 20:08:33 last_committed=6205 sequence_number=6207
Parallel Scheduling on Replicas [1 of 4]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
16
A transaction can be started when its last_committed and all
previous transactions have been executed
The intervals below mean that:
• When transaction #6203 (and all previous) have been executed,
transactions #6204, #6205 and #6206 can be started
• When transaction #6205 (and all previous) have been executed,
transaction #6207 can be started
#170206 20:08:33 last_committed=6201 sequence_number=6203
#170206 20:08:33 last_committed=6203 sequence_number=6204
#170206 20:08:33 last_committed=6203 sequence_number=6205
#170206 20:08:33 last_committed=6203 sequence_number=6206
#170206 20:08:33 last_committed=6205 sequence_number=6207
Parallel Scheduling on Replicas [2 of 4]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
17
Trx #6204 and #6205 (and #6206) can be run in parallel but #6205
could complete before #6204
slave_preserve_commit_order (SPCO) does what you expect:
• If SPCO is OFF, #6205 can commit before #6204
• If SPCO is ON , #6205 waits for #6204 completion before committing
SPCO to OFF generates data states that never existed on the primary
• Use with care (you probably want ON most of the time)
#170206 20:08:33 last_committed=6201 sequence_number=6203
#170206 20:08:33 last_committed=6203 sequence_number=6204
#170206 20:08:33 last_committed=6203 sequence_number=6205
#170206 20:08:33 last_committed=6203 sequence_number=6206
#170206 20:08:33 last_committed=6205 sequence_number=6207
Parallel Scheduling on Replicas [3 of 4]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
SPCO to OFF generates data states that never existed on the primary
• Use with care (you probably want ON most of the time)
Also, SPCO to OFF generates temporary gaps in GTIDs
• For filling the gaps, use START SLAVE UNTIL SQL_AFTER_MTS_GAPS
SPCO to ON needed log_slave_updates until 8.0.19 (Bug#75396)
Before 5.7.33 and 8.0.23, SPCO to ON could deadlock (Bug#89247)
• If replication gets stuck, kill -9 mysqld (no data lost unless trx_commit = 0)
• Thanks Percona / Venkatesh for the fixing this, but…
• But I have recently seen deadlocks in PS 5.7.36: not fully fixed 18
Parallel Scheduling on Replicas [4 of 4]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
19
When run in parallel, trx could fail to execute (below from the doc):
• If replication fails to execute a trx because of an InnoDB deadlock or
the trx's execution time exceeded innodb_lock_wait_timeout…
Such failed trx will be retried (below more from the doc):
• …it automatically retries slave_transaction_retries times before
stopping with an error.
The column COUNT_TRANSACTIONS_RETRIES from the the P_S
table replication_applier_status indicates how many retries
have happened, and the table repl._app._status_by_worker
contains details about the retries
Reminder: What is LOGICAL_CLOCK ?
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
From the MySQL Reference Manual for LOGICAL_CLOCK:
• Transactions that are part of the same binary log group commit on a
source are applied in parallel on a replica.
• But this is not 100% exact (Bug#85977)
When using LOGICAL_CLOCK, MTR is scheduling trx on replica
using dependency tracking information from the binary logs:
• We will name these information Intervals
• By default, trx intervals are generated with Commit Timestamps
(which is close, but not strictly the same as group commit)
• Much better intervals can be generated with Write Set
(and Write Set is completely different to group commit) 20
Commit Timestamps [1 of 3]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Commit Timestamps (CTS) is the default for 5.7 and 8.0
(binlog_transaction_dependency_tracking = COMMIT_ORDER)
It is a dependency tracking method exploiting parallelism on the
source to generate the intervals:
• While running a DML, the current trx notes the last committed trx
(this is where last_committed from mysqlbinlog is coming from)
(this is close to Group Commit, but not strictly the same)
Without tunning, CTS is not producing efficient intervals
(by default, MTR will not give much better replication speed) 21
Commit Timestamps [2 of 3]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
In below, the update id=3 ran while the update id=1 was not yet committed:
• Insert id=4 and update id=1 can run in parallel on a replica
• (Note that nothing commit at the same time à no group commit)
Session #1: CREATE TABLE t(id INT PRIMARY KEY, v INT);
Session #1: INSERT INTO t VALUES (1,1),(2,2),(3,3);
Session #2: BEGIN; UPDATE t SET v = 11 WHERE id = 1;
Session #3: UPDATE t SET v = 33 WHERE id = 3;
Session #1: INSERT INTO t VALUES (4,4);
Session #2: COMMIT;
#220410 11:21:47 last_committed=0 sequence_number=1 -- CREATE
#220410 11:21:49 last_committed=1 sequence_number=2 –- INSERT id=1,2,3
#220410 11:21:55 last_committed=2 sequence_number=3 -- UPDATE id=3
#220410 11:21:59 last_committed=3 sequence_number=4 –- INSERT id=4
#220410 11:22:03 last_committed=2 sequence_number=5 -- UPDATE id=1 22
Commit Timestamps [3 of 3]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
More binlog: note that the DML order is different from the commit order
mysqlbinlog $f |
awk '$11 ~ "last_committed"{print $1,$2,$11,$12} $10 ~ "_rows|Xid"{print $1,$2,$10}'
#220410 11:21:47 last_committed=0 sequence_number=1 -- CREATE
#220410 11:21:49 last_committed=1 sequence_number=2 -- INSERT id=1,2,3
#220410 11:21:49 Write_rows:
#220410 11:21:49 Xid
#220410 11:21:55 last_committed=2 sequence_number=3 -- UPDATE id=3
#220410 11:21:55 Update_rows:
#220410 11:21:55 Xid
#220410 11:21:59 last_committed=3 sequence_number=4 -- INSERT id=4
#220410 11:21:59 Write_rows:
#220410 11:21:59 Xid
#220410 11:22:03 last_committed=2 sequence_number=5 -- UPDATE id=1
#220410 11:21:52 Update_rows:
#220410 11:22:03 Xid
23
Tunning Commit Timestamp
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
If very few trx run in parallel, or if commit is quick (auto-commit, very
fast disks or low durability), CTS do not identify much parallelism
(low durability: sync_binlog != 1 and trx_commit != 1)
A solution is to delay commit for having more transactions running
in parallel, which leads to better interval generation:
• binlog_group_commit_sync_delay
• binlog_group_commit_sync_no_delay_count
I call this slowing-down the primary to speed-up the replicas
But performing this tunning just looking at replica speed is tedious24
Intervals Quality
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
To ease the tunning, I came-up with a metric for interval quality:
• the Average Modified Interval Length (AMIL)
It is based on the following observation:
• the larger the intervals are, the better MTR throughput will be
All the details about the AMIL in
http://jfg-mysql.blogspot.com/2017/02/metric-for-tuning-parallel-replication-mysql-5-7.html
Computing the AMIL needs parsing the output of mysqlbinlog
for extracting interval information, MySQL should make this easier:
• Bug#85965: Expose, on the master, counters for monitoring // information quality
• Bug#85966: Expose, on slaves, counters for monitoring // information quality 25
Tunning Commit Timestamps with AMIL [1 of 2]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Monitoring the AMIL provides an immediate tunning feedback:
• After increasing binlog_group_commit_sync_delay, the AMIL
should increase (if not, increase more for better parallel replication,
or rollback for not penalizing commit latency)
But while doing this, also look at the commit-rate of the primary:
• If the commit-rate noticeably drop, consider reverting the tuning
(it is penalizing the application throughput)
But penalizing throughput might be ok for solving replication lag:
• lowering the primary throughput from 200 to 100 tps might be
acceptable to increase replica throughput from 50 to 150 tps 26
Tunning Commit Timestamps with AMIL [2 of 2]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
AMIL without and with tuning on four primaries:
(speed-up the replicas by increasing binlog_group_commit_sync_delay)
27
Other Commit Timestamp Tunning
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
28
To alleviate drop in throughput caused by delay,
increase the number of operations done in parallel in MySQL
Also, more transactions run in parallel by the application leads to
more parallelism identified by Commit Timestamps and better
replication speed on replicas
CTS and Non-Primary Sources
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Without MTR, CTS leads to no parallelism identification on replicas
Similarly, even with MTR and compared to the primary,
less parallelism is identified on replicas by Commit Timestamps:
• MTR loses efficiency when replicating from a non-primary source
Also, after promoting a replica as the new primary, its binlogs have
smaller intervals than the one from the old primary:
• Using these leads to less efficient MTR (eg: after restore backup)
With SBR, no good solutions to that (except Binlog Servers)
https://medium.com/booking-com-infrastructure/better-parallel-replication-for-mysql-14e2d7857813 29
Write Set [1 of 4]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
If using RBR, a better dependency tracking method can be used:
• Write Set was developed for Group Replication (in MySQL 5.7)
• It can also be used for dependency tracking (introduced in 8.0)
• It was back-ported in 5.7.22 (thanks Oracle for this back-port)
The Write Set is a data structure used to generate intervals:
• It stores a hash of all primary keys (and unique keys) modified by trx
(tunable size with binlog_transaction_dependency_history_size)
The intervals generated by Write Set on replicas are as good as on
the primary (and better if the primary is using CTS)
(Write Set can be used on a RBR replica of a SBR source) 30
Write Set [2 of 4]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Enabling Write Set dependency tracking (on both primary and replicas):
• Set BTDT to WRITESET (or WRITESET_SESSION)
(BTDT for binlog_transaction_dependency_tracking)
• This needs TWSE to XXHASH64 (or MURMUR32)
(TWSE for transaction_write_set_extraction)
• Which also needs binlog_format to ROW
With BTDT to WRITESET, a second trx run by a single session does
not depend on the first:
• With BTDT to WRITESET, SPCO should be enabled on replica
• Or set BTDT to WRITESET_SESSION on the primary
(not a good setting for replicas, we will come back to this) 31
Write Set [3 of 4]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
In below, updates id=3 and id=1, and insert id=4 can execute in parallel:
• Without SPCO (and the update), insert id=4 could commit before id=1,2,3
SET GLOBAL binlog_transaction_dependency_tracking = WRITESET;
Session #1: CREATE TABLE t(id INT PRIMARY KEY, v INT);
Session #1: INSERT INTO t VALUES (1,1),(2,2),(3,3);
Session #2: BEGIN; UPDATE t SET v = 11 WHERE id = 1;
Session #3: UPDATE t SET v = 33 WHERE id = 3;
Session #1: INSERT INTO t VALUES (4,4);
Session #2: COMMIT;
#220410 12:01:19 last_committed=0 sequence_number=1 -- CREATE
#220410 12:01:21 last_committed=1 sequence_number=2 -- INSERT id=1,2,3
#220410 12:01:29 last_committed=2 sequence_number=3 -- UPDATE id=3
#220410 12:01:34 last_committed=1 sequence_number=4 -- INSERT id=4
#220410 12:01:38 last_committed=2 sequence_number=5 -- UPDATE id=1 32
Barriers in Parallel Scheduling on Replicas
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
#220410 12:01:19 last_committed=0 sequence_number=1 -- CREATE
#220410 12:01:21 last_committed=1 sequence_number=2 -- INSERT id=1,2,3
#220410 12:01:29 last_committed=2 sequence_number=3 -- UPDATE id=3
#220410 12:01:34 last_committed=1 sequence_number=4 -- INSERT id=4
#220410 12:01:38 last_committed=2 sequence_number=5 -- UPDATE id=1 33
In below, the update id=3 (seq_no 3) depends on the multi-value insert:
• Parallel transaction scheduling blocks until the dependent trx completes
A dependent transaction acts as a barrier in parallel scheduling:
• Even if insert id=4 could be run at the same time as the multi-value insert,
the update id=3 prevents this
This might change in a future MySQL version
(it is also the M in AMIL – Modified)
Write Set [4 of 4]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
If running with slave_preserve_commit_order to OFF,
WRITESET_SESSION avoid inconsistencies: insert id=4 runs after id=1,2,3
SET GLOBAL binlog_transaction_dependency_tracking = WRITESET_SESSION;
Session #1: CREATE TABLE t(id INT PRIMARY KEY, v INT);
Session #1: INSERT INTO t VALUES (1,1),(2,2),(3,3);
Session #2: BEGIN; UPDATE t SET v = 11 WHERE id = 1;
Session #3: UPDATE t SET v = 33 WHERE id = 3;
Session #1: INSERT INTO t VALUES (4,4);
Session #2: COMMIT;
#220410 12:12:30 last_committed=0 sequence_number=1 -- CREATE
#220410 12:12:32 last_committed=1 sequence_number=2 -- INSERT id=1,2,3
#220410 12:12:39 last_committed=2 sequence_number=3 -- UPDATE id=3
#220410 12:12:43 last_committed=2 sequence_number=4 -- INSERT id=4
#220410 12:12:47 last_committed=2 sequence_number=5 -- UPDATE id=1 34
Write Set and Non-Primary Sources
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
We can enable Write Set on intermediary source:
• Even on a single-threaded replica
• But we should use BTDT to WRITESET because
WRITESET_SESSION would create new dependencies
(without MTR, all intervals of WRITESET_SESSION are of size one)
• This solves the CTS interval degradation on intermediary source
This was done to get the first (E*) benchmark results (in 2017):
• Primary in 5.7 (pre .22),
intermediary source in 8.0 with WRITESET, replica with MTR
This is useful to “try” MTR before big migration:
• SBR to RBR for Write Set, or 5.6 to 5.7 for MTR 35
AMIL for Write Set [1 of 2]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
AMIL from a CTS primary and a Write Set intermediary source:
• The primary does not need RBR, only the intermediary source does
• Write Set AMIL is usually better than untuned and tunned CTS
36
AMIL for Write Set [2 of 2]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Sometimes, combining delay with Write Set gives a better AMIL:
• Discovered by accident, tunning opportunity?, more research needed
(Write Set AMIL below 3x better with delay compared to without)
37
More Tunning: Smaller Transaction Sizes
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Because transaction scheduling sometimes/often blocks (barriers
and dependencies), some replica workers might be idle or waiting
on another transaction completion:
• In the worse case, many workers are waiting for a single big trx
These big trx reduce the potential for speedups:
• Avoiding big transactions in the application will make MTR faster
Ironically, when commit was expensive, or to amortize round-trip to
database, doing many things in a single (big) trx was / is the right
thing to do, but this makes MTR run slower 38
Recommended Settings for MTR Readiness
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Without RBR, CTS is the only available dependency tracking
à not worth “preparing” for MTR (implement and tune when needed)
With RBR and MySQL 8.0:
• binlog_transaction_dependency_tracking to WRITESET
• slave_parallel_type to LOGICAL_CLOCK (new default in 8.0.27)
• slave_preserve_commit_order to ON (new default in 8.0.27)
With RBR and MySQL 5.7, all above plus:
• transaction_write_set_extraction to XXHASH64
(this is the default in 8.0)
With these, faster repl. only needs updating slave_parallel_workers
Thoughts on MySQL 8.0.27 new Defaults
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
In MySQL 8.0.27 we have:
• slave_parallel_type to LOGICAL_CLOCK
• slave_preserve_commit_order to ON (was OFF before)
• and slave_parallel_workers to 4 in 8.0.27 (was 0 before)
But intervals are still generated with Commit Timestamps:
• Having 4 workers will probably not improve replication speed much
• But this opens possibility for MTR rough-edges
Enabling parallel replication by default might be a little early
and when these bugs will be fixed, Write Set should also be enabled
40
Benchmark
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Benchmarks done a few years ago and at HubSpot:
• Used 0 to 256 threads (slave_parallel_workers)
• E1 to E8: different environments / workloads (2017-2018)
• F1 to F4: more environments / workloads (from HubSpot)
• HD: High Durability: sync_binlog = 1 and trx_commit = 1
• ND: No Durability: sync_binlog = 0 and trx_commit = 2
• IS: Intermediary Source: with log_slave_updates
• NO: No Order: slave_preserve_commit_order = OFF
• (WO: with order: slave_preserve_commit_order = ON)
• (RB: replica with binary logs: without log_slave_updates)
41
Benchmark Presentation [1 of 2]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
42
E5 IS-HD Single-Threaded: 6138 seconds
E5 IS-ND Single-Threaded: 2238 seconds
Going from HD to ND for E5 IS gives a speedup of 2.74
Benchmark Presentation [2 of 2]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
All in a single graph:
• HD et ND: previous speedups
• HD vs ND single threaded: the speedup from HD to ND
43
Old Benchmark Setup
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
The test environments involve three nodes:
+---+ +---+ +---+
| A | -------> | B | -------> | C |
+---+ +---+ +---+
• A is a production primary (MySQL 5.6 or 5.7: this was in 2017-2018)
• Some of the primaries are SBR, some are RBR
• B is a MySQL 8.0.3 Intermediary Source in RBR with Write Set
(transaction_write_set_extraction = XXHASH64)
(binlog_transaction_dependency_tracking = WRITESET)
• C is a replica where timing is done 44
Previous Benchmark Results [1 of 2]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Previous Benchmark Results [2 of 2]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Observations:
• High variation of speedups, very workload dependent
(probably possible to do better if spending time optimizing the application)
• HD gives better speedups than ND (normal because group commit)
• ND gives some interesting speedups, but do not expect 16x faster
not possible to use all CPUs of modern hardware on replica for writes
HubSpot Benchmark Setup
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
The test environments involve two nodes:
+---+ +---+
| A | -------> | B |
+---+ +---+
• A is a production primary (Percona Server 5.7.36 with Write Set)
(transaction_write_set_extraction = XXHASH64)
(binlog_transaction_dependency_tracking = WRITESET)
• B is a replica where timing is done
47
HubSpot Benchmarks
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Observations:
• Higher HD speedups for F{1,2} than E*
• Better HD vs ND single-threaded
for F* than E* (F1 being much better)
Something is different between E and F.
Benchmark Discussion
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Previous was on local SSD, HubSpot is EBS in AWS
• Disk-sync more expensive for HubSpot
• Which explains better HD results
• Also explains better HD vs ND single-threaded speedup
• (Also, make benchmarking hard because AWS variance)
(HubSpot is running production with ND + semi-sync)
Also, HubSpot is running with the previously recommended settings:
• Write Set enabled on primaries and replicas
• slave_parallel_type to LOGICAL_CLOCK
• slave_preserve_commit_order to ON
à Ready for MTR by updating slave_parallel_workers 49
Other Benchmarks Results
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
I also have results for:
• WO, and NO vs WO
• RB, and IS vs RB
Too long to present everything, contact me if interested
TL&DR:
• NO is faster than WO
• RB is faster than IS
• Both highly variable depending on workload and durability
50
Other Parallel Replication Subjects
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
MySQL Parallel Replication Simulator
https://blog.koehntopp.info/2021/11/08/mysql-parallel-replication.html
Replica Group Commit (Slave Group Commit in MariaDB)
https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-2-slave-group-commit-459026a141d2
• Replicating trx sequentially, in many threads for Group Committing
• Almost as good as ND single-threaded speed, with benefit of HD
Optimistic Parallel Replication (in MariaDB)
https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-4-more-benchmarks-in-production-49ee255043ab
• Go beyond scheduling barriers on replica, achieving replication prefetching
Facebook partial transaction parallel execution in MySQL 5.6
https://www.percona.com/live/e18/sessions/faster-mysql-replication-using-row-dependencies
• Using Write Set on replicas for fine grain parallel statement excution 51
Links [1 of 3]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Content related to this talk from the speaker:
• A Metric for Tuning Parallel Replication in MySQL 5.7
https://jfg-mysql.blogspot.com/2017/02/metric-for-tuning-parallel-replication-mysql-5-7.html
• Better Parallel Replication for MySQL (Binlog Server)
https://medium.com/booking-com-infrastructure/better-parallel-replication-for-mysql-14e2d7857813
• An update on Write Set (parallel replication) bug fix in MySQL 8.0
https://jfg-mysql.blogspot.com/2018/01/an-update-on-write-set-parallel-replication-bug-fix-in-mysql-8-0.html
Binlog Server content from the speaker:
• MySQL Slave Scaling (and more)
https://medium.com/booking-com-infrastructure/mysql-slave-scaling-and-more-a09d88713a20
• Abstracting Binlog Servers and MySQL Master Promotion without Reconfiguring all Slaves
https://medium.com/booking-com-infrastructure/abstracting-binlog-servers-and-mysql-master-promotion-without-reconfiguring-all-slaves-44be1febc8a0
Other parallel replication content from the speaker:
• Slave Group Commit
https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-2-slave-group-commit-459026a141d2
• Optimistic Parallel Replication
https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-4-more-benchmarks-in-production-49ee255043ab 52
Links [2 of 3]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
Related bugs (some fixed, some still open) for the full history:
• Bug#75396: Allow slave_preserve_commit_order without log-slave-updates.
• Bug#81840: Automatic Replication Recovery Does Not Handle Lost Events (applies to MTR)
• Bug#85965: Expose, on the master, counters for monitoring // information quality.
• Bug#85966: Expose, on slaves, counters for monitoring // information quality.
• Bug#85977: The doc. for LOGICAL_CLOCK wrongly references Group Commit
• Bug#89247: Deadlock with MTS when slave_preserve_commit_order = ON
• PS-8030: Multithreaded replica crashes [at STOP SLAVE] inside rli::cannot_safely_rollback()
• I think there is still a deadlock bug in 5.7 (at least in Percona Server 5.7.36),
troubleshooting in progress at HubSpot
• Also, there might be other bugs
53
Links [3 of 3]
(MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
And more related content:
• Solving MySQL Replication Lag with LOGICAL_CLOCK and Calibrated Delay
https://orangematter.solarwinds.com/2017/01/13/solving-mysql-replication-lag-with-logical_clock-and-calibrated-delay/
• How to Fix a Lagging MySQL Replication
https://thoughts.t37.net/fixing-a-very-lagging-mysql-replication-db6eb5a6e15d
• Estimating potential for MySQL 5.7 parallel replication
https://www.percona.com/blog/2016/02/10/estimating-potential-for-mysql-5-7-parallel-replication/
• How Binary Logs (and Filesystems) Affect MySQL Performance
https://www.percona.com/blog/2018/05/04/how-binary-logs-and-filesystems-affect-mysql-performance/
• A Dive Into MySQL Multi-Threaded Replication
https://www.percona.com/blog/a-dive-into-mysql-multi-threaded-replication/
• MySQL Parallel Replication Simulator
https://blog.koehntopp.info/2021/11/08/mysql-parallel-replication.html
54
Thanks !
by Jean-François Gagné
System and MySQL Expert at HubSpot
Presented at Percona Live Austin 2022
jfg.mysql AT gmail DOT com
Twitter: @jfg956

More Related Content

What's hot

MySQL Shell for DBAs
MySQL Shell for DBAsMySQL Shell for DBAs
MySQL Shell for DBAs
Frederic Descamps
 
Galera cluster for high availability
Galera cluster for high availability Galera cluster for high availability
Galera cluster for high availability
Mydbops
 
MySQL Parallel Replication by Booking.com
MySQL Parallel Replication by Booking.comMySQL Parallel Replication by Booking.com
MySQL Parallel Replication by Booking.com
Jean-François Gagné
 
Redo log improvements MYSQL 8.0
Redo log improvements MYSQL 8.0Redo log improvements MYSQL 8.0
Redo log improvements MYSQL 8.0
Mydbops
 
MySQL Database Monitoring: Must, Good and Nice to Have
MySQL Database Monitoring: Must, Good and Nice to HaveMySQL Database Monitoring: Must, Good and Nice to Have
MySQL Database Monitoring: Must, Good and Nice to Have
Sveta Smirnova
 
The consequences of sync_binlog != 1
The consequences of sync_binlog != 1The consequences of sync_binlog != 1
The consequences of sync_binlog != 1
Jean-François Gagné
 
Replication Troubleshooting in Classic VS GTID
Replication Troubleshooting in Classic VS GTIDReplication Troubleshooting in Classic VS GTID
Replication Troubleshooting in Classic VS GTID
Mydbops
 
MySQL InnoDB Cluster - Group Replication
MySQL InnoDB Cluster - Group ReplicationMySQL InnoDB Cluster - Group Replication
MySQL InnoDB Cluster - Group Replication
Frederic Descamps
 
Percona Xtrabackup - Highly Efficient Backups
Percona Xtrabackup - Highly Efficient BackupsPercona Xtrabackup - Highly Efficient Backups
Percona Xtrabackup - Highly Efficient Backups
Mydbops
 
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
Altinity Ltd
 
[2018] MySQL 이중화 진화기
[2018] MySQL 이중화 진화기[2018] MySQL 이중화 진화기
[2018] MySQL 이중화 진화기
NHN FORWARD
 
MaxScale이해와활용-2023.11
MaxScale이해와활용-2023.11MaxScale이해와활용-2023.11
MaxScale이해와활용-2023.11
NeoClova
 
My sql failover test using orchestrator
My sql failover test  using orchestratorMy sql failover test  using orchestrator
My sql failover test using orchestrator
YoungHeon (Roy) Kim
 
MySQL InnoDB Cluster - Advanced Configuration & Operations
MySQL InnoDB Cluster - Advanced Configuration & OperationsMySQL InnoDB Cluster - Advanced Configuration & Operations
MySQL InnoDB Cluster - Advanced Configuration & Operations
Frederic Descamps
 
Query logging with proxysql
Query logging with proxysqlQuery logging with proxysql
Query logging with proxysql
YoungHeon (Roy) Kim
 
MySQL GTID Concepts, Implementation and troubleshooting
MySQL GTID Concepts, Implementation and troubleshooting MySQL GTID Concepts, Implementation and troubleshooting
MySQL GTID Concepts, Implementation and troubleshooting
Mydbops
 
The Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScale
The Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScaleThe Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScale
The Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScale
Colin Charles
 
MySQL Performance Schema in Action
MySQL Performance Schema in ActionMySQL Performance Schema in Action
MySQL Performance Schema in Action
Sveta Smirnova
 
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
Jean-François Gagné
 
MySQL Router REST API
MySQL Router REST APIMySQL Router REST API
MySQL Router REST API
Frederic Descamps
 

What's hot (20)

MySQL Shell for DBAs
MySQL Shell for DBAsMySQL Shell for DBAs
MySQL Shell for DBAs
 
Galera cluster for high availability
Galera cluster for high availability Galera cluster for high availability
Galera cluster for high availability
 
MySQL Parallel Replication by Booking.com
MySQL Parallel Replication by Booking.comMySQL Parallel Replication by Booking.com
MySQL Parallel Replication by Booking.com
 
Redo log improvements MYSQL 8.0
Redo log improvements MYSQL 8.0Redo log improvements MYSQL 8.0
Redo log improvements MYSQL 8.0
 
MySQL Database Monitoring: Must, Good and Nice to Have
MySQL Database Monitoring: Must, Good and Nice to HaveMySQL Database Monitoring: Must, Good and Nice to Have
MySQL Database Monitoring: Must, Good and Nice to Have
 
The consequences of sync_binlog != 1
The consequences of sync_binlog != 1The consequences of sync_binlog != 1
The consequences of sync_binlog != 1
 
Replication Troubleshooting in Classic VS GTID
Replication Troubleshooting in Classic VS GTIDReplication Troubleshooting in Classic VS GTID
Replication Troubleshooting in Classic VS GTID
 
MySQL InnoDB Cluster - Group Replication
MySQL InnoDB Cluster - Group ReplicationMySQL InnoDB Cluster - Group Replication
MySQL InnoDB Cluster - Group Replication
 
Percona Xtrabackup - Highly Efficient Backups
Percona Xtrabackup - Highly Efficient BackupsPercona Xtrabackup - Highly Efficient Backups
Percona Xtrabackup - Highly Efficient Backups
 
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
 
[2018] MySQL 이중화 진화기
[2018] MySQL 이중화 진화기[2018] MySQL 이중화 진화기
[2018] MySQL 이중화 진화기
 
MaxScale이해와활용-2023.11
MaxScale이해와활용-2023.11MaxScale이해와활용-2023.11
MaxScale이해와활용-2023.11
 
My sql failover test using orchestrator
My sql failover test  using orchestratorMy sql failover test  using orchestrator
My sql failover test using orchestrator
 
MySQL InnoDB Cluster - Advanced Configuration & Operations
MySQL InnoDB Cluster - Advanced Configuration & OperationsMySQL InnoDB Cluster - Advanced Configuration & Operations
MySQL InnoDB Cluster - Advanced Configuration & Operations
 
Query logging with proxysql
Query logging with proxysqlQuery logging with proxysql
Query logging with proxysql
 
MySQL GTID Concepts, Implementation and troubleshooting
MySQL GTID Concepts, Implementation and troubleshooting MySQL GTID Concepts, Implementation and troubleshooting
MySQL GTID Concepts, Implementation and troubleshooting
 
The Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScale
The Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScaleThe Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScale
The Proxy Wars - MySQL Router, ProxySQL, MariaDB MaxScale
 
MySQL Performance Schema in Action
MySQL Performance Schema in ActionMySQL Performance Schema in Action
MySQL Performance Schema in Action
 
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
MySQL Parallel Replication (LOGICAL_CLOCK): all the 5.7 (and some of the 8.0)...
 
MySQL Router REST API
MySQL Router REST APIMySQL Router REST API
MySQL Router REST API
 

Similar to MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)

MySQL Replication Evolution -- Confoo Montreal 2017
MySQL Replication Evolution -- Confoo Montreal 2017MySQL Replication Evolution -- Confoo Montreal 2017
MySQL Replication Evolution -- Confoo Montreal 2017
Dave Stokes
 
Demystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash SafetyDemystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash Safety
Jean-François Gagné
 
Demystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash SafetyDemystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash Safety
Jean-François Gagné
 
ConFoo MySQL Replication Evolution : From Simple to Group Replication
ConFoo  MySQL Replication Evolution : From Simple to Group ReplicationConFoo  MySQL Replication Evolution : From Simple to Group Replication
ConFoo MySQL Replication Evolution : From Simple to Group Replication
Dave Stokes
 
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale by ...
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale  by ...[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale  by ...
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale by ...
Insight Technology, Inc.
 
MySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated EnvironmentMySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated Environment
Jean-François Gagné
 
Exploring mysql cluster 7.4
Exploring mysql cluster 7.4Exploring mysql cluster 7.4
Exploring mysql cluster 7.4
Ivan Ma
 
User Camp High Availability Presentation
User Camp High Availability PresentationUser Camp High Availability Presentation
User Camp High Availability Presentation
sankalita chakraborty
 
Oss4b - pxc introduction
Oss4b   - pxc introductionOss4b   - pxc introduction
Oss4b - pxc introduction
Frederic Descamps
 
MariaDB 5.5 and what comes next - Percona Live NYC 2012
MariaDB 5.5 and what comes next - Percona Live NYC 2012MariaDB 5.5 and what comes next - Percona Live NYC 2012
MariaDB 5.5 and what comes next - Percona Live NYC 2012
Colin Charles
 
NoSQL on MySQL - MySQL Document Store by Vadim Tkachenko
NoSQL on MySQL - MySQL Document Store by Vadim TkachenkoNoSQL on MySQL - MySQL Document Store by Vadim Tkachenko
NoSQL on MySQL - MySQL Document Store by Vadim Tkachenko
Data Con LA
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Miguel Araújo
 
My sql 56_roadmap_april2012
My sql 56_roadmap_april2012My sql 56_roadmap_april2012
My sql 56_roadmap_april2012
sqlhjalp
 
My sql 56_roadmap_april2012_zht2
My sql 56_roadmap_april2012_zht2My sql 56_roadmap_april2012_zht2
My sql 56_roadmap_april2012_zht2Ivan Tu
 
MySQL 5.7 what's new
MySQL 5.7 what's newMySQL 5.7 what's new
MySQL 5.7 what's new
Ricky Setyawan
 
MySQL Tech Tour 2015 - 5.7 Whats new
MySQL Tech Tour 2015 - 5.7 Whats newMySQL Tech Tour 2015 - 5.7 Whats new
MySQL Tech Tour 2015 - 5.7 Whats new
Mark Swarbrick
 
Postgres Vienna DB Meetup 2014
Postgres Vienna DB Meetup 2014Postgres Vienna DB Meetup 2014
Postgres Vienna DB Meetup 2014
Michael Renner
 
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)
Aurimas Mikalauskas
 
OSDC 2018 | Scaling & High Availability MySQL learnings from the past decade+...
OSDC 2018 | Scaling & High Availability MySQL learnings from the past decade+...OSDC 2018 | Scaling & High Availability MySQL learnings from the past decade+...
OSDC 2018 | Scaling & High Availability MySQL learnings from the past decade+...
NETWAYS
 
MySQL Replication Update -- Zendcon 2016
MySQL Replication Update -- Zendcon 2016MySQL Replication Update -- Zendcon 2016
MySQL Replication Update -- Zendcon 2016
Dave Stokes
 

Similar to MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK) (20)

MySQL Replication Evolution -- Confoo Montreal 2017
MySQL Replication Evolution -- Confoo Montreal 2017MySQL Replication Evolution -- Confoo Montreal 2017
MySQL Replication Evolution -- Confoo Montreal 2017
 
Demystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash SafetyDemystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash Safety
 
Demystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash SafetyDemystifying MySQL Replication Crash Safety
Demystifying MySQL Replication Crash Safety
 
ConFoo MySQL Replication Evolution : From Simple to Group Replication
ConFoo  MySQL Replication Evolution : From Simple to Group ReplicationConFoo  MySQL Replication Evolution : From Simple to Group Replication
ConFoo MySQL Replication Evolution : From Simple to Group Replication
 
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale by ...
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale  by ...[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale  by ...
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale by ...
 
MySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated EnvironmentMySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated Environment
 
Exploring mysql cluster 7.4
Exploring mysql cluster 7.4Exploring mysql cluster 7.4
Exploring mysql cluster 7.4
 
User Camp High Availability Presentation
User Camp High Availability PresentationUser Camp High Availability Presentation
User Camp High Availability Presentation
 
Oss4b - pxc introduction
Oss4b   - pxc introductionOss4b   - pxc introduction
Oss4b - pxc introduction
 
MariaDB 5.5 and what comes next - Percona Live NYC 2012
MariaDB 5.5 and what comes next - Percona Live NYC 2012MariaDB 5.5 and what comes next - Percona Live NYC 2012
MariaDB 5.5 and what comes next - Percona Live NYC 2012
 
NoSQL on MySQL - MySQL Document Store by Vadim Tkachenko
NoSQL on MySQL - MySQL Document Store by Vadim TkachenkoNoSQL on MySQL - MySQL Document Store by Vadim Tkachenko
NoSQL on MySQL - MySQL Document Store by Vadim Tkachenko
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
My sql 56_roadmap_april2012
My sql 56_roadmap_april2012My sql 56_roadmap_april2012
My sql 56_roadmap_april2012
 
My sql 56_roadmap_april2012_zht2
My sql 56_roadmap_april2012_zht2My sql 56_roadmap_april2012_zht2
My sql 56_roadmap_april2012_zht2
 
MySQL 5.7 what's new
MySQL 5.7 what's newMySQL 5.7 what's new
MySQL 5.7 what's new
 
MySQL Tech Tour 2015 - 5.7 Whats new
MySQL Tech Tour 2015 - 5.7 Whats newMySQL Tech Tour 2015 - 5.7 Whats new
MySQL Tech Tour 2015 - 5.7 Whats new
 
Postgres Vienna DB Meetup 2014
Postgres Vienna DB Meetup 2014Postgres Vienna DB Meetup 2014
Postgres Vienna DB Meetup 2014
 
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)
MySQL Performance Tuning. Part 1: MySQL Configuration (includes MySQL 5.7)
 
OSDC 2018 | Scaling & High Availability MySQL learnings from the past decade+...
OSDC 2018 | Scaling & High Availability MySQL learnings from the past decade+...OSDC 2018 | Scaling & High Availability MySQL learnings from the past decade+...
OSDC 2018 | Scaling & High Availability MySQL learnings from the past decade+...
 
MySQL Replication Update -- Zendcon 2016
MySQL Replication Update -- Zendcon 2016MySQL Replication Update -- Zendcon 2016
MySQL Replication Update -- Zendcon 2016
 

More from Jean-François Gagné

Autopsy of a MySQL Automation Disaster
Autopsy of a MySQL Automation DisasterAutopsy of a MySQL Automation Disaster
Autopsy of a MySQL Automation Disaster
Jean-François Gagné
 
MySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated EnvironmentMySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated Environment
Jean-François Gagné
 
MySQL/MariaDB Parallel Replication: inventory, use-case and limitations
MySQL/MariaDB Parallel Replication: inventory, use-case and limitationsMySQL/MariaDB Parallel Replication: inventory, use-case and limitations
MySQL/MariaDB Parallel Replication: inventory, use-case and limitations
Jean-François Gagné
 
The two little bugs that almost brought down Booking.com
The two little bugs that almost brought down Booking.comThe two little bugs that almost brought down Booking.com
The two little bugs that almost brought down Booking.com
Jean-François Gagné
 
Autopsy of an automation disaster
Autopsy of an automation disasterAutopsy of an automation disaster
Autopsy of an automation disaster
Jean-François Gagné
 
How Booking.com avoids and deals with replication lag
How Booking.com avoids and deals with replication lagHow Booking.com avoids and deals with replication lag
How Booking.com avoids and deals with replication lag
Jean-François Gagné
 
MySQL Parallel Replication: inventory, use-case and limitations
MySQL Parallel Replication: inventory, use-case and limitationsMySQL Parallel Replication: inventory, use-case and limitations
MySQL Parallel Replication: inventory, use-case and limitations
Jean-François Gagné
 
MySQL Parallel Replication: inventory, use-cases and limitations
MySQL Parallel Replication: inventory, use-cases and limitationsMySQL Parallel Replication: inventory, use-cases and limitations
MySQL Parallel Replication: inventory, use-cases and limitations
Jean-François Gagné
 
Riding the Binlog: an in Deep Dissection of the Replication Stream
Riding the Binlog: an in Deep Dissection of the Replication StreamRiding the Binlog: an in Deep Dissection of the Replication Stream
Riding the Binlog: an in Deep Dissection of the Replication Stream
Jean-François Gagné
 

More from Jean-François Gagné (9)

Autopsy of a MySQL Automation Disaster
Autopsy of a MySQL Automation DisasterAutopsy of a MySQL Automation Disaster
Autopsy of a MySQL Automation Disaster
 
MySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated EnvironmentMySQL Scalability and Reliability for Replicated Environment
MySQL Scalability and Reliability for Replicated Environment
 
MySQL/MariaDB Parallel Replication: inventory, use-case and limitations
MySQL/MariaDB Parallel Replication: inventory, use-case and limitationsMySQL/MariaDB Parallel Replication: inventory, use-case and limitations
MySQL/MariaDB Parallel Replication: inventory, use-case and limitations
 
The two little bugs that almost brought down Booking.com
The two little bugs that almost brought down Booking.comThe two little bugs that almost brought down Booking.com
The two little bugs that almost brought down Booking.com
 
Autopsy of an automation disaster
Autopsy of an automation disasterAutopsy of an automation disaster
Autopsy of an automation disaster
 
How Booking.com avoids and deals with replication lag
How Booking.com avoids and deals with replication lagHow Booking.com avoids and deals with replication lag
How Booking.com avoids and deals with replication lag
 
MySQL Parallel Replication: inventory, use-case and limitations
MySQL Parallel Replication: inventory, use-case and limitationsMySQL Parallel Replication: inventory, use-case and limitations
MySQL Parallel Replication: inventory, use-case and limitations
 
MySQL Parallel Replication: inventory, use-cases and limitations
MySQL Parallel Replication: inventory, use-cases and limitationsMySQL Parallel Replication: inventory, use-cases and limitations
MySQL Parallel Replication: inventory, use-cases and limitations
 
Riding the Binlog: an in Deep Dissection of the Replication Stream
Riding the Binlog: an in Deep Dissection of the Replication StreamRiding the Binlog: an in Deep Dissection of the Replication Stream
Riding the Binlog: an in Deep Dissection of the Replication Stream
 

Recently uploaded

Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
Safe Software
 
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
Sri Ambati
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Tobias Schneck
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
KatiaHIMEUR1
 
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdfFIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance
 
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdfSmart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
91mobiles
 
Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
Alison B. Lowndes
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
Product School
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Jeffrey Haguewood
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
Jemma Hussein Allen
 
PCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase TeamPCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase Team
ControlCase
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
DianaGray10
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
RTTS
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Product School
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
Elena Simperl
 
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
Ana-Maria Mihalceanu
 
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
DianaGray10
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
James Anderson
 
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance
 

Recently uploaded (20)

Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
 
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
 
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdfFIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdf
 
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdfSmart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
 
Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
 
The Future of Platform Engineering
The Future of Platform EngineeringThe Future of Platform Engineering
The Future of Platform Engineering
 
PCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase TeamPCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase Team
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
 
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
 
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
 
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...
 
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdfFIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
 
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdfFIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf
 

MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)

  • 1. MySQL Parallel Replication: all the 5.7 and 8.0 details (LOGICAL_CLOCK) by Jean-François Gagné System and MySQL Expert at HubSpot Presented at Percona Live Austin 2022 jfg.mysql AT gmail DOT com Twitter: @jfg956
  • 2. Benchmark Preview (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) 2 Replication throughput (TPS) without and with parallel replication
  • 3. Terminology [1 of 2] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) MySQL Terminology Updates in 2020: https://dev.mysql.com/blog-archive/mysql-terminology-updates/ • master / slave à primary / replica • the master of a slave à the source of a replica This presentation uses above terminology, with below additions: • Multi-Threaded Slave (MTS) à Multi-Threaded Replication (MTR) • intermediary master à intermediary source or non-primary source (Exceptions for citing content from before the terminology change) 3
  • 4. Terminology [2 of 2] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) From MySQL 8.0.22 (not in 5.7), new commands were introduced: • START/STOP SLAVE à START/STOP REPLICA • And more… • Some missing: START REPLICA UNTIL SQL_AFTER_MTS_GAPS From 8.0.26 (not in 5.7 either), new variables were introduced: • log_slave_updates à log_replica_updates • slave_parallel_workers à replica_parallel_workers • And more… Because this talk applies to 5.7, and not everyone is on 8.0-latest, this presentation uses old command and variable names
  • 5. Summary (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) • Replication reminders • Why parallel replication is important (and complicated) • How to enable parallel replication and what is LOGICAL_CLOCK • How parallel replication works on replicas • How dependency tracking works on primaries • Benchmarks • Closing thoughts 5
  • 6. MySQL Replication (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) • The primary records transactions in a journal (binary logs), and replicas • downloads the journal and saves them locally in the relay logs (IO thread) • executes the relay logs on their local database (SQL thread) • could also write binlogs to be themselves a source (log-slave-updates)
  • 7. Other Replication Concepts: Durability (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Persistence / Durability (D of ACID): • Is there data lost after a crash ? (different types of crash: MySQL process (mysqld) or OS (Linux)) • Parameters controlling durability: sync_binlog and trx_commit (trx_commit is short for innodb_flush_logs_at_trx_commit) • High Durability (HD) or no data lost after crashes (sync_binlog = 1 and trx_commit = 1) (most expensive / slower because flush to disk after each trx) • No Durability (ND sometimes called low durability) (sync_binlog = 0 and trx_commit = 2) (faster, with data lost on OS crash, but no loss on mysqld crash) • (Data lost for mysqld crash in case trx_commit = 0 but not covered here) 7
  • 8. Other Replication Concepts: Group Commit (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Group Commit is an optimization making high durability faster • Before Group Commit (GC), each trx needed its disk flush • With GC, many trx can be persisted in the same flush à better write throughput on primaries with concurency • (GC was introduced in MariaDB xx and then in MySQL 5.6) • (Single threaded replication cannot Group Commit) • (MariaDB 10.0 introduced Replica Group Commit) 8
  • 9. From Single to Multi-Threaded Replication (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Without MTR, a single CPU is used for applying writes on replicas: • on the primary, many CPUs are used for writes (one per session) WO MTR, a single read IO can be “in flight” for writes on replicas: • this is a bottleneck, especially for high latency disks (cloud) • on primary, many IOs can be in flights for writes (one per session) WO MTR, no group commit on replicas: • on the p., a single flush can persist many trx in binlogs and redo logs • on replicas, flushing each trx is a bottleneck Multi-Threaded Replication is a solution to all above problems 9
  • 10. Challenges of Multi-Threaded Replication (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) But running transactions in parallel on replica is not trivial: • Updating a row cannot be run in parallel with the insert of this row • MTR implementation needs to keep data consistency Enabling MTR is also not straightforward: • Many parameters and complex tunning MTR also still has some rough edges: • In 5.7, START SLAVE UNTIL not supported with MTR (ok from 8.0.0) • Latest -1 Percona Server 5.7 crashes on STOP SLAVE with MTR (PS-8030: 5.7.27 to .36 affected and 8.0.17 to .21, Oracle MySQL not affected) • I have also seen deadlock with PS 5.7.36, and rumor of another crash (to be investigated, unknown if both in Percona Server and Oracle MySQL)
  • 11. Solutions for MTR Data Consistency (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) • MySQL 5.6: schema based parallel replication • MariaDB 10.0: group commit based parallel replication • MariaDB 10.1: optimistic parallel replication • Improvement on 10.0, probably the fastest solution, but high CPU cost • MySQL 5.7: same schema parallel replication (Logical Clock) • Not as fast as optimistic, but not wasteful in CPU • MySQL 8.0: Write-Set improvement (still Logical Clock) • Lot to say, including need for RBR and back-ported in 5.7.22 This talk is about Logical Clock in 5.7 and 8.0 11
  • 12. Solutions for MTR Data Consistency (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) • MySQL 5.6: schema based parallel replication • MariaDB 10.0: group commit based parallel replication • MariaDB 10.1: optimistic parallel replication • Improvement on 10.0, probably the fastest solution, but high CPU cost • MySQL 5.7: same schema parallel replication (Logical Clock) • Not as fast as optimistic, but not wasteful in CPU • MySQL 8.0: Write-Set improvement (still Logical Clock) • Lot to say, including need for RBR and back-ported in 5.7.22 This talk is about Logical Clock in 5.7 and 8.0 12
  • 13. Enabling Parallel Replication (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Set slave_parallel_type to LOGICAL_CLOCK • This is the new default in 8.0.27 (was DATABASE before (schema-based solution of 5.6), including in 5.7) (in 8.0.26, this variable is deprecated for dropping DATABASE support) And set slave_preserve_commit_order to ON • This is also the new default in 8.0.27(was OFF before, including in 5.7) (more about this parameter later in the talk) And set slave_parallel_workers to 2 or more • New default to 4 in 8.0.27 (was 0 before including in 5.7) These enable MTR, but are not sufficient for best throughput 13
  • 14. What is LOGICAL_CLOCK ? (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) From the MySQL Reference Manual for LOGICAL_CLOCK: • Transactions that are part of the same binary log group commit on a source are applied in parallel on a replica. • But this is not 100% exact (Bug#85977) When using LOGICAL_CLOCK, MTR is scheduling trx on replica using dependency tracking information from the binary logs: • We will name these information Intervals • By default, trx intervals are generated with Commit Timestamps (which is close, but not strictly the same as group commit) • Much better intervals can be generated with Write Set (and Write Set is completely different to group commit) 14
  • 15. Intervals (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Each trx is tagged with two numbers in the binlogs (starting with 5.7): • sequence_number: increasing id for each trx (not to confuse with GTID) • last_committed: id of the latest trx on which this trx depends The last_committed / sequence_number is the trx interval Here an example of intervals from a real system: 15 mysqlbinlog $f | awk '$11 ~ "last_committed"{print $1,$2,$11,$12}' [...] #170206 20:08:33 last_committed=6201 sequence_number=6203 #170206 20:08:33 last_committed=6203 sequence_number=6204 #170206 20:08:33 last_committed=6203 sequence_number=6205 #170206 20:08:33 last_committed=6203 sequence_number=6206 #170206 20:08:33 last_committed=6205 sequence_number=6207
  • 16. Parallel Scheduling on Replicas [1 of 4] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) 16 A transaction can be started when its last_committed and all previous transactions have been executed The intervals below mean that: • When transaction #6203 (and all previous) have been executed, transactions #6204, #6205 and #6206 can be started • When transaction #6205 (and all previous) have been executed, transaction #6207 can be started #170206 20:08:33 last_committed=6201 sequence_number=6203 #170206 20:08:33 last_committed=6203 sequence_number=6204 #170206 20:08:33 last_committed=6203 sequence_number=6205 #170206 20:08:33 last_committed=6203 sequence_number=6206 #170206 20:08:33 last_committed=6205 sequence_number=6207
  • 17. Parallel Scheduling on Replicas [2 of 4] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) 17 Trx #6204 and #6205 (and #6206) can be run in parallel but #6205 could complete before #6204 slave_preserve_commit_order (SPCO) does what you expect: • If SPCO is OFF, #6205 can commit before #6204 • If SPCO is ON , #6205 waits for #6204 completion before committing SPCO to OFF generates data states that never existed on the primary • Use with care (you probably want ON most of the time) #170206 20:08:33 last_committed=6201 sequence_number=6203 #170206 20:08:33 last_committed=6203 sequence_number=6204 #170206 20:08:33 last_committed=6203 sequence_number=6205 #170206 20:08:33 last_committed=6203 sequence_number=6206 #170206 20:08:33 last_committed=6205 sequence_number=6207
  • 18. Parallel Scheduling on Replicas [3 of 4] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) SPCO to OFF generates data states that never existed on the primary • Use with care (you probably want ON most of the time) Also, SPCO to OFF generates temporary gaps in GTIDs • For filling the gaps, use START SLAVE UNTIL SQL_AFTER_MTS_GAPS SPCO to ON needed log_slave_updates until 8.0.19 (Bug#75396) Before 5.7.33 and 8.0.23, SPCO to ON could deadlock (Bug#89247) • If replication gets stuck, kill -9 mysqld (no data lost unless trx_commit = 0) • Thanks Percona / Venkatesh for the fixing this, but… • But I have recently seen deadlocks in PS 5.7.36: not fully fixed 18
  • 19. Parallel Scheduling on Replicas [4 of 4] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) 19 When run in parallel, trx could fail to execute (below from the doc): • If replication fails to execute a trx because of an InnoDB deadlock or the trx's execution time exceeded innodb_lock_wait_timeout… Such failed trx will be retried (below more from the doc): • …it automatically retries slave_transaction_retries times before stopping with an error. The column COUNT_TRANSACTIONS_RETRIES from the the P_S table replication_applier_status indicates how many retries have happened, and the table repl._app._status_by_worker contains details about the retries
  • 20. Reminder: What is LOGICAL_CLOCK ? (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) From the MySQL Reference Manual for LOGICAL_CLOCK: • Transactions that are part of the same binary log group commit on a source are applied in parallel on a replica. • But this is not 100% exact (Bug#85977) When using LOGICAL_CLOCK, MTR is scheduling trx on replica using dependency tracking information from the binary logs: • We will name these information Intervals • By default, trx intervals are generated with Commit Timestamps (which is close, but not strictly the same as group commit) • Much better intervals can be generated with Write Set (and Write Set is completely different to group commit) 20
  • 21. Commit Timestamps [1 of 3] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Commit Timestamps (CTS) is the default for 5.7 and 8.0 (binlog_transaction_dependency_tracking = COMMIT_ORDER) It is a dependency tracking method exploiting parallelism on the source to generate the intervals: • While running a DML, the current trx notes the last committed trx (this is where last_committed from mysqlbinlog is coming from) (this is close to Group Commit, but not strictly the same) Without tunning, CTS is not producing efficient intervals (by default, MTR will not give much better replication speed) 21
  • 22. Commit Timestamps [2 of 3] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) In below, the update id=3 ran while the update id=1 was not yet committed: • Insert id=4 and update id=1 can run in parallel on a replica • (Note that nothing commit at the same time à no group commit) Session #1: CREATE TABLE t(id INT PRIMARY KEY, v INT); Session #1: INSERT INTO t VALUES (1,1),(2,2),(3,3); Session #2: BEGIN; UPDATE t SET v = 11 WHERE id = 1; Session #3: UPDATE t SET v = 33 WHERE id = 3; Session #1: INSERT INTO t VALUES (4,4); Session #2: COMMIT; #220410 11:21:47 last_committed=0 sequence_number=1 -- CREATE #220410 11:21:49 last_committed=1 sequence_number=2 –- INSERT id=1,2,3 #220410 11:21:55 last_committed=2 sequence_number=3 -- UPDATE id=3 #220410 11:21:59 last_committed=3 sequence_number=4 –- INSERT id=4 #220410 11:22:03 last_committed=2 sequence_number=5 -- UPDATE id=1 22
  • 23. Commit Timestamps [3 of 3] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) More binlog: note that the DML order is different from the commit order mysqlbinlog $f | awk '$11 ~ "last_committed"{print $1,$2,$11,$12} $10 ~ "_rows|Xid"{print $1,$2,$10}' #220410 11:21:47 last_committed=0 sequence_number=1 -- CREATE #220410 11:21:49 last_committed=1 sequence_number=2 -- INSERT id=1,2,3 #220410 11:21:49 Write_rows: #220410 11:21:49 Xid #220410 11:21:55 last_committed=2 sequence_number=3 -- UPDATE id=3 #220410 11:21:55 Update_rows: #220410 11:21:55 Xid #220410 11:21:59 last_committed=3 sequence_number=4 -- INSERT id=4 #220410 11:21:59 Write_rows: #220410 11:21:59 Xid #220410 11:22:03 last_committed=2 sequence_number=5 -- UPDATE id=1 #220410 11:21:52 Update_rows: #220410 11:22:03 Xid 23
  • 24. Tunning Commit Timestamp (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) If very few trx run in parallel, or if commit is quick (auto-commit, very fast disks or low durability), CTS do not identify much parallelism (low durability: sync_binlog != 1 and trx_commit != 1) A solution is to delay commit for having more transactions running in parallel, which leads to better interval generation: • binlog_group_commit_sync_delay • binlog_group_commit_sync_no_delay_count I call this slowing-down the primary to speed-up the replicas But performing this tunning just looking at replica speed is tedious24
  • 25. Intervals Quality (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) To ease the tunning, I came-up with a metric for interval quality: • the Average Modified Interval Length (AMIL) It is based on the following observation: • the larger the intervals are, the better MTR throughput will be All the details about the AMIL in http://jfg-mysql.blogspot.com/2017/02/metric-for-tuning-parallel-replication-mysql-5-7.html Computing the AMIL needs parsing the output of mysqlbinlog for extracting interval information, MySQL should make this easier: • Bug#85965: Expose, on the master, counters for monitoring // information quality • Bug#85966: Expose, on slaves, counters for monitoring // information quality 25
  • 26. Tunning Commit Timestamps with AMIL [1 of 2] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Monitoring the AMIL provides an immediate tunning feedback: • After increasing binlog_group_commit_sync_delay, the AMIL should increase (if not, increase more for better parallel replication, or rollback for not penalizing commit latency) But while doing this, also look at the commit-rate of the primary: • If the commit-rate noticeably drop, consider reverting the tuning (it is penalizing the application throughput) But penalizing throughput might be ok for solving replication lag: • lowering the primary throughput from 200 to 100 tps might be acceptable to increase replica throughput from 50 to 150 tps 26
  • 27. Tunning Commit Timestamps with AMIL [2 of 2] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) AMIL without and with tuning on four primaries: (speed-up the replicas by increasing binlog_group_commit_sync_delay) 27
  • 28. Other Commit Timestamp Tunning (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) 28 To alleviate drop in throughput caused by delay, increase the number of operations done in parallel in MySQL Also, more transactions run in parallel by the application leads to more parallelism identified by Commit Timestamps and better replication speed on replicas
  • 29. CTS and Non-Primary Sources (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Without MTR, CTS leads to no parallelism identification on replicas Similarly, even with MTR and compared to the primary, less parallelism is identified on replicas by Commit Timestamps: • MTR loses efficiency when replicating from a non-primary source Also, after promoting a replica as the new primary, its binlogs have smaller intervals than the one from the old primary: • Using these leads to less efficient MTR (eg: after restore backup) With SBR, no good solutions to that (except Binlog Servers) https://medium.com/booking-com-infrastructure/better-parallel-replication-for-mysql-14e2d7857813 29
  • 30. Write Set [1 of 4] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) If using RBR, a better dependency tracking method can be used: • Write Set was developed for Group Replication (in MySQL 5.7) • It can also be used for dependency tracking (introduced in 8.0) • It was back-ported in 5.7.22 (thanks Oracle for this back-port) The Write Set is a data structure used to generate intervals: • It stores a hash of all primary keys (and unique keys) modified by trx (tunable size with binlog_transaction_dependency_history_size) The intervals generated by Write Set on replicas are as good as on the primary (and better if the primary is using CTS) (Write Set can be used on a RBR replica of a SBR source) 30
  • 31. Write Set [2 of 4] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Enabling Write Set dependency tracking (on both primary and replicas): • Set BTDT to WRITESET (or WRITESET_SESSION) (BTDT for binlog_transaction_dependency_tracking) • This needs TWSE to XXHASH64 (or MURMUR32) (TWSE for transaction_write_set_extraction) • Which also needs binlog_format to ROW With BTDT to WRITESET, a second trx run by a single session does not depend on the first: • With BTDT to WRITESET, SPCO should be enabled on replica • Or set BTDT to WRITESET_SESSION on the primary (not a good setting for replicas, we will come back to this) 31
  • 32. Write Set [3 of 4] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) In below, updates id=3 and id=1, and insert id=4 can execute in parallel: • Without SPCO (and the update), insert id=4 could commit before id=1,2,3 SET GLOBAL binlog_transaction_dependency_tracking = WRITESET; Session #1: CREATE TABLE t(id INT PRIMARY KEY, v INT); Session #1: INSERT INTO t VALUES (1,1),(2,2),(3,3); Session #2: BEGIN; UPDATE t SET v = 11 WHERE id = 1; Session #3: UPDATE t SET v = 33 WHERE id = 3; Session #1: INSERT INTO t VALUES (4,4); Session #2: COMMIT; #220410 12:01:19 last_committed=0 sequence_number=1 -- CREATE #220410 12:01:21 last_committed=1 sequence_number=2 -- INSERT id=1,2,3 #220410 12:01:29 last_committed=2 sequence_number=3 -- UPDATE id=3 #220410 12:01:34 last_committed=1 sequence_number=4 -- INSERT id=4 #220410 12:01:38 last_committed=2 sequence_number=5 -- UPDATE id=1 32
  • 33. Barriers in Parallel Scheduling on Replicas (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) #220410 12:01:19 last_committed=0 sequence_number=1 -- CREATE #220410 12:01:21 last_committed=1 sequence_number=2 -- INSERT id=1,2,3 #220410 12:01:29 last_committed=2 sequence_number=3 -- UPDATE id=3 #220410 12:01:34 last_committed=1 sequence_number=4 -- INSERT id=4 #220410 12:01:38 last_committed=2 sequence_number=5 -- UPDATE id=1 33 In below, the update id=3 (seq_no 3) depends on the multi-value insert: • Parallel transaction scheduling blocks until the dependent trx completes A dependent transaction acts as a barrier in parallel scheduling: • Even if insert id=4 could be run at the same time as the multi-value insert, the update id=3 prevents this This might change in a future MySQL version (it is also the M in AMIL – Modified)
  • 34. Write Set [4 of 4] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) If running with slave_preserve_commit_order to OFF, WRITESET_SESSION avoid inconsistencies: insert id=4 runs after id=1,2,3 SET GLOBAL binlog_transaction_dependency_tracking = WRITESET_SESSION; Session #1: CREATE TABLE t(id INT PRIMARY KEY, v INT); Session #1: INSERT INTO t VALUES (1,1),(2,2),(3,3); Session #2: BEGIN; UPDATE t SET v = 11 WHERE id = 1; Session #3: UPDATE t SET v = 33 WHERE id = 3; Session #1: INSERT INTO t VALUES (4,4); Session #2: COMMIT; #220410 12:12:30 last_committed=0 sequence_number=1 -- CREATE #220410 12:12:32 last_committed=1 sequence_number=2 -- INSERT id=1,2,3 #220410 12:12:39 last_committed=2 sequence_number=3 -- UPDATE id=3 #220410 12:12:43 last_committed=2 sequence_number=4 -- INSERT id=4 #220410 12:12:47 last_committed=2 sequence_number=5 -- UPDATE id=1 34
  • 35. Write Set and Non-Primary Sources (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) We can enable Write Set on intermediary source: • Even on a single-threaded replica • But we should use BTDT to WRITESET because WRITESET_SESSION would create new dependencies (without MTR, all intervals of WRITESET_SESSION are of size one) • This solves the CTS interval degradation on intermediary source This was done to get the first (E*) benchmark results (in 2017): • Primary in 5.7 (pre .22), intermediary source in 8.0 with WRITESET, replica with MTR This is useful to “try” MTR before big migration: • SBR to RBR for Write Set, or 5.6 to 5.7 for MTR 35
  • 36. AMIL for Write Set [1 of 2] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) AMIL from a CTS primary and a Write Set intermediary source: • The primary does not need RBR, only the intermediary source does • Write Set AMIL is usually better than untuned and tunned CTS 36
  • 37. AMIL for Write Set [2 of 2] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Sometimes, combining delay with Write Set gives a better AMIL: • Discovered by accident, tunning opportunity?, more research needed (Write Set AMIL below 3x better with delay compared to without) 37
  • 38. More Tunning: Smaller Transaction Sizes (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Because transaction scheduling sometimes/often blocks (barriers and dependencies), some replica workers might be idle or waiting on another transaction completion: • In the worse case, many workers are waiting for a single big trx These big trx reduce the potential for speedups: • Avoiding big transactions in the application will make MTR faster Ironically, when commit was expensive, or to amortize round-trip to database, doing many things in a single (big) trx was / is the right thing to do, but this makes MTR run slower 38
  • 39. Recommended Settings for MTR Readiness (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Without RBR, CTS is the only available dependency tracking à not worth “preparing” for MTR (implement and tune when needed) With RBR and MySQL 8.0: • binlog_transaction_dependency_tracking to WRITESET • slave_parallel_type to LOGICAL_CLOCK (new default in 8.0.27) • slave_preserve_commit_order to ON (new default in 8.0.27) With RBR and MySQL 5.7, all above plus: • transaction_write_set_extraction to XXHASH64 (this is the default in 8.0) With these, faster repl. only needs updating slave_parallel_workers
  • 40. Thoughts on MySQL 8.0.27 new Defaults (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) In MySQL 8.0.27 we have: • slave_parallel_type to LOGICAL_CLOCK • slave_preserve_commit_order to ON (was OFF before) • and slave_parallel_workers to 4 in 8.0.27 (was 0 before) But intervals are still generated with Commit Timestamps: • Having 4 workers will probably not improve replication speed much • But this opens possibility for MTR rough-edges Enabling parallel replication by default might be a little early and when these bugs will be fixed, Write Set should also be enabled 40
  • 41. Benchmark (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Benchmarks done a few years ago and at HubSpot: • Used 0 to 256 threads (slave_parallel_workers) • E1 to E8: different environments / workloads (2017-2018) • F1 to F4: more environments / workloads (from HubSpot) • HD: High Durability: sync_binlog = 1 and trx_commit = 1 • ND: No Durability: sync_binlog = 0 and trx_commit = 2 • IS: Intermediary Source: with log_slave_updates • NO: No Order: slave_preserve_commit_order = OFF • (WO: with order: slave_preserve_commit_order = ON) • (RB: replica with binary logs: without log_slave_updates) 41
  • 42. Benchmark Presentation [1 of 2] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) 42 E5 IS-HD Single-Threaded: 6138 seconds E5 IS-ND Single-Threaded: 2238 seconds Going from HD to ND for E5 IS gives a speedup of 2.74
  • 43. Benchmark Presentation [2 of 2] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) All in a single graph: • HD et ND: previous speedups • HD vs ND single threaded: the speedup from HD to ND 43
  • 44. Old Benchmark Setup (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) The test environments involve three nodes: +---+ +---+ +---+ | A | -------> | B | -------> | C | +---+ +---+ +---+ • A is a production primary (MySQL 5.6 or 5.7: this was in 2017-2018) • Some of the primaries are SBR, some are RBR • B is a MySQL 8.0.3 Intermediary Source in RBR with Write Set (transaction_write_set_extraction = XXHASH64) (binlog_transaction_dependency_tracking = WRITESET) • C is a replica where timing is done 44
  • 45. Previous Benchmark Results [1 of 2] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022)
  • 46. Previous Benchmark Results [2 of 2] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Observations: • High variation of speedups, very workload dependent (probably possible to do better if spending time optimizing the application) • HD gives better speedups than ND (normal because group commit) • ND gives some interesting speedups, but do not expect 16x faster not possible to use all CPUs of modern hardware on replica for writes
  • 47. HubSpot Benchmark Setup (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) The test environments involve two nodes: +---+ +---+ | A | -------> | B | +---+ +---+ • A is a production primary (Percona Server 5.7.36 with Write Set) (transaction_write_set_extraction = XXHASH64) (binlog_transaction_dependency_tracking = WRITESET) • B is a replica where timing is done 47
  • 48. HubSpot Benchmarks (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Observations: • Higher HD speedups for F{1,2} than E* • Better HD vs ND single-threaded for F* than E* (F1 being much better) Something is different between E and F.
  • 49. Benchmark Discussion (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Previous was on local SSD, HubSpot is EBS in AWS • Disk-sync more expensive for HubSpot • Which explains better HD results • Also explains better HD vs ND single-threaded speedup • (Also, make benchmarking hard because AWS variance) (HubSpot is running production with ND + semi-sync) Also, HubSpot is running with the previously recommended settings: • Write Set enabled on primaries and replicas • slave_parallel_type to LOGICAL_CLOCK • slave_preserve_commit_order to ON à Ready for MTR by updating slave_parallel_workers 49
  • 50. Other Benchmarks Results (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) I also have results for: • WO, and NO vs WO • RB, and IS vs RB Too long to present everything, contact me if interested TL&DR: • NO is faster than WO • RB is faster than IS • Both highly variable depending on workload and durability 50
  • 51. Other Parallel Replication Subjects (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) MySQL Parallel Replication Simulator https://blog.koehntopp.info/2021/11/08/mysql-parallel-replication.html Replica Group Commit (Slave Group Commit in MariaDB) https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-2-slave-group-commit-459026a141d2 • Replicating trx sequentially, in many threads for Group Committing • Almost as good as ND single-threaded speed, with benefit of HD Optimistic Parallel Replication (in MariaDB) https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-4-more-benchmarks-in-production-49ee255043ab • Go beyond scheduling barriers on replica, achieving replication prefetching Facebook partial transaction parallel execution in MySQL 5.6 https://www.percona.com/live/e18/sessions/faster-mysql-replication-using-row-dependencies • Using Write Set on replicas for fine grain parallel statement excution 51
  • 52. Links [1 of 3] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Content related to this talk from the speaker: • A Metric for Tuning Parallel Replication in MySQL 5.7 https://jfg-mysql.blogspot.com/2017/02/metric-for-tuning-parallel-replication-mysql-5-7.html • Better Parallel Replication for MySQL (Binlog Server) https://medium.com/booking-com-infrastructure/better-parallel-replication-for-mysql-14e2d7857813 • An update on Write Set (parallel replication) bug fix in MySQL 8.0 https://jfg-mysql.blogspot.com/2018/01/an-update-on-write-set-parallel-replication-bug-fix-in-mysql-8-0.html Binlog Server content from the speaker: • MySQL Slave Scaling (and more) https://medium.com/booking-com-infrastructure/mysql-slave-scaling-and-more-a09d88713a20 • Abstracting Binlog Servers and MySQL Master Promotion without Reconfiguring all Slaves https://medium.com/booking-com-infrastructure/abstracting-binlog-servers-and-mysql-master-promotion-without-reconfiguring-all-slaves-44be1febc8a0 Other parallel replication content from the speaker: • Slave Group Commit https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-2-slave-group-commit-459026a141d2 • Optimistic Parallel Replication https://medium.com/booking-com-infrastructure/evaluating-mysql-parallel-replication-part-4-more-benchmarks-in-production-49ee255043ab 52
  • 53. Links [2 of 3] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) Related bugs (some fixed, some still open) for the full history: • Bug#75396: Allow slave_preserve_commit_order without log-slave-updates. • Bug#81840: Automatic Replication Recovery Does Not Handle Lost Events (applies to MTR) • Bug#85965: Expose, on the master, counters for monitoring // information quality. • Bug#85966: Expose, on slaves, counters for monitoring // information quality. • Bug#85977: The doc. for LOGICAL_CLOCK wrongly references Group Commit • Bug#89247: Deadlock with MTS when slave_preserve_commit_order = ON • PS-8030: Multithreaded replica crashes [at STOP SLAVE] inside rli::cannot_safely_rollback() • I think there is still a deadlock bug in 5.7 (at least in Percona Server 5.7.36), troubleshooting in progress at HubSpot • Also, there might be other bugs 53
  • 54. Links [3 of 3] (MySQL Parallel Replication Logical Clock – Percona Live Austin 2022) And more related content: • Solving MySQL Replication Lag with LOGICAL_CLOCK and Calibrated Delay https://orangematter.solarwinds.com/2017/01/13/solving-mysql-replication-lag-with-logical_clock-and-calibrated-delay/ • How to Fix a Lagging MySQL Replication https://thoughts.t37.net/fixing-a-very-lagging-mysql-replication-db6eb5a6e15d • Estimating potential for MySQL 5.7 parallel replication https://www.percona.com/blog/2016/02/10/estimating-potential-for-mysql-5-7-parallel-replication/ • How Binary Logs (and Filesystems) Affect MySQL Performance https://www.percona.com/blog/2018/05/04/how-binary-logs-and-filesystems-affect-mysql-performance/ • A Dive Into MySQL Multi-Threaded Replication https://www.percona.com/blog/a-dive-into-mysql-multi-threaded-replication/ • MySQL Parallel Replication Simulator https://blog.koehntopp.info/2021/11/08/mysql-parallel-replication.html 54
  • 55. Thanks ! by Jean-François Gagné System and MySQL Expert at HubSpot Presented at Percona Live Austin 2022 jfg.mysql AT gmail DOT com Twitter: @jfg956