This document summarizes new features in MySQL replication introduced in versions 5.6 and 5.7. Key features discussed include binary log group commit for improved performance, optimized row-based replication with partial binary logging, multi-threaded slave replication, global transaction identifiers for topologies with multiple masters, transactional metadata storage, and binary log event checksums. The document provides examples and explanations of how these features improve high availability, scalability and reliability of MySQL replication deployments.

1. Advanced
MySQL
ReplicaCon
Architectures
and
Latest
Developments
Mark
Swarbrick
Technical
Presales
Consultant
MySQL
UK&I
London
October,
2014
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|

2. Safe
Harbor
Statement
The
following
is
intended
to
outline
our
general
product
direcCon.
It
is
intended
for
informaCon
purposes
only,
and
may
not
be
incorporated
into
any
contract.
It
is
not
a
commitment
to
deliver
any
material,
code,
or
funcConality,
and
should
not
be
relied
upon
in
making
purchasing
decisions.
The
development,
release,
and
Cming
of
any
features
or
funcConality
described
for
Oracle’s
products
remains
at
the
sole
discreCon
of
Oracle.
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
2

3. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Agenda
1
2
3
4
5
Background
Generally
Available
Features
in
MySQL
5.6
Next
GeneraCon
ReplicaCon
Features
in
MySQL
5.7
DMR
What
is
Next?
Summary
3

4. Background:
ReplicaCon
Components
relay log
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Insert...
Insert...
B
binary log
Insert...
Insert...
A
binary log
Client
Sender
thread
Receiver
thread
Applier
Threads
Network
Master
info
Relay
Log
Info

5. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
§ Binary
Log
§ File
based
logical
log
that
records
the
changes
on
the
master.
§ Statement
or
Row
based
format
(may
be
intermixed).
§ TransacCons
are
split
into
groups
of
events.
§ Control
events:
Rotate,
Format
DescripCon,
GCd,
...
Layout of a
Binary Log File
Background:
ReplicaCon
Components
BEGIN E1 E2 ... COMMIT BEGIN E1 E2 ... COMMIT

6. Background:
What
is
ReplicaCon
Used
For?
write clients
More
reads?
More
slaves!
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
§ Read
scale-­‐out
M S
S
S
S
M
write clients read clients read clients

7. What
about
Write
scale-­‐out?
More
writes?
More
Masters?
M?
M M?
write clients write clients
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
M?
MySQL
Fabric

8. Uh Oh! Phew!
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
§ Redundancy:
If
master
crashes,
promote
slave
to
master
C
B
A
Background:
What
is
ReplicaCon
Used
For?
C
B
CraAs h
C
B
A
B is the
new master

9. Background:
What
is
ReplicaCon
Used
For?
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
§ On-­‐line
Backup
and
ReporBng
M
S
n write
clients
n business
intelligent
client
apps
n reporCng
client
apps
n big
queries
client
apps

10. CB
AC BA
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Image
from
www.ginkgomaps.com
Background:
What
is
ReplicaCon
Used
For?

11. Generally
Available
Features
in
MySQL
5.6
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|

12. relay log
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Insert...
Insert...
B
binary log
Insert...
Insert...
A
binary log
Client
Network
Master
info
Relay
Log
Info
Binary
Log
Group
Commit
Sender
thread
Receiver
thread
Applier
Threads

13. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Trx1
…
TrxN
Trx1 . . . TrxN
A
binary log
§ TransacCons
are
flushed
and
synced
together
and
then
commided
in
the
engine!
§ Binary
Log
is
the
conceptual
commit
point.
§ Resilience,
crash-­‐safeness
and
durability
with
good
performance
(sync_binlog=1).
§ No
impact
on
single-­‐threaded
workloads.
§ Loss-­‐less,
automaCc
recovery.
§ Corrupt
tail
of
the
binary
log
is
trimmed
on
recovery.
Binary
Log
Group
Commit
mysql> SET sync_binlog=1

14. n 5x
faster
than
MySQL
5.5
n 4.5x
even
when
sync_binlog=1
hdps://blogs.oracle.com/MySQL/entry/mysql_5_6_replicaCon_performance
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Binary
Log
Group
Commit

15. n Days
of
losing
50%+
master
performance
are
gone!
n 10%
overhead
when
configuring
sync_binlog=0
n AddiConal
5%
when
configuring
sync_binlog=1
hdps://blogs.oracle.com/MySQL/entry/mysql_5_6_replicaCon_performance
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Binary
Log
Group
Commit

16. OpCmized
Row-­‐based
ReplicaCon
§ Smaller
storage,
network
and
memory
footprints.
§ Each
record
change
may
require
its
before
and/or
aker
images
to
be
stored
in
the
RBR
event.
§ Before
MySQL
5.6
the
server
would
log
full
images
only.
§ In
MySQL
5.6
and
later,
the
server
can
log
images
with
just
the
needed
data.
§ A
PK-­‐less
table
on
the
slave
is
much
quicker
to
change.
§ Applier
does
at
most
one
table
scan
per
Update
or
Delete
event.
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|

17. OpCmized
Row-­‐based
ReplicaCon:
ParCal
Images
Before Image After Image
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Full Rows
Rows without Blobs
Minimal Rows
Blob
Primary
Key
Changed
Columns
mysql> binlog_row_image={MINIMAL|NOBLOB|FULL}

18. OpCmized
Row-­‐based
ReplicaCon:
Improved
Rows
Lookup
by
RBR
Applier
mysql> SET slave_rows_search_algorithm={INDEX_SEARCH | TABLE_SCAN | HASH_SCAN }
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
§ slave_rows_search_algorithm
§ INDEX_SEARCH
–
use
an
index
(if
available)
to
locate
the
row.
§ TABLE_SCAN
–
do
a
table
scan
to
locate
the
row.
§ HASH_SCAN
–
do
at
most
ONE
table
scan
to
locate
ALL
rows
in
the
event.
§ Result:
large
tables
without
primary
keys
are
now
updated
quicker
by
the
slave
thread.
NEW

19. relay log
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Insert...
Insert...
B
binary log
Insert...
Insert...
A
binary log
Client
Network
Master
info
Relay
Log
Info
MulC-­‐Threaded
Slave
Applier
Sender
thread
Receiver
thread
Applier
Threads

20. relay log
MulC-­‐Threaded
Slave
Applier
Worker
threads
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Insert...
B
binary log
Insert...
Client Client
Insert...
A
binary log
Network
Client
Coordinator
thread (reading) SQL
thread
Multiple worker
threads! (executing)
One thread reads
And schedules
Transactions!

21. MulC-­‐Threaded
Slave
Applier
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
§ Parallelize
by
schema
(database)
§ Different
worker
threads
never
update
same
schema.
§ EffecCve
if
you
parBBon
data
by
schema.
§ Consistency
§ SequenCal
consistency
criteria
within
schema.
§ Eventually
consistent
between
schemas.
§ Commit
order
may
differ
between
master
and
slave,
among
slaves.
SET GLOBAL SLAVE_PARALLEL_WORKERS = 4;

22. MulC-­‐Threaded
Slave
Applier
5x
Higher
Performance
with
MySQL
5.6
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|

23. relay log
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Tx2: Update...
Tx1: Insert...
Trx1
B
binary log
Trx1
Trx2
Trx1
Trx2
A
binary log
Client
Network
Master
info
Relay
Log
Info
Global
TransacCon
IdenCfiers
Client
Sender
thread
Receiver
thread
Applier
Threads

24. A
B
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
§ Handle
servers
unavailability
with
minimal
disrupBon.
§ Switch-­‐over/Fail-­‐over
§ Topologies:
§ Tree
§ Line
§ Circle
§ …
§ Arbitrary
Topology
§ Log
coordinate:
filename
+
offset
§ Not
good
enough
–
The
need
for
a
logical
idenCfier.
C
D
E
F
Global
TransacCon
IdenCfiers

25. Global
TransacCon
IdenCfiers
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
§ Master
assigns
Global
TransacBon
IdenBfier
(GTID)
to
every
transacCon:
§ server_uuid:number
a61678ba-­‐4889-­‐4279-­‐9e58-­‐45ba840af334:1
§ server_uuid
idenCfies
the
server
and
is
globally
unique;
§ number
is
incremented
by
1
for
each
transacCon
on
this
server.
§ GTIDs
are
wriden
to
the
binary
log.
GTID BEGIN E1 E2 ... COMMIT GTID BEGIN E1 E2 ... COMMIT
One Id per transaction!

26. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
§ Purpose:
Fail-­‐over
/
Switch-­‐over
§ Very
simple
usage:
§ Fail-­‐over
has
very
small
admin
overhead
§ Basic
monitoring
is
simple
(Applied
transacCons)
(Retrieved
and
Executed)
(Applied
but
purged
from
binary
logs
already)
Global
TransacCon
IdenCfiers
mysql> CHANGE MASTER TO MASTER_AUTO_POSITION=1, MASTER_HOST='...';
mysql> select @@global_gtid_executed
mysql> SHOW SLAVE STATUS
mysql> select @@global.gtid_purged

27. relay log
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Insert...
Insert...
B
binary log
Insert...
Insert...
A
binary log
Client
Network
Master
info
Relay
Log
Info
TransacConal
ReplicaCon
Metadata
Sender
thread
Receiver
thread
Applier
Threads

28. relay log
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
§ Slave
threads
maintain
their
current
posiBons
§ Receiver
thread:
where
to
read
from
master
§ Applier
thread:
where
to
read
from
relay
log
§ PosiCons
stored
on
disk
§ Before:
in
file
§ 5.6:
in
table
insert...
B
binary log
insert...
Receiver
thread
Applier
thread
TransacConal
ReplicaCon
Metadata
Master
info
Relay
Log
Info

29. TransacConal
ReplicaCon
Metadata
§ Consistent
aker
server
crash
§ Requires
(InnoDB)
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
§ File:
updated
aker
transacCon
commit
Trans-action
Update
position
Trans-action
Upd ate
position
Crash
possible
§ Table:
updated
atomically
at
transacCon
commit
transacBonal
table
master-info-repository=TABLE
relay-log-info-repository=TABLE

30. Binary
Log
Events
Checksums
relay log
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Insert...
Insert...
B
binary log
Insert...
SHOW BINLOG ...
Client
Insert...
A
binary log
Client
Sender
thread
Network
Master
info
Relay
Log
Info
CRC creation point!
CRC verification point!
Receiver
thread
Applier
Threads

31. Binary
Log
Events
Checksums
§ Reduce
the
Cme
it
takes
to
discover
the
cause
of
binary
log
corrupCon.
§ Hard
disk
failure
(?),
network
problems
(?),
memory
sCcky
bits
(?)
…
or
just
plain
server
bugs
(?).
§ Each
event
is
appended
a
32bits
cyclic
redundancy
check
(CRC32).
§ MulCple
verificaCons
points
(on
the
master
and/or
on
the
slave).
§ Can
be
turned
on/off
independently.
One CRC per Event.
BEGIN E1 E2 ... COMMIT BEGIN E1 E2 ... COMMIT
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
C
R
C
C
R
C
C
R
C
C
R
C
C
R
C
C
R
C
C
R
C
C
R
C

32. High
Availability
+
Sharding-­‐Based
Scale-­‐out
§ High
Availability:
§ Server
monitoring
with
auto-­‐promoCon
and
transparent
applicaCon
failover
§ OpConally
scale-­‐out
through
sharding
§ ApplicaCon
provides
shard
key
§ Range
or
Hash
§ Tools
for
resharding
§ Global
updates
&
tables
§ Fabric-­‐aware
connectors
rather
than
proxy:
Python,
Java
&
PHP
§ Lower
latency,
bodleneck-­‐free
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
Fabric
MySQL Fabric
ApplicaCon
Connector
ApplicaCon
Read-slaves
mappings
SQL
HA group
Read-slaves
HA group
Connector

33. Next
GeneraBon
ReplicaBon
Features
in
MySQL
5.7
DMR
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|

34. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
What
is
New?
§ MySQL
5.7.4
-­‐
Development
Milestone
Release,
April
2014
§ Higher
throughput
§ Slave
:
applies
transacCons
in
parallel
even
within
same
database
§ Master:
beder
synchronizaCon
between
replicaCon-­‐user
sessions
§ DMR4
improves
master
performance
for
Semi-­‐synchronous
ReplicaCon
§ Lossless
ReplicaCon
through
enhanced
Semi-­‐sync
§ Performance
Schema
tables
for
monitoring
slave
§ Dynamic
ReplicaCon
Filters:
change
filters
on
the
fly

35. MySQL
MulC-­‐Threaded
Slave
§ The
evoluCon…
§ 2010:
Inter-­‐schema
MTS
(labs)
§ 2013:
Inter-­‐schema
MTS
(5.6
GA)
§ 2013:
Intra-­‐schema
MTS
(5.7.2
DMR)
§ Today:
Tune
size
of
group
commit
buckets
-­‐
trading
commit
latency
on
master
(labs)
§ 6X
slave
throughput
Slave
keeps
pace
with
the
master
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
labs.mysql.com
4500"
4000"
3500"
3000"
2500"
2000"
1500"
1000"
500"
0"
Slave)Throughput)
0" 4" 8" 12" 16" 22" 24" 28" 32" 36" 40" 44" 48"
Transac'ons)per)second)
Slave)Worker)Threads)
Larger)BGC)bucket)
No)Master)Impact)
Baseline)

36. Higher
Master
Throughput:
Sender,
aka
Dump,
Thread
Enhancement
Concurrent
reads
by
the
sender
thread
with
ongoing
writes
from
user
threads.
§ Sender
thread
does
not
block
user
sessions
more
than
necessary.
§ Higher
throughput
for
both
sender
threads
and
user
sessions.
Sender Thread Reads Binary Log User Thread Writes to Binary Log
Sender Thread Reads Binary Log
User Thread Writes to Binary Log
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Time
pre 5.7.2
5.7.2+

37. Higher
Slave
Throughput:
Timestamp
based
MulC-­‐
threaded
Applier
relay log
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Insert...
Insert...
B
binary log
Insert...
Insert...
A
binary log
Client
Network
Master
info
Relay
Log
Info
Sender
thread
Receiver
thread
Applier
Threads

38. Higher
Slave
Throughput:
Timestamp
based
MulC-­‐
threaded
Applier
§ Leverage
parallelizaCon
informaCon
obtained
from
the
execuCon
on
the
master.
§ TransacCons
that
prepare
on
the
same
“version”
of
the
database,
are
assigned
the
same
Cmestamp.
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
§ Meanwhile,
at
the
slave:
§ TransacCons
with
the
same
Cmestamp
can
be
executed
in
parallel;
§ Concurrent
transacCons
commit
independently,
thus
no
waiCng
involved.

39. Higher
Slave
Throughput:
Timestamp
based
MulC-­‐
threaded
Applier
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
T1
T2
T3
Time
Commit
Prepare
Concurrent
ExecuBon
History
on
the
Master
timestamp++
Execution
Prepare
Commit

40. Higher
Slave
Throughput:
Timestamp
based
MulC-­‐
threaded
Applier
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
T1
T2
T3
Time
Commit
Prepare
Concurrent
ExecuBon
History
on
the
Master
timestamp++
Execution
Prepare
Commit
Parallel
on the Slave.
Not Executed in
Parallel.

41. Higher
Slave
Throughput:
Timestamp
based
MulC-­‐
threaded
Applier
§ Supports
statement-­‐based
or
row-­‐based
formats.
§ Scheduling
policy
controlled
through:
mysql> SET slave_parallel_type= [logical_clock|database]
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
§ Logical_clock
-­‐
means
schedule
based
on
the
prepare
Cmestamp
§ database
-­‐
the
scheduling
policy
from
5.6
(concurrency
control
done
per
database).
§ Work
to
improve
slave
scalability
conCnues,
does
not
stop
here.

42. Beder
ReplicaCon
Monitoring:
P_S
ReplicaCon
Tables
relay log
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Insert...
Insert...
B
binary log
Insert...
Insert...
A
binary log
Client
Network
Master
info
Relay
Log
Info
Sender
thread
Receiver
thread
Applier
Threads

43. Beder
ReplicaCon
Monitoring:
P_S
ReplicaCon
Tables
n Access
monitoring
informaCon
through
an
SQL
interface.
n Write
stored
funcCons
or
procedures
with
input
from
replicaCon
internals.
n Logically
unrelated
informaCon
into
different
places.
n Flexible
and
easier
to
extend
and
adapt
as
new
feature
get
into
the
server.
n More
user
friendly
names
idenCfying
the
monitoring
fields.
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|

44. Beder
ReplicaCon
Monitoring:
P_S
ReplicaCon
Tables
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Connection
Configuration
Connection
Status
Execution
Configuration
Execution
Status
Slave Status
Applier / Coordinator
Status
Workers
Status

45. Better, Flexible, Loss-less Semi-sync Replication
relay log
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Insert...
Insert...
B
binary log
Insert...
Insert...
A
binary log
Client
Network
Master
info
Relay
Log
Info
Sender
thread
Receiver
thread
Applier
Threads

46. Beder,
Flexible,
Loss-­‐less
Semi-­‐sync
ReplicaCon
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
§ Master
does
not
commit
transacCon
unCl
it
receives
an
ACK
from
the
slave.
§ (as
opposed
to:
Master
does
not
release
the
session
unCl
it
receives
an
ACK
from
the
slave.)
§ Therefore,
concurrent
transacCons
do
not
externalize
changes
while
waiCng
for
ACK.
§ Should
a
master
fail,
then
any
transacCon
that
it
may
have
externalized
is
also
persisted
on
a
slave.
§ User
can
choose
between
the
original
semisync
behavior
and
the
new
one.
mysql> SET rpl_semi_sync_master_wait_point= [AFTER_SYNC|AFTER_COMMIT]

47. Beder,
Flexible,
Loss-­‐less
Semi-­‐sync
ReplicaCon
§ Master
can
wait
for
more
than
one
slave.
§ Increases
resiliency
to
consecuCve
failures
§ Increases
durability
–
transacCon
is
persisted
in
N
places
mysql> SET rpl_semi_sync_master_wait_slave_count= 1..1024
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|

48. relay log
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
Insert...
Insert...
B
binary log
Insert...
Insert...
A
binary log
Client
Network
Master
info
Relay
Log
Info
Dynamic
ReplicaCon
Filters
Sender
thread
Receiver
thread
Applier
Threads

49. Dynamic
ReplicaCon
Filters
n Update
filters
without
shu{ng
down
server
instance.
mysql> CHANGE REPLICATION FILTER REPLICATE_DO_DB=(db1);
ERROR 1896 (HY000): This operation cannot be performed with
a running slave sql thread; run STOP SLAVE SQL_THREAD first
mysql> STOP SLAVE SQL_THREAD;
Query OK, 0 rows affected (0.05 sec)
mysql> CHANGE REPLICATION FILTER REPLICATE_DO_DB=(db1);
Query OK, 0 rows affected (0.00 sec)
mysql> START SLAVE SQL_THREAD;
Query OK, 0 rows affected (0.05 sec)
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|

50. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MySQL
Labs

51. Master
N
Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MulC-­‐Source
ReplicaCon
§ Consolidate
updates
from
mulCple
Masters
into
one
Slave
§ Consolidated
view
of
all
shards
§ More
flexible
topologies
§ Centralized
point
for
backups,
easier
reporCng
§ CompaCble
with
Semi-­‐Synchronous
ReplicaCon
&
enhanced
MTS
§ Master-­‐specific
slave
filters
planned
for
GA
§ ApplicaCon
must
keep
data
sets
disjoint
between
sources
Master
1
Binlog
Master
2
Binlog
…
…
Binlog
IO
1
Relay
1
Coordinator
W1
W2
…
WX
IO
2
Relay
2
Coordinator
W1
W2
…
WX
…
…
Coordinator
W1
W2
…
WX
IO
N
Relay
N
Coordinator
W1
W2
…
WX
Slave
labs.mysql.com

52. Copyright
©
2014
Oracle
and/or
its
affiliates.
All
rights
reserved.
|
MulC-­‐Source
ReplicaCon
M M
S
Slave can have more than one master.
Feature preview based on 5.7 is available
on labs.mysql.com .
M M
The
need
for
gathering
data
in
a
central
server:
n Integrated
backup;
n Complex
queries
for
analyCcs
purposes;
n Data
HUB
for
inter-­‐cluster
replicaCon.