MySQL High Availability with Replication New Features

Open Source India 2014
MySQL High Availability with
Replication New Features
Shivji Kumar Jha (shivji.jha@oracle.com)
Manish Kumar (manish.4.kumar@oracle.com)
Software Developer, MySQL Replication Team
2 Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, 8th November, 2014. |

Safe Harbor Statement
The following is intended to outline our general product direction. It is
intended for information purposes only, and may not be incorporated
into any contract. It is not a commitment to deliver any material, code,
or functionality, and should not be relied upon in making purchasing
decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole
discretion of Oracle.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Open Source India | Bangalore, India, 8th November, 2014. |

Program Agenda
Background
Performance
Usability
Flexibility
Consistency
Extras
0
1
2
3
4
5

Background: Replication Components
relay log
Insert...
Insert...
B
binary log
Insert...
Insert...
A
binary log
Client
Sender
thread
Receiver
thread
Applier
Threads
Network

Background: What is Replication Used For?
S
Read scale-out
M S
S
S
M
write clients read clients
read clients
write clients
More
reads?
More
slaves!

What about write scale-out?
More
writes?
More
Masters?
M?
M M?
write clients write clients
M?
MySQL Fabric helps sharding your data with MySQL...

Redundancy: If master crashes, promote slave to master
C
B
A
C
B
CraAsh
C
B
A
B is the
new master
Uh Oh! Whew!

On-line Backup and Reporting
M
S
write clients
 business intelligent client apps
 reporting client apps
 big queries client apps

CB
CA AB
Image from
www.ginkgomaps.com

Program Agenda
Background
Performance
Usability
Flexibility
Consistency
Extras
1
2
3
4
5
6

Performance
— Higher Master Throughput.
 Improved concurrency between Sender and user threads,
 More Sender thread enhancements...
— Higher Slave Throughout.
 MySQL- 5.7.2: Multi-Threaded Slave (MTS) Timestamp based Applier.
 MySQL Labs: Enhanced MTS Timestamp based Applier.
 Options to Fine Tune Binlog Group Commit and Multi-Threaded Applier

Improved concurrency b/w Sender and user threads
relay log
Insert...
Insert...
B
binary log
Insert...
Insert...
A
binary log
Client
Network
Sender
thread
Receiver
thread
Applier
Threads

Improved concurrency b/w Sender and user threads
Concurrent reads by the sender thread with ongoing writes from user
threads.
— Sender thread does not block user sessions more than necessary.
— Higher throughput for both sender threads and user sessions.
Sender Thread Reads Binary Log User Thread Writes to Binary Log
Sender Thread Reads Binary Log
User Thread Writes to Binary Log
Time
pre 5.7.2
5.7.2+

More Sender Thread Enhancements
relay log
Insert...
Insert...
B
binary log
Insert...
Insert...
A
binary log
Client
Network
Sender
thread
Receiver
thread
Applier
Threads

 Send buffer is not allocated and freed every time an event is sent.
 When sender threads require larger send buffer → buffer grows as needed.
 When buffer is larger than needed→ buffer shrinks dynamically.
 Together with the sender thread enhancements released on MySQL 5.7.2:
— increase master scalability;
— Master, with dump threads attached, copes better with peak loads;
— Separate semisync ACK receiver thread (details later!)

 Ad hoc microbenchmark using
mysqlslap
 1M queries, concurrency=200,
commit=1
 N slaves attached (fake slaves
using remote mysqlbinlogs)
 48 cores HT / 512 GB RAM / HDD
0 1 5 10 20 30 40 50 100
16000
14000
12000
10000
8000
6000
4000
2000
 Small number of slaves: no difference between 5.6 and 5.7.
 Larger number of slaves: 5.7 is able to sustain the same throughput while
5.6 is not.
0
Master's Throughput
MySQL 5.6.16
MySQL 5.7.4
Number of Slaves
QpS

This was the case when we released MySQL 5.6 GA.
 Up to 5x more throughput
with 10 sysbench schemas.
 Changes to each schema
applied and committed
independently by worker
threads (both in SBR and
RBR).
0 1 2 3 4 5 6 10 24
300
250
200
150
100
50
0
Slave Throughput
Number Worker Threads
QpS
Higher Slave Throughput (MySQL-5.6)

“It's the bee's knees! We've been
using it at VividCortex in
production and it works great.”
– Baron Schwartz, CEO, VividCortex
https://vividcortex.com/blog/2014/03/17/multi-threaded-replication-in-mysql/

relay log
Insert...
Insert...
B
binary log
Insert...
Insert...
A
binary log
Client
Network
Sender
thread
Receiver
thread
Applier
Threads

Higher Slave Throughput
Timestamp based Multi-threaded Slave (MTS) Applier
● Leverage parallelization information obtained from the execution on the
master.
● Transactions that prepare on the same “version” of the database, are
assigned the same timestamp.
● Meanwhile, at the slave:
● Transactions with the same timestamp can be executed in parallel;
● Concurrent transactions commit independently, thus no waiting involved.

Higher Slave Throughput
 Supports statement-based or row-based formats.
 Scheduling policy controlled through:
mysql> SET slave_parallel_type= [logical_clock|database]
— logical_clock - means schedule based on the prepare timestamp.
— database - the scheduling policy from 5.6 (concurrency control done per
database).
 Work to improve slave scalability continues, does not stop here.

relay log
Insert...
Insert...
B
binary log
Insert...
Insert...
A
binary log
Client
Network
Sender
thread
Receiver
thread
Applier
Threads
Enhanced Scheduling Policy (MTS)

Enhanced Scheduling Policy (MTS)
Sysbench / oltp test / 48 clients /
10M rows
SSD / 48 core HT / 512 GB RAM
GTIDs / Crash-safe
Numbers from
mysql-5.7.5-labs-preview
UPDATED
TO BE 0 4 16 50 200
Up to 6X performance improvement vs single threaded slave
without any changes to the application
5000
4500
4000
3500
3000
2500
2000
1500
1000
500
0
Transactions per Second (TPS)
TPS while replaying 300 seconds of accumulated load from the master.
Baseline
100 clients – 5.7.5-labs
200 clients – 5.7.5-labs
Number of Worker Threads
Transactions Per Second

BGC tuning options – MTS Enhancers
relay log
Insert...
Insert...
B
binary log
Insert...
Insert...
A
binary log
Client
Network
Sender
thread
Receiver
thread
Applier
Threads

Sysbench / oltp test / 48 clients /
10M rows
SSD / 48 core HT / 512 GB RAM
GTIDs / Crash-safe
Numbers from
mysql-5.7.4-labs-mts
small commit delay
large commit delay
0 5 10 15 20 25 30 35 40 45 50
4500
4000
3500
3000
2500
2000
1500
1000
500
0
Number of Slave Worker Threads
TPS

By tuning BGC on the master, we
get more parallelism on the slave.
Tuning means more transactions
preparing together (no magic
formula – highly dependent on
hardware and workload).
Ongoing work on improving MTS and
on the analysis of experimental
results.
small commit delay
large commit delay
0 5 10 15 20 25 30 35 40 45 50
4500
4000
3500
3000
2500
2000
1500
1000
500
0
Number of Slave Worker Threads
TPS
 6X slave throughput – large commit delay
 3X slave throughput – small commit delay

Better Replication Monitoring
relay log
Insert...
Insert...
B
binary log
Insert...
Insert...
A
binary log
Client
Network
Sender
thread
Receiver
thread
Applier
Threads

Performance Schema Replication Tables
● Access monitoring information through an SQL interface.
● Write stored functions or procedures with input from replication internals.
● Logically unrelated information into different places.
● Flexible and easier to extend and adapt as new feature get into the
server.
● More user friendly names identifying the monitoring fields.

Connection
Configuration
Connection
Status
Execution
Configuration
Execution
Status
Slave Status
Applier / Coordinator
Status
Workers
Status

Semi-sync Replication – Wait for Multiple ACKs
relay log
Insert...
Insert...
B
binary log
Insert...
Insert...
A
binary log
Client
Network
Sender
thread
Receiver
thread
Applier
Threads

Semi-sync Replication – Wait for Multiple ACKs
 Master does not commit transaction until it receives N ACKs from N slaves.
 Dynamically settable:
mysql> SET GLOBAL rpl_semi_sync_master_wait_for_slave_count= N
Master
T1: COMMIT
Slave 2 relaylog
ACK
Time
Slave 1 relaylog
ACK
T1: COMMIT
succeeds
mysql> SET GLOBAL rpl_semi_sync_master_wait_for_slave_count= 2

relay log
Storing GTIDs In a Table
Insert...
B
Insert...
Insert...
A
binary log
Client
Network
Sender
thread
Receiver
thread
Applier
Threads
gtid_executed gtid_executed

Use Cases
 Slaves can use GTIDs with binary logs disabled.
— Slaves that are never candidates to become a master can still use GTIDs
for auto positioning.
 GTIDs are more resilient to unexpected deletion of binary logs.
— If the DBA decides to delete all binlogs but the most recent one, GTID
history is still preserved.

Mechanics
 GTIDs are saved into a mysql system table.
CREATE TABLE gtid_executed(
source_uuid CHAR(36) NOT NULL,
interval_start BIGINT NOT NULL,
interval_end BIGINT NOT NULL,
PRIMARY KEY(source_uuid, interval_start)
);

Mechanics
 If binary log is enabled, gtids are first stored on the binary log and then
copied to the table on binlog rotation or shutdown.
 If binary log is disabled, gtids are stored in the table on commit.

Preserve Commit Order (MTS)
relay log
Insert...
Insert...
B
binary log
Insert...
Insert...
A
binary log
Client
Network
Sender
thread
Receiver
thread
Applier
Threads

Preserve Commit Order (MTS)
 Multi-threaded Applier May be Instructed to Preserve Commit Order.
— Applier Threads will commit in the same order transactions appear in the
relay log.
— Order is enforced at the expense of performance degradation.
— This is controllable through the new global option/variable:
--slave-preserve-commit-order. Default is FALSE.
mysql> SET GLOBAL slave_preserve_commit_order= [TRUE|FALSE]

Redirect to new Master without interrupting the
Applier
relay log
Insert...
Insert...
B
binary log
Insert...
Insert...
A
binary log
Client
Network
Sender
thread
Receiver
thread
Applier
Threads

Applier
Change master to B from A without stopping the applier threads.
C
B
A
C
B
CraAsh
C
B
A
B is the
new master
Uh Oh! Whew!

Applier
Change master to M2 from M1 without stopping the applier threads.
Uh Oh! Whew!
C
B
A
C
B
CraAsh
C
B
A
DBA did not need C to stop applying its
relay log to change C's master from A to B.

Applier
 More “onlineness” during fail-over:
mysql> STOP SLAVE IO_TRHEAD;
mysql> CHANGE MASTER TO MASTER_HOST='master2', …;
mysql> START SLAVE IO_THREAD;
 Stopping, changing master and restarting the receiver thread is all done
while the applier thread is running.
 Change applier properties while the receiver thread is working:
mysql> STOP SLAVE SQL_TRHEAD;
mysql> CHANGE MASTER TO MASTER_DELAY=3600, …;
mysql> START SLAVE SQL_THREAD;

Dynamic Slave Replication Filters
relay log
Insert...
Insert...
B
binary log
Insert...
Insert...
A
binary log
Client
Network
Sender
thread
Receiver
thread
Applier
Threads

Dynamic Slave Replication Filters
Change Slave's Replication Filters dynamically.
— No need to stop and restart slave for establishing new replication filtering
rules.
— All slave filters are supported.
mysql> CHANGE REPLICATION FILTER REPLICATE_DO_DB= (db1, db2)

Multi-Source Replication
M M
S
Slave can have more than one master.
Feature preview based on 5.7 is available
on labs.mysql.com .
M M
The need for gathering data in a central server:
 Integrated backup;
 Complex queries for analytics purposes;

 A server (slave) can replicate from multiple sources (masters).
 Multiple channels (channel: connection thread, relay log, applier threads) that
can be stopped started individually.
 Integrated with Multi-threaded Slave: each channel has its own multi-threaded
applier set of threads.
 Integrated with the new P_S tables.
— replication_execute_status_by_coordinator shows multiple entries, one per
channel/source applier.
— replication_connection_status shows multiple entries, one per connection to
different sources.

 Integrated with GTIDs.
 Integrated with crash-safe tables.
— Progress state is stored in these tables for recoverability purposes.
 Works with semisync replication.
— Working on further improving the integration.
 Virtually no limit on the number of sources (capped to 256, but can be changed
if server binary is rebuilt).
 Able to manage each source separately.

Loss-less Semi-sync Replication
relay log
Insert...
Insert...
B
binary log
Insert...
Insert...
A
binary log
Client
Network
Sender
thread
Receiver
thread
Applier
Threads

 Master waits for slave's ACK before committing (as opposed to: master waits
for slave's ACK after committing).
— Therefore, concurrent transactions do not externalize changes while waiting
for ACK.
 Should a master fail, then any transaction that it may have externalized is also
persisted on a slave.
 User can choose between the original semisync behavior and the new one.
mysql> SET rpl_semi_sync_master_wait_point= [AFTER_SYNC|AFTER_COMMIT]

prepare commit
Master
Slave
execute binlog
relaylog
T1: INSERT INTO t1 VALUES (1000)
ACK
Time
T1
T2: SELECT * FROM t1;
T2
empty set

Semi-sync Replication – ACK Receiver Thread
relay log
Insert...
Insert...
B
binary log
Insert...
Insert...
A
binary log
Client
Network
Sender
thread
Receiver
thread
Applier
Threads
ACK
Receiver
thread

Semi-sync Replication – ACK Receiver Thread
 Consecutive transactions do not block each other while waiting for ACKs
— Transaction t1 and t2 are sent immediately to the slave by the sender thread
— ACKs are received only by a special thread
— Transaction t2 does not include t1 round trip in its semisync overall latency
 Thread starts when semisync is activated
mysql> SET GLOBAL rpl_semi_master_enabled= ON
 Thread stops when semisync is deactivated
mysql> SET GLOBAL rpl_semi_master_enabled= ON

Semi-sync Replication in MySQL 5.7
“Semi-sync in MySQL 5.7 looks like a great
improvement. [...] With MySQL 5.7, I’d be
much more apt to recommend semi-sync as
an option than in previous releases. Thanks
to Oracle for investing here.”
– Jay Janssen, MySQL Consulting Manager, Percona Inc
http://www.percona.com/blog/2014/05/27/semi-sync-replication-performance-mysql-5-7-4-dmr/

Introducing MySQL Group Replication
 Multi-master update everywhere with conflict resolution.
 Automatic distributed member recovery
— When a server joins the group, it gets all the missing transactions before actually
start serving requests.

Group-Based Replication Plugin
M M M M M
MySQL
Server
API
Replication
Plugin
Group Comm.
System (Corosync)
Group Comm.
System (Corosync)

Update ...
M M M M M
Application issues an update.

M M M M M
Replication
Events
Master executes and multi/broadcasts
the update to the other servers.

M M M M M
Servers receive same transaction in the
same order and check for conflicts.

M M M M M
All servers, independently, decide to
commit the transaction – no conflicts.

All servers, independently, decide to
commit the transaction – no conflicts.
Originating master replies to the
application.
M M M M M

MySQL Group Replication: Use Cases
● Automatic group membership management and failure detection
- No need for server fail-over
● No single point of failure
- Automatic reconfiguration

Under the hood
● Each server can belong to a group.
● The Group is identified by a name which is the Group UUID string, so group behaves
like a single server.
● The Group UUID is used to log transactions.
● Whenever a server joins or leaves, it will start a new stable period in time on which
servers can agree on distributed decisions. A special message called View Message
is sent to each group member notifying that.
● Server join/leave can be issue manually or by scripts by DBA.
● Server leave can be caused also by node failures.

Under the hood
 Current used group communication toolkit is Corosync. It provides:
— Virtual synchrony guarantees for creating replicated state machines.
 Replicated state machine simple definition:
— If a set of processes with the same initial state receives the same set of deterministic
messages at the same order, them will reach the same state.
 All write operations are sent to all group members for certification.
 Read operations are only local.

COMMIT dbv: 2
OK
OK
OK
Certification - Positive
s1
s2
dbv: 1 UPDATE
dbv: 2
s3
dbv: 2
COMMIT
COMMIT
cv: 1 1 >= 1
1 >= 1
1 >= 1
cv: 1
cv: 1
dbv: 1
dbv: 1
cv: 2
cv: 2
cv: 2
s ← server
v ← snapshot version
sn ← certifier sequence number (write set version)

T1: COMMIT dbv: 2
Certification - Negative
OK
OK
T2: UPDATE cv: 2
OK
KO
KO
KO
s1
s2
dbv: 1 T1: UPDATE
dbv: 2
s3
dbv: 2
T1: COMMIT
T1: COMMIT
cv: 1
cv: 1
cv: 1
1 >= 1
1 >= 1
1 >= 1
cv: 2
cv: 2
1 >= 2
1 >= 2
1 >= 2
dbv: 1
dbv: 1
s ← server
v ← snapshot version
sn ← certifier sequence number (write set version)

Automated distributed member recovery
R1
R2
R3
R4
GCS
R1
R2
R3
R4
R4 joins the group.
GCS

R1
R2
R3
R4
R1 is R4 Donor
GCS
R1
R2
R3
R4
GCS

R1
R2
R3
R4
GCS
R1
R2
R3
R4
GCS

R1
R2
R3
R4
GCS
R1
R2
R3
R4
R4 has caught up
with the view!
GCS

R1
R2
R3
R4
GCS
R1
R2
R3
R4
GCS
Execute buffered
transactions from the view.

PERFORMANCE SCHEMA TABLE
– REPLICATION_NODE_STATUS
● Statistics regarding Applier and Certifier module.
– REPLICATION_CONNECTION_NODES
● Information regarding different nodes in the group and its
connection details
– REPLICATION_CONNECTION_STATUS
● Extended table to give details about the group connection status.

Replication Monitoring: replication_node_status
node1> SELECT * FROM performance_schema.replication_node_statusG
*************************** 1. row ***************************
GROUP_NAME: 8a84f397-aaa4-18df-89ab-c70aa9823561
NODE_ID: 2af09cb1-3e46-11e4-9e45-ecf4bb227f3b
TRANSACTIONS_IN_QUEUE: 0
CERTIFIED_TRANSACTIONS: 12
POSITIVELY_CERTIFIED: 7
NEGATIVELY_CERTIFIED: 5
CERTIFICATION_DB_SIZE: 6
STABLE_SET: 8a84f397-aaa4-18df-89ab-c70aa9823561:1-7
LAST_CERTIFIED_TRANSACTION: 8a84f397-aaa4-18df-89ab-c70aa9823561:7
APPLIER_MODULE_STATUS: ON

node1> SELECT * FROM performance_schema.replication_connection_nodesG
*************************** 1. row ***************************
GROUP_NAME 8a94f357-aab4-11df-86ab-c80aa9429573
NODE_ID 597dbb72-3e2c-11e4-9d9d-ecf4bb227f3b
NODE_HOST nightfury
NODE_PORT 13000
NODE_STATE ONLINE
*************************** 2. row ***************************
NODE_ID 59efb8a1-3e2c-11e4-9d9d-ecf4bb227f3b
NODE_HOST nightfury
NODE_PORT 13001
NODE_STATE ONLINE
*************************** 3. row ***************************
NODE_ID 5a706f6b-3e2c-11e4-9d9d-ecf4bb227f3b
NODE_HOST nightfury
NODE_PORT 13002
NODE_STATE RECOVERING

node1> SELECT * FROM performance_schema.replication_connection_statusG
*************************** 1. row ***************************
GROUP_NAME: 8a94f357-aab4-11df-86ab-c80aa9429563
SOURCE_UUID: 8a94f357-aab4-11df-86ab-c80aa9429563
THREAD_ID: NULL
SERVICE_STATE: ON
COUNT_RECEIVED_HEARTBEATS: 0
LAST_HEARTBEAT_TIMESTAMP: 0000-00-00 00:00:00
RECEIVED_TRANSACTION_SET:
LAST_ERROR_NUMBER: 0
LAST_ERROR_MESSAGE:
LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00
TOTAL_MESSAGES_RECEIVED: 9
TOTAL_MESSAGES_SENT: 5
TOTAL_BYTES_RECEIVED: 1891
TOTAL_BYTES_SENT: 1065
LAST_MESSAGE_TIMESTAMP: 2014-09-23 09:20:26
MAX_MESSAGE_LENGTH: 358
MIN_MESSAGE_LENGTH: 51
VIEW_ID: 2
NUMBER_OF_NODES: 2

 Only transactional engines are supported (currently only InnoDB).
 All tables must have primary key
 GTID must be enabled.
 Only deterministic DDL is allowed.
 Optimistic execution: transactions may abort on COMMIT.
Limitations

References
 Read a feature preview of MySQL Group replication on Luis' blog:
http://mysqlhighavailability.com/mysql-group-replication-hello-world/
 Read about how to get started with MySQL Group replication on Pedro's blog:
http://mysqlhighavailability.com/getting-started-with-mysql-group-replication/
 Read about Transaction life cycle in MySQL Group replication on Nuno's blog:
http://mysqlhighavailability.com/mysql-group-replication-transaction-life-cycle-explained/
 Read about Distributed recovery in MySQL Group Replication on Pedro's blog:
http://mysqlhighavailability.com/distributed-recovery-behind-the-scenes/
 Read about Monitoring for MySQL Group Replication on Manish's blog:
http://mysqlhighavailability.com/mysql-group-replication-monitoring/
 Read about Corosync and Group Communication details on Tiago's blog:
http://mysqlhighavailability.com/group-communication-behind-the-scenes/
http://mysqlhighavailability.com/mysql-group-replication-a-small-corosync-guide/
 Read about Testing of MySQL Group Replication on Manish's blog:
http://mysqlhighavailability.com/mysql-group-replication-testing/

References: Read about the 5.7 DMRs Replication
Features
 Read more about P_S tables for replication on Shiv's blog:
http://shivjijha.blogspot.com/2013/09/Monitoring-Replication-with-the-NEW-performance-schema-tables.html
 Read more about Intra-Schema Multi-threaded Slave on Rohit's blog:
http://geek.rohitkalhans.com/2013/09/enhancedMTS-deepdive.html
◾ Read more about Semisync enhancements on Libing's blog:
http://my-replication-life.blogspot.com/2013/09/dump-thread-enhancement.html
http://my-replication-life.blogspot.pt/2013/09/loss-less-semi-synchronous-replication.html

Features
◾ Read more about mysqlbinlog idempotent mode on Rohit's blog:
http://binlogtorelaylog.blogspot.com/2013/05/mysqlbinlog-idemmpotent-mode.html
◾ Read more about mysqlbinlog –rewrite-db option on Manish's blog:
http://manishthe.blogspot.com/2013/05/introduction-with-wonderful-and-best.html

Features
 Read more on Dynamic Replication Filters on Venkat's blog:
http://my-s-q-l.blogspot.com/2013/12/Making-MySQL-Slave-Replication-Filters-Dynamic.html
 Read more on Semi-sync Multiple ACKs on Libing's blog:
http://my-replication-life.blogspot.com/2013/12/enforced-semi-synchronous-replication.html
 Read more on SSL options for mysqlbinlog on João's blog:
http://jmysqlrep.blogspot.com/2013/12/mysql-57-mysqlbinlog-now-supports-ssl.html
 Read more on Semisync ACK thread receiver on Libing's blog:
http://my-replication-life.blogspot.com/2014/03/faster-semisync-replication.html

Features
 Read more on Flexible Change Master on Shiv's blog:
http://work.shivjijha.com/2014/04/change-master-without-stopping-slave.html

Features
 Read more about Multi-source replication on Rith's blog:
http://on-mysql-replication.blogspot.com/2013/09/feature-preview-mysql-multi-source-replication.html
 Read a lot more on : mysqlhighavailability.com

MySQL High Availability with Replication New Features

MySQL High Availability with Replication New Features

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (17)

Similar to MySQL High Availability with Replication New Features

Similar to MySQL High Availability with Replication New Features (20)

More from Shivji Kumar Jha

More from Shivji Kumar Jha (20)

Recently uploaded

Recently uploaded (20)

MySQL High Availability with Replication New Features