20 Real-World Use Cases to help pick a better MySQL Replication scheme (2012)

MySQL Replication: Pros and Cons
Achieve Higher Performance, Uptime, Reliability and Simplicity
for Real-World Use Cases.
Darpan Dinker
VP of Engineering
Schooner Information Technology

p2 | © 2011 Schooner Information Technology. All rights reserved.
© 2012 Schooner Information Technology, p2
Agenda
• Quick tour of MySQL asynchronous, semi-synchronous and
synchronous replication schemes
• 20 real-world use cases: which replication mechanisms to choose
and why
• SchoonerSQL Q&A

• MySQL Master-Master
• DRBD
• Resilience
• Point-in-time recovery (PITR)
• Delayed Slave
• Very large databases
• Automated replication failover
• Mixed hardware
• Virtualized or Cloud env.
• Elasticity requirements
• High write rate
• Read scaling
• Use of Flash memory
• High Availability
• Server sprawl, inefficient HW util.
• Full and incremental backup
• WAN, multi-DC deployments
• Online schema updates
• Online maintenance & upgrades
• Minimize administration
20 Real-World Use Cases

• Concepts that effect MySQL replication
– Parallelism
– Flow control
– Consistency & Serializability
• Quick tour of replication schemes for MySQL
– Asynchronous
– Semi-synchronous
– Synchronous
Part One

Parallelism, or the Lack of
• Commodity hardware: 4-10 core processors
– MySQL Slave uses <1 core to apply replication
– Can MySQL Master sustain write transactions
using >1 core ?
• Hard-disks have >5ms latency (random)
– Single threaded Slave cannot make >200
random reads/sec in foreground
– Can a MySQL Slave use more disk throughput ?
• Flash memory offers >50k IOPS
– Add Flash memory to improve performance of a
single thread
– At what price/performance ?

Answer to Parallelism: SchoonerSQL
2-10X throughput on
SchoonerSQL Slaves
1,693,476
5,326,066
5,080,958
1,782,231
0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
Percona 5.5.20 Schooner
ASYNC
Schooner SYNC Stock 5.5.21
commit/1800s
Slave commits on Sysbench
3,235,410 3,239,277
11,174,862
9,456,770
0
2,000,000
4,000,000
6,000,000
8,000,000
10,000,000
12,000,000
Percona 5.5.20 Stock 5.5.21 Schooner SYNC Schooner
ASYNC
Commit/
3900
sec
Slave commits on DBT2
2200
1598
2190
700
0
500
1000
1500
2000
2500
Schooner 5.1.3 Async Percona 5.5.20 Async
Commits
/
sec
TPCC-mysql Throughput @ 5000W
Master
Slave

Flow Control

No flow control
• Slave lags the Master
– Stale reads on Slave
– Failover onto Slave is tricky
– Action: make Slave faster
(add Flash, more memory)
– Action: shard database
without hitting H/W limits
– Action: use SchoonerSQL
parallel async replication &
reduce slave-lag
With flow control
• Slave is in lock-step with
Master
– Reads are near or fully
consistent
– Failover is low or no risk
– Slave may throttle Master
– Action: use SchoonerSQL
parallel sync replication to
eliminate data-loss and slave-
lag
– Action: use similar H/W within
a cluster
Flow Control in Replication

Serializability and Consistency
• Serializability: a transaction schedule is serializable if its outcome
(e.g., the resulting database state) is equal to the outcome of its
transactions executed serially, i.e., sequentially without overlapping
in time.
• MySQL Slave is single threaded, i.e. executes and commits in a
serial order, hence serializable.
• SchoonerSQL Slave (async/semi-sync*/sync)
– Checks for conflicts
– Executes in parallel
– Commits transactions in the same order as the Master
– Strong consistency, 100% compatibility, Binlog in identical sequence

Things to Watch Out for in Parallel Replication Implementations
Transaction Ordering (1/2): Master
T5
T3
T1
T4
T2
Commit order
T3
T1 T2
T1 T2 T3 T4 T5
(In InnoDB and binary log)
(Transactions on Master)
MASTER
#1

Things to Watch Out for in Parallel Replication Implementations
Transaction Ordering (2/2): Slave or 2nd Master
T1
T2
T3
T4
T5
Commit order
T3
T1 T2
T5 T4 T3 T2 T1
(In InnoDB and binary log)
(order in relay log or sync replication buffers)
MASTER
#2

MySQL Replication Technology
1. Asynchronous
– Oldest, most popular and widely deployed
2. Semi-synchronous
– Introduced in v5.5.
– Objective: avoid many data-loss situations
3. Synchronous (“Virtual Synchrony”, not 2PC)
– Not available from Oracle/ MySQL
– SchoonerSQL: >1 year in production
– XtraDB Cluster: announced last week

#1 MySQL Asynchronous Replication
• Loosely coupled master/slave
relationship
• Master does not wait for Slave
• Slave determines how much to read
and from which point in the binary log
• Slave can be arbitrarily behind master in
reading and applying changes
• Read on slave can give old data
• No checksums in binary or relay log
stored on disk, data corruption possible
• Upon a Master’s failure
• Slave may not have latest committed
data resulting in data loss
• Fail-over to a slave is stalled until all
transactions in relay log have been
committed – not instantaneous
Master mysqld Slave mysqld
Relay
log
InnoDB
Tx log
DB
MySQL
Bin log
InnoDB
Tx log
DB
Last tx=100 Last tx=100 Last tx=70 Last tx=50
Log events pulled by Slave
Tx.Commit(101) Repl.apply(51)
tx=101 tx=101 tx=51 tx=51
Read version based on tx=50
Read Stale data on Slave
No flow control
Corruption
Master failure = mess
Data loss
Single-threaded Slave

#2 MySQL Semi-synchronous Replication
• Semi-coupled master/slave relationship
• On commit, Master waits for an ACK
from Slave
• Slave logs the transaction event in relay
log and ACKs (may not apply yet)
• Slave can be arbitrarily behind master in
applying changes
• Read on slave can give old data
• No checksums in binary or relay log
stored on disk, data corruption possible
• Fail-over to a slave is stalled until all
transactions in relay log have been
committed – not instantaneous
Relay
log
InnoDB
Tx log
DB
MySQL
Bin log
InnoDB
Tx log
DB
Last tx=100 Last tx=100 Last tx=100 Last tx=50
Log for tx=100 pulled by Slave
Tx.Commit(101) Repl.apply(51)
tx=101 tx=101 tx=51 tx=51
Slave ACK for tx=100
No flow control
Corruption
Data loss
Single threaded Slave

#3 SchoonerSQL Synchronous Replication
• Tightly-coupled master/slave
relationship
• After commit, all Slaves guaranteed to
receive and commit the change
• Slave in lock-step with Master
• Read on slave gives latest committed
data
• Checksums in replication paths
• Fail-over to a slave is fully integrated
and automatic
• Application writes continue on new
master instantaneously
• No data loss
InnoDB
Tx log
DB
MySQL
Bin log
InnoDB
Tx log
DB
Last tx=100 Last tx=100 Last tx=100
Log for tx=100 pushed to Slave
Tx.Commit(101)
tx=101
tx=100
Slave ACK for tx=100 Repl.apply(100)
No flow control
Corruption
Data loss
Single threaded Slave

Qualitative Comparison
MySQL 5.5
Asynchronous
SchoonerSQL
Asynchronous
MySQL 5.5
Semi-
Synchronous
SchoonerSQL
Semi-
Synchronous*
XtraDB Cluster
Synchronous
SchoonerSQL
Synchronous
Parallel replication (for same schema) N Y N Y Y Y
Throughput Medium High Medium High Medium High
High network latency tolerant Y Y N N N N
Multi-level failure detection N Y N Y N Y
Global transaction IDs N Y N Y N Y
Capacity with transient failures Y Y Y Y N Y
Commit consistency Y Y Y Y N Y
Auto recovery with high consistency Medium High Medium High Medium High
Auto replication failover and repair N Y N Y Y Y
Large update/ insert/ LOAD INFILE Y Y Y Y N Y
Unexpected aborts & deadlocks N N N N Y N
Multi-Master Cluster N N N N Y N
* Future release

20 real-world use cases: which replication
mechanisms to choose and why?
Part Two

#1 High Write Rate
• Async/ semi-sync
– Issue: Slave lags the Master
– Issue: Master hits limits (code + H/W)
• Typical solutions
– Use more main-memory and/or Flash memory
– Shard database
• SchoonerSQL
– SchoonerSQL is optimized for multi-cores, Flash memory and fast
network
– Master scales vertically with H/W resource (CPU, memory, storage).
– Async and sync Slave have parallel threads for replication that match
the speed of the Master.

#2 Read Scaling
0
50000
100000
150000
200000
250000
300000
1 2 4 8
Nodes in Schooner Cluster
Sysbench Read/Write
Reads/s
Writes/s
• Async/ semi-sync/ sync
– Issue: Scale read queries for a read
intensive application
• Typical solution
– MySQL replication allows unlimited
number of Slaves
• SchoonerSQL
– Schooner sync supports consistent reads from up to 7 Slaves
– Schooner async allows same unlimited number of Slaves

#3 Use of Flash Memory
– Issue: MySQL/InnoDB is IO latency sensitive, adding dozens of hard disks
does not scale well
– Issue: Slave is unable to keep-up with Master
– Issue: InnoDB flushing & check-pointing has unstable performance
– Use Flash to provide more IOPS to single threaded Slave
– Use Flash for faster random access to read and write database files
• SchoonerSQL
– SchoonerSQL is designed for fast storage (such as Flash memory) to reduce
writes, increase flushing and checkpoint efficiencies, executing more read
and write IOs in parallel.
– Slave parallelism and vertical scalability helps avoid sharding and provides
factors of better price/performance, order of magnitude when compared with
disks

#4 High Availability (HA)
– Issue: MySQL provides weak out-of-the-box availability
– Issue: Replication and application connections are not failed over
• Sync
– Issue: Requires failover of application connections
– Use a proxy
– Use Virtual IPs (MMM)
• SchoonerSQL for mission-critical applications
– Includes integrated VIP management and automatic failover
– Global transaction IDs in binary log allow sync, semi-sync* and async
replication connections to be automatically repaired/ redirected

#4 High Availability (HA): SchoonerSQL
Base
Sync
replication
(LAN)
Auto
client
connection
failover
Sync
node
failures
(up
to
7)
Auto
sync
connection
repair
Fault
tolerance
(sync
->
async
->
sync)
Async
WAN
connection
repair
WAN
level
failover
Two
tier
failure
detection
Self
healing

#5 Server Sprawl, Inefficient Hardware Utilization
– Issue: Software bottlenecks force premature sharding
Issue: Servicing reads or taking backup
on Slave conflicts with single
threaded Slave
– Issue: DRBD stand-by servers
– Use a Slave solely for backups
– Add Slave servers
• SchoonerSQL
– Provides high consolidation via high
vertical scalability and 2-10X faster Slave replication
1,693,476
5,326,066
5,080,958
1,782,231
0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
Percona 5.5.20 Schooner
ASYNC
Schooner SYNC Stock 5.5.21
commit/1800s
Slave commits on Sysbench

#5 Server Sprawl, Inefficient Hardware Utilization
SchoonerSQL Vertical Scalability
0
200
400
600
800
1000
1200
1400
1600
1 4 16 64
Utilization
(CPU
threads)
Total concurrent connections
CPU Scalability
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
1 4 16 64
IOPS
IO Scalability
Read IOPS
Write IOPS
0
5
10
15
20
25
30
1 4 16 64
Bandwidth
(MBps)
Network Scalability
Network in
Network out
SchoonerSQL scales on
multi-cores & Flash
Benchmark: 5000 warehouse TPCC-MySQL

#6 Full and Incremental Backup
– Issue: Slave instance may lag due to backup
• Sync
– Issue: Flow control may be triggered slowing down the commits
– Reserve a slave solely for backup (non-sync mode)
– Schedule or execute backup during periods with low write activity
– Throttle down backup speed (increasing the time taken to backup)
– Separate storage device and controller for backup target
• SchoonerSQL
– Slave parallelism reduces or eliminates these issues
– Schooner backup includes adaptive throttling to reduce contention

#7 WAN and Multi-Data Center Deployments
• Semi-sync/ Sync
– Issue: Increases in network latency slow down commits
– Issue: Failure detection over higher latency networks is slow
– Use semi-sync or async within DC (or with metro-area networks)
– Use async for high latency networks (WAN)
• SchoonerSQL
– Use sync and/or async within DC (or with metro-area networks)
– Use async for WAN (automatic failover supported)
– High throughput maintained across WAN, as permitted by network
bandwidth & quality

MASTER
Read Master
READS
REPL
Read
Master
Read Master
REPL
READS
READS
READS
Data Center #1
WRITES
WRITES
REPL
Synchronous
Cluster
MASTER
Read Master
READS
REPL
Read
Master
Read Master
REPL
REPL
READS
READS
READS
Data Center #2
Synchronous
Cluster
Parallel
Async
#7 WAN and Multi-Data Center Deployments: SchoonerSQL

#7 WAN and Multi-Data Center Deployments: SchoonerSQL
SchoonerSQL asynchronous replication performance

#8 Online Schema Updates
– Issue: Certain schema changes take minutes-hours and lock table for
the duration (blocking write transactions that write to this table)
– Issue: Adding index has similar effects
• Sync
– Issue: Requires extra machinery if async is not supported
– Use Flash memory for small-medium tables
– Leverage one of several solutions (open-ark, pt-online-schema..)
• SchoonerSQL
– Supports asynchronous and synchronous for an instance.
– Fully compatible with existing mechanisms to update schema.

#9 Online Maintenance and Upgrades
– Issue: Application failover is required before taking a MySQL instance
offline
• Sync
– Issue: To service load during maintenance, a temporary server may
need to be used and require sufficient manual work
– Use scripts, manual steps
• SchoonerSQL
– GUI and CLI includes features to migrate an instance to another
server and moving application connections. SchoonerSQL supports
automated full and incremental database provisioning and recovery.
– Automation reduces errors.

#10 Minimize Administration
– Issue: MySQL requires considerable configuration and administration
for replication, failure-detection, failover, backup/restore
– Add MySQL experienced members on the dev/ops team
• SchoonerSQL
– Auto provisioning of a Peer/ Slave
– Mark a sync instance as Master
– Auto-sync after an instance is restarted
– Schedule full & incremental backups
– Alerts for notification
– State and progress of startup and shutdown

SchoonerSQL: GUI for Management
• Integrated GUI simplifies administration
– Powerful 1-click actions for
• On-line provisioning of servers and MySQL instances
• On-line database migration, upgrades
• Master role is set on an instance with a click
• Create / Start / Stop / Remove MySQL instances
• Easy Server Management
– Assign VIPs to Masters, Read Masters and
Asynchronous Masters and Slaves
– Create Synchronous and Asynchronous groups or
clusters
– Change the configuration parameters through GUI

SchoonerSQL: Monitoring & Backup Schedules
• Monitoring and optimization:
• Extensive displays with resource utilization statistics
• Physical (cores, storage, network) and logical
(buffers, locks,…)
• CPU, Disk, RAM usage
• IO Read and Write kb/s
• Bytes In & Out kb/s
• Integrated full & incremental online backup
– Scheduled backup ( daily, monthly)
– Supports full database restore
– Adaptive backup that minimizes effect on database

SchoonerSQL: Alerts
• SchoonerSQL provides email based alerts that are very useful in
monitoring the database environment. Alerts include name, date & time,
severity, description and have configurable thresholds.
• Sample alerts
– Instance created/deleted
– Instance up/down
– Instance attached/detached
– Group created/removed
– Async failover
– VIP configuration

#11 MySQL Master-Master Setup (Active/Passive)
– Issue: Require quick failover upon active Master’s failure
• Async
– Issue: May loose transactions upon active Master’s failure
– Configure two MySQL instances as Master-Master (setup to replicate
to and from each other)
– Use external failure-detection and load (re-) direction mechanisms
(e.g. MMM, Flipper, HAProxy)
• SchoonerSQL
– SchoonerSQL sync cluster setup is similar to an active-passive
Master-Master setup. Automatic failure detection and application
failover is integrated in the solution.

#11 MySQL Master-Master Setup (Active/Passive)
MASTER
Read Master
READS
REPL
Read
Master
Read Master
REPL
READS
READS
READS
SchoonerSQL
WRITES
WRITES
REPL
Synchronous
Cluster
MASTER
Read Master
READS
Read
Master
Read Master
REPL
READS
READS
READS
SchoonerSQL
WRITES
WRITES
REPL
Synchronous
Cluster
Instantaneous Failover

#12 MySQL with DRBD Setup
• Async
– Issue: Possibility of data loss when active Master fails & immediate
failover is required
– Issue: Reconnecting Slaves to new Master is not straightforward
– Use semi-sync replication in v5.5
– Use DRBD to replicate at physical storage block device level
• SchoonerSQL
– Instead of wasting resources on a stand-by and long failover time
(recovery and warm-up), SchoonerSQL sync cluster provides logical
replication (avoiding corruption propagation in DRBD) with no data
loss, instantaneous failover and automatic failover of replication
connections (sync or async).

#13 Resilience
• Sync
– Issue: Sensitive to N/W quality, splits cluster at any fault
– Issue: Read capacity compromised
– Semi-sync falls back to async mode
– Async Slaves disconnect without affecting the Master, and reconnect
at a later time
• SchoonerSQL
– Sync falls back to async* without affecting the Master or
compromising the read capacity
* As of SchoonerSQL 5.1.5

#14 Point In Time Recovery (PITR)
• Sync
– Issue: Binary logs within a cluster may not contain transactions in the
same sequence. PITR becomes difficult if the same instance that
created backup files is unavailable.
– Backup database and binary log for all or majority of cluster instances
(increases disk space requirements and may cause slow down
during backup)
• SchoonerSQL
– SchoonerSQL Slave instances using any replication type commit in
the same order as Master, even when parallelizing writes. Global
transaction IDs help stitch state of any cluster member. Backup from
one instance of a cluster is sufficient.

#15 Delayed Slave
• Sync
– Issue: Instances in the cluster are in lock-step with Master and do not
read binary log (as an async Slave does)
• SchoonerSQL
– SchoonerSQL supports multiple replication types in the same
database cluster. An async Slave instance can be used for this
purpose.

#16 Very Large Databases
– Issue: Working set may be larger than
main-memory -> slow Slave replication
– Issue: Large databases take longer
to backup
• Typical solution
– Throttle backup and control database size
– Use snapshots for backup
• SchoonerSQL
– Adaptive backup helps with file copy based backup
– Parallel Slave replication helps to hide high rate of reads
– Gains seen hard disks as well as Flash
2200
1598
2190
700
0
500
1000
1500
2000
2500
Schooner 5.1.3
Async
Percona 5.5.20
Async
Commits
/
sec
TPCC-mysql Sample @ 0.5 TB
Master
Slave

#17 Automated Replication Failover
– Issue: MySQL does not include failover of replication
– Use custom external tool-kits
– Manual work when instances fail
• SchoonerSQL
– Integrated support for failover of all replication types

#17 Automated Replication Failover (with Sync replication)
MASTER
Read Master
READS
REPL
Read
Master
Read Master
REPL
READS
READS
READS
SchoonerSQL
WRITES
WRITES
REPL
Synchronous
Cluster
MASTER
Read Master
READS
Read
Master
Read Master
REPL
READS
READS
READS
SchoonerSQL
WRITES
WRITES
REPL
Synchronous
Cluster
Instantaneous Failover

#17 Automated Replication Failover (with Async replication) –
1/2
MASTER
Read Master
READS
REPL
Read
Master
Read Master
REPL
READS
READS
READS
Data Center #1
WRITES
WRITES
REPL
Synchronous
Cluster
MASTER
Read Master
READS
REPL
Read Master
REPL
READS
READS READS
Data Center #2
Synchronous
Cluster
Async

#17 Automated Replication Failover (with Async replication) –
2/2
MASTER
Read Master
READS
Read
Master
Read Master
REPL
READS
READS
READS
Data Center #1
WRITES
WRITES
REPL
Synchronous
Cluster
MASTER
Read Master
READS
REPL
Read Master
REPL
READS
READS READS
Data Center #2
Synchronous
Cluster
Replication Failover

#18 Mixed Hardware
– Issue: Smaller hardware for Slave causes it to lag
• Sync
– Issue: Cluster executes commits at the pace of the weakest H/W
– Use better (or identical) hardware for Slave servers
– Use Flash memory in Slave servers (alleviate single threaded Slave)
• SchoonerSQL
– SchoonerSQL sync commits at the pace of weakest H/W
– SchoonerSQL async with parallel replication has higher chances of
staying close with Master’s commit speed

#19 Virtualized or Cloud Environments
– Issue: IO latency and throughput is typically poor
• Sync
– Issue: Frequent failure detections stemming from resource starvation
– Working-set or database is maintained in main-memory (avoid read
IOs)
– Databases are heavily sharded (to reduce read and write IOs)
• SchoonerSQL
– Support sync only on well provisioned servers, typically with elevated
failure detection timeouts.
– Async provides speedup (parallel threads hide longer latencies)

#20 Elasticity Requirements
– Issue: Dynamically add/remove Slaves based on load or trends
• Sync
– Issue: Disables an instance while a new instances is provisioned
– Manual work, scripts, proxies
• SchoonerSQL
– Sync instance is online while provisioning another instance
– A click in the GUI provisions a Slave that starts servicing read load. A
click removes an instance from the cluster
– Dynamically add virtual IPs that load balance between instances

What is SchoonerSQL ?
Part Three
• SchoonerSQL is a complete
distribution of MySQL+InnoDB
– mysqld smells & feels the same
– 100% client compatible, uses same
database files
• Installer
• Cluster manager
– GUI, CLI
– Administer and monitor
• Backup & schedules
• System and MySQL metric
collection & display
• Utility to capture an incident
(optionally call-home)
• High performance & high
availability
• Minimizes replication
administration
– Automatic app failover
– Automatic repl connection repair
– Single click provisioning
• Online upgrades
• Online maintenance
• Elastic (user-driven) cluster
• Alerts

SchoonerSQL Benefits Summary
• 4-20x more throughput/server vs. MySQL 5.5
•Highest performance synchronous and
asynchronous replication clusters
High Performance and Scalability
• No service interruption for planned or
unplanned database downtime
•Instant automatic fail-over
• On-line upgrade and migration
• 90% less downtime vs. MySQL 5.5
• WAN/DR auto-failover
High Availability
Ease of Administration
• No error-prone manual processes
• Monitoring and Optimization
• No lost data
• Highest data consistency
High Data Integrity
• Full MySQL + InnoDB: not a toolkit
• Free your staff to build your business,
not a custom database
•TCO 50% cheaper than MySQL 5.5
Compelling Economics
Out-of-the-box Product
100% MySQL Enterprise
InnoDB Compatible
Broad Industry Deployment
• eCommerce, Social Media, Telco, Financial
Services, Education
• High volume web sites
• Geographically distributed websites

Thank You!
www.schoonersql.com
@schoonerinfo

20 Real-World Use Cases to help pick a better MySQL Replication scheme (2012)

More Related Content

What's hot

Similar to 20 Real-World Use Cases to help pick a better MySQL Replication scheme (2012)

Recently uploaded

20 Real-World Use Cases to help pick a better MySQL Replication scheme (2012)

Editor's Notes