A history of database replication methods, the MariaDB Replication Manager and it's use of HAProxy and MaxScale

1. 1
MariaDB-HaProxy

2. 2
1.
2.
3.

3. 13.5 Billion year ago was nothing !
For backup and Point In Time recovery the
BINARY LOGS
was born

4. Early after the replication was born in MySQL 3.23.
Replicas track binlog files position and get notified for new events !
IO Thread for copying events
SQL thread for applying events
BINARY LOGS ARE CROSS ENGINES
AND STATEMENT BASED

5. But the story get more complex
GOTCHA WITH BINARY LOGS
DON’T SCALE FOR WRITES
# FSYNC IS LIMITED ON DISK
Workaround was not to sync bin logs making repli
CRASH UNSAFE
MariaDB Fixed it in 5.5 with
TC + GROUP COMMIT

6. WRITE PERFORMANCE IS BACK
REPLICAS CRASH SAFE in 10.0
According to Facebook first implementation of google patch for group commit

7. But the story get more complex
NOW LEADER IS TO FAST REPLICAS CAN’T CATCH
DON’T SCALE FOR WRITES
Need Parallel Replication
Workaround was using Tungsten replicator, Galera Group replication , implement
prefetch events on slaves
MariaDB Fixed it in 10.0 and improved it 10.1 with
In order parallel replication and Optimistic Commit

8. WRITE PERF IS BACK ON REPLICAS
According to Booking slave group commits enable to go faster vs the workload x4
http://www.slideshare.net/JeanFranoisGagn/mysql-parallel-replication-inventory-usecase-and-limitations

9. In the last microsecond of evolution !
REPLICATION
BECOME CRASH SAFE AND REPLICAS CAN CATCH
BINARY LOGS
STILL NO GOOD FOR REPLICAS ELECTION

10. But the story get more complex
Replicas track position in files and not unique TX ID
Need GTID
Workaround: pseudo GTID, parse diff like MHA, TUNGSTEN or BINLOG server
MariaDB 10.0 Fixed it without gotcha
Domain ID - Server ID - Transaction seq #
No downtime implementation

11. But the story get more complex
TRX on Leader not guaranteed to be on any replicas
Fixing can be seen as regression ?
Workaround: Galera , NDB cluster, MongoDB
MariaDB 10.1 Not fixing only with Galera but improved Semi-Sync
Google patches for parallel Semi-Sync with group commit

12. In the last nanosecond of evolution !
MariaDB was feature ready for LEADER
ELECTION and so was born
Replication-Manager

13. 13
Replication-Manager

14. Switchover Workflow

15. Failover - False Positive Detection

16. Failover Monitoring Workflow

17. Maxscale
•
•
•
•
•
•
•
HaProxy
•
•
•
•

18. Features
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•

19. Settings
# REPLICATION
#skip_slave_start = 1
plugin_load = "semisync_master.so;semisync_slave.so;sql_errlog.so"
rpl_semi_sync_master = ON
rpl_semi_sync_slave = ON
loose_rpl_semi_sync_master_enabled = ON
loose_rpl_semi_sync_slave_enabled = ON
log_slave_updates = ON
slave_parallel_mode = optimistic
slave_domain_parallel_threads = %%ENV:NODES_CPU_CORES%%
slave_parallel_threads = %%ENV:NODES_CPU_CORES%%
relay_log = ./.system/repl/relay-bin
relay_log_index = ./.system/repl/relay-bin.index
relay_log_space_limit = 1G
log_bin = ./.system/repl/mariadb-bin
log_bin_index = ./.system/repl/mariadb-bin.index
binlog_format = ROW
binlog_checksum = 1
binlog_cache_size = 1M
binlog_stmt_cache_size = 1M
expire_logs_days = 5
sync_binlog = 1
replicate_annotate_row_events = 1
report_host='host_%%ENV:SERVER_ID%%'
# TOPOLOGY
hosts = "%%ENV:SVC_CONF_ENV_BACKEND_IPS%%"
user = "root:%%ENV:SVC_CONF_ENV_MYSQL_ROOT_PASSWORD%%"
rpluser = "root:%%ENV:SVC_CONF_ENV_MYSQL_ROOT_PASSWORD%%"
# LOG
logfile = "./dashboard/replication-manager.log"
#verbose = true
# HTTP & ALERTS
http-server = true
http-bind-address = "0.0.0.0"
http-port = "%%ENV:SVC_CONF_ENV_PORT_HTTP%%"
http-root = "./dashboard"
mail-from = "mrm@localhost"
mail-smtp-addr = "localhost:25"
mail-to = "mrm@mariadb.com"
# FAILOVER
autorejoin = true
readonly = true
wait-kill = 5000
post-failover-script = ""
pre-failover-script =""
# CHECK
check-type = "tcp"
failcount = 5
failover-limit = 3
failover-time-limit = 10
gtidcheck = true
maxdelay = 30
# HA PROXY WRAPPER MODE
# ---------------------
haproxy = true
haproxy-binary-path = "/usr/sbin/haproxy"
haproxy-write-port= %%ENV:SVC_CONF_ENV_PORT_RW%%
haproxy-read-port=%%ENV:SVC_CONF_ENV_PORT_R_LB%%

20. Downloads
•
Source & Documentation
•
Roadmap and Features Management
•
Talk to us
•

21. 21
Demo

22. •
•
•
•
○
○
○
○
○
22
MariaDB
Replication
Manager

23. Agent Stack
•
•
•
•
•
•
Collector Stack
•
•
•
•

24. Step 1 : User account bootstrap
•

25. Step 2 : Agent bootstrap
•
wget -O/tmp/opensvc.deb http://repo.opensvc.com/deb/current
sudo dpkg -i /tmp/opensvc.deb
•
sudo nodemgr set --param node.repopkg --value http://repo.opensvc.com/
sudo nodemgr set --param node.repocomp --value
http://repo.opensvc.com/compliance/
sudo nodemgr set --param node.dbopensvc --value
https://collector.opensvc.com

26. Step 2 : Agent bootstrap is generating a node conf file
•
[node]
repopkg = http://repo.opensvc.com/
repocomp = http://repo.opensvc.com/compliance/
dbopensvc = https://collector.opensvc.com/feed/default/call/xmlrpc
dbcompliance = https://collector.opensvc.com/init/compliance/call/xmlrpc
host_mode = PRD
uuid = d5bccb78-a2b2-4809-b036-ac7c7bfa7101
[compliance]
auto_update = true
schedule = @1440
[stats]
schedule = @60
[gcedisks]
scheduler = @120

27. Step 2 : Agent bootstrap
•
sudo nodemgr register --user my.self@corp.com
sudo nodemgr set --param node.uuid --value
d5bccb78-a2b2-4809-b036-ac7c7bfa7101
sudo nodemgr pushasset
sudo nodemgr pushdisks
sudo nodemgr pushpkg
sudo nodemgr pushpatch
sudo nodemgr checks
sudo nodemgr sysreport

28. Step 3 : Node configuration
sudo nodemgr compliance fix --attach --moduleset mariadb.node
sudo nodemgr compliance attach --moduleset mariadb.node
sudo nodemgr compliance check --moduleset mariadb.node
sudo nodemgr compliance fix --moduleset mariadb.node

29. Step 4 : Service deployment via agent template
docker_user [] > svar
subnet_cidr [10.0.0.0/24] >
backend_ips
[10.0.0.231,10.0.0.232,10.0.0.233] >
replication_manager_img
[tanji/replication-manager] >
ip_pod01 [10.0.0.229] >
mysql_root_password [mariadb] >
vip_netmask [10.0.0.1] >
ip_pod02 [10.0.0.230] >
port_r_lb [3308] >
subnet_name [spdnet] >
vip_addr [10.0.0.1] >
nodes [{nodename}] >
base_dir
[/Users/{env.docker_user}/{svcname}] >
maxscale_img [tanji/maxscale:keepalived] >
port_http [10001] >
port_rw [3306] >
port_rw_split [3307] >

30. Step 4 : Service deployment via collector
•
•

31. Step 4 : Service deployment
•
sudo svcmgr -s mysvc pull
•
sudo svcmgr -s mysvc provision

32. 32
Q&A

33. Thank You
33