This presentation covers our move to Google Cloud, the decisions that we made to use ProxySQL and MHA for MySQL to provide a high availability solution.

1. S
High Availability in
GCE
By Allan and Carmen Mason

2. Who We Are
S Allan Mason
Advance Auto Parts
“The largest retailer of automotive replacement parts and accessories in the United States.”

3. Who We Are
S Carmen Mason
VitalSource Technologies
"VitalSource is a global leader in building, enhancing, and delivering e-learning content.”

4. Agenda
S Current Infrastructure and Failover solution.
S HA requirements
S Something to manage our traffic
S Topology Managers
S Execution
S Final Infrastructure and Failover solution.

5. S
Infrastructure & Failover
Solution
Current Data Center (LaVerne)

7. Local Data Center
Solution Provides
• Saved Binary Logs
• Differential Relay Logs
• Notification
• VIP Handling

8. What a Failover looks like
S 7 to 11 seconds
S Automatically assigns most up to date replica as new master
S Automatically updates VIP.
S Replicas are slaved to the new master.
S A CHANGE MASTER statement is logged to bring the old master in
line once it’s fixed.
S Alerts by email, with the decisions that were made during the failover.

9. Herding Cats
(noisy neighbors)

10. “Firemen just ran into our datacenter
with a hose.”
- Bossman

11. S
Requirements & Issues

12. S

13. Why Google Cloud?
S Amazon is our competition.
S Our biggest reason.
S Amazon seems to cost more?
S It is hard to be sure.
S Reserved Instances
S Lost interest and capital opportunity cost though
S Lost flexibility
S Probably fine if you need a lot of instances, with a stable load.

14. Project Requirements
S Move our existing applications and databases to GCE.
S Infrastructure (INF) team CANNOT be the bottleneck.
S The initial request was to move to GCE in one year.
S Given 1 month to test and decide on MHA w/ MHA
Helper wrapper replacement.

15. Boss’s Mandate
S “Do things the right way.”

16. Second Generation High Availability
with CloudSQL!
Misconceptions
S CloudSQL high availability is only if the entire zone fails,
not the instance.
S This failover could take 5 minutes to even begin to trigger.
S Server options are MySQL and PostgreSQL (beta)
S No Percona Server?

17. Why Not ‘Lift and Ship’
S MHA with MHA Helper wrapper was not a viable option at
the time.
S GCE instances are CentOS 7, which MHA Helper didn’t
support.
S No VIP support in Google Cloud.

18. Our Requirements &
‘nice to have’s
S Try to at least reach our current uptime with a hands free
automatic failover.
S Availability: Min 99.9% (<8.76 hours/yr application uptime).
S Prefer 99.999% (<31sec. downtime/yr). Doesn’t everybody?
S VIP handling or traffic control to direct the apps to the database.
S All slaves are brought up to date with differential relay logs.
S Slaves still reachable are issued CHANGE MASTER statements.
S Old master is fenced off until we can get to it.

19. Traffic Control & Topology
S Two problems…
S Traffic control : How do the applications find the Master?
S Topology: Who’s the new Master, after current Master fails?

20. S
Traffic Control

21. The Players: Traffic Control
S MaxScale
S https://mariadb.com/products/mariadb-maxscale
S HAProxy
S http://www.haproxy.org/
S ProxySQL
S http://www.proxysql.com/

22. Why Not MaxScale?
S MaxScale
S New License: BSL
S Research showed that it does not scale above 6 threads as well
as ProxySQL.

23. Why Not HAProxy?
S HAProxy
S Although a viable solution, we wanted something written
specifically for MySQL, that provides additional functionality.

24. Why ProxySQL?
S ProxySQL
S Written specifically for MySQL.
S Monitors topology and quickly recognizes changes, forwarding
queries accordingly.
S Provides functionality that we liked, such as:
S Rewriting queries on the fly – bad developers!
S Query routing – read / write splitting
S Load balancing – balance reads between replicas
S Query throttling – If you won’t throttle the API…

25. ProxySQL Install
S Can be found in the Percona repo
S Each app is in it’s own Google Container Engine (GKE)
cluster.
S Every app gets it’s own container cluster.
S ProxySQL is in each of those containers ( aka tiled )
S No network latency
S No single point of failure

26. ProxySQL Config
S Do not run with the default config as is. Change the default
admin_credentials in
/etc/proxysql.cnf
S admin_credentials = "admin:admin”
S “SETEC ASTRONOMY” (Google it!)
S Mount the config file as a Kubernetes Secret

27. ProxySQL Config
S We use the remaining defaults in the configuration file for
now, with the following exceptions:
S mysql-monitor_ping_interval = 5000 # default = 60000 (ms)
S connect_retries_on_failure = 5 # default = 10

28. ProxySQL Failover
S Create the monitor user in MySQL.
S GRANT REPLICATION CLIENT ON *.* TO 'monitor'@'%'
IDENTIFIED BY 'password';
S ProxySQL monitors the read_only flag on the server to
determine hostgroup. In our configuration, if a server is in
hostgroup 0 it is writable. If it is in hostgroup 1 it is read-
only.
S hostgroup = 0 (writers)
S hostgroup = 1 (readers)

29. S
Replication Topology

30. The Players: Topology
S Master High Availability Manager with tools for MySQL
(MHA)
S https://github.com/yoshinorim/mha4mysql-manager/wiki
S Orchestrator
S https://github.com/github/orchestrator

31. Wooo Shiny!
Put a pin in it…
S Orchestrator
S Does not apply differential relay log events.
S Lack of extensive in-house experience with it.
S Set up in test environment to consider for near-term use.

32. What is Known
S Why MHA?
S Easy to automate the configuration and installation
S Easy to use.
S Very fast failovers: 7 to 11 seconds of downtime, proven several
times in production.
S Brings replication servers current using the logs from the most
up to date slave, or the master (if accessible).

33. Setup MHA

34. Setup MHA
S One manager node, three MHA nodes.
S Install a few perl dependencies.
S Install MHA node on all DB servers.
S Including on the manager node.
S Install MHA Manager.
S MHA assumes it runs as root.
S Create /etc/sudoers.d/mha_sudo
Cmnd_Alias VIP_MGMT = /sbin/ip, /sbin/arping
Defaults:mha !requiretty
mha ALL=(root) NOPASSWD: VIP_MGMT

35. MHA Requirements
S SSH with passphraseless authentication.
S Set databases as read_only (all but Master).
S log_bin must be enabled on candidate masters.
S Replication filtering rules must be the same on all MySQL
servers.
S Preserve relay logs: relay_log_purge = OFF;

36. MHA Sanity Check
S MHA Manager has built in Sanity Checks.
S If it is all good? It just listens...
[info] Ping(SELECT) succeeded, waiting until MySQL doesn't
respond..

37. MHA Failover
[warning] Got error on MySQL select ping: 2006 (MySQL server
has gone away)
S If the MHA Manager can ssh to the Master, it will save the
binlogs to the MHA Manager’s directory.
S Otherwise it copies the Relay Log from the most up to date slave.
S Polls 4 more times to see if the Master is really really not there.
S Polls the servers. Who’s dead? Who’s alive?
S Begins Phase 1: Configuration Check
S Phase 2: Dead Master Shutdown
S Phase 3: Master Recovery Phase

38. S
Execution of Our
Solution

40. “I have this TERRIBLE idea…”
DB Slave in
Datacenter
Server in
Old DR site
GCE DR
instance
Desperately trying to upload Terabytes of data
And then replicate it from the datacenter!

41. Moving the BIG guys
S Tried to use the VPN from the datacenter to GCE.
S Too unstable.
S Tried direct to GCE.
S Too slow
S Aspera to the rescue!
S http://www.asperasoft.com/

42. Xtrabackup
S On Source Server
S innobackupex
S --compress --compress-threads
S --stream=xbstream --parallel
S --slave-info
S On Destination Instance
S Install xbstream and qpress
S xbstream -x
S innobackupex --decompress

44. Trickle
S Trickle to throttle, and not kill our delicate network.
S -s run as a one-off vs trickled service
S -u set the upload rate (KB/s)
S GSUtil
S -m Parallel copy (multithreaded and multiprocessing)
S gs:// Google Storage bucket.
S trickle -s -u 75000 gsutil -m cp 2017mmddDB.xbstream gs://
mybucket/

45. Aspera
S Aspera
S -Q fair (not trickle) transfer policy
S -T Max throughput, no encryption
S -v Verbose
S -m 400M minimum transfer rate
S -l 800M target transfer rate
S -i key file
S ascp -Q -T -v -m 400M -l 800M -i /home/ascp/.ssh/id_rsa /path/
to/bkup/2017mmddDB.xbstream ascp@192.168.0.1:/target/dir

46. Pt-heartbeat
CREATE DATABASE dbatools;
CREATE TABLE `heartbeat` (
`ts` varchar(26) NOT NULL,
`server_id` int(10) unsigned NOT NULL,
`file` varchar(255) DEFAULT NULL,
`position` bigint(20) unsigned DEFAULT NULL,
`relay_master_log_file` varchar(255) DEFAULT NULL,
`exec_master_log_pos` bigint(20) unsigned DEFAULT NULL,
PRIMARY KEY (`server_id`)
pt-heartbeat -D recovery --update -h localhost --daemonize
pt-heartbeat -D recovery --monitor h=masterIP --master-server-id masterID -u
dbatools -p Password

47. Failing Prod over to GCE
S Database slave in GCE production environment.
S Run parallel builds, where possible, to test against.
S Cut over in our time, during a maintenance window.

48. Switch to GCE Master
S Stop writes to current master:
S screen
S FLUSH TABLES WITH READ LOCK;
S SET GLOBAL READ_ONLY = 1;
S Keep this window open.
S Stop slave on new master:
S SHOW SLAVE STATUSG to check that the slave is waiting for the master to
send another event. (Not currently updating)
S STOP SLAVE;
S RESET SLAVE ALL;
S Stop service on current master:
S service mysql stop
S DNS changes are made here in both Google and our local DNS servers.
S Start MHA manager now that we are no longer replicating back to DC

49. S
Final Infrastructure &
Failover solution
Google Cloud

50. Final Infrastructure
Solution Provides
• Saved Binary Logs
• Differential Relay Logs
• Notification
• No silos
• No SPOF
• Beautifully Scalable

51. What a Failover Looks Like
S 10-20 seconds
S Automatically assigns most up to date replica as new master
S ProxySQL directs traffic to master.
S Replicas are slaved to the new master.
S A CHANGE MASTER statement is logged to bring the old master in
line once it’s fixed.
S Alerts by email and notifications in HipChat, with the decisions that
were made during the failover.

52. New Issues
S ~10ms lag between regions, which means…
S All database instances are in the same region as the application
GKE clusters.
S Avoid allowing Central instance to be master to prevent lag, as
all GKE clusters are in an East region.
S no_master

53. Contact Info
S Allan Mason
S allan@digital-knight.com
S Carmen Mason
S Carmen.mason@ingramcontent.com
S http://www.vitalsource.com
S @CarmenMasonCIC

54. S
Thanks!
go away