The document details a case study presented by Vinoth Kanna RS at a MyDBOps open-source database meetup, focusing on MySQL database transformation. Key highlights include the discussion of database observability challenges, high availability solutions, replication strategies, and significant operational savings achieved by optimizing infrastructure. The presentation emphasizes the importance of regular audits and defining recovery objectives (RPO and RTO) as part of database management best practices.

1. 07 October 2023
Mydbops Open Source Database Meetup 14
Vinoth Kanna RS
Co-Founder, Mydbops
MySQL Database Transformation Case Study

2. Interested in Open Source technologies
▪
MySQL, MongoDB Certified Professional
▪
Active MySQL Community Contributor
▪
Tech Speaker/Blogger
▪
Co-Founded, Mydbops IT Solutions, 2016
▪
Vinoth Kanna RS
About Me

3. Mydbops
at a
glance
300+
Clients
3000+
Servers
ISO & PCI DSS
Certified
7 Yrs of
Expertise
AWS Advanced
Consulting Partner
TiDB PingCap
Partner

4. Mydbops Services
Consulting
Services
Managed
Services
MySQL, MongoDB, PostgreSQL, TiDB & Cassandra
Targeted
Engagement
24 x 7
DBA Team

5. Observability
Challenges and Solutions
Implementation
Case Brief
Agenda

6. Case Brief
Audit DB infrastructure
▪
Reduce TCO without impacting Performance, Availability
▪
Plan for version upgrade
▪
Decouple application and database
▪
Propose and Implement best practices (HA, Backups, etc)
▪

7. Case Brief - MySQL DB Infra
DC 1
PDC Source PDC R 01 PDC R 02 PDC R 03 PDC R 700
DC 2
SDC Relay SDC R1 SDC R2 SDC R3 SDC R 700
DC 7
DC7 Relay DC7 R1 DC7 R2 DC7 R3 DC7 R 700
PDC - Primary DC SDC - Secondary DC R - Replica App + DB Hosted On All Servers
MySQL Version - 5.6

8. Observability
Generic Database Monitoring
Provide database stats
▪
Informative
▪
Organised based on component
or technology
▪
Difficulties in navigation
▪
Battlecat
Built to suit
▪
Decisive
▪
Translate Industry expertise to a
product that can be used to
manage databases at scale
▪
OS Exporters + Custom Exporters
▪

9. Observability
Concurrency
▪
Connections
▪
Bandwidth usage
▪
Query performance metrics
▪
Varying percentile of queries on the volume being queried
▪
Hotdata volume to be served from application
▪
Other OS metrics
▪

10. Observability
Average of 150 connections established from each application
▪
1,00,000 - 1,50,000 connections / DC
▪
Critical queries, Approx 10% returned data around 64MB of data / Query
▪
Avg. Active threads and QPS metrics were collected too.
▪
More you consolidate, More the concurrency, Lesser the time to react during issue
▪
Multiple applications and Additional databases on replica
▪

11. Choice Of Technology
Consistency
▪
Performance
▪
Resilency
▪
Fit to the product requirements
▪

12. Choice Of Technology - MySQL HA Solutions
Completely Managed Via MySQL Shell
Percona XtraDB Cluster / MariaDB Cluster (Galera Plugin Based)
Async Replication
▪
Semi Sync Replication
▪
Group Replication Cluster, 2016
▪
InnoDB Cluster, 2016
▪
InnoDB Replica Set, 2020
▪
InnoDB Clusterset, 2021
▪
Goal:
RPO = 0
RTO ~ Seconds

13. Choice Of Technology - Async Replication

14. Choice Of Technology - Semi Sync Replication

15. Choice Of Technology - InnoDB Cluster

16. Choice Of Technology - InnoDB Cluster Set

17. Choice Of Technology - MySQL HA Solutions
RTO (Recovery Time Objective)
RPO (Recovery Point Objective)
Failure Types:
How long does it take to recover from a single failure
▪
How much data can be lost when a failure occurs
▪
High Availability: Single Server Failure, Network Partition
▪
Disaster Recovery: Full Region/Network Failure
▪
Human Errors
▪

18. Choice Of Technology - Group Replication Cluster
MySQL's Native HA Solution
Synchronous Replication
Group replication has turned more stable, improved well with 8.0 release
Tuneable Consistency
Scale out reads effectively
Configurable Writer Modes
Built on robust replication framework uses binlogs with GTID for replication

19. Choice Of Technology - MySQL Compatible Load Balancers
MySQL Router
▪
Maxscale
▪
ProxySQL
▪
HA Proxy
▪
TCP / DNS based load balancing
▪

20. Choice Of Technology - Database Aware Load Balancers
Schema Based Routing
SQL Regex Based Routing
User Based Routing
Load Balancing
Replication, Group Replication,
Galera Replication Awareness
Connection Pooling
Audit Logging
Native Read Write Split
Query Caching
Query Mirroring

21. Choice Of Technology - ProxySQL
Ability to scale better in multi core systems
Clustering, Config management can be simplified
Measurable performance (qps or connections) / vpcu
Audit Logging, JSON Format, Introduced On v 2.0.6 Synchronous Replication
Dynamic / Runtime configuration changes, Provides granular control
Routing and Query statistics exposed are extensive and useful

22. Choice Of Technology - ProxySQL Scaling
Connection Pooling
Audit Logging
Multiplexing

23. Multiplexing
Choice Of Technology - ProxySQL Scaling
Application Servers ProxySQL
EC2 Instance
MySQL
Connection
Pool

24. Challenges
Multiple database engines
Issues in Schema, Charset, Collations
Instance bandwidth saturation
Connector level issues
GTID logging disabled

25. Callenges - MySQL 5.6 with GTID OFF
Source
MySQL 8.0.34
MySQL 5.6
GTID: OFF
MySQL 5.7
GTID: OFF GTID: ON
assign_gtids_to_anonymous
_transactions=LOCAL;
Node 1
Node 2 Node 3
MySQL
8.0.34
GR Cluster
Actual Implementation

26. Callenges - Auto Local GTID Creation
Master MySQL 8.0.34
GTID: OFF GTID: ON
Log Slave updates: ON
change replication source to
. . . . . . . . . . . . . . . . . . . . .
assign_gtids_to_anonymous_
transactions=LOCAL;
#210212 23:00:42 server id 320
end_log_pos 399 . . . . . Xid = 1245
use `test`/*!*/;
truncate table event_logs
/*!*/;
# at 399
SET @@SESSION.GTID_NEXT= 'bb71f03a-
1ab9-42ba-b3a2-1803de73944f:1224'/*!*/;
#210212 23:00:42 server id 320 Xid = 1245
use `test`/*!*/;
truncate table event_logs
/*!*/;
# at 399
Replicate To
GR Cluster

27. Implementation - PDC and SDC
Node 1
Node 2 Node 3
MySQL
8.0.34
GR Cluster
R1 R2 R3
R4 R5
ProxySQL
Cluster
From DNS LB
Weighted,
User Based
Routing
Writes
Consistent
Reads
App 1 Read
App 2 Read
App 3 Read

28. Implementation - Infra Savings
Before
~ 3500 servers Across 7 DC's
▪
EOL MySQL Version
▪
Unorganised data platform
▪
No Defined HA
▪
Operational complexity
▪
After
10 - 15 Servers / DC
▪
Est. Saving of 1 Milliion $
▪
Simplified Data Platform
▪
MySQL 8 Upgrade
▪
Measurable RTO, RPO
▪

29. Key Takeaways
Audit regularly
Scale with technology upgrades
Ensure DB Observability
Test Failover Strategies, Backups
System failure is inevitable even in database managed services
Define RPO, RTO
MySQL 5.7, End Of Life - 31st Oct 2023

30. Thank You!