You store some of your most critical data at AWS. That data needs to be available to the application no matter what, and it must not be lost. In this session, we explain how to take your conventional data center's high-availability architecture to AWS, and we show you how you can rely on Amazon Relational Database Service (Amazon RDS) to help you do it. We go into detail on how to use the features in Amazon RDS MySQL and Amazon Aurora MySQL for unparalleled high availability and disaster recovery to keep your applications available and responsive. 

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MySQL High Availability and
Disaster Recovery
FromDoingitManuallytoAmazonRDSandAmazonAurora
Peter Zaitsev, CEO, Percona
D A T 3 6 1

3. © 2018 Percona.
3
About This Presentation
High Availability and Disaster Recovery
Implementing HA and DR manually
MySQL HA and DR with AWS DBaaS offerings

4. © 2018 Percona.
4
What Do We Want From Applications?
Work
correctly
Be “Up”
Have good
performance

5. © 2018 Percona.
5
From the Engineer’s Standpoint
• Many practices from design to proper
QACorrectness
• Benchmarks, load testing, capacity
planningPerformance
• Design, chaos engineeringHigh Availability

6. © 2018 Percona.
6
High Availability is Hard
Hard to foresee everything that can go wrong
Long tail of low probability problems
A lot of environment specifics
Depends on technology and operational practices

7. © 2018 Percona.
7
Engineered Systems
Like security, High Availability
favors well designed and
tested “Engineered Systems”

8. © 2018 Percona.
8
Pro Tip
Consider having expert
evaluation of your HA and DR
design, implementation and
operational practices.

9. © 2018 Percona.
9
High Availability and Disaster Recovery
High Availability
• High/medium event frequency
• Low/medium event impact
• No/minimum downtime
• No data loss
• No manual intervention needed
Disaster Recovery
• Rare event frequency
• High event impact
• Downtime may be allowed by
design
• Data loss may be allowed by design
• Manual intervention may be
needed

10. © 2018 Percona.
10
High Availability Versus Disaster Recovery
High Availability
•Failure of a single
server
•Failure of a single
Availability Zone
Disaster Recovery
•Multiple AZ failure
within region
•Major software issues
•Major security
incidents

11. © 2018 Percona.
11
High Availability Planning
What is the estimated time between events (MTBF)?
What is the acceptable impact to the system?
What is the mean time to recover (MTTR)?

12. © 2018 Percona.
12
Example: RAID5 Volume Disk Failure
Single SSD may have 1Mil+ hours between failures (100+ years)
With 1000s of disks in the data center, failure is not a rare event
Impact to the system: performance impact, loss of redundancy
Time to recover: hours

13. © 2018 Percona.
13
Achieving High Availability for a Database
Compute redundancy
Data redundancy and replication
Failover management
Service endpoint provisioning

14. © 2018 Percona.
14
HA and DR for MySQL
MySQL is often your “system of
record”
Appropriate HA and DR planning is
critically important

15. © 2018 Percona.
15
Choices for MySQL at AWS
Build your
own
Use AWS
DBaaS
services

16. © 2018 Percona.
16
Build Your Own
Advantages
•More flexibility
•More control
•Potentially lower
infrastructure costs
Disadvantages
•Additional time and
effort required
•Potentially
additional risks

17. © 2018 Percona.
17
Use DBaaS
Advantages
•Save time
•Reduce risk
•Empower
development team
Disadvantages
•Potentially higher
infrastructure costs
•Less flexibility
•Less control

18. © 2018 Percona.
18
Build Your Own HA Options for MySQL

19. © 2018 Percona.
19
Achieving Data Redundancy
Storage level Database level
External/application
level

20. © 2018 Percona.
20
Storage Level Redundancy
Mount EBS volume to another instance
DRBD
Clustered file systems

21. © 2018 Percona.
21
Database Level Redundancy
Classic MySQL replication
MySQL group replication/MySQL InnoDB Cluster
Percona XtraDB Cluster and other Galera-based technologies
MySQL Cluster (NDB)

22. © 2018 Percona.
22
External/Application Level
Manual
application
replication
Trigger-based
replication
Using Kafka as
message bus

23. © 2018 Percona.
23
Classic MySQL Replication

24. © 2018 Percona.
24
Classic MySQL Replication Properties
Asynchronous or semi-synchronous
Parallel (since MySQL 5.7)
Many masters to many slaves (MySQL 5.7)
No conflict resolution or “protection”
No built-in failover

25. © 2018 Percona.
25
Advanced MySQL Replication Topologies

26. © 2018 Percona.
26
MySQL Group Replication

27. © 2018 Percona.
27
MySQL Group Replication (New in 5.7)
“Group of peers”
Write-anywhere or dedicated writer
Asynchronous replication with flow control
Conflicts prevented through certification
Built in failover
No automated provisioning

28. © 2018 Percona.
28
Percona XtraDB Cluster/Galera

29. © 2018 Percona.
29
Percona XtraDB Cluster Topology

30. © 2018 Percona.
30
Percona XtraDB Cluster/Galera Properties
Write to any node
Certification based replication
Virtually synchronous; can ensure no stale reads
Built-in fail-over
Built-in node provisioning

31. © 2018 Percona.
31
Transaction Commit Flow
Node 1 (master) Node 2 (slave)
User Interaction
Time contributing
to response time
BEGIN
COMMIT
Responsetime
ofCOMMIT
Return
result of
COMMIT
query1
query2
query3
User transaction
Certification
Physical COMMIT (or
ROLLBACK)
Network
Event sent to
other node(s)
Certification
Apply event
(slave thread) Time out-
of-sync
Legend:

32. © 2018 Percona.
32
MySQL Cluster Properties
Niche use only
Synchronous replication
Initially designed for in-memory only
Built-in sharding
Very high network performance requirements
WAN replication with conflict resolution

33. © 2018 Percona.
33
MySQL NDB Cluster High Level Architecture

34. © 2018 Percona.
34
Failover Management
MySQL classic replication does not have
built-in failover management
Many custom made scripts were created
through the years; most of them are bad

35. © 2018 Percona.
35
• Highly available
• Manual and automated
failovers
• Auditing
• Plugin interface
MySQL Orchestrator

36. © 2018 Percona.
36
Service Point Provisioning
MySQL client library connects to
single host
How do you manage it during
failover?

37. © 2018 Percona.
37
Multiple Solutions
DNS based
Elastic load balancer
HAProxy
MySQL router
ProxySQL

38. © 2018 Percona.
38
100% open source
Well supported
Read-write splitting
Caching
Query rewrites
Load balancing
10k+ connection support
ProxySQL

39. © 2018 Percona.
39
3
9
Dedicated Shared ProxySQL

40. © 2018 Percona.
40
4
0
ProxySQL on Application Side

41. © 2018 Percona.
41
Backup and Restore
•MySQL Enterprise Backup
•Percona Xtrabackup
•MySQLDump/MyDumper
No HA
solution
replaces
need for
backup

42. © 2018 Percona.
42
Backup: PITR (Point-in-Time Recovery)
•MySQL binary log
archiving
•Storage level Continuous
Data Protection (CDP)
Do you need
to restore to
a specific
point in
time?

43. © 2018 Percona.
43
Flashback
•Can be done with some hacks
when using ROW binlog
format
•Does not work for table drops
or DDLs
Undo
unwanted
update/delete

44. © 2018 Percona.
44
Kubernetes?
Use Amazon Elastic Container Service for Kubernetes
(Amazon EKS) (Managed Kubernetes Service)
Roll your own Kubernetes on Amazon Elastic
Compute Cloud (Amazon EC2)

45. © 2018 Percona.
45
MySQL on Kubernetes
•Official MySQL Operator (Alpha)
•Presslabs MySQL Operator
•Percona is working on Operator
for Percona XtraDB Cluster
Early
Stage

46. © 2018 Percona.
46
100% free and open source
Based on industry leading technologies
Supports both self made and Amazon Relational Database
Service (Amazon RDS)
General health
Query performance
Replication and ProxySQL
Understanding Your Environment - PMM

47. © 2018 Percona.
47
Oh My…. It Sounds Complicated
Do it yourself MySQL High Availability at AWS is not easy
Percona provides Professional Services and Support to
assist
There is a modern way…

48. © 2018 Percona.
48
MySQL High Availability with DBaaS

49. © 2018 Percona.
49
DBaaS Options
• MySQL and MariaDB
• Minimal differences from open source
implementation
Amazon
RDS
• High performance engine designed for the cloud
• Additional features not present in open source MySQL
• May lag behind most recent MySQL release
Amazon
Aurora

50. © 2018 Percona.
50
Amazon RDS High Availability (Multi-AZ)

51. © 2018 Percona.
51
Amazon RDS High Availability Properties
Synchronous replication (no data loss on failover)
Failover time depends on transactional load (60-120 sec)
DNS record changes to healthy instance

52. © 2018 Percona.
52
Amazon RDS Read Replicas
Scaling reads
Reporting queries
Disaster recovery

53. © 2018 Percona.
53
Amazon RDS Backup and Snapshots
Backups are taken automatically
You can chose backup policy
Secondary is automatically used for backups if available
Backups suspend IO for a few seconds to few minutes
Specify backup window to manage it better
Create manual snapshots
Point in time recovery is available

54. © 2018 Percona.
54
Amazon Aurora offers
state-of-the-art High
Availability and DR
options for MySQL
Amazon Aurora

55. © 2018 Percona.
55
Single Instance Deployment
Data is replicated 6 ways across 3 availability zones
Any AZ failure causes no data loss and only brief
interruption of service
New instance is automatically created in case of instance
failure

56. © 2018 Percona.
56
Amazon Aurora Replicas
Replicas automatically promoted to primary upon failure
Faster than single instance recovery
Replicas can be used for read queries
Replication lag is lower than for MySQL replication

57. © 2018 Percona.
57
Amazon Aurora Deployment Overview

58. © 2018 Percona.
58
Amazon Aurora Read Replicas
Replication between Amazon Aurora clusters
.. or between Amazon RDS and Amazon Aurora
Often done between regions
Great feature for disaster recovery

59. © 2018 Percona.
59
Roll back the state of the
cluster
Supports DDL
Requires cluster restart
Amazon Aurora Backtrack

60. © 2018 Percona.
60
Built In Monitoring: Amazon CloudWatch

61. © 2018 Percona.
61
Built In Monitoring: Enhanced Monitoring

62. © 2018 Percona.
62
Performance Insights

63. © 2018 Percona.
63
Summary
You can build advanced custom MySQL HA architectures on MySQL
For many companies, Amazon RDS and Amazon is a great choice
Beware of “Unknown unknowns” in self-made MySQL HA architectures
Whatever path you take, Percona is ready to support you

64. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Peter Zaitsev, CEO, Percona

65. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.