Airbnb has served over 200,000,000 customers across 191 countries and is one of the largest database consumers on AWS. They have heavily adopted MySQL and have recently completed a migration to Amazon Aurora. In this session, Airbnb shares their story, including design considerations for operating at Airbnb scale, tips, tricks, and advice for others startups, and thoughts on why they decided to run on Aurora.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Airbnb Runs on Aurora
S i r i s h C h a n d r a s e k a r a n – P r i n c i p l e P r o d u c t M a n a g e r , A m a z o n A u r o r a
X i n y a o H u – E n g i n e e r M a n a g e r , A i r b n b
R u o b i n g L i – S o f t w a r e E n g i n e e r , A i r b n b
D A T 3 3 1
November 29, 2017

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Airbnb Overview

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Airbnb Overview
3,000,000
listings
65,000
cities
191
countries
An online marketplace and hospitality service

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Airbnb Architecture Overview

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database Business Requirement
• Avg latency: 1-2ms
• P95 latency: 10ms
• Average DB queries on a listing detail page: ~30
• QPS: 100K for the biggest cluster

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Airbnb’s Database Deployment Overview
• Mostly MySQL with very few Postgres / Oracle.
• All MySQL deployed on RDS@AWS from 2011.
• Hundreds of RDS clusters deployed. One of the largest RDS deployments in AWS.
• Managed by terraform.

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Challenges

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Weekly Traffic vs. Response Time
October 2013 to Present (Last 4 Years)

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Number of Services / Databases Running In Prod
Jan 2016 to Present (Past 2 years)

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data Growth
Data Growth for one cluster from Feb 2017 to Aug 2017

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Operational Challenges when Running at Web-Scale
• Takes more than a day to create a replica for big clusters.
• Takes more than 16 hours to restore a replica for big clusters.
• Need to warm up the disk layer for RDS.
• Frequent jitter.

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Operational Challenges when Running at Web-Scale
Jitter

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Operational Challenges when Running at Web-Scale
Jitter

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Operational Challenges when Running at Web-Scale
Jitter
10% Airbnb’s down-time can be
attributed to latency jitter.

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Alternatives
EC2 MySQL + DBA Team Aurora
• High Performance Local SSD.
• More controls and visibility to the system.
• Need to handle sharding in the near term.
• Need to staff a DBA team.
• No jitter
• Scale up to 64TB.
• Fully managed by AWS.
• More appealing features which are not
available in standard MySQL builds.

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Evaluation Metrics
Performance Compatibility Failure Scenarios Operational Overhead
• Latency
• Throughput
• Read / write
headroom
Evaluating on real production workload and use cases
• Ruby / Java Clients
• Transaction
isolation
• Durability setting
• Monitoring
• Tooling support
• Disk failure
• Network blip
• Master failover
• Create cluster
• Create read replica
• Change parameter
group
• Point in time
recovery (PITR)

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DBReplay
• In-house developed load testing framework.
• Replay production workload on restored production DB.
• Stress test DB with 1x ~ 10x workload.
• Help database headroom projection.
• Verify new infrastructure project (Aurora).

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DBReplay
Overview

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DBReplay
User Interface

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DBReplay
MySQL / Aurora side by side comparison
Aurora
MySQL 5.7

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Performance Metrics Summary*
2x
Read head room
4x
Write head room
- 75 %
Write Latency
* based on a write heavy workload, result may vary for different workload

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Non-Performance Metrics Summary
Compatibility Ruby / Java Clients Fully compatible with Ruby clients and
JDBC
Compatibility Transaction isolation Same behavior as MySQL
Compatibility Durability settings Does not lose binlog in crash
Compatibility Monitoring Comparable CloudWatch and MySQL
metrics
Compatibility Tooling support Fully compatible with mysqlbinlog /
mydumper / terraform
Failure Scenarios Network blip none
Failure Scenarios Time to do a master failover 2 minutes -> 20 seconds
Operational Overhead Time to create a read replica 1 day -> 5 minutes
Operational Overhead PITR 10 hours -> 1 hour

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Jitter
0% No jitter was observed during
Aurora evaluation.

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Requirements
• Minimize disruptions on end users and our product teams.
• Scalable enough to handle 100+ clusters.
• Migration should be revertible.

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration Timeline
July 2017 Sep 2017 Oct 2017
Start Aurora
Evaluation.
Finish the
evaluation. Start
running replica
traffic on Aurora
Migrate a few of
the most important
masters to Aurora.
Still keep RDS
MySQL fleet. Can
revert the
migration if
necessary.
Failover all
masters while
keeping RDS
MySQL fleet.
Dec 2017 Jan 2018
Delete all RDS
MySQL instances.

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Terraform
• Manages RDS resources with code.
• Automates DB cluster creation, update and deletion.
• Tracks complete history of DB changes.

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration Stage 1
Create Aurora Replicas

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration Stage 2
Shift Slave Traffic from RDS MySQL to Aurora

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DBProxy
• Maxscale open source MySQL database proxy.
• In-house developed features.
• Configurable connection pooling
• Requests throttling and back pressure
• Blacklist queries rejection
• Fast connections refresh
• Drop-in deployment as dbproxy services for core RDS databases.
• Allow scaling web service tier independently from database tier
• Proxy tier scale horizontally with capacity increase

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DBProxy
Fast Connection Refresh

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DBProxy
Fast Connection Refresh Cont’d

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration Stage 3
Master Promotion

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration Stage 3
Master Promotion cont’d

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration Stage 3
Master Promotion cont’d
• < 1 minute of write down time. No read down time.
• Stage 3 is revertible.
• Repeat this process for every cluster.

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migration Progress
25 %
Clusters migrated
70 %
Traffic migrated

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Conclusion & Future Work

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Benefits of Aurora
RDS MySQL Aurora
Read headroom 1x 2x
Write headroom 1x 4x
Jitter Yes No
Replica lag Up to minutes 20ms
Time to do a master
failover
2 minutes 20 seconds
Time to create replica 1 day 5 minutes
PITR 10 hours 1 hour

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Leverage Aurora features
• Clone
• Fast DDL
• Load data into S3 for disaster recovery

40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank You!

41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.