Scaling up to and beyond 10M users

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Adrian Hornsby
Cloud Architecture Evangelist – Amazon Web Services
Scaling up to and beyond 10M
users
@adhorn

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Now that’s a lot
of things to read!
This is NOT
where we
want to start!

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What are we scaling?

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
horizontally
vertically

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
People
Application
Network & Data
Infrastructure
Scalabilty 101

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Fundamentals

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Region and Availability Zones
Availability Zone a
Availability Zone bAvailability Zone c
Region

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Region (# of Zones)
Coming Soon!
AWS Global Infrastructure
55 Availability Zones within 18 geographic Region
+ 116 Points of Presence
(105 Edge Locations and 11
Regional Edge Caches)

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s start from…

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The “Must” from Day 1
• High quality code
• Version controlled
• Automation when possible
• Infrastructure as code
• CI/CD pipeline
• Security at every layer
• Cost conscious
• Test everything
• DR procedure
Operational Excellence

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Build Measure
Learn

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Users > 1

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Start simple, basic

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Simple Storage Service (S3)
Amazon S3

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon S3
App v0.1
Simple Static Website
http://poliko.adhorn.me.s3-website-us-east-1.amazonaws.com/
Bucket name: poliko.adhorn.me

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Route53
• Traffic Policies
• Latency Based Routing
• Geo DNS
• Weighted Round Robin
• DNS Failover
• Private DNS for Amazon VPC
• Health Checks and Monitoring
• Domain Registration
Highly available and scalable DNS web service.

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon S3
App v0.2
http://poliko.adhorn.me.s3-website-us-east-1.amazonaws.com/
Simple Static Website
Amazon
Route53 http://poliko.adhorn.me

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Custom backend

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
App v1a
Amazon
EC2
instance
Elastic IP
User
Amazon
Route 53
EC2 backend
www.example.com
54.223.92.16

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
App v1b
Docker
Container
Elastic IP
User
Amazon
Route 53
Containerized backend
www.example.com
54.223.92.16

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Managed
API Gateway
cache
Amazon
CloudWatch
API Gateway
Endpoints on
Amazon EC2
Any other publicly
accessible endpoint
AWS Lambda
functions
Amazon
CloudFront
API Gateway
User Amazon
Route 53
App v1c
Serverless backend

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Application
Elastic IP
Database
User
Amazon
Route 53
App v2.0
Adding the database www.example.com
54.223.92.16

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Self-managed Fully managed
Amazon EC2 Amazon
DynamoDB
Amazon RDS
Database options
SQL vs NoSQL

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
User Sign-Up, Sign-In
Access Control

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
User Sign-Up, Sign-In, and Access Control
Amazon Cognito
Secure and scalable user
directory
Social and enterprise
identity federation

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Application
Elastic IP
Database
User
Amazon
Route 53
App v2.1
with Authentication www.example.com
54.223.92.16
Amazon
Cognito

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Users > 1000

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Separation of content type

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Delivery Network
Amazon CloudFront
• Cache content at the edge
• Lower load on origin
• Dynamic and static content
• Streaming video
• Custom SSL certificates
• Low TTLs

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Application
Elastic IP
Database
User
Amazon
Route 53
App v3.0
Separate static assets from dynamic content
Amazon S3
Amazon
CloudFront
*.js
*.jpeg
*.mp4
Amazon
Cognito

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availability and Redundancy

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availability in Parallel
Component Availability Downtime
X 99% (2-nines) 3 days 15 hours
Two X in parallel 99.99% (4-nines) 52 minutes
Three X in parallel 99.9999% (6-nines) 31 seconds

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availability Zone 1 Availability Zone 2 Availability Zone n
Multi-AZ
Support Instance Failure
Application

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Highly available
• Health checks
• Session stickiness
• Monitoring / logging
• Content-based routing
• Container-based apps
• WebSockets
• HTTP/2
Elastic Load Balancer

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Web
Instance
RDS DB Instance
Active (Multi-AZ)
Availability Zone Availability Zone
Web
Instance
RDS DB Instance
Standby (Multi-AZ)
Load
balancer
App v4.0
Available & redundant application
User
Amazon
Route 53
Amazon
CloudFront
Amazon S3
Amazon
Cognito

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
App v4.0
Serverless option (be aware of limitations)
Amazon
DynamoDB
AWS LambdaAmazon API
Gateway
Browser
Amazon
CloudFront
Amazon
S3
Amazon
Cognito

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Caching layer

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon
ElastiCache
RDS DB Instance
Active (Multi-AZ)
Availability Zone Availability Zone
RDS DB Instance
Standby (Multi-AZ)
ELB
User
Amazon
Route 53
Amazon
CloudFront
Amazon S3
App v5.0
Stateless application
Web
Instance
Web
Instance

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Elasticache
Redis and Memcached
• Transient session data
• Shared state
• High-frequency counters
• Queues
• Leaderboards
• Lists, sets, …
In-memory data store and cache

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Microseconds response times at
millions of reads/sec from single DAX
cluster
• Reduce provisioned read capacity for
DynamoDB tables for tables with hot
data
Amazon DynamoDB Accelerator (DAX)
Read- and Write-Through
Application
DAX
DynamoDB

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Users > 100,000

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sunday Monday Tuesday Wednesday Thursday Friday Saturday
Typical weekly traffic to Amazon.com

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sunday Monday Tuesday Wednesday Thursday Friday Saturday
Typical weekly traffic to Amazon.com
Provisioned capacity

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
November
November traffic to Amazon.com

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
November
November traffic to Amazon.com
Provisioned capacity

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
November
November traffic to Amazon.com
Provisioned capacity
76%
24%

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Auto-Scaling • Compute efficiency
• Node failure
• Traffic spikes
• Performance bugs

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
November
November traffic to Amazon.com
with auto-scaling

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
RDS DB Instance
Active (Multi-AZ)
Availability Zone Availability Zone
RDS DB Instance
Standby (Multi-AZ)
ELB
App v6.0
Auto scaling groups
User
Amazon
Route 53
Amazon
CloudFront
Amazon S3
Web
Instances
Web
Instances ElastiCache
Auto-Scaling group
STATELESS

51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Users > 1,000,000

52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Databases

53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Read / Write Sharding
RDS DB Instance
Read Replica
App
Instance
App
Instance
App
Instance
RDS DB Instance
Master (Multi-AZ)
RDS DB Instance
Read Replica
RDS DB Instance
Read Replica

54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Database Federation
Users
DB
Products
DB
App
Instance
App
Instance
App
Instance

55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Database Sharding
User ShardID
002345 A
002346 B
002347 C
002348 B
002349 A
CBA
App
Instance
App
Instance
App
Instance

56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Transient state does not belong
in the database
BUT is perfect in the cache!

57. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Little story of Databases at Amazon
“A deep dive on how we were using our existing databases
revealed that they were frequently not used for their
relational capabilities. About 70 percent of operations were
of the key-value kind, where only a primary key was used
and a single row would be returned. About 20 percent would
return a set of rows, but still operate on only a single table.”
Werner Vogels

58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Prime Day 2017 Metrics
Block Storage – Use of Amazon Elastic Block Store (EBS) grew by 40% year-over-year, with aggregate
data transfer jumping to 52 petabytes (a 50% increase) for the day and total I/O requests rising to 835
million (a 30% increase).
NoSQL Database – Amazon DynamoDB requests from Alexa, the Amazon.com sites, and the Amazon
fulfillment centers totaled 3.34 trillion, peaking at 12.9 million per second.
Stack Creation – Nearly 31,000 AWS CloudFormation stacks were created for Prime Day in order to
bring additional AWS resources on line.
API Usage – AWS CloudTrail processed over 50 billion events and tracked more than 419 billion, all in
support of Prime Day.
Configuration Tracking – AWS Config generated over 14 million Configuration items for AWS
resources.

59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Users > 10,000,000

60. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Asynchronous patterns

61. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CAP Theorem
Consistency Availability Partition Tolerance
Data is consistent.
All nodes see the same state.
Every request is non-failing. Service still responds as expected
if some nodes crash.
Distributed System
In the presence of a network partition, you must
choose between consistency and availability!

62. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Eventual Consistency
… if no new updates are
made to a given data item,
eventually all accesses to that
item will return the last
updated value.
Availability
An eventually consistent system can
return any value before it converges!!
https://en.wikipedia.org/wiki/Eventual_consistency
Distributed System
Every request is non-failing.

63. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Process A Process B Process A Process B
Synchronous Asynchronous
Waiting
Working
Continues
get or fetch resultGet result

64. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Non-blocking UI
https://medium.com/@sophie_paxtonUX/stop-getting-in-my-way-non-blocking-ux-5cbbfe0f0158

65. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Message passing for async. patterns
A
Queue
B
A
Queue
BListener
Pub-Sub

66. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Web
Instances
Worker
Instance
Worker
Instance
Queue
API
Instance
API
Instance
API
Instance
API: {DO foo}
PUT JOB: {JobID: 0001, Task: DO foo}
API: {JobID: 0001}
GET JOB: {JobID: 0001, Task: DO foo}
Cache
Result:
{
JobID: 0001,
Result: bar
}

67. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Worker
Instance
Worker
Instance
Queue
API
Instance
API
Instance
API
Instance
Cache
Amazon SNS
Push Notification
User

68. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
RDS DB Instance
Active (Multi-AZ)
Availability Zone
Elastic Load
Balancer
Web
Instance
Web
Instance
Amazon
Route 53User
Amazon S3
Amazon
Cloudfront
ElastiCache
Worker
Instance
Worker
Instance
App v7.0
Full Decoupling
Queue Amazon SNS

69. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Microservices Architecture

70. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Event-driven patterns to the rescue

71. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How Lambda works
S3 event
notifications
DynamoDB
Streams
Kinesis
events
Cognito
events
SNS
events
Custom
events
CloudTrail
events
LambdaDynamoDB
Kinesis S3
Any custom
Invoked in response to events
- Changes in data
- Changes in state
Redshift
SNS
Access any service,
including your own
Such as…
Lambda functions
CloudWatch
events

72. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Event-driven using Lambda and S3
AWS Lambda:
Resize Images
Users upload photos
S3:
Source Bucket
S3:
Destination Bucket
Triggered on
PUTs

73. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Event-driven using Lambda and DynamoDB
Amazon
DynamoDB
AWS
Lambda
Amazon
DynamoDB Streams
Actions
Item Create
Item Update
Item Delete

74. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Monitoring & Steady State

75. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• Monitoring, metrics, and logging
• What are customers saying?
• Try to squeeze as much performance
out of each service/component
Make it ridiculously easy to start monitoring a new metric!
1 line of code rule

76. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
AGGREGATE
LEVEL
METRICS
LOG
ANALYSIS
EXTERNAL
SITE PERFORMANCE
HOST
LEVEL
METRICS

77. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Steady State
• ”normal” behavior of your system
https://www.elastic.co/blog/timelion-tutorial-from-zero-to-hero

78. Steady State
• ”normal” behavior of your system
• Business Metric
https://medium.com/netflix-techblog/sps-the-pulse-of-netflix-streaming-ae4db0e05f8a

79. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Increasing Resiliency

80. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Circuit Breaker
• Wrap a protected function call in a
circuit breaker object, which
monitors for failures.
• If failures reach a certain
threshold, the circuit breaker trips.
https://martinfowler.com/bliki/CircuitBreaker.html

81. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Service Degradation & Fallbacks

82. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Save yourself a rollback ..

83. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Immutable Infrastructure
No updates on live systems
• Always start from a new resource being provisioned
• Deploy the new software
• Test in different environments (dev, staging)
• Deploy to prod (inactive)
• Change references (DNS or Load Balancer)
• Keep old version around (inactive)
• Fast rollback if things go wrong

84. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
New Version
Users
Canary deployment for Immutable Infra.
Old Version
99%
Users
1%
Users
Start with ..
Dynamic Routing
(Route53)

85. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Time to Go Global!

86. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
James Hamilton – 2016 re:Invent
Vice President & Distinguished Engineer

87. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-Region Multi-VPC Connectivity

88. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S3 - Cross-Region Replication

89. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cross-Region Read Replicas for Amazon RDS
** For Aurora, MySQL,
MariaDB and PostgreSQL
engines.

90. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cross-Region Usage Pattern
• Regional Reads
• All critical writes traffic
to a single master

91. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Aurora multi-master - scale out reads & writes
First MySQL compatible DB service with scale-out across multiple data centers
Zero application downtime from ANY instance failure
Zero application downtime from ANY AZ failure
Faster write performance and higher scale
Sign up for single-region multi-master preview today;
Multi-Region Multi-Master coming in 2018
Availability
Zone 1
Scale out both reads and writes
Availability
Zone 2
Availability
Zone 3
Application
Read/Write
Master 1
Shared distributed storage volume
Read/Write
Master 2
Read/Write
Master 3

92. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon DynamoDB Global Tables (GA)
First fully managed, multi-master, multi-region database
Build high performance, globally distributed applications
Low latency reads & writes to locally available tables
Disaster proof with multi-region redundancy
Easy to set up and no application rewrites required
Globally dispersed users
Replica (N. America)
Replica (Europe)
Replica (Asia)
Global App
Global Table

93. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Users beyond 10,000,000

94. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fire Drills

95. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Chaos Engineering

96. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Testing to the next level: Failure Injection
• Start small & build confidence
• Application level
• Host failure
• Resource attacks (CPU, memory, …)
• Network attacks (dependencies, latency, …)
• Region attacks!

98. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
… at the Edge

99. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!
@adhorn
https://medium.com/@adhorn