How Architecture and Operations can Support Continuous Deployment: Architecting and operating a highly-available (HA) system without considering how a Continuous Deployment (CD) pipeline might affect it can lead to more risk than you're willing to accept. Building a flexible architecture that can degrade gracefully and remain operating gives a team confidence to deploy frequently. If you're moving to CD, your organization is likely realigning too: operators and support staff for production systems are more involved with developers and product releases. Operations planning is now an integrated effort with developers. This talk covers hints and tips for how to architect and operate HA systems that are under CD.

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
DevOps and Highly-Available Systems
Rich Alberth

2. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Approach
• Part 1: Understand High-Availability
• Part 2: Measure Deployment Risk
• Part 3: Minimize Probability and Impact

3. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
A system is
Highly-Available if …
Part 1:

4. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Degraded

5. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Trick Question: a system degrades when…
A. Out of capacity (ASG maxed)
B. Network outage (us-east-1a down?)
C. SNS unavailable (temporary credential
expired)

6. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Cause & Effect
Cause internallyis what happens to your application.
Effect externallyis what your app does to your users.
Database
failure
Can’t accept
orders
Can’t accept
orders
Delayed
orders

7. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Rule #1:
Internal ≠ customer
failure impact

8. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
High-Availability

9. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Risk Management
• Risk
• Probability
• Impact
• P × I
Part 2:
Low Med High
Low
Med
High
Amazon
SQS
AWS Elastic
Beanstalk
AWS
Lambda

10. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Reduce
Probability
Reduce
Impact

11. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Rule #2:
Reduce P × I

12. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Part 3: Architecture that Degrades Gracefully
Proactive: Reduce Probability
Reactive: Reduce the Impact
No single point
of failure
Build
horizontally
Build
vertically
Local cache Retry Reroute
1

13. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Proactive: No Single Point of Failure1
Fault-Tolerant System  Remove single points of failure
Highly-Available System  Remove single points of business failure

14. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Proactive: Build Horizontally
Amazon
RDS
Amazon EC2
Instances
Elastic Beanstalk Amazon S3
Bucket

15. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Proactive: Build Horizontally
Amazon
RDS
Amazon EC2
Instances
Amazon S3
Bucket
Amazon S3
Bucket
Amazon API
Gateway
AWS
Lambda
Web
browser

16. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Proactive: Build Horizontally
Web
browser
Amazon Simple
Queue Service
Amazon
ElastiCache

17. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Proactive: Build Vertically
Amazon
RDS
Amazon EC2
Instances
Elastic Beanstalk
Amazon S3
Bucket
External
Web Service
External
Web Service

18. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Proactive: Build Vertically
Amazon
RDS
Amazon EC2
Instances
Elastic Beanstalk
Amazon
S3
External
Web
Service
Amazon SWF
Worker Worker Worker Worker
External
Web
Service

19. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Rule #1:
Build ‘em small.

20. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Reactive: Local Cache
Other approaches concentrate on “all or nothing” semantics:
• Publish to SNS works, or fail-over to something else.
• Database there or not
• Web server up or fail-over to backup region
Local Cache:
• Serving some data is better than serving no data
• We degrade by using stale data until data store RTS

21. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Reactive: Local Cache
Amazon RDS
Database
Amazon EC2
Web
browser
AWS Elastic
Beanstalk
Application
Amazon
ElastiCache
Local file
cache
HTML5
Local
Storage
Amazon Simple
Queue Service
Mobile
Application
Amazon
CloudFront

22. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Rule #3:
Put some data from some
systems somewhere else too.

23. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Reactive: Retry
Wait a bit, try same request again
Try something else
Schedule it for later

24. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Reactive: Retry
AWS Elastic
Beanstalk
Amazon Simple
Queue Service
Mobile
Application
Amazon
S3
DynamoDB
Amazon
CloudSearch
Web
browser
AWS
Lambda
AWS
Lambda
Other Web
Service

25. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Reactive: Reroute
decider worker
Route 53 Elastic Load
Balancing EC2
Instances
Amazon SWF
worker
Amazon SQS
Amazon
Aurora
Dead
Letter
Queue

26. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Reactive: Reroute
AWS Elastic
Beanstalk
Amazon Simple
Queue Service
Mobile
Application
Web
browser
Amazon RDS
DynamoDB

27. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Reactive: Local Cache + Reroute
us-east-2
Elastic
Beanstalk
application
Route 53
Hosted Zone
Amazon
S3
DynamoDBAmazon
CloudSearch
sa-east-1
Elastic
Beanstalk
application
Route 53
Hosted Zone
Amazon
S3
DynamoDBAmazon
CloudSearch
+

28. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Rule #3:
Have a “Plan B”

29. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Operations to Degrade Gracefully
Proactive
Reactive
Canary
System
Monitors
Gremlins
Detection Playbooks

30. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Canary
Measure the user
experience,
Alarm when
distasteful.

31. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
System Monitors
Measure the internal
health of the system,
Alarm when it is not
within control.
Amazon
SQS
AWS Elastic
Beanstalk
AWS
Lambda
Monitor

32. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Custom System Monitor
Amazon Simple
Queue Service
Mobile
Application
Amazon
S3
AWS
Lambda
Other Web
Service

33. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Gremlins
Separate components
Create controlled failure
Audit what they do
Elastic
Load
Balancing
EC2
Instances
Amazon
Aurora
Amazon Simple
Queue Service
AWS
Lambda

34. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Detection
Time breakdown:
• Alert when a problem
probably will occur
• Alarm when a problem
happens
Control Range:
• Alarm over and under
• Alarm when data missing
0
20
40
60
80
100
1 6 11 16
CPUUtilization

35. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Playbooks
Document how to RTS: Return To Service
Versioned along with code
Reviewed and approved with code changes
Not about diagnosing problems, all about RTS

36. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Playbook: “Website 5xx errors”
Route 53 Elastic
Load
Balancing
EC2
Instances
Runbook:
1. Dig DNS name
2. Ping ELB IP
3. Ssh EC2 instances
4. Kill off EC2 instances one at a time
5. Fail-over to us-west-2

37. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Playbook: database intermittent errors
EC2
Instances
Amazon
Aurora
Amazon
ElastiCache
Runbook:
1. Connect to the Aurora endpoint
2. Create 2 new replicas
3. Isolate all databases

38. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
SWF worker EC2 instances keep rebooting
decider worker
Amazon SWF
worker
Amazon
Aurora
Runbook:
1. Don’t touch anything!
2. Monitor other metrics to look
for customer impact.

39. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Thank you