In this workshop, we illustrate how to set up your own resiliency testing. We set up a simple three-tier architecture and explore the failure modes with Bash and Python scripts. To participate, you need an account that can run AWS CloudFormation, AWS Step Functions, AWS Lambda, Application Load Balancers, Amazon EC2, Amazon RDS (MySQL), and the AWS Database Migration Service, and Route53.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Resiliency Testing: Verify That Your
System Is as Reliable as You Think
Rodney Lester
Reliability Lead
AWS Well Architected
A R C 4 0 3

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Deploy testing infrastructure
AWS well-architected reliability
pillar
Scenario walkthrough
Lab work
Summary of tests and best
practices

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Related breakouts
Thursday, November 29
Chaos Engineering and Scalability at Audible.com
1 p.m. – 2 p.m. | Aria West, Level 3, Ironwood 5
Tuesday, November 27
Globalizing Player Accounts at Riot Games While Maintaining
Availability
1:45 p.m. – 2:45 p.m. | Aria East, Plaza Level, Orovada 2
Tuesday, November 27
How Intuit TurboTax Ran Entirely on AWS for 2017 Taxes
10:45 a.m. – 11:45 a.m. | Venetian, Level 2, Venetian F

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Download the lab guide
• Navigate to and download the lab guide
• https://bit.ly/2QjJASo
• Execute the first section: Deploying the Infrastructure
• We can start testing when the web servers are available in the Ohio Region
• This will take five minutes to start the execution and about 30 minutes until you can perform
tests
• The replication to a second region will be available about 25 minutes later

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is it?
“The reliability pillar encompasses the ability of a system to recover from
infrastructure or service disruptions, dynamically acquire computing
resources to meet demand, and mitigate disruptions such as
misconfigurations or transient network issues.” – AWS Well-Architected
Framework whitepaper
Design principles
• Test recovery procedures
• Automatically recover from failures
• Scale horizontally to increase aggregate system availability
• Stop guessing capacity
• Manage change using automation

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is it?
“The reliability pillar encompasses the ability of a system to recover from
infrastructure or service disruptions, dynamically acquire computing
resources to meet demand, and mitigate disruptions such as
misconfigurations or transient network issues.” – AWS Well-Architected
Framework whitepaper
Design principles
• Test recovery procedures
• Automatically recover from failures
• Scale horizontally to increase aggregate system availability
• Stop guessing capacity
• Manage change using automation

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Failure management
“Everything fails, all the time.” – Werner Vogels, Amazon CTO
We are going to work on withstanding component failures and planning
for recovery
Netflix OSS Simian Army, specifically “Chaos” suite, is an example of how
to test fault tolerance
• Chaos Monkey – injects failure on instances/containers
• Chaos Gorilla (deprecated) – simulates an AZ failure
• Chaos Kong – simulates an AWS Region failure

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Automated recovery
Manual recovery will take much longer
Use Amazon Web Services (AWS) to automate
• AWS Auto Scaling
• Amazon Relational Database Service (Amazon RDS)
• Amazon Route 53
• Amazon Simple Storage Service (Amazon S3)

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why are we here?
Just use Netflix OSS, right?
Wait, Chaos Monkey is written in Java or Go?
Wait, I need to share SSH keys to inject failure?
What do you mean there is no AZ failure?
(No more Chaos Gorilla?)
This creates fear, uncertainty, and doubt
Can I do this?

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Yes, you can do this!
You can write code to perform these scenarios in any language
You can use AWS Systems Manager and Amazon Elastic Compute Cloud
Run Command to inject failure without using SSH
You can write your own AZ failure simulation
You can simulate your own regional failure

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scenario for testing
AWS US-EAST-2 Region
Virtual private cloud
Availability zoneAvailability zone Availability zone
IGW routed subnet IGW routed subnet IGW routed subnet
Private subnet Private subnet Private subnet
Application
Load Balancer
MySQL DB
Multi-AZ
MySQL DB
Multi-AZ
Instance Instance
Auto Scale
app tier
Bucket
Instance Instance Instance Instance

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
But wait, there’s more….
AWS US-WEST-2 Region
Virtual private cloud
Availability zoneAvailability zone Availability zone
IGW routed subnet IGW routed subnet IGW routed subnet
Private subnet Private subnet Private subnet
Application
Load Balancer
MySQL DB
Multi-AZ
MySQL DB
Multi-AZ
Instance Instance
Auto Scale
app tier
Bucket
Instance Instance
Instance Instance
RDS DB
read
replica
AWS DMS

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
And even more
AWS US-EAST-2 Region
Virtual private cloud
Availability zoneAvailability zone Availability zone
IGW routed subnet IGW routed subnet IGW routed subnet
Private subnet Private subnet Private subnet
Application
Load Balancer
MySQL DB
Multi-AZ
MySQL DB
Multi-AZ
Instance Instance
Auto Scale
app tier
Bucket
Instance Instance Instance Instance
RDS DB
read
replica

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Start at “Discussion and Example Failure Scenarios”
Minimum requirements to start testing
Step Functions state machine should have completed “WaitForWebApp” state
No need to wait for read replicas or database migration instance
Decide which language you are comfortable using
• Bash
• PowerShell
• Java
• C#
• Python
Ask for help if you need it

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Start at “Cross Region Failure Scenarios”
Minimum requirements to start testing
Step Functions state machine should have completed “WaitForDMS” state, which is the last
state
Scenarios are limited due to features and student environment
requirements
Follow instructions
Ask for help if you need it

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Simulated failures
AWS US-EAST-2 Region
Virtual private cloud
Availability zoneAvailability zone Availability zone
IGW routed subnet IGW routed subnet IGW routed subnet
Private subnet Private subnet Private subnet
Application
Load Balancer
MySQL DB
Multi-AZ
MySQL DB
Multi-AZ
Instance Instance
Auto Scale
app tier
Bucket
Instance Instance Instance Instance
RDS DB
read
replica

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What did you learn?
We’d like you to lose the fear of writing these tests
The simulation might not be obvious – discuss and think about the effects
before implementation and revise based on results, but writing this code
is not difficult
On very large implementations, you’ll need to use orchestration to have
things happen almost simultaneously

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Things to watch out for
You will find problems and resolve them, but real failure might not look
like the simulation
Expect this to be an ongoing effort
Add these tests to your pipeline/acceptance testing
Some failure modes are destructive
Automated deployment will help bring the environment back
Failover is easier than failback
Transactions will be in flight
Flapping can be worse than an outage
Game day exercises are essential
Practice, practice, practice

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
This code is available for download
Download it at (this is also in the lab guide):
https://bit.ly/2Nb0XHg
It is licensed under the Apache License, Version 2.0
https://aws.amazon.com/apache2.0

29. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Rodney Lester
rodneyle@amazon.com

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.