This hands-on, instructor-led lab will guide you through the steps to perform basic resiliency testing of a simple web application. You will use the AWS Management Console to understand how to view what has been created, and what the effect of your testing is. This lab will also help you understand how to use the AWS command line interface. The skills learned will help provide you the ability to start a resiliency test repository for chaos engineering.

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
How to Determine if You Are Well-Architected for Reliability
Rodney Lester,
Reliability Pillar Lead, Well-Architected

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Setting Up Your Test Environment

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Downloading the Lab Guide
• Navigate to or download the lab guide: https://bit.ly/2rgjFAh
• Execute the first section: Deploying the Infrastructure.
• This will take about 5 minutes to start the execution and about 30 minutes to
deploy everything.

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Background on the AWS Well-Architected Reliability Pillar

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
What is it?
• “The reliability pillar encompasses the ability of a system to recover from
infrastructure or service disruptions, dynamically acquire computing resources
to meet demand, and mitigate disruptions such as misconfigurations or
transient network issues.” – AWS Well Architected Framework Whitepaper
Design principles
• Test recovery procedures
• Automatically recover from failures
• Scale horizontally to increase aggregate system availability
• Stop guessing capacity
• Manage change using automation

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
What is it?
• “The reliability pillar encompasses the ability of a system to recover from
infrastructure or service disruptions, dynamically acquire computing resources
to meet demand, and mitigate disruptions such as misconfigurations or
transient network issues.” – AWS Well Architected Framework Whitepaper
Design principles
• Test recovery procedures
• Automatically recover from failures
• Scale horizontally to increase aggregate system availability
• Stop guessing capacity
• Manage change using automation

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Failure Management
• “Everything fails, all the time.” - Werner Vogels, Amazon CTO
• We are going to work on withstanding COMPONENT failures and planning
for RECOVERY
• Netflix OSS Simian Army, specifically “Chaos” suite, is an example of how
to test fault tolerance
• Chaos Monkey—injects failures on instances/containers
• Chaos Gorilla (deprecated)—simulates an AZ failure
• Chaos Kong—simulates an AWS Region failure

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Automated Recovery
• Manual recovery will take much longer
• Use AWS services to automate
• Auto Scaling
• Amazon Relational Database Service (Amazon RDS)
• Amazon Route 53
• Amazon Simple Storage Service (Amazon S3)
Auto
Scaling
group

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Why are we here?
• Just use Netflix OSS, right?
• Wait, Chaos Monkey is written in Java or Go?
• Wait, I need to share SSH keys to inject failure?
• Wait, what about simulating an AZ Failure
• (no more Chaos Gorilla)?
• This creates fear, uncertainty, and doubt
• Can I do this?

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Yes, you can do this!
• You can write code to perform these scenarios in any language
• You can use AWS Systems Manager and EC2 Run Command to inject
failure without using SSH
• You can write your own AZ Failure simulation
• You can simulate your own regional failure

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Scenario: Simple three tier web application

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Scenario for testing
AWS US-EAST-2 Region
Virtual private cloud
Availability zoneAvailability zone Availability zone
IGW routed subnet IGW routed subnet IGW routed subnet
Private subnet Private subnet Private subnet
Application Load
Balancer
MySQL DB
Multi-AZ
MySQL DB
Multi-AZ
Instance Instance Auto Scale
App Tier
Bucket
Instance Instance Instance Instance

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Lab Time

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Start at Discussion and Example Failure Scenarios Section in
the Lab Guide

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Summary of Tests

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Simulated Failures
AWS US-EAST-2 Region
Virtual private cloud
Availability zoneAvailability zone Availability zone
IGW routed subnet IGW routed subnet IGW routed subnet
Private subnet Private subnet Private subnet
Application Load
Balancer
MySQL DB
Multi-AZ
MySQL DB
Multi-AZ
Instance Instance Auto Scale
App Tier
Bucket
Instance Instance Instance Instance

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
What did you learn?
We want you to lose the fear of writing these tests!

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Failure and Resiliency Testing
• The simulation might not be obvious—discuss and think about it before
implementation
• Writing this code is not difficult
• On very large implementations, you’ll need to use orchestration to have
things happen almost simultaneously
AWS Step
Functions
Lambda function
Lambda function
Lambda function

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Things to watch out for
• You will find problems and resolve them, but real failure might not look
exactly like the simulation
• Expect this to be an ongoing effort
• Add to your acceptance testing
• Some failure modes are destructive
• Automated deployment will help bring the environment back to
original deployment configuration
• Failover is easy; failback is more difficult
• Game day testing is essential
• Practice, practice, practice

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
This Code is Available for Download
• Download it at:
• https://bit.ly/2low3dY
• Which expands to:
• https://s3.us-east-2.amazonaws.com/aws-well-architected-labs-
ohio/Reliability/AWSLoftReliabilityLabCode.zip
• It is licensed under the Apache License, Version 2.0
• https://aws.amazon.com/apache2.0

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Thank you!