Use Amazon EC2 Systems Manager to Perform Automated Resilience Testing in Your CI/CD Pipeline - DEV338 - re:Invent 2017

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Use Amazon EC2 Systems Manager to
Perform Automated Resilience
Testing in Your CI/CD Pipeline
AWS re:INVENT
D a v e B a k s h a n i
S e n i o r S o f t w a r e E n g i n e e r @ E x p e d i a
W i l l i e W h e e l e r
P r i n c i p a l A p p l i c a t i o n s E n g i n e e r @ E x p e d i a
N o v e m b e r 2 9 , 2 0 1 7

In 2016, Expedia, Inc., was
the 11th largest internet
company by revenue at
$8.77B.
THE PRICE OF DOWNTIME—A BACK-OF-THE-
ENVELOPE CALCULATION
Source: https://en.wikipedia.org/wiki/List_of_largest_Internet_companies

Uptime Downtime
Potential lost
revenue
99% 3.65d $87.7M
99.9% 8.76h $8.77M
99.99% 52.56m $877K
THE PRICE OF DOWNTIME—A BACK-OF-THE-
ENVELOPE CALCULATION

Keeping our site up protects
tens of millions of dollars per year

OUR TALK
• [Overview] Resilience engineering
• [Overview] Amazon EC2 Systems Manager
• [Demo] Attack your app with Amazon EC2 Systems
Manager
• [Demo] Run resilience tests in a CI/CD pipeline
• Implementing other attacks
• Q&A

RESILIENCE ENGINEERING
O V E R V I E W O F

RESILIENCE ENGINEERING GOALS
Keep the site up
Even when components fail

STORAGE MEMORY POWER
TRAFFIC CPU NETWORK
WHAT GOES WRONG (A PARTIAL LIST)

Overprovisioning and
redundancy
Auto-scaling Load-shedding
CAPACITY STRATEGIES

Fallback/failover Fail silent Fail fast
FAULT-TOLERANCE STRATEGIES
Circuit breaker Circuit breakerCircuit breaker

ANATOMY OF A RESILIENCE EXPERIMENT
Attack
system
Identify
desired
response
Verify
response

P R O D : R A N D O M I Z E DT E S T : C O N T R O L L E D
RESILIENCE EXPERIMENTS BY ENVIRONMENT

RECAP
 Goal: Keep the site up
 Many strategies available
 Run experiments in test and
prod

AMAZON EC2 SYSTEMS MANAGER
O V E R V I E W O F

• Automation
• Inventory
• Maintenance windows
• Parameter store
• Patch management
• State management
• Run command
WHAT I S AMAZON EC 2 SYST EMS MANAGER ?
Capabilities:
Systems Manager is a service to help manage your
Amazon Elastic Compute Cloud (Amazon EC2) and
on-premises instances

SYSTEMS MANAGER OVERVIEW
Systems Manager Service
EC2
Instance
Systems
Manager Agent
EC2
Instance
On-Prem
Instance
Systems
Manager Agent
Systems
Manager Agent

Step 1:
Create an IAM role with the
AmazonEC2RoleforSSM
policy
Step 2:
Attach the IAM role to your
EC2 instances
Step 3:
Install the Systems Manager
agent (if it’s not already
there)
SYSTEMS MANAGER SETUP

SYSTEMS MANAGER IN A NUTSHELL
• Systems Manager documents
• Run command

{
"schemaVersion": "2.0",
"description": "Install or uninstall the latest version or specified version of an AWS package.",
"parameters": {
"action": {
"description": "(Required) Specify whether or not to install or uninstall the package.",
"type": "String",
"allowedValues": [ "Install", "Uninstall" ]
},
... other parameters ...
},
"mainSteps": [{
"action": "aws:configurePackage",
"name": "configurePackage",
"inputs": {
"action": "{{ action }}",
"name": "{{ name }}",
"version": "{{ version }}"
}
}]
}
SYSTEMS MANAGER DOCUMENTS

SYSTEMS MANAGER RUN COMMAND
Supports remote management of instance configuration
Choose
Command
Document
Choose
target
instances
Specify
document
params
Execute

RECAP
We learned about:
 Systems Manager capabilities, architecture, and setup
 Systems Manager documents and Run command

Now the fun stuff

ATTACK YOUR APP WITH
AMAZON EC2 SYSTEMS MANAGER

ATTACKING SYSTEMS WITH SYSTEMS
MANAGER
Create
Command
Document
for attack
Execute
Run
Command
Systems
Manager
Agent runs
command

{
"schemaVersion": "2.0",
"description": "Blackhole an outbound port on an instance",
"parameters": {
"port": {
"type": "String",
"description": "(Required) Specify an outbound port to blackhole."
}
},
"mainSteps": [{
"action": "aws:runShellScript",
"name": "runShellScript",
"inputs": {
"runCommand": [
"iptables -A OUTPUT -p tcp --destination-port {{ port }} -j DROP"
]
}
}]
}
A SYSTEMS MANAGER ATTACK DOCUMENT

[DEMO]
ATTACKING YOUR APP WITH SYSTEMS
MANAGER

SAMPLE APPLICATION: WEATHER REPORT
Weather Report UI
Weather Report API
Weather
Report DB
OpenWeatherMap
API

Two versions of the app:
• Non-resilient
• Resilient (has breakers)
Weather Report UI
Weather Report API
Weather
Report DB
OpenWeatherMap
API

• Non-resilient
What we’ll cover:
Weather Report UI
Weather Report API
Weather
Report DB
OpenWeatherMap
API

• Non-resilient
What we’ll cover:
• Blackhole attack against DB
SAMPLE APP: WEATHER REPORT
Weather Report UI
Weather Report API
Weather
Report DB
OpenWeatherMap
API

[DEMO]

 We learned how to use
Systems Manager to
execute an attack
 We saw both
non-resilient and
resilient responses
RECAP

RUN RESILIENCE TESTS IN A CI/CD
PIPELINE

Use CI/CD pipeline for
ongoing verification
RESILIENCE TESTING

RESILIENCE TESTING IN A CI/CD PIPELINE
build
deploy
to test
security
scan
perf
tests
release
to test
deploy
to prod
release
to prod
resilience
tests

[DEMO]
RESILIENCE TESTING IN A CI/CD PIPELINE

WHAT WE’LL COVER
• Build pipeline with three resilience tests:
• Pre-test (happy path)
• Blackhole attack & fallback test
• Post-test (recovery)

RECAP
 We embedded our resilience test in a CI/CD pipeline

IMPLEMENTING OTHER ATTACKS

• Burn CPU
• Burn I/O
• Fail DNS
• Fail Amazon
DynamoDB
• Fail EC2
• Fail S3
• Fill disk
• Kill processes
• Network corruption
• Network latency
• Network loss
• Null route
SIMIAN ARMY ATTACKS
Source: https://github.com/Netflix/SimianArmy/tree/master/src/main/resources/scripts

Attack scripts are straightforward to implement

#!/bin/bash
cat << EOF > /tmp/infiniteburn.sh
#!/bin/bash
while true;
do openssl speed;
done
EOF
for i in {1..`nproc --all`}
do
nohup /bin/bash /tmp/infiniteburn.sh &
done
BURN CPU

#!/bin/bash
cat << EOF > /tmp/loopburnio.sh
#!/bin/bash
while true;
do
dd if=/dev/urandom of=/burn bs=1M count=1024 iflag=fullblock
done
EOF
nohup /bin/bash /tmp/loopburnio.sh &
BURN I/O

#!/bin/bash
# Block all traffic on port 53
iptables -A INPUT -p tcp -m tcp --dport 53 -j DROP
iptables -A INPUT -p udp -m udp --dport 53 -j DROP
FAIL DNS

 We learned how
Systems Manager helps
keep your site up
 Start testing and
enhancing your site
resilience today
WRAPPING UP

thankYou([
{
name: "Kuldeep Chowhan",
reasons: [
"Original concept"
]
}, {
name: "Jay Spang",
reasons: [
"Initial implementation in our resilience testing framework",
"Wrote the proposal that led to this presentation"
]
}
])
ACKNOWLEDGMENTS

Thank you!
F o r m o r e i n f o r m a t i o n :
h t t p s : / / t e c h b l o g . e x p e d i a . c o m / 2 0 1 7 / 1 1 / 2 1 / r e i n v e n t - 2 0 1 7 - u s e -
e c 2 - s y s t e m s - m a n a g e r - t o - p e r f o r m - a u t o m a t e d - r e s i l i e n c y -
t e s t i n g - i n - y o u r - c i c d - p i p e l i n e /

Use Amazon EC2 Systems Manager to Perform Automated Resilience Testing in Your CI/CD Pipeline - DEV338 - re:Invent 2017

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Use Amazon EC2 Systems Manager to Perform Automated Resilience Testing in Your CI/CD Pipeline - DEV338 - re:Invent 2017

Similar to Use Amazon EC2 Systems Manager to Perform Automated Resilience Testing in Your CI/CD Pipeline - DEV338 - re:Invent 2017 (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Use Amazon EC2 Systems Manager to Perform Automated Resilience Testing in Your CI/CD Pipeline - DEV338 - re:Invent 2017