Failure Testing: Automating a Series of Unfortunate Events Presentation by: Alper Kokmen PagerDuty Software Engineer

1. @alperkokmen
Failure Testing
AUTOMATING A SERIES OF UNFORTUNATE EVENTS
#OSCON

2. Alper Kokmen
PRESENT
Software Engineer at PagerDuty
Surrounded by smart people
PAST
Start-ups, Microsoft
Surrounded by smart people
#OSCON

3. #OSCON

4. Goals
Start manually injecting failures.
Start automating your manual tests.
#OSCON

5. CHAOS ENGINEERING
“[T]he discipline of experimenting on a
distributed system in order to build confidence
in the system’s capability to withstand
turbulent conditions in production.”
Principles of Chaos Engineering
http://principlesofchaos.org
#OSCON

6. Netflix Simian Army
DIFFERENT SIMIANS FOR DIFFERENT FAILURES
#OSCON

7. PagerDuty Simian Army?
Multiple cloud providers (AWS and Azure)
Experimentation
Application-specific failure scenarios
#OSCON

8. PagerDuty Simian Human Army
FAILURE FRIDAY
Time-boxed recurring meeting
Pre-announced agenda
Break things
Sign-off from service owners
Attendance
GROUND RULES
Keep monitoring & alerting
Abort if needed
#OSCON

9. Failure Friday: Agenda
#OSCON

10. Failure Friday: Process
#OSCON
Inject Failure
Monitor
Repeat

11. Failure Friday: Monitoring
#OSCON

12. 2 Years Later
BENEFITS
System design
Knowledge sharing
Incident response training
#OSCON

13. 2 Years Later
ACCOMPLISHMENTS
Whole DC outages
Target multiple services at once
Distribute failure testing to teams
Automation (in progress)
#OSCON

14. Automation: Rationale
#OSCON
“MANY” HOSTS
- Distribute tasks to multiple people and keep executing manually.
- Watch Operations team with envy while they use chef and knife.
- Start automating.

15. PagerDuty/blender
A MODULAR ORCHESTRATION ENGINE
Ruby DSL
Host Discovery (blender-chef, blender-serf)
Ranjib Dey (@RanjibDey)
#OSCON

16. PagerDuty/blender
CODE
#OSCON
# example.rb
ssh_task 'update' do
execute 'sudo apt-get update -y'
members ['ubuntu01', 'ubuntu02', 'ubuntu03']
end

17. PagerDuty/blender
EXECUTION
#OSCON
blend -f example.rb
Run[example.rb] started
3 job(s) computed using 'Default' strategy
Job 1 [update on ubuntu01] finished
Job 2 [update on ubuntu02] finished
Job 3 [update on ubuntu03] finished
Run finished (42.228923876 s)

18. PagerDuty/smoothie
A SIMPLE LIBRARY OF BLENDER RECIPES
Chef Integration
Recipes for Disaster
CLI to Specify Recipes
#OSCON

19. PagerDuty/smoothie
REBOOT RECIPE
#OSCON
def recipe__reboot(hosts)
ssh_task 'reboot' do
members hosts
execute 'sudo /sbin/reboot'
# shutdown will break ssh connection.
ignore_failure true
end
end

20. PagerDuty/smoothie
UNICORN SUSPEND & RESUME RECIPES
#OSCON
def recipe__unicorn_suspend_master(hosts)
ssh_task 'suspend unicorn[master] immediately' do
members hosts
execute 'sudo kill -s STOP `cat /u/.../pids/unicorn.pid`'
end
end
def recipe__unicorn_resume_master(hosts)
ssh_task 'resume unicorn[master] immediately' do
members hosts
execute 'sudo kill -s CONT `cat /u/.../pids/unicorn.pid`'
end
end

21. PagerDuty/smoothie
LATENCY RECIPE
#OSCON
def recipe__tc_add_latency(hosts)
ssh_task 'add network latency using tc' do
members hosts
execute 'sudo tc qdisc add dev eth0 root netem delay 500ms 100ms loss 20%'
end
end
def recipe__tc_remove_latency(hosts)
ssh_task 'remove network latency using tc' do
members hosts
execute 'sudo tc qdisc del dev eth0 root netem'
end
end

22. PagerDuty/smoothie
EXECUTION
#OSCON
HOSTFILTER=app1 RECIPE=reboot blend -f smoothie.rb
def recipe__reboot(hosts)

23. PagerDuty/smoothie
EXECUTION
#OSCON
ZONE=us-west-2a RECIPE=reboot blend -f smoothie.rb
def recipe__reboot(hosts)

24. Failure Friday: Blender
#OSCON
ZONE=us-west-2a ROLE=web-app RECIPE=monit_unmonitor
ZONE=us-west-2a ROLE=web-app RECIPE=monit_monitor
ZONE=us-west-2a ROLE=web-app RECIPE=unicorn_stop_master_gracefully
ZONE=us-west-2b ROLE=web-app RECIPE=unicorn_suspend_master
ZONE=us-west-2b ROLE=web-app RECIPE=unicorn_resume_master
ZONE=us-west-2c ROLE=web-app RECIPE=reboot
ZONE=us-west-2a ROLE=web-app RECIPE=iptables_network_isolate
ZONE=us-west-2a ROLE=web-app RECIPE=iptables_rebuild
ZONE=us-west-2b ROLE=web-app RECIPE=tc_add_latency
ZONE=us-west-2b ROLE=web-app RECIPE=tc_remove_latency

25. Future
AUTOMATION
Build more automation for service-specific scenarios.
Scheduled runs (similar to Netflix).
#OSCON

26. Future
CHATOPS
Inject failures by invoking chat commands.
Share metrics and graphs to help people follow along.
Collect TODOs during Failure Fridays and generate a report.
#OSCON

27. Future
NEW TYPES OF FAILURES
Distributed Denial of Service (DDoS) attacks for services.
Impediments that come up during Incident Response.
#OSCON

28. Summary
FAILURES WILL HAPPEN
Anything that can go wrong, will go wrong.
Proactively test failure handling now.
Start simple.
#OSCON

29. #OSCON
PROPOSED EDIT
“Experiments that aren’t introducing new
insights should be automated and used to
monitor ongoing health of the system. New
experiments should be devised to continue to
push the bounds of the system.”
Culture From Chaos by @dougbarth
https://speakerdeck.com/dougbarth/culture-from-chaos

30. Thank you.
#OSCON
@alperkokmen