Downloaded 13 times
Uploaded byPagerDuty
Failure Friday: Start Injecting Failure Today!
The document discusses insights from the 'Failure Friday' event at DevOpsDays Toronto 2014, emphasizing the importance of testing failure handling in systems to prepare for inevitable failures. It highlights various attack strategies, the cultural impact of knowledge sharing on failure management, and encourages proactive automation of failure testing. Overall, it advocates for continuous improvement in reliability and response to outages within development and operations teams.
More Related Content
Similar to Failure Friday: Start Injecting Failure Today!
Failure Friday: Start Injecting Failure Today!
- 1.
- 2.
- 3.
- 4. 9/15/14 “DO NOT FEARFAILURE” BYTOMASZ STASIUK
- 5. 9/15/14FAILURE FRIDAY! How isbabby PagerDuty formed?
- 6.
- 7. 9/15/14 Designed for reliability FAILUREFRIDAY! Downstream providers fail 3 phone providers 3 email providers 6 SMS providers PagerDuty providers fail 2 cloud providers 3 data centers
- 8. 9/15/14 Hung up ondetails FAILURE FRIDAY! Bugs in exceptional code paths Systems not recovering as quickly as expected What is normal when things are abnormal?
- 9.
- 10. 9/15/14 Simian Army FAILURE FRIDAY! ChaosMonkey Latency Monkey Chaos Gorilla Chaos Kong “WP7WALLPAPER_EVIL_MONKEY_09” BY SKYLER817
- 11. 9/15/14 Keep it simple FAILUREFRIDAY! “KISS BAND MEMBER CUPCAKES” BY CLEVER CUPCAKES
- 12. 9/15/14 Process FAILURE FRIDAY! “HOW TODRAW AN OWL” BY CHESTER
- 13. 9/15/14 Get buy in FAILUREFRIDAY! “ANGRY BOSS” BY KAUSHAL KARKHANIS
- 14. 9/15/14 Schedule FAILURE FRIDAY! 1 hourrecurring meeting Developers & Operations List of attacks and identify victim Finish as much as possible
- 15. 9/15/14 Before starting FAILURE FRIDAY! Disablecron jobs & CM system Announce the start Open up relevant dashboards Leave alarms enabled
- 16. 9/15/14 Attacks FAILURE FRIDAY! Test asingle host and then DC 5 minutes Return to a working state Stop if things break
- 17. 9/15/14 Keep a log FAILUREFRIDAY! Keep track of actions taken Times are super important Also track discoveries and TODOs Share dashboards/metrics Chat rooms make this easy
- 18.
- 19. 9/15/14 Finishing up FAILURE FRIDAY! Soundthe all clear Enable crons & CM Move TODOs to issue tracker
- 20. 9/15/14 Attack Strategies FAILURE FRIDAY! “UNICORNATTACK!” BY SAM HOWZIT
- 21. 9/15/14FAILURE FRIDAY! SERVICE STOPCASSANDRA
- 22.
- 23. 9/15/14FAILURE FRIDAY! IPTABLES -IINPUT 1 -P TCP --DPORT 9160 -J DROP IPTABLES -I INPUT 1 -P TCP --DPORT 7000 -J DROP ! IPTABLES -I OUTPUT 1 -P TCP --SPORT 9160 -J DROP IPTABLES -I OUTPUT 1 -P TCP --SPORT 7000 -J DROP
- 24. 9/15/14FAILURE FRIDAY! TC QDISCADD DEV ETH0 ROOT NETEM DELAY 500MS 100MS LOSS 5%
- 25. 9/15/14 “RESULTS READER BOARD”BY ROSA SAY
- 26. 9/15/14 Issues fixed FAILURE FRIDAY! Aggressiverestarts by monit Large files on ext3 volumes Failing to restart due to bad /etc/fstab file High latency from network isolated cache Low capacity with a lost DC Missing alerts/metrics
- 27. 9/15/14 Cultural impact FAILURE FRIDAY! Knowledgesharing Highlights untestable systems Keeps failure handling on everyone’s mind
- 28. 9/15/14 Future plans “ROBOT SWORDSMANFIGHT.” BY PATRICK GAGE KELLEY
- 29. 9/15/14 Break more things FAILUREFRIDAY! Start testing whole DC outages Break multiple services at once Distribute failure testing to teams Automate
- 30. 9/15/14 Break more things FAILUREFRIDAY! Start testing whole DC outages Break multiple services at once Distribute failure testing to teams Automate
- 31. 9/15/14 Summary FAILURE FRIDAY! Failures willhappen Proactively test failure handling now Choose something easy: app server, cache Automate later
- 32.