Rebooting a Cloud
•
8 likes•2,718 views
Video of this talk is here: http://www.opscode.com/blog/2012/02/14/automate-all-the-things/
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (11)
Similar to Rebooting a Cloud
Similar to Rebooting a Cloud (20)
Recently uploaded
Recently uploaded (20)
Rebooting a Cloud
- 1. “Rebooting a Cloud” Jesse Robbins Cofounder, Opscode @jesserobbins http://www.flickr.com/photos/mwichary/2355832413/
- 3. “Rebooting a Cloud” http://www.flickr.com/photos/mwichary/2355832413/
- 5. Press button, Receive candy! (who here wants candy?) (there is always someone.) http://www.flickr.com/photos/mwichary/2355832413/
- 6. “Rebooting a Cloud” http://www.flickr.com/photos/mwichary/2355832413/
- 8. Complexity Increases, Latent Defects Accumulate, & FAILURE HAPPENS!
- 9. define: The Nines (roughly) 99% 5256 min (3.5 days) 99.9% 528 min ( 8.8 hours ) 99.99% 53 min 99.999% 5 min 99.9999% 30 Seconds 99.99999% 3 Seconds
- 10. 99.9% * 99.9% * 99.9% = 99.7% (oops!)
- 11. Everyone goes through the same thing...
- 13. ANGER “HOW THE $^@* COULD THIS BE HAPPENING!!!”
- 14. (more anger) “#@(* #$#$ %@%!!#$@ #* $#@$ @$*&#($*&@#(*$(”
- 15. Bargaining If we do this one thing, it will stop happening.
- 16. Depression What’s the point if this is just going to happen?
- 17. Acceptance “Failure Happens. We’ll just have to design for it.”
- 18. Successful companies say: “Failure Happens” “Embrace Failure” “Design For Failure” “Healthy attitude about Failure” “Resilient (to Failure)” THE OUTAGES WILL CONTINUE UNTIL THE APPROACH IMPROVES ;-)
- 19. GameDay Slide Courtesy of John Allspaw - http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr http://www.flickr.com/photos/dnorman/2678090600
- 20. define: GameDay An exercise designed to increase Resilience through large-scale fault injection across critical systems. Part of a larger discipline called Resilience Engineering. Not new, just new to us ;-)
- 21. Resilience is a product of People & Technology
- 22. GameDay increases Resilience in 3 ways Preparation ‣ Identification and mitigation of risks and impact from failure ‣ Reduces frequency of failure (MTBF) ‣ Reduces duration of recovery (MTTR) Participation ‣ Builds confidence & competence responding to failure and under stress. ‣ Strengthens individual and cultural ability to anticipate, mitigate, respond to, and recover from failures of all types. Exercises ‣ Trigger and expose “latent defects” ‣ Choose when discover them, instead of letting that be determined by the next real disaster.
- 26. full scale, live fire exercises http://tacomafiredepartment.blogspot.com/2010/05/west-slope-training-burn.html
- 27. safety standards & “building codes” http://www.flickr.com/photos/peregrinari/3801964067
- 28. no substitutes for experience... Failure free operations require experience with failure. Ana Grillo © Ana Grillo Photography
- 29. Lessons Learned: Every Post-mortem, Ever...
- 30. Root Cause: “a perfect storm of impossible events”
- 31. Lesson #1 “we have a bunch of manual processes which we need to automate”
- 32. (thanks to Hyperbole & A Half))
- 33. Infrastructure as Code: Enable the reconstruction of the business from nothing but a source code repository, an application data backup, and bare resources.
- 34. Lesson #1.5 “Golden Images don’t work in dynamic (ie: Cloud) Environments”
- 35. Golden Images are not the answer • Gold is heavy & expensive • Hard to transport • Hard to mold • Easy to lose configuration detail http://www.flickr.com/photos/garysoup/2977173063/
- 36. When this Varnish Jboss App Memcache Postgres Slaves Postgres Master
- 37. Becomes this... Varnish Jboss App Memcache Postgres Slaves Postgres Master
- 38. • Load balancer Change is only Constant config • Nagios host ping Varnish • Nagios host ssh • Nagios host HTTP • Nagios host app Jboss App health • Graphite CPU • Graphite Memcache Memory • Graphite Disk Postgres Slaves • Graphite SNMP • Memcache • Postgres Master 12+ resource changes for 1 node addition firewall • Postgres firewall
- 39. Scale & Complexity increase! AZ2 DC1 AZ3
- 40. CLONING CANNOT COPE WITH THIS http://www.flickr.com/photos/evelynishere/2798236471/
- 41. (thanks to Hyperbole & A Half )
- 42. Lesson #2 “We need better Incident Management”
- 45. ♥ Dev+Ops Culture Slide Courtesy of John Allspaw - http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
- 46. make it fun!
- 47. Lesson #3 “(load balancing|website|DNS| database|deployment|storage| provisioning|cloud|etc) failover didn’t work. We need to test & maintain our emergency tools & processes”
- 48. use it or lose it.
- 49. Give yourself lots of knobs & levers & use them as part of regular process.
- 50. MTTR > MTBF
- 51. I ❤ MTTR
- 52. (thanks to Hyperbole & A Half )
- 53. GameDay Slide Courtesy of John Allspaw - http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr http://www.flickr.com/photos/dnorman/2678090600
- 55. Thank You!!! Jesse Robbins Cofounder, Opscode @jesserobbins jesse@opscode.com