Download as PDF, PPTX
Uploaded byJesse Robbins
Cloud Operations Bootcamp: Culture - Jesse Robbins
The document discusses operations culture and the importance of being prepared for failures. Some key points include: - Operations is the culture of an organization and the ability to consistently deploy software to unreliable platforms. - Failure happens and any system promising no failures is lying. Operations teams need to accept failures will occur. - The OODA (Observe, Orient, Decide, Act) loop is an important model for operations to quickly respond to issues and failures. - Teams should conduct regular "fire drills" to practice responding to hypothetical failures and ensure preparedness.
Cloud Operations Bootcamp: Culture - Jesse Robbins
- 1. Operations Culture Speaker: Jesse RobbinsCEO ‣ jesse@opscode.com ‣ @jesserobbins ‣ www.opscode.com 1
- 2. Today 2
- 3. Today ‣ Operations isCulture 2
- 4. Today ‣ Operations isCulture ‣ Failure Happens 2
- 5. Today ‣ Operations isCulture ‣ Failure Happens ‣ The OODA Loop 2
- 6. Today ‣ Operations isCulture ‣ Failure Happens ‣ The OODA Loop ‣ Do Fire Drills 2
- 7.
- 8. “You don’t choosethe moment, the moment chooses you. You only get to choose how prepared you are when it does.” -Fire Chief Mike Burtch 4
- 9. Cloud Operations isthe ability to consistently create and deploy reliable software to an unreliable platform that scales horizontally. http://radar.oreilly.com/2007/10/operations-is-a-competitive-ad.html 5
- 10. “It’s not mycode, it’s your machines! http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr 6
- 11. “It’s not mycode, it’s your machines! http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr 6
- 12. “It’s not mycode, it’s your machines! Spock Scotty Little bit weird Pulls levers & turns knobs Sits closer to the boss Easily excited Thinks too hard Yells a lot in emergencies http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr 6
- 13. No ngerpointing http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr http://www. ickr.com/photos/rocketjim54/2955889085/Reserved Copyright © 2010 Opscode, Inc - All Rights 7
- 14. Fingerpointyness problem!!! argggh! time http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
- 15. Fingerpointyness problem!!! argggh! freaking out, not talking, finding fault time http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
- 16. Fingerpointyness problem!!! argggh! freaking out, blaming, not talking, covering finding fault ass time http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
- 17. Fingerpointyness problem!!! argggh! freaking out, blaming, not talking, covering whining, finding fault ass hiding. hurt egos time http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
- 18. Fingerpointyness problem!!! argggh! freaking out, blaming, figuring it not talking, covering whining, out finding fault ass hiding. hurt egos time http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
- 19. Fingerpointyness problem!!! argggh! fixed freaking out, blaming, figuring it fixing things not talking, covering whining, out finding fault ass hiding. hurt egos time http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
- 20. Being productive problem!!! argggh! time http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
- 21. Being productive problem!!! argggh! figuring it out time http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
- 22. Being productive problem!!! argggh! fixed figuring it fixing things out time http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
- 23. Being productive problem!!! argggh! fixed figuring it fixing things feeling out guilty time http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
- 24. Being productive problem!!! argggh! fixed figuring it fixing things feeling move out guilty on with life time http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr
- 25. This will beon the test: FAILURE HAPPENS!
- 26.
- 27. Catastrophic Potential Simple Complexity Complex Tight Coupling Loose Created by Jesse Robbins "Catastrophic Potential" adapted from Normal Accidents by Charles Perrow 12
- 28. Catastrophic Potential Simple Complexity Complex Tight KEEP OUT!!! Coupling Loose Created by Jesse Robbins "Catastrophic Potential" adapted from Normal Accidents by Charles Perrow 12
- 29.
- 30. define: Nines (roughly) 99% 5256 min (3.5 days)
- 31. define: Nines (roughly) 99% 5256 min (3.5 days) 99.9% 528 min ( 8.8 hours )
- 32. define: Nines (roughly) 99% 5256 min (3.5 days) 99.9% 528 min ( 8.8 hours ) 99.99% 53 min
- 33. define: Nines (roughly) 99% 5256 min (3.5 days) 99.9% 528 min ( 8.8 hours ) 99.99% 53 min 99.999% 5 min
- 34. define: Nines (roughly) 99% 5256 min (3.5 days) 99.9% 528 min ( 8.8 hours ) 99.99% 53 min 99.999% 5 min 99.9999% 30 Seconds
- 35. define: Nines (roughly) 99% 5256 min (3.5 days) 99.9% 528 min ( 8.8 hours ) 99.99% 53 min 99.999% 5 min 99.9999% 30 Seconds 99.99999% 3 Seconds
- 36. 99.9% * 99.9% * 99.9% = 99.7% 14
- 37.
- 42. ;''-1(<"=/-)"3.1>0?-'"@'-': !"#$$%"&'(')*)"+,-.,-/01,( +/.01210*"345467"89: #
- 43.
- 44.
- 45. YOU Copyright © 2010Opscode, Inc - All Rights Reserved 21
- 46. Continuous Power... isn’t
- 47.
- 48.
- 49.
- 50.
- 51. Failure happens Asingle datacenter is the problem • Since they all fail at some point Recovery procedures after failure • Power was gone ~45 minutes • Most services took hours to come back • Some unnamed ones more than 12 hours
- 52. Geography is a SinglePoint of Failure
- 54. Copyright © 2010Opscode, Inc - All Rights Reserved 30
- 55.
- 56. Copyright © 2010Opscode, Inc - All Rights Reserved 32
- 57. Failure Happens. Anyone promisingotherwise is either foolish or lying (or both).
- 58.
- 59. OODA: Observe, Orient,Decide, Act http://en.wikipedia.org/wiki/OODA_loop 35
- 60.
- 61. Speaker: Jesse Robbins CEO ‣ jesse@opscode.com ‣ @jesserobbins ‣ www.opscode.com 37