Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

2012 Velocity London: DevOps Patterns Distilled


Published on

2012 Velocity London,
Presentation by Patrick Debois (@patrickdebois), 
Damon Edwards (@damonedwards), Gene Kim (@realgenekim), 
John Willis (@botchagalupe)

  • Be the first to comment

2012 Velocity London: DevOps Patterns Distilled

  1. 1. DevOps Patterns Distilled Patrick Debois (@patrickdebois)Damon Edwards (@damonedwards) Gene Kim (@realgenekim) John Willis (@botchagalupe) Velocity Europe 2012 1
  2. 2. Source: Allspaw/Hammond (2009) 2
  3. 3. Source: Allspaw/Hammond (2009) 3
  4. 4. Source: Allspaw/Hammond (2009) 4
  5. 5. Source: Allspaw/Hammond (2009) 5
  6. 6. Source: John Jenkins, 6
  7. 7. Every Company Is An IT Company… 95% of all capital projects have an IT component… 50% of all capital spending is technology-related Where we need to be… IT is always in the way (again…) We are here…
  8. 8. The DevOps Cookbook (Coming H1 2013) John Allspaw (@allspaw) Patrick Debois (@patrickdebois)Damon Edwards (@damonedwards) Gene Kim (@realgenekim) Mike Orzen (@mikeorzen_leanit) John Willis (@botchagalupe) 8
  9. 9. The First Way:Systems Thinking
  10. 10. The First Way:Systems Thinking (Left To Right) Understand the flow of work Always seek to increase flow Never unconsciously pass defects downstream Never allow local optimization to cause global degradation Achieve profound understanding of the system
  11. 11. The Second Way:Amplify Feedback Loops
  12. 12. The Second Way:Amplify Feedback Loops (Right to Left) Understand and respond to the needs of all customers, internal and external Shorten and amplify all feedback loops: stop the line when necessary Create quality at the source Create and embed knowledge where we need it
  13. 13. The Third Way:Culture Of Continual Experimentation & Learning
  14. 14. “Devops Areas”a way to ‘codify’ problems/solutions
  15. 15. Area 1: Extend delivery to production “think Jez Humble” Area 1DEV OPS
  16. 16. Area 2: Extend operations feedback to project think “John Allspaw” DEV OPS Area2
  17. 17. Area 3: Embed Dev into Ops think “Adrian Cockcroft” Area 3DEV OPS
  18. 18. Area 4: Embed Ops into Dev think “Chris Read”DEV OPS Area 4
  19. 19. Area 3: Embed Project knowledge into Operations Area 1: Extend delivery to productionDEV OPS Area 2: Extend operations feedback to project Area 4: Embed Operations knowledge into Project
  20. 20. The Third Way:Culture Of Continual Experimentation & Learning Foster a culture that rewards:  Experimentation (taking risks) and learning from failure  Repetition is the prerequisite to mastery Why?  You need a culture that keeps pushing into the danger zone  And have the habits that enable you to survive in the danger zone
  21. 21. Area 1:Extend Continuous Deliver Into Production Patrick Debois @patrickdebois 21
  22. 22. GOALS Big Goal Refocus on Business View End-To-End DEV Practical Goal QAOPS Get conversation started Business Bring the pain forward 22 22
  23. 23. Step #1 - Re-Establish Trust Co-location Teams Face to Face meetings IRC, Chat, Group feeling Align Management Goals HR Policies 23
  24. 24. Step #2 - Understand your bottleneck •Understand (a) ‘shared’ goal •Value Stream Mapping •Bug Reports, Backlog •Where? •Tools, Process, People 24
  25. 25. Step #3 - Think Continuous Integration + Infra as Code •Version Control Everything •Single Repository of Truth •One step Dev, Test, Prod Environment build process “Technology” Focused 25
  26. 26. Step #4 - Think Continuous Delivery •Extend Release into Prod •Reduce Technical Debt •Definition of Done •Visualize Tasks/Bugs Mind Shift 26
  27. 27. Step #5 - Integrate other roles in the process QA CAB Security Management 27
  28. 28. Devops Anti Patterns 28
  29. 29. Anti Pattern #1 - Config Mgmt = Devops Tools Process Culture 29
  30. 30. Especially for JohnCULTURE 30
  31. 31. Repeat after me 31
  32. 32. Anti-Pattern #2 - Guerrilla Devops Integrate into the process Only local changes don’t have much effect 32
  33. 33. Anti-Pattern #3- Start a separate devops group(Yet Another Silo)Act as a change agent 33
  34. 34. Anti-Pattern #4 - Silo X takes overIf devs take over, theyhave to (re)-learnproduction Symmetry of Ignorance/Arrogance 34
  35. 35. Anti-Pattern #5 - Sell it as a buzzword 35
  36. 36. Anti-Pattern #6 - Organizational Inertia Group experiment: > 80% people invest: Investors 80$ , other 0$ < 80% people invest: Investors -10$ , other 0$ Convergance to invest or not invest depends on initial group decision Nash Equilibrium - Game Theory 36
  37. 37. Outcomes Business Goal(s) Shared Process Trust People Robust Technology 37
  38. 38. Area 2:Provide Feedback and Visibility Damon Edwards @damonedwards 38
  39. 39. GOAL Provide feedback and visibility 39
  40. 40. GOAL Provide feedback and visibility ...but why? 40
  41. 41. GOAL Provide feedback and visibilityto align your organization’s improvement efforts 41
  42. 42. HOW DO YOU ALIGN YOUR ORGANIZATION? 1. Clear goals and operating instructions 2. Shared situational awareness 42
  44. 44. 44 People & Process DataApplication Situational Infrastructure Data Awareness Data Business Data FOUR TYPES OF DATA YOU NEED
  45. 45. Step 1: MAKE ALL INFRASTRUCTURE DATA VISIBLE DaBusi • Network, Disk I/O, Memory, Utilization, etc... • Present data in context of the application • Standardize and extend to allational Data environmentsreness Application • Create awareness of deviations from norm Dat& Pro Peo 45
  46. 46. STEP 2: MAKE ALL APPLICATION DATA VISIBLE ata iness• Performance, faults, availability, logs, etc...• Dev takes ownership of instrumenting their applications, but anyone can view or Data eness Infrastructure extend tional• Enable self-service metric creation (“one line of code”)• Increase signal, decrease noise ata ocess ople 46
  47. 47. STEP 3: BREAK BUSINESS DATA OUT OF IT’S SILO Data Awareness Data Infrastructure Situational Application Data Process • Sales, signups, churn, clickstream, etc... Key Business Metric • Make goals explicit (KPIs, one metric that matters) Secondary Business Metric • Link all other metrics to business metrics Technical/Process Metric • Empower improvement by showing cause and effect My activity 47
  48. 48. STEP 4: COLLECT AND VISUALIZE ORGANIZATION &PROCESS DATA • Change activity, quality, cycle time,effectiveness, etc... • Focus on effectiveness, not efficiency • Visualize the flow across the entire lifecycle • Capture change data and enable overlays on any graph Data Business 48
  49. 49. 49 Organization & Process DataApplication Situational Infrastructure Data Awareness Data Business Data USE TO DRIVE CONTINUOUS IMPROVEMENT
  50. 50. “PAINT THE WALLS” 50
  51. 51. Area 3:Embed Dev into Ops Gene Kim @realgenekim 51
  52. 52. 52
  53. 53. Goals Shorten and amplify feedback loops Create knowledge and capabilities where we need it Ensure that we’re optimizing for the entire system 53
  54. 54. “We found that when we woke up developersat 2am, defects got fixed faster than ever” Patrick LightbodyFounder/CEO, BrowserMob 54
  55. 55. IT Operations As The Developers’ Best Friend Tom Limoncelli Patrick Debois Adrian Cockcroft 55
  56. 56. Require That Dev Initially Maintain Their Own ServiceSource: Tom Limoncelli, Google (Usenix 2012) 56
  57. 57. Test Whether Developers Qualify For IT Operations Resources Types/frequency of pager alerts Maturity of monitoring System architecture review Release process Defect counts and severity Production hygieneSource: Tom Limoncelli, Google (Usenix 2012) 57
  58. 58. Return Fragile Services Back To DevSource: Tom Limoncelli, Google (Usenix 2012) 58
  59. 59. Integrate Dev Into IT Operations Integrate Dev into IT Operations escalation processes Have Dev cross-train IT Operations staff Have Dev improve the environment 59
  60. 60. Eric Passmore 60
  61. 61. Area 4:Embed Ops Into Dev John Willis @botchagalupe 61
  62. 62. Devops Alignment 62
  63. 63. Embed Ops into Dev 63
  64. 64. Why• Seeing End to End• Sharing the Pain• Operations Andon Cord• Create a Common Language• Educate Dev to Think Like Ops• Flattening Knowledge Chain• Create Patterns of Fault Tolerance• Manage Technical Debt 64
  65. 65. Engagement Models for Embedding• One Off• Cross Functional Teams• Mercenaries• Specialized Teams• NoOps 65
  66. 66. Design for Operations• Improve Application• Config files• Instrumentation• logging• Improve Environment• Configuration Management• Immune system (BDD) 66
  67. 67. @cread• DevOps Facilitator at DRW• London, United Kingdom• 67
  68. 68. Institutionalize IT Operations Knowledge• Building Reusable IT Operations• Embedded Operations• Design• Architecture• Controls• Monitoring• Deployment 68
  69. 69. Break Things Early And Often “Do painful things more frequently, so you can make it less painful… We don’t get pushback from Dev, because they know it makes rollouts smoother.” -- Adrian Cockcroft, Architect, Netflix 69
  70. 70. You Don’t Choose Chaos Monkey…Chaos Monkey Chooses You 70
  71. 71. Outcomes• Improved Operational Readiness• Improved Deployment Success• Decreased Cycle Time 71
  72. 72. Anti Patterns• Embedding is a Social Experiment• Understand Motivation• Previous Relationships 72
  73. 73. 73
  74. 74. Why Do I Think This Is An Important Problem? 74
  75. 75. Help The Business Win…
  76. 76. With Support From Your Peers…
  77. 77. And Do More With Less Effort…
  78. 78. 78
  79. 79. When IT Fails: A Business Novel andThe DevOps Cookbook Coming January 15, 2013 and Q1 2013 “The lessons in When IT Fails might just save your business if IT fails for you. Every IT executive should share this book with their business peers.” -James Turnbull, VP Operations, Puppet Labs and author of “Pro Puppet” “The greatest IT management book of our generation.” –Branden Williams, CTO Marketing, RSA “This book will have a profound effect on IT, just as The Goal did for manufacturing.’ - Jez Humble, co-author of the Jolt award-winning book Continuous Delivery, and Principal at ThoughtWorks Studios.
  80. 80. Our Mission: Positively Impact The LivesOf One Million IT Workers By 2017  For these slides, the “Top 10 Things You Need To Know About DevOps,” Rugged DevOps resources, and updates on the books:  Or text “[email_address] 75271” to +1 (858) 598-3980  Or signup at:  Or email