Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

2012 SxSW When IT Says No by Gene Kim


Published on

Published in: Business, Technology

2012 SxSW When IT Says No by Gene Kim

  1. 1. When IT Says No:How To Create Fast Feature Flow (Without Breaking Everything) Gene Kim IT Revolution Press March 10, 2012 @RealGeneKim,
  2. 2. Logistics• Gene Kim: @realgenekim• Twitter hashtag: #devops @RealGeneKim,
  3. 3. About Gene Kim – Researcher, Author – Industry: • Invented and founded Tripwire, CTO (1997-2010) • Co-author: “Visible Ops Handbook”(2006), “Visible Ops Security” (2008) • Co-author: “When IT Fails: The Novel,” “The DevOps Cookbook” (Coming May 2012) – Things I’ve been researching: • Benchmarked 1300+ IT organizations to test effectiveness of IT controls vs. IT performance • DevOps, Rugged DevOps • Scoping PCI Cardholder Data Environment3 @RealGeneKim,
  4. 4. Where Did The High Performers Come From? @RealGeneKim,
  5. 5. The Downward SpiralOperations Sees… Dev Sees…• Fragile applications are prone to failure • More urgent, date-driven projects put into the queue• Long time required to figure out “which bit got flipped” • Even more fragile code (less secure) put into production• Detective control is a salesperson • More releases have increasingly• Too much time required to restore service “turbulent installs”• Too much firefighting and unplanned work • Release cycles lengthen to amortize• Urgent security rework and remedation “cost of deployments”• Planned project work cannot complete • Failing bigger deployments more difficult to diagnose• Frustrated customers leave • Most senior and constrained IT ops• Market share goes down resources have less time to fix underlying process problems• Business misses Wall Street commitments • Ever increasing backlog of work that• Business makes even larger promises to cold help the business win Wall Street • Ever increasing amount of tension between IT Ops, Development, Design… These aren’t IT or Design problems… These are business problems! @RealGeneKim,
  6. 6. My Mission: Figure Out How Break The IT Core Chronic Conflict • Every IT organization is pressured to simultaneously: – Respond more quickly to urgent business needs – Provide stable, secure and predictable IT service Words often used to describe process improvement: “hysterical, irrelevant, bureaucratic, bottleneck, difficult to understand, not aligned with the business, immature, shrill, perpetually focused on irrelevant technical minutiae…” Source: The authors acknowledge Dr. Eliyahu Goldratt, creator of the Theory of Constraints and author of The Goal, has written extensively on the theory and practice of identifying and resolving core, chronic conflicts.6 @RealGeneKim,
  7. 7. Good News: It Can Be DoneBad News: You Can’t Do It Alone @RealGeneKim,
  8. 8. Ops @RealGeneKim,
  9. 9. QA And TestSource: Flickr: vandyll @RealGeneKim,
  10. 10. Development @RealGeneKim,
  11. 11. Infosec @RealGeneKim,
  12. 12. Product Management And DesignSource: Flickr: birdsandanchors @RealGeneKim,
  13. 13. DevOps:The Shining Beacon Of Hope @RealGeneKim,
  14. 14. Source: John Allspaw @RealGeneKim,
  15. 15. Source: John Allspaw @RealGeneKim,
  16. 16. @RealGeneKim,
  17. 17. Source: John Allspaw @RealGeneKim,
  18. 18. Source: John Allspaw @RealGeneKim,
  19. 19. Source: Theo Schlossnagle @RealGeneKim,
  20. 20. Source: Theo Schlossnagle @RealGeneKim,
  21. 21. Source: Theo Schlossnagle @RealGeneKim,
  22. 22. Source: John Jenkins, @RealGeneKim,
  23. 23. Ludicrous Speed 23 @RealGeneKim,
  24. 24. @RealGeneKim,
  25. 25. Source: James Wickett @RealGeneKim,
  26. 26. Source: James Wickett @RealGeneKim,
  27. 27. Source: James Wickett @RealGeneKim,
  28. 28. DevOps: It’s A Real Movement• I would never do another startup that didn’t employ DevOps-like principles• It’s not just startups – it’s happening in the enterprise and in public sector, too• I believe working in DevOps environments will be a necessary skillset 5 years from now @RealGeneKim,
  29. 29. The Prescriptive DevOps Cookbook • “DevOps Cookbook” Authors – Patrick DeBois, Mike Orzen, John Willis • Goals – Codify how to start and finish DevOps transformations – How does Development, IT Operations and Infosec become dependable partners – Describe in detail how to replicate the transformations describe in “When IT Fails: The Novel” @RealGeneKim,
  30. 30. Philosophies And Outcomes: The Three Ways @RealGeneKim,
  31. 31. The First Way: Systems ThinkingDev Ops @RealGeneKim,
  32. 32. The First Way: Systems Thinking (Left To Right)• Never pass defects to downstream work centers• Never allow local optimization to create global degradation• Increase flow to elevate bottlenecks, reduce WIP, throttle release of work, reduce batch sizes• Eradicate blockages in the flow @RealGeneKim,
  33. 33. The First Way: Outcomes• Determinism in the release process• Packaging responsibility moves to development• Consistent Dev, QA, Int, and Staging environments, all properly built before deployment begins• Decrease cycle time – Reduce deployment times from 6 hours to 45 minutes – Refactor deployment process that had 1300+ steps spanning 4 weeks• Faster release cadence @RealGeneKim,
  34. 34. The Second Way:Amplify Feedback LoopsDev Ops @RealGeneKim,
  35. 35. The Second Way:Amplify Feedback Loops (Right to Left)• Expose visual data everyone can see how their decisions affect the entire system• Protect the integrity of the entire system of work, versus completion of tasks @RealGeneKim,
  36. 36. The Second Way: Outcomes• Andon cords that stop the production line• Eradicating “quick fixes” that circumvent the process• Ops user stories are part of the Agile planning process• Better build and deployment systems• More stable environment• Happier and more productive Ops staff @RealGeneKim,
  37. 37. The Third Way:Culture Of Continual Experimentation And Learning Dev Ops @RealGeneKim,
  38. 38. The Third Way: Culture Of Continual Experimentation And Learning• Foster a culture that rewards: – Experimentation (taking risks) and learning from failure – Repetition is the prerequisite to mastery• Why? – You need a culture that keeps pushing into the danger zone – And have the habits that enable you to survive in the danger zone when youre doing 1000 deploys/hour @RealGeneKim,
  39. 39. The Third Way: Outcomes• 15 minutes/day spent on improving daily work• Continual reduction of unplanned• More cycles for planned work• Projects completed to pay down technical debt and increase flow• Elimination of needless complexity• More resilient code and environments @RealGeneKim,
  40. 40. Some Prescriptive Steps @RealGeneKim,
  41. 41. Meeting The DevOps Leadership Team• Typically led by Dev, QA, IT Operations and Product Management/Design• Our ultimate goal is to add value at every step in the flow of work – See the end-to-end value flow – Shorten and amplify feedback loops – Help break silos (e.g., server, networking, database) @RealGeneKim,
  42. 42. Definition: Agile Sprints• The basic unit of development in Agile Scrums, typically between one week and one month• At the end of each sprint, team should have potentially deliverable product Aha Moment: shipping product implies not just code – 42 it’s the environment, too! @RealGeneKim,
  43. 43. Help Dev And Ops Build Code And Environments• Dev and Ops work together in Sprint 0 and 1 to create code and environments – Create environment that Dev deploys into – Create downstream environments: QA, Staging, Production – Create testable migration procedures from Dev all the way to production• Integrate Infosec and QA into daily sprint activities @RealGeneKim,
  44. 44. Definition: Andon Cord 44 @RealGeneKim,
  45. 45. Integrate Ops Into Dev• Embed Ops person into Dev structure – Describes non-functional requirements, use cases and stories from Ops – Responsible for improving “quality at the source” (e.g., reducing technical debt, fix known problems, etc.) – Has special responsibility for pulling the Andon cord • No ability to restart service without rebooting • Configuration settings impossible to find @RealGeneKim,
  46. 46. Integrate Dev Into Ops• MobBrowser case study: “Waking up developers at 3am is a great feedback loop: defects get fixed very quickly”• Goal is to get Dev closer to the customer – Infosec can help determine when it’s too close (and when SOD is a requirement) @RealGeneKim,
  47. 47. Keep Shrinking Batch Sizes• Waterfall projects often have cycle time of one year• Sprints have cycle time of 1 or 2 weeks• When IT Operations work is sufficiently fast and capable, we may decide to decouple deployments from sprint boundaries (e.g., Kanbans) @RealGeneKim,
  48. 48. Definition: Kanban Board• Signaling tool to reduce WIP and increase flow 48 @RealGeneKim,
  49. 49. IT Operations Increases Process Rigor• Standardize deployment• Standardize how unplanned work is prosecuted: make it repeatable• Modify first response: ensure constrained resources have all data at hand to diagnose• Elevate preventive activities to reduce incidents @RealGeneKim,
  50. 50. Letter to Development• Seek the downstream effects of your actions – Unplanned work comes at the expense of planned work – Technical debt retards feature throughput – Environment matters as much as the code• Allocate time for fault modeling, asking “what could go wrong?” and implementing countermeasures @RealGeneKim,
  51. 51. Letter To QA• Ensure test plans cover not only code functionality, but also: – Suitability of the environment the code runs in – The end-to-end deployment process• Help find variance… – Functionality, performance, configuration – Duration, wait time and handoff errors, rework, … @RealGeneKim,
  52. 52. Letter To IT Operations • “The best way to avoid failure is to fail constantly” • Harden the production environment • Have scheduled drills to “crash the data center” • Create your “chaos monkeys” to introduce faults into the system (e.g., randomly kill processes, take out servers, etc.) • Rehearse and improve responding to unplanned work – NetFlix: Hardened AWS service – StackOverflow – Amazon firedrills (Jesse Allspaw) – The Monkey (Mac) @RealGeneKim,
  53. 53. You Don’t Choose Chaos Monkey… Chaos Monkey Chooses You @RealGeneKim,
  54. 54. Letter To Product ManagementLesson: Allocate 20% of Dev cycles to paying down technical debt @RealGeneKim,
  55. 55. To Designers• Help IT Operations codify their work and requirements into great and ever increasing library of user stories• Realize that IT processes are likely the largest impediment preventing your great ideas from making it to market @RealGeneKim,
  56. 56. When IT Fails: The Novel and The DevOps Cookbook • Coming in July 2012 • “In the tradition of the best MBA case studies, this book should be mandatory reading for business and IT graduates alike.” -Paul Muller, VP Software Marketing, Hewlett-Packard • “The greatest IT management book of ourGene Kim, Tripwire generation.” –Branden Williams, CTO Marketing, RSAfounder, Visible Ops co-author @RealGeneKim,
  57. 57. When IT Fails: The Novel and The DevOps Cookbook • Coming in July 2012 • If you would like the “Top 10 Things You Need To Know About DevOps,” sample chapters and updates on the book: Sign up at http://itrevolution.comGene Kim, Tripwirefounder, Visible Ops co-author Email Give me your business card @RealGeneKim,
  58. 58. To Join The Movement• Sunday 7pm: The Ginger Man –• If you would like the “Top 10 Things You Need To Know About DevOps,” sample chapters and updates on the book: Sign up at Email Give me your business card @RealGeneKim,
  59. 59. @RealGeneKim,
  60. 60. Other Resources• From the IT Process Institute – Both Visible Ops Handbooks – ITPI IT Controls Performance Study• Rugged Software by Corman, et al:• “Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation” by Humble, Farley• Follow us… – @JoshCorman, @RealGeneKim – – @RealGeneKim,