Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Winnipeg ISACA Security is Dead, Rugged DevOps

1,523 views

Published on

Published in: Business, Technology
  • Be the first to comment

  • Be the first to like this

Winnipeg ISACA Security is Dead, Rugged DevOps

  1. 1. Infosec In The New WorldOrder:Rugged DevOps and More…Gene KimWinnipeg ISACAApril 26, 2012Session ID:
  2. 2. Where Did The High Performers Come From?
  3. 3. Agenda Background of research The big unsolved problem What is Rugged? What is DevOps? How do you do Rugged DevOps? Things you can do right away 3
  4. 4. High Performing IT Organizations High performers maintain a posture of compliance  Fewest number of repeat audit findings  One-third amount of audit preparation effort High performers find and fix security breaches faster  5 times more likely to detect breaches by automated control  5 times less likely to have breaches result in a loss event When high performers implement changes…  14 times more changes  One-half the change failure rate  One-quarter the first fix failure rate  10x faster MTTR for Sev 1 outages When high performers manage IT resources…  One-third the amount of unplanned work  8 times more projects and IT services  6 times more applications Source: IT Process Institute, 2008
  5. 5. Visible Ops: Playbook of High Performers The IT Process Institute has been studying high-performing organizations since 1999  What is common to all the high performers?  What is different between them and average and low performers?  How did they become great? Answers have been codified in the Visible Ops Methodology www.ITPI.org
  6. 6. 2007: Three Controls Predict 60% OfPerformance To what extent does an organization define, monitor and enforce the following?  Standardized configuration strategy  Process discipline  Controlled access to production systems Source: IT Process Institute, 2008
  7. 7. The Downward Spiral Operations Sees… Dev Sees…  Fragile applications are prone to  More urgent, date-driven projects failure put into the queue  Long time required to figure out “which  Even more fragile code (less bit got flipped” secure) put into production  Detective control is a salesperson  More releases have increasingly “turbulent installs”  Too much time required to restore service  Release cycles lengthen to amortize “cost of deployments”  Too much firefighting and unplanned work  Failing bigger deployments more difficult to diagnose  Urgent security rework and remediation  Most senior and constrained IT ops resources have less time to  Planned project work cannot complete fix underlying process problems  Frustrated customers leave  Ever increasing backlog of work  Market share goes down that cold help the business win  Business misses Wall Street  Ever increasing amount of commitments tension between IT Ops, Development, Design…  Business makes even larger promises to Wall Street These aren’t IT or Infosec problems… These are business problems!
  8. 8. My Mission: Figure Out How Break The IT Core Chronic Conflict  Every IT organization is pressured to simultaneously:  Respond more quickly to urgent business needs  Provide stable, secure and predictable IT service Words often used to describe process improvement: “hysterical, irrelevant, bureaucratic, bottleneck, difficult to understand, not aligned with the business, immature, shrill, perpetually focused on irrelevant technical minutiae…” Source: The authors acknowledge Dr. Eliyahu Goldratt, creator of the Theory of Constraints and author of The Goal, has written extensively on the theory and practice of identifying and resolving10 core, chronic conflicts.
  9. 9. Good News: It Can Be DoneBad News: You Can’t Do It Alone
  10. 10. Ops
  11. 11. QA And Test Source: Flickr: vandyll
  12. 12. Development
  13. 13. Infosec
  14. 14. Product Management And Design Source: Flickr: birdsandanchors
  15. 15. But… 18
  16. 16. Ludicrous Speed? 19
  17. 17. Ludicrous Speed 20
  18. 18. Ludicrous Speed! 21
  19. 19. Ludicrous Fail?! 22
  20. 20. DevOps:The Shining Beacon Of Hope
  21. 21. Source: John Allspaw
  22. 22. Source: John Allspaw
  23. 23. Source: John Allspaw
  24. 24. Source: John Allspaw
  25. 25. Source: Theo Schlossnagle
  26. 26. Source: Theo Schlossnagle
  27. 27. Source: Theo Schlossnagle
  28. 28. Source: John Jenkins, Amazon.com
  29. 29. What Is Rugged? 33
  30. 30. Rugged Software DevelopmentJoshua Corman, David Rice, Jeff Williams2010
  31. 31. RUGGED SOFTWARE
  32. 32. …so software not only needs to be…
  33. 33. FAST
  34. 34. AGILE
  35. 35. Are You Rugged?
  36. 36. HARSH
  37. 37. UNFRIENDLY
  38. 38. THE MANIFESTO
  39. 39. I recognize that my code will be used in ways Icannot anticipate, in ways it was not designed, and for longer than it was ever intended.
  40. 40. www.ruggedsoftware.org CrossTalkhttp://www.crosstalkonline.org/issues/marchapril-2011.html
  41. 41. What Is Rugged DevOps? 49
  42. 42. Source: James Wickett
  43. 43. Source: James Wickett
  44. 44. DevOps: It’s A Real Movement I would never do another startup that didn’t employ DevOps like principles It’s not just startups – it’s happening in the enterprise and in public sector, too I believe working in DevOps environments will be a necessary skillset 5 years from now Just as Agile helped Dev regain trust with the business, DevOps will help all of IT
  45. 45. How Do You DoDevOps? 67
  46. 46. The Prescriptive DevOps Cookbook  “DevOps Cookbook” Authors  Patrick DeBois, Mike Orzen, John Willis  Goals  Codify how to start and finish DevOps transformations  How does Development, IT Operations and Infosec become dependable partners  Describe in detail how to replicate the transformations describe in “When IT Fails: The Novel”
  47. 47. “The Goal” by Dr. Eliyahu Goldratt
  48. 48. 71
  49. 49. 72
  50. 50. The First Way:Systems Thinking
  51. 51. The First Way:Systems Thinking(Business) (Customer)
  52. 52. The First Way:Systems Thinking (Left To Right) Never pass defects to downstream work centers Never allow local optimization to create global degradation Increase flow: elevate bottlenecks, reduce WIP, throttle release of work, reduce batch sizes Understanding where reliance is placed
  53. 53. Phase 1: Extend the Agile CI/CR Processes Assign Ops person into Dev team Create one-step Dev, Test and Production environment creation procedure in Sprint 0 Create the one-step automated code deployment procedure Define roles of Dev, QA, Prod Mgmt and Infosec
  54. 54. The First Way:Systems Thinking: Infosec Insurgency Have infosec attend the daily Agile standups  Gain awareness of what the team is working on Find the automated infrastructure project team (e.g., puppet, chef)  Provide hardening guidance  Integrate and extend their production configuration monitoring Find where code packaging is performed  Integrate security testing pre- and post-deployment Integrate into continuous integration and release process  Add security test scripts to automated test library
  55. 55. The First Way:Outcomes Determinism in the release process Continuation of the Agile and CI/CR processes Creating single repository for code and environments Packaging responsibility moves to development Consistent Dev, QA, Int, and Staging environments, all properly built before deployment begins Decrease cycle time  Reduce deployment times from 6 hours to 45 minutes  Refactor deployment process that had 1300+ steps spanning 4 weeks Faster release cadence
  56. 56. The Second Way:Amplify Feedback Loops
  57. 57. The Second Way:Amplify Feedback Loops (Right to Left) Protect the integrity of the entire system of work, versus completion of tasks Expose visual data so everyone can see how their decisions affect the entire system
  58. 58. Phase 2: Extend Release Process And CreateRight -> Left Feedback Loops Embed Dev into Ops escalation process Invite Dev to post-mortems/root cause analysis meeting Create necessary rollback procedures (instead of fixing forward) Create application monitoring/metrics to aid in Ops work (e.g., incident/problem management) Actively manage flow of work across org boundaries
  59. 59. The Second Way:Amplify Feedback Loops: Infosec Insurgency Extend criteria of what changes/deploys cannot be made without triggering full retest Create reusable Infosec use and abuse stories that can be added to every project  “Handle peak traffic of 4MM users and constant 4-6 Gb/sec Anonymous DDoS attacks” Integrate Infosec and IR into the Ops/Dev escalation processes (e.g., RACI) Pre-enable, shield streamline successful audits  Document separation of duty and compensating controls  Don’t let them disrupt the work
  60. 60. The Second Way:Outcomes Andon cords that stop the production line Kanban to control work Project freeze to reduce work in process Eradicating “quick fixes” that circumvent the process Ops user stories are part of the Agile planning process Better build and deployment systems More stable environment Happier and more productive staff
  61. 61. Definition: Kanban Board Signaling tool to reduce WIP and increase flow 84
  62. 62. The Third Way:Culture Of Continual Experimentation AndLearning
  63. 63. The Third Way:Culture Of Continual Experimentation AndLearning Foster a culture that rewards:  Experimentation (taking risks) and learning from failure  Repetition is the prerequisite to mastery Why?  You need a culture that keeps pushing into the danger zone  And have the habits that enable you to survive in the danger zone
  64. 64. You Don’t Choose Chaos Monkey…Chaos Monkey Chooses You
  65. 65. Phase 3: Organize Dev and Ops To AchieveOrganizational Goals Allocate 20% of Dev cycles to non-functional requirements Build Ops user stories and environments in Dev that can be reused across all projects (e.g., deployment, capacity, security) Integrate fault injection and resilience into design, development and production (e.g., Chaos Monkey) Prioritize backlog to manage technical debt
  66. 66. The Third Way:Culture Of Continual Experimentation AndLearning: Infosec Add Infosec fixes to the Agile backlog  Make technical debt visible  Help prioritize work against features and other non-functional requirements Weaponize the Security Monkey  Evil/Fuzzy/Chaotic Monkey  Eridicate SQLi and XSS defects in our lifetime Let loose the Security Monkies and the Simian Army Eliminate needless complexity Become the standard bearer: 20% of Dev cycles spent on non-functional requirements Take work out of the system Keep decreasing cycle time: it increases work that the system can achieve
  67. 67. The Third Way:Outcomes Dedicated time spent on improving daily work (best practice: 20% of Dev dedicated to non-functional requirements) Continual reduction of unplanned work More cycles for planned work Projects completed to pay down technical debt and increase flow Elimination of needless complexity More resilient code and environments Balancing nimbleness and practiced repetition Enabling wider range of risk/reward balance
  68. 68. What Does Transformation Feel Like? 92
  69. 69. Find What’s Most Important First
  70. 70. Quickly Find What Is Different…
  71. 71. Before Something Bad Happens…
  72. 72. Find Risk Early…
  73. 73. Communicate It Effectively To Peers…
  74. 74. Hold People Accountable…
  75. 75. Based On Objective Evidence…
  76. 76. Answer Important Questions…
  77. 77. Recognize Compounding Technical Debt…
  78. 78. That Gets Worse…
  79. 79. And Fixing It… Source: Pingdom
  80. 80. Have What We Need, When When We NeedIt…
  81. 81. Big Things Get Done Quickly…
  82. 82. Ever Increasing Situational Mastery…
  83. 83. Help The Business Win…
  84. 84. With Support From Your Peers…
  85. 85. And Do More With Less Effort…
  86. 86. This Is An Important Problem Operations Sees… Dev Sees…  Fragile applications are prone to  More urgent, date-driven projects failure put into the queue  Long time required to figure out “which  Even more fragile code (less bit got flipped” secure) put into production  Detective control is a salesperson  More releases have increasingly “turbulent installs”  Too much time required to restore service  Release cycles lengthen to amortize “cost of deployments”  Too much firefighting and unplanned work  Failing bigger deployments more difficult to diagnose  Urgent security rework and remediation  Most senior and constrained IT ops resources have less time to  Planned project work cannot complete fix underlying process problems  Frustrated customers leave  Ever increasing backlog of work  Market share goes down that cold help the business win  Business misses Wall Street  Ever increasing amount of commitments tension between IT Ops, Development, Design…  Business makes even larger promises to Wall Street
  87. 87. When IT Fails: The Novel and The DevOps Cookbook  Coming in July 2012  “In the tradition of the best MBA case studies, this book should be mandatory reading for business and IT graduates alike.” Paul Muller, VP Software Marketing, Hewlett- PackardGene Kim, Tripwire founder,  “The greatest IT management book of ourVisible Ops co-author generation.” Branden Williams, CTO Marketing, RSA
  88. 88. When IT Fails: The Novel and The DevOps Cookbook  Our mission is to positively affect the lives of 1 million IT workers by 2017  If you would like the “Top 10 Things You Need To Know About DevOps,” sample chapters and updates on the book:  Sign up at http://itrevolution.comGene Kim, Tripwirefounder, Visible Ops co-  Email genek@realgenekim.meauthor  Hand me a business card
  89. 89. Thank You 113
  90. 90. Appendix 114
  91. 91. Resources From the IT Process Institute www.itpi.org  Both Visible Ops Handbooks  ITPI IT Controls Performance Study Rugged Software by Corman, et al: http://ruggedsoftware.org “Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation” by Humble, Farley Follow us…  @JoshCorman, @RealGeneKim  mailto:genek@realgenekim.me  http://realgenekim.me/blog
  92. 92. Common Traits of High PerformersCulture of…Change management Integration of IT operations/security via problem/change management Processes that serve both organizational needs and business objectives Highest rate of effective changeCausality Highest service levels (MTTR, MTBF) Highest first fix rate (unneeded rework)Compliance and continual reduction ofoperational variance Production configurations Highest level of pre-production staffing Effective pre-production controls Effective pairing of preventive and detective controls Source: IT Process Institute
  93. 93. Visible Ops: Playbook of High Performers The IT Process Institute has been studying high-performing organizations since 1999  What is common to all the high performers?  What is different between them and average and low performers?  How did they become great? Answers have been codified in the Visible Ops Methodology The “Visible Ops Handbook” is available from the ITPI www.ITPI.org
  94. 94. IT Operations Increases Process Rigor Standardize deployment Standardize unplanned work: make it repeatable Modify first response: ensure constrained resources have all data at hand to diagnose Elevate preventive activities to reduce incidents
  95. 95. Help Development… Help them see downstream effects  Unplanned work comes at the expense of planned work  Technical debt retards feature throughput  Environment matters as much as the code Allocate time for fault modeling, asking “what could go wrong?” and implementing countermeasures
  96. 96. Help QA… Ensure test plans cover not only code functionality, but also:  Suitability of the environment the code runs in  The end-to-end deployment process Help find variance…  Functionality, performance, configuration  Duration, wait time and handoff errors, rework, …
  97. 97. John Pesche, CISO  CISO for 12 years  39 years old  Aggressive career climber  Ex-Big Four auditor
  98. 98. John Pesche, CISO
  99. 99. John Pesche, CISO
  100. 100. John Pesche, CISO

×