Remediation patterns


Published on

There's a recording of this presentation from QCon London 2011:

Published in: Technology
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Remediation patterns

  1. 1. Remediation Patterns October 12, 2011, GOTO Amsterdam Jez Humble, ThoughtWorks Studios @jezhumble #continuousdelivery © 2012 ThoughtWorks, Inc.
  2. 2. remediationITIL: “Recovery to a known state a er afailed Change or Release.”Recovery: “Returning a Con guration Item oran IT Service to a working state.”Jez: “Fixing shit when it breaks”
  3. 3. strategies for remediationpreventionpatterns for low-risk releasepatterns for incremental delivery
  4. 4. 1oz of prevention Diagram invented by Brian Marick
  5. 5. deployment pipelineDelivery team Version control Build & unit Automated User acceptance Release tests acceptance tests tests Check in Trigger Feedback Check in Trigger Feedback Trigger Feedback Check in Trigger Feedback Trigger Feedback Approval Feedback Approval
  6. 6. the hard bitstesting on production environmentscreating maintainable acceptance teststesting cross-functional requirements
  7. 7. reducing release riskoptimize for resilienceautomate provisioning and deploymentdevs, testers and ops collaborate throughout
  8. 8. optimize for TTR MTBF MTTR Thanks to John Allspaw:
  9. 9. infrastructure as code Destroy works of art
  10. 10. low risk releases are incremental DEPENDENT SERVICE 1.0 1.1 Database Abstraction layer Abstraction layer STATIC CONTENT /static/1.1 APPLICATION /static/1.0 Router / Load balancer Interwebs
  11. 11. canary releasing Diagram by Martin Fowler
  12. 12. canary releasing Diagram by Martin Fowler
  13. 13. canary releasingreduce risk of releaseA/B testingperformance testing
  14. 14. immune systemwhat if someone replaced your“buy” button with spacer.gif? T cells
  15. 15. monitoringBusiness metrics - revenue, # orders, # usersOps metrics - changes, incidents, TTD, TTR,TBFTechnical metrics - TPS, response time, hits
  16. 16. the hard bitsanalyzing root causescollaborationdata
  17. 17. incremental delivery John Allspaw: “Ops Metametrics”
  18. 18. incremental deliverydevelop on mainlinefeature toggles and branch by abstractiondark launching
  19. 19. feature toggles Config File[featureToggles] some.jspwobblyFoobars: trueflightyForkHandles: false <toggle name=wobblyFoobars> ... various UI elements </toggle> forkHandle = (featureConfig.isOn(‘flightlyForkHandles)) ? new FlightyForkHander(aCandle) : new ForkHandler(aCandle) Stolen from Martin Fowler
  20. 20. branch by abstraction Component A A Component Seam Component B
  21. 21. branch by abstraction Component A A Component Abstraction layerComponent B Component B’
  22. 22. dark launching Diagram by Martin Fowler
  23. 23. dark launching Diagram by Martin Fowler
  24. 24. measuring effectivenessHow long would it take you to release achange to a single line of code?Ops metrics - changes, incidents, TTD, TTR,TBFIf your data center blew up, how long wouldyou take to restore service?
  25. 25. questionsjez@thoughtworks.com #continuousdelivery © 2012 ThoughtWorks, Inc.
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.