Remediation patterns

5,141 views
4,980 views

Published on

There's a recording of this presentation from QCon London 2011: http://www.infoq.com/presentations/Remediation-Patterns

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,141
On SlideShare
0
From Embeds
0
Number of Embeds
141
Actions
Shares
0
Downloads
96
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Remediation patterns

  1. 1. Remediation Patterns October 12, 2011, GOTO Amsterdam Jez Humble, ThoughtWorks Studios @jezhumble #continuousdelivery http://continuousdelivery.com/ © 2012 ThoughtWorks, Inc.
  2. 2. remediationITIL: “Recovery to a known state a er afailed Change or Release.”Recovery: “Returning a Con guration Item oran IT Service to a working state.”Jez: “Fixing shit when it breaks”
  3. 3. strategies for remediationpreventionpatterns for low-risk releasepatterns for incremental delivery
  4. 4. 1oz of prevention Diagram invented by Brian Marick
  5. 5. deployment pipelineDelivery team Version control Build & unit Automated User acceptance Release tests acceptance tests tests Check in Trigger Feedback Check in Trigger Feedback Trigger Feedback Check in Trigger Feedback Trigger Feedback Approval Feedback Approval
  6. 6. the hard bitstesting on production environmentscreating maintainable acceptance teststesting cross-functional requirements
  7. 7. reducing release riskoptimize for resilienceautomate provisioning and deploymentdevs, testers and ops collaborate throughout
  8. 8. optimize for TTRhttp://www.flickr.com/photos/chuecy/2629242132/ MTBF MTTR Thanks to John Allspaw: http://slideshare.net/jallspaw/
  9. 9. infrastructure as code Destroy works of art
  10. 10. low risk releases are incremental DEPENDENT SERVICE 1.0 1.1 Database Abstraction layer Abstraction layer STATIC CONTENT /static/1.1 APPLICATION /static/1.0 Router / Load balancer Interwebs
  11. 11. canary releasing Diagram by Martin Fowler
  12. 12. canary releasing Diagram by Martin Fowler
  13. 13. canary releasingreduce risk of releaseA/B testingperformance testing
  14. 14. immune systemwhat if someone replaced your“buy” button with spacer.gif? T cells http://www.flickr.com/photos/gehealthcare/3326186490/
  15. 15. monitoringBusiness metrics - revenue, # orders, # usersOps metrics - changes, incidents, TTD, TTR,TBFTechnical metrics - TPS, response time, hits http://www.flickr.com/photos/wwarby/3296379139/
  16. 16. the hard bitsanalyzing root causescollaborationdata
  17. 17. incremental delivery John Allspaw: “Ops Metametrics” http://slidesha.re/dsSZIr
  18. 18. incremental deliverydevelop on mainlinefeature toggles and branch by abstractiondark launching
  19. 19. feature toggles Config File[featureToggles] some.jspwobblyFoobars: trueflightyForkHandles: false <toggle name=wobblyFoobars> ... various UI elements </toggle> other.java forkHandle = (featureConfig.isOn(‘flightlyForkHandles)) ? new FlightyForkHander(aCandle) : new ForkHandler(aCandle) Stolen from Martin Fowler http://martinfowler.com/bliki/FeatureToggle.html
  20. 20. branch by abstraction Component A A Component Seam Component B
  21. 21. branch by abstraction Component A A Component Abstraction layerComponent B Component B’
  22. 22. dark launching Diagram by Martin Fowler
  23. 23. dark launching Diagram by Martin Fowler
  24. 24. measuring effectivenessHow long would it take you to release achange to a single line of code?Ops metrics - changes, incidents, TTD, TTR,TBFIf your data center blew up, how long wouldyou take to restore service?
  25. 25. questionsjez@thoughtworks.comhttp://continuousdelivery.com/@jezhumble #continuousdelivery © 2012 ThoughtWorks, Inc. http://thoughtworks-studios.com/

×