Your SlideShare is downloading. ×
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Remediation patterns
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Remediation patterns

3,468

Published on

There's a recording of this presentation from QCon London 2011: http://www.infoq.com/presentations/Remediation-Patterns

There's a recording of this presentation from QCon London 2011: http://www.infoq.com/presentations/Remediation-Patterns

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,468
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
89
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Remediation Patterns October 12, 2011, GOTO Amsterdam Jez Humble, ThoughtWorks Studios @jezhumble #continuousdelivery http://continuousdelivery.com/ © 2012 ThoughtWorks, Inc.
  • 2. remediationITIL: “Recovery to a known state a er afailed Change or Release.”Recovery: “Returning a Con guration Item oran IT Service to a working state.”Jez: “Fixing shit when it breaks”
  • 3. strategies for remediationpreventionpatterns for low-risk releasepatterns for incremental delivery
  • 4. 1oz of prevention Diagram invented by Brian Marick
  • 5. deployment pipelineDelivery team Version control Build & unit Automated User acceptance Release tests acceptance tests tests Check in Trigger Feedback Check in Trigger Feedback Trigger Feedback Check in Trigger Feedback Trigger Feedback Approval Feedback Approval
  • 6. the hard bitstesting on production environmentscreating maintainable acceptance teststesting cross-functional requirements
  • 7. reducing release riskoptimize for resilienceautomate provisioning and deploymentdevs, testers and ops collaborate throughout
  • 8. optimize for TTRhttp://www.flickr.com/photos/chuecy/2629242132/ MTBF MTTR Thanks to John Allspaw: http://slideshare.net/jallspaw/
  • 9. infrastructure as code Destroy works of art
  • 10. low risk releases are incremental DEPENDENT SERVICE 1.0 1.1 Database Abstraction layer Abstraction layer STATIC CONTENT /static/1.1 APPLICATION /static/1.0 Router / Load balancer Interwebs
  • 11. canary releasing Diagram by Martin Fowler
  • 12. canary releasing Diagram by Martin Fowler
  • 13. canary releasingreduce risk of releaseA/B testingperformance testing
  • 14. immune systemwhat if someone replaced your“buy” button with spacer.gif? T cells http://www.flickr.com/photos/gehealthcare/3326186490/
  • 15. monitoringBusiness metrics - revenue, # orders, # usersOps metrics - changes, incidents, TTD, TTR,TBFTechnical metrics - TPS, response time, hits http://www.flickr.com/photos/wwarby/3296379139/
  • 16. the hard bitsanalyzing root causescollaborationdata
  • 17. incremental delivery John Allspaw: “Ops Metametrics” http://slidesha.re/dsSZIr
  • 18. incremental deliverydevelop on mainlinefeature toggles and branch by abstractiondark launching
  • 19. feature toggles Config File[featureToggles] some.jspwobblyFoobars: trueflightyForkHandles: false <toggle name=wobblyFoobars> ... various UI elements </toggle> other.java forkHandle = (featureConfig.isOn(‘flightlyForkHandles)) ? new FlightyForkHander(aCandle) : new ForkHandler(aCandle) Stolen from Martin Fowler http://martinfowler.com/bliki/FeatureToggle.html
  • 20. branch by abstraction Component A A Component Seam Component B
  • 21. branch by abstraction Component A A Component Abstraction layerComponent B Component B’
  • 22. dark launching Diagram by Martin Fowler
  • 23. dark launching Diagram by Martin Fowler
  • 24. measuring effectivenessHow long would it take you to release achange to a single line of code?Ops metrics - changes, incidents, TTD, TTR,TBFIf your data center blew up, how long wouldyou take to restore service?
  • 25. questionsjez@thoughtworks.comhttp://continuousdelivery.com/@jezhumble #continuousdelivery © 2012 ThoughtWorks, Inc. http://thoughtworks-studios.com/

×