Remediation patterns

remediationITIL: “Recovery to a known state a er afailed Change or Release.”Recovery: “Returning a Con guration Item oran IT Service to a working state.”Jez: “Fixing shit when it breaks”

strategies for remediationpreventionpatterns for low-risk releasepatterns for incremental delivery

1oz of prevention Diagram invented by Brian Marick

deployment pipelineDelivery team Version control Build & unit Automated User acceptance Release tests acceptance tests tests Check in Trigger Feedback Check in Trigger Feedback Trigger Feedback Check in Trigger Feedback Trigger Feedback Approval Feedback Approval

the hard bitstesting on production environmentscreating maintainable acceptance teststesting cross-functional requirements

reducing release riskoptimize for resilienceautomate provisioning and deploymentdevs, testers and ops collaborate throughout

optimize for TTRhttp://www.flickr.com/photos/chuecy/2629242132/ MTBF MTTR Thanks to John Allspaw: http://slideshare.net/jallspaw/

infrastructure as code Destroy works of art

low risk releases are incremental DEPENDENT SERVICE 1.0 1.1 Database Abstraction layer Abstraction layer STATIC CONTENT /static/1.1 APPLICATION /static/1.0 Router / Load balancer Interwebs

canary releasing Diagram by Martin Fowler

canary releasingreduce risk of releaseA/B testingperformance testing

immune systemwhat if someone replaced your“buy” button with spacer.gif? T cells http://www.flickr.com/photos/gehealthcare/3326186490/

monitoringBusiness metrics - revenue, # orders, # usersOps metrics - changes, incidents, TTD, TTR,TBFTechnical metrics - TPS, response time, hits http://www.ﬂickr.com/photos/wwarby/3296379139/

the hard bitsanalyzing root causescollaborationdata

incremental delivery John Allspaw: “Ops Metametrics” http://slidesha.re/dsSZIr

incremental deliverydevelop on mainlinefeature toggles and branch by abstractiondark launching

feature toggles Config File[featureToggles] some.jspwobblyFoobars: trueflightyForkHandles: false <toggle name=wobblyFoobars> ... various UI elements </toggle> other.java forkHandle = (featureConfig.isOn(‘flightlyForkHandles)) ? new FlightyForkHander(aCandle) : new ForkHandler(aCandle) Stolen from Martin Fowler http://martinfowler.com/bliki/FeatureToggle.html

branch by abstraction Component A A Component Seam Component B

branch by abstraction Component A A Component Abstraction layerComponent B Component B’

dark launching Diagram by Martin Fowler

measuring eﬀectivenessHow long would it take you to release achange to a single line of code?Ops metrics - changes, incidents, TTD, TTR,TBFIf your data center blew up, how long wouldyou take to restore service?

http://www.scoop.it	43
http://www.linkedin.com	3
http://paper.li	2
http://twitter.com	1
http://www.mefeedia.com	1

Remediation patterns

by Jez Humble

Accessibility

Categories

Upload Details

Usage Rights

5 Embeds 50

Statistics

Remediation patterns Presentation Transcript