Download as ZIP, PPTX
Uploaded byJohn Allspaw
Ops Meta-Metrics: The Currency You Pay For Change
The document discusses tracking operations metrics to build confidence in an organization's ability to handle changes. It recommends tracking: 1) Types of changes made, including code, configuration, infrastructure, etc. 2) Frequency and size of changes through metrics like number of code commits. 3) Results of changes, including number of incidents related to changes and their root causes, time to detect incidents, and time to resolve them. Tracking these metrics over time can provide insight into what types of changes are most likely to cause incidents and how the organization responds to incidents on average. The data can help identify areas to improve processes and build confidence that changes do not necessarily lead to outages.
More Related Content
Similar to Ops Meta-Metrics: The Currency You Pay For Change
Ops Meta-Metrics: The Currency You Pay For Change
- 1. Ops Meta-Metrics The CurrencyYou Use to Pay For Change John Allspaw VP Operations Etsy.com http://www.flickr.com/photos/wwarby/3296379139
- 2. Warning Graphs and numbersin this presentation are sort of made up
- 3.
- 4. How R UDoing? http://www.flickr.com/photos/a4gpa/190120662/
- 5. We track bugsalready... Example: https://issues.apache.org/jira/browse/TS
- 6. We should track these, too...
- 7. We should track these, too... Changes (Who/What/When/Type)
- 8. We should track these, too... Changes (Who/What/When/Type) Incidents (Type/Severity)
- 9. We should track these, too... Changes (Who/What/When/Type) Incidents (Type/Severity) Response to Incidents (TTR/TTD)
- 10. trepidation noun 1 a feelingof fear or agitation about something that may happen : the men set off in fear and trepidation. 2 archaic trembling motion. DERIVATIVES trepidatious adjective ORIGIN late 15th cent.: from Latin trepidatio(n-), from trepidare ‘be agitated, tremble,’ from trepidus ‘alarme
- 11. Change Required. Often feared. Why? http://www.flickr.com/photos/20408885@N03/3570184759/
- 12. This is why OMGWTF OUTAGES!!!1!! la de da, everything’s fine change happens
- 13. Change PTSD? http://www.flickr.com/photos/tzofia/270800047/
- 14.
- 15.
- 16. But wait.... (OMGWTF) la de da, everything’s fine change happens
- 17. But wait.... (OMGWTF) la de da, } everything’s fine How much change is this? change happens
- 18. But wait.... (OMGWTF) la de da, } everything’s fine How much change is this? What kind of change? change happens
- 19. But wait.... (OMGWTF) la de da, } everything’s fine How much change is this? What kind of change? How often does this happen? change happens
- 20. Need to raiseconfidence that change != outage
- 21. ...incidents can be handled well http://www.flickr.com/photos/axiepics/3181170364/
- 22. ...root causes canbe fixed quick enough http://www.flickr.com/photos/ljv/213624799/
- 23. ...change can be safe enough http://www.flickr.com/photos/marksetchell/43252686/
- 24. But how? How dowe have confidence in anything in our infrastructure? We measure it. And graph it. And alert on it.
- 25. Tracking Change 1. Type 2.Frequency/Size 3. Results of those changes
- 26. Types of Change Layers Examples App code PHP/Rails/etc or ‘front-end’ code Apache, MySQL, DB schema, Services code PHP/Ruby versions, etc. OS/Servers, Switches, Routers, Infrastructure Datacenters, etc. (you decide what these are for your architecture)
- 27. Code Deploys: Who/What/When WHEN WHO WHAT (guy who pushed the button) (link to diff) (http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/)
- 28. Code Deploys: Who/What/When Last 2 prod deploys Last 2 Chef changes
- 29. other changes (insert whateverticketing/tracking you have)
- 30.
- 31.
- 32.
- 33.
- 34. Tracking Incidents http://www.flickr.com/photos/47684393@N00/4543311558/
- 35.
- 36. Incident Size Big Outage TTR still going
- 37. Tracking Incidents 1. Frequency 2.Severity 3. Root Cause 4. Time-To-Detect (TTD) 5. Time-To-Resolve (TTR)
- 38. The How Doesn’t Matter http://www.flickr.com/photos/matsuyuki/2328829160/
- 39. Incident/Degradation Tracking Start Detect Resolve Root PostMortem Date Severity Done? Time Time Time Cause 1/2/08 12:30 ET 12:32 ET 12:45 ET Sev1 DB Change Yes 3/7/08 18:32 ET 18:40 ET 18:47 ET Sev2 Capacity Yes 5/3/08 17:55 ET 17:55 ET 18:14 ET Sev3 Hardware Yes
- 40. Incident/Degradation Tracking Start Detect Resolve Root PostMortem Date Severity Time These Time give you will Time context Cause Done? for your rates of change. (You’ll need them for postmortems, anyway.)
- 41.
- 42. Change:Incident Ratio Important.
- 43. Change:Incident Ratio Important. Not because all changes are equal.
- 44. Change:Incident Ratio Important. Not because all changes are equal. Not because all incidents are equal, or change-related.
- 45. Change:Incident Ratio But because humanswill irrationally make a permanent connection between the two. http://www.flickr.com/photos/michelepedrolli/449572596/
- 46.
- 47. Severity Not all incidentsare created equal.
- 48. Severity Not all incidentsare created equal. Something like:
- 49. Severity Not all incidentsare created equal. Something like:
- 50. Severity Not all incidentsare created equal. Something like: SEV1 Full outage, or effectively unusable.
- 51. Severity Not all incidentsare created equal. Something like: SEV1 Full outage, or effectively unusable. SEV2 Significant degradation for subset of users.
- 52. Severity Not all incidentsare created equal. Something like: SEV1 Full outage, or effectively unusable. SEV2 Significant degradation for subset of users. SEV3 Minor impact on user experience.
- 53. Severity Not all incidentsare created equal. Something like: SEV1 Full outage, or effectively unusable. SEV2 Significant degradation for subset of users. SEV3 Minor impact on user experience. SEV4 No impact, but time-sensitive failure.
- 54. Root Cause? (Not all incidents are change related) Something like: Note: this can be difficult to categorize. http://en.wikipedia.org/wiki/Root_cause_analysis
- 55. Root Cause? (Not all incidents are change related) Something like: 1. Hardware Failure 2. Datacenter Issue 3. Change: Code Issue 4. Change: Config Issue 5. Capacity/Traffic Issue 6. Other Note: this can be difficult to categorize. http://en.wikipedia.org/wiki/Root_cause_analysis
- 56. Recording Your Response (worth the hassle) http://www.flickr.com/photos/mattblaze/2695044170/
- 57.
- 58. la de da, everything’s fine Time
- 59. la de da, everything’s fine Time change happens
- 60. Noticed there was a problem la de da, everything’s fine Time change happens
- 61. Noticed there was a problem Figured out la de da, what the cause is everything’s fine Time change happens
- 62. Fixed the problem Noticed there •rolled back was a problem •rolled forward •temporary solution •etc Figured out la de da, what the cause is everything’s fine Time change happens
- 63. Fixed the problem Noticed there •rolled back was a problem •rolled forward •temporary solution •etc Figured out la de da, what the cause is everything’s fine Time change happens
- 64. • Coordinate troubleshooting/diagnosis Fixed the problem Noticed there •rolled back was a problem •rolled forward •temporary solution •etc Figured out la de da, what the cause is everything’s fine Time change happens
- 65. • Coordinate troubleshooting/diagnosis • Communicate to support/community/execs Fixed the problem Noticed there •rolled back was a problem •rolled forward •temporary solution •etc Figured out la de da, what the cause is everything’s fine Time change happens
- 66. Fixed the problem Noticed there •rolled back was a problem •rolled forward •temporary solution •etc Figured out la de da, what the cause is everything’s fine Time Time change happens
- 67. • Coordinate responses* Fixed the problem Noticed there •rolled back was a problem •rolled forward •temporary solution •etc Figured out la de da, what the cause is everything’s fine Time Time change happens * usually, “One Thing At A Time” responses
- 68. • Coordinate responses* • Communicate to support/community/execs problem Fixed the Noticed there •rolled back was a problem •rolled forward •temporary solution •etc Figured out la de da, what the cause is everything’s fine Time Time change happens * usually, “One Thing At A Time” responses
- 69. Fixed the problem Figured out what the cause is Noticed there •rolled back was a problem •rolled forward •temporary solution •etc la de da, everything’s fine Time Time change happens
- 70. • Confirm stability,resolving steps Fixed the problem Figured out what the cause is Noticed there •rolled back was a problem •rolled forward •temporary solution •etc la de da, everything’s fine Time Time change happens
- 71. • Confirm stability,resolving steps • Communicate to support/community/execs Fixed the problem Figured out what the cause is Noticed there •rolled back was a problem •rolled forward •temporary solution •etc la de da, everything’s fine Time Time change happens
- 72.
- 73. Fixed the problem Figured out what the cause is Noticed there •rolled back was a problem •rolled forward •temporary solution •etc la de da, everything’s fine Time Time change happens PostMortem
- 74. Time To Detect (TTD) Time To Resolve la de da, (TTR) la de da, everything’s fine everything’s fine Time change happens
- 75. Hypothetical Example: “We’reSo Nimble!”
- 76.
- 77. Is There AnyPattern?
- 78.
- 79.
- 80. Maybe this istoo Maybe you’re much suck? } changing too much at once? } Happening too often?
- 81. What percentage ofincidents are related to change? http://www.flickr.com/photos/78364563@N00/2467989781/
- 82. What percentage ofchange- related incidents are “off-hours”? http://www.flickr.com/photos/jeffreyanthonyrafolpiano/3266123838
- 83. What percentage ofchange- related incidents are “off-hours”? Do they have higher or lower TTR? http://www.flickr.com/photos/jeffreyanthonyrafolpiano/3266123838
- 84. What types ofchange have the worst success rates? http://www.flickr.com/photos/lwr/2257949828/
- 85. What types ofchange have the worst success rates? Which ones have the best success rates? http://www.flickr.com/photos/lwr/2257949828/
- 86. Does your TTD/TTR increase depending on the: - SIZE? - FREQUENCY? http://www.flickr.com/photos/45409431@N00/2521827947/
- 87. Side effect is that you’re also tracking successful changes to production as well http://www.flickr.com/photos/wwworks/2313927146
- 88. Q2 2010 Incident Success Type Successes Failures Minutes(Sev1 Rate /2) App code 420 5 98.81 8 Config 404 3 99.26 5 DB Schema 15 1 93.33 10 DNS 45 0 100 0 Network (misc) 5 0 100 0 Network (core) 1 0 100 0
- 89. Q2 2010 Incident Success Type Successes Failures Minutes(Se ! Rate v1/2) App code 420 5 98.81 8 Config 404 3 99.26 5 DB Schema 15 1 93.33 10 DNS 45 0 100 0 Network (misc) 5 0 100 0 Network (core) 1 0 100 0
- 90.
- 91. Incident Observations Morale Length of Incident/Outage
- 92. Incident Observations Mistakes Length of Incident/Outage
- 93. Change Observations Change Size Change Frequency
- 94. Change Observations Huge changesets deployed rarely Change Size Change Frequency
- 95. Change Observations Huge changesets (high TTR) deployed rarely Change Size Change Frequency
- 96. Change Observations Huge changesets (high TTR) deployed rarely Change Size Tiny changesets deployed often Change Frequency
- 97. Change Observations Huge changesets (high TTR) deployed rarely Change Size Tiny changesets deployed often (low TTR) Change Frequency
- 98. Specifically.... la de da, What if this was only 5 } everything’s fine lines of code that were changed? Does that feel safer? change happens (it should)
- 99. Pay attention tothis stuff http://www.flickr.com/photos/plasticbag/2461247090/
- 100. We’re Hiring Ops! SF& NYC In May: - $22.9M of goods were sold by the community - 1,895,943 new items listed - 239,340 members joined
- 101.
- 102.
- 103. Continuous Deployment Described in 6 graphs (Originally Cal Henderson’s idea)
Editor's Notes
- #2 This is about metrics about YOU! Metrics *about* the metrics-makers!
- #3 They are basically taken from both Flickr and Etsy.
- #4 HOW MANY: write public-facing app code? maintain the release tools? release process? respond to incidents? have had an outage or notable degradation this month? that was change-related?
- #5 Too fast? Too loose? Too many issues? Too many upset and stressed out humans?
- #6 Everyone is used to bug tracking, it’s something worthwhile....
- #10 If this is a feeling you have often, please read on.
- #12 All you need is to see this happen once, and it’s hard to get out of your memory. No wonder why some people can start to think “code deploy = outage”.
- #13 Mild version of “Critical Incident Stress Management”? Change = risk, and sometimes risk = outage. And outages are stressful.
- #14 Not supposed to feel like this.
- #16 Details about the change play a huge role in your ability to respond to change-related incidents.
- #17 Details about the change play a huge role in your ability to respond to change-related incidents.
- #18 Details about the change play a huge role in your ability to respond to change-related incidents.
- #20 We do this by tracking our responses to outages and incidents.
- #21 We can do this by tracking our change, and learning from the results.
- #22 We need to raise confidence that we’re moving as fast as we can while still being safe enough to do so. And we can adjust the change to meet our requirements...
- #23 Why should change and results of changes be any different?
- #24 Type = code, schema, infrastructure, etc. Frequency/Size = how often each type is changed, implies risk Results = how often each change results in an incident/degradation
- #25 Lots of different types here. Might be different for everyone. Not all types of change bring the same amount of risk.
- #26 This info should be considered mandatory. This should also be done for db schema changes, network changes, changes in any part of the stack, really.
- #27 The header of our metrics tools has these statistics, too.
- #28 The tricky part: getting all prod changes written down without too much hassle.
- #29 Here’s one type of change....
- #30 Here’s another type of change....
- #31 Here’s yet another type of change...
- #32 Size does turn out to be important. Size = lines of code, level of SPOF risk, etc.
- #33 This seems like something you should do. Also: “incidents” = outages or degradations.
- #34 Just an example. This looks like it’s going well! Getting better!
- #35 Maybe I can’t say that it’s getting better, actually....
- #37 Some folks have Techcrunch as their incident log keeper. You could just use a spreadsheet.
- #38 An example!
- #39 You *are* doing postmortems on incidents that happen, right? Doing them comes at a certain point in your evolution.
- #43 Without the statistics, even a rare but severe outage can make the impression that change == outage.
- #44 Just examples. It’s important to categorize these things so you can count the ones that matter for the user’s experience. #4 Loss of redundancy
- #45 Just examples. It’s important to categorize these things so you can count the ones that matter for the user’s experience. #4 Loss of redundancy
- #46 Just examples. It’s important to categorize these things so you can count the ones that matter for the user’s experience. #4 Loss of redundancy
- #47 Just examples. It’s important to categorize these things so you can count the ones that matter for the user’s experience. #4 Loss of redundancy
- #48 Just examples. It’s important to categorize these things so you can count the ones that matter for the user’s experience. #4 Loss of redundancy
- #49 Just examples. It’s important to categorize these things so you can count the ones that matter for the user’s experience. #4 Loss of redundancy
- #50 Just examples. It’s important to categorize these things so you can count the ones that matter for the user’s experience. #4 Loss of redundancy
- #51 Just examples. This normally comes from a postmortem meeting. A good pointer on Root Cause Analysis is Eric Ries’ material on Five Whys, and the wikipedia page for RCA.
- #52 http://www.flickr.com/photos/mattblaze/2695044170/
- #53 What happens in our response to a change-related incident is just as important as the occurrence of the incident.
- #54 What happens in our response to a change-related incident is just as important as the occurrence of the incident.
- #55 What happens in our response to a change-related incident is just as important as the occurrence of the incident.
- #56 What happens in our response to a change-related incident is just as important as the occurrence of the incident.
- #57 What happens in our response to a change-related incident is just as important as the occurrence of the incident.
- #58 Th
- #59 Th
- #60 This might also be known as a ‘diagnose’ point.
- #61 This might also be known as a ‘diagnose’ point.
- #62 These events usually spawn other events.
- #63 These events usually spawn other events.
- #64 This should be standard operating procedure at this point,
- #65 These events usually spawn other events.
- #66 Some folks might notice a “Time To Diagnose” missing here. ALSO: it’s usually more complex than this, but this is the gist of it.
- #69 Do incidents increase with size of change? With frequency? With frequency/size of different types?
- #72 If you don’t track: Change, Incidents, and Responses, you’ll never have answers for these questions.
- #75 Reasonable questions.
- #76 *YOU* get to decide what is “small” and “frequent”.
- #79 THIS is what can help give you confidence. Or not.
- #81 The longer an outage lasts, the bigger of a bummer it is for all those who are working on fixing it.
- #82 The longer an outage lasts, the more mistakes people make. (and, as the night gets longer) Red herrings...
- #83 put two points on this graph
- #84 put two points on this graph
- #85 put two points on this graph
- #86 put two points on this graph
- #87 It should, because it is.
- #88 How we feel about change and how it can (or not) cause outages is important. Some of the nastiest relationships emerge between dev and ops because of these things.
- #93 “Normal” = lots of change done at regular intervals, change = big, time = long.
- #94 2 weeks? 5000 lines?
- #95 Scary Monster of Change! Each incident-causing deploy has only one recourse: roll it all back. Even code that was ok and unrelated to the incident. Boo!
- #96 Silly Monster of Nothing to Be Afraid Of Because His Teeth Are Small.
- #97 Problem? Roll that little piece back. Or better yet, roll it forward!
- #98 This looks like an adorable monster. Like a Maurice Sendak monster.