Ops Meta-Metrics: The Currency You Pay For Change

1. Ops Meta-Metrics The Currency You Use to Pay For Change John Allspaw VP Operations Etsy.com http://www.ﬂickr.com/photos/wwarby/3296379139

2. Warning Graphs and numbers in this presentation are sort of made up

3. /usr/nagios/libexec/check_ops.pl

4. How R U Doing? http://www.ﬂickr.com/photos/a4gpa/190120662/

5. We track bugs already... Example: https://issues.apache.org/jira/browse/TS

6. We should track these, too...

7. We should track these, too... Changes (Who/What/When/Type)

8. We should track these, too... Changes (Who/What/When/Type) Incidents (Type/Severity)

9. We should track these, too... Changes (Who/What/When/Type) Incidents (Type/Severity) Response to Incidents (TTR/TTD)

10. trepidation noun 1 a feeling of fear or agitation about something that may happen : the men set off in fear and trepidation. 2 archaic trembling motion. DERIVATIVES trepidatious adjective ORIGIN late 15th cent.: from Latin trepidatio(n-), from trepidare ‘be agitated, tremble,’ from trepidus ‘alarme

11. Change Required. Often feared. Why? http://www.ﬂickr.com/photos/20408885@N03/3570184759/

12. This is why OMGWTF OUTAGES!!!1!! la de da, everything’s ﬁne change happens

13. Change PTSD? http://www.ﬂickr.com/photos/tzoﬁa/270800047/

14. Brace For Impact?

15. Brace For Impact?

16. But wait.... (OMGWTF) la de da, everything’s ﬁne change happens

17. But wait.... (OMGWTF) la de da, } everything’s ﬁne How much change is this? change happens

18. But wait.... (OMGWTF) la de da, } everything’s ﬁne How much change is this? What kind of change? change happens

19. But wait.... (OMGWTF) la de da, } everything’s ﬁne How much change is this? What kind of change? How often does this happen? change happens

20. Need to raise conﬁdence that change != outage

21. ...incidents can be handled well http://www.ﬂickr.com/photos/axiepics/3181170364/

22. ...root causes can be ﬁxed quick enough http://www.ﬂickr.com/photos/ljv/213624799/

23. ...change can be safe enough http://www.ﬂickr.com/photos/marksetchell/43252686/

24. But how? How do we have conﬁdence in anything in our infrastructure? We measure it. And graph it. And alert on it.

25. Tracking Change 1. Type 2. Frequency/Size 3. Results of those changes

26. Types of Change Layers Examples App code PHP/Rails/etc or ‘front-end’ code Apache, MySQL, DB schema, Services code PHP/Ruby versions, etc. OS/Servers, Switches, Routers, Infrastructure Datacenters, etc. (you decide what these are for your architecture)

27. Code Deploys: Who/What/When WHEN WHO WHAT (guy who pushed the button) (link to diff) (http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/)

28. Code Deploys: Who/What/When Last 2 prod deploys Last 2 Chef changes

29. other changes (insert whatever ticketing/tracking you have)

30. Frequency

31. Frequency

32. Frequency

33. Size

34. Tracking Incidents http://www.ﬂickr.com/photos/47684393@N00/4543311558/

35. Incident Frequency

36. Incident Size Big Outage TTR still going

37. Tracking Incidents 1. Frequency 2. Severity 3. Root Cause 4. Time-To-Detect (TTD) 5. Time-To-Resolve (TTR)

38. The How Doesn’t Matter http://www.ﬂickr.com/photos/matsuyuki/2328829160/

39. Incident/Degradation Tracking Start Detect Resolve Root PostMortem Date Severity Done? Time Time Time Cause 1/2/08 12:30 ET 12:32 ET 12:45 ET Sev1 DB Change Yes 3/7/08 18:32 ET 18:40 ET 18:47 ET Sev2 Capacity Yes 5/3/08 17:55 ET 17:55 ET 18:14 ET Sev3 Hardware Yes

40. Incident/Degradation Tracking Start Detect Resolve Root PostMortem Date Severity Time These Time give you will Time context Cause Done? for your rates of change. (You’ll need them for postmortems, anyway.)

41. Change:Incident Ratio

42. Change:Incident Ratio Important.

43. Change:Incident Ratio Important. Not because all changes are equal.

44. Change:Incident Ratio Important. Not because all changes are equal. Not because all incidents are equal, or change-related.

45. Change:Incident Ratio But because humans will irrationally make a permanent connection between the two. http://www.ﬂickr.com/photos/michelepedrolli/449572596/

46. Severity

47. Severity Not all incidents are created equal.

48. Severity Not all incidents are created equal. Something like:

49. Severity Not all incidents are created equal. Something like:

50. Severity Not all incidents are created equal. Something like: SEV1 Full outage, or effectively unusable.

51. Severity Not all incidents are created equal. Something like: SEV1 Full outage, or effectively unusable. SEV2 Signiﬁcant degradation for subset of users.

52. Severity Not all incidents are created equal. Something like: SEV1 Full outage, or effectively unusable. SEV2 Signiﬁcant degradation for subset of users. SEV3 Minor impact on user experience.

53. Severity Not all incidents are created equal. Something like: SEV1 Full outage, or effectively unusable. SEV2 Signiﬁcant degradation for subset of users. SEV3 Minor impact on user experience. SEV4 No impact, but time-sensitive failure.

54. Root Cause? (Not all incidents are change related) Something like: Note: this can be difﬁcult to categorize. http://en.wikipedia.org/wiki/Root_cause_analysis

55. Root Cause? (Not all incidents are change related) Something like: 1. Hardware Failure 2. Datacenter Issue 3. Change: Code Issue 4. Change: Config Issue 5. Capacity/Traffic Issue 6. Other Note: this can be difficult to categorize. http://en.wikipedia.org/wiki/Root_cause_analysis

56. Recording Your Response (worth the hassle) http://www.ﬂickr.com/photos/mattblaze/2695044170/

57. Time

58. la de da, everything’s ﬁne Time

59. la de da, everything’s ﬁne Time change happens

60. Noticed there was a problem la de da, everything’s ﬁne Time change happens

61. Noticed there was a problem Figured out la de da, what the cause is everything’s ﬁne Time change happens

62. Fixed the problem Noticed there •rolled back was a problem •rolled forward •temporary solution •etc Figured out la de da, what the cause is everything’s ﬁne Time change happens

63. Fixed the problem Noticed there •rolled back was a problem •rolled forward •temporary solution •etc Figured out la de da, what the cause is everything’s ﬁne Time change happens

64. • Coordinate troubleshooting/diagnosis Fixed the problem Noticed there •rolled back was a problem •rolled forward •temporary solution •etc Figured out la de da, what the cause is everything’s ﬁne Time change happens

65. • Coordinate troubleshooting/diagnosis • Communicate to support/community/execs Fixed the problem Noticed there •rolled back was a problem •rolled forward •temporary solution •etc Figured out la de da, what the cause is everything’s ﬁne Time change happens

66. Fixed the problem Noticed there •rolled back was a problem •rolled forward •temporary solution •etc Figured out la de da, what the cause is everything’s ﬁne Time Time change happens

67. • Coordinate responses* Fixed the problem Noticed there •rolled back was a problem •rolled forward •temporary solution •etc Figured out la de da, what the cause is everything’s ﬁne Time Time change happens * usually, “One Thing At A Time” responses

68. • Coordinate responses* • Communicate to support/community/execs problem Fixed the Noticed there •rolled back was a problem •rolled forward •temporary solution •etc Figured out la de da, what the cause is everything’s ﬁne Time Time change happens * usually, “One Thing At A Time” responses

69. Fixed the problem Figured out what the cause is Noticed there •rolled back was a problem •rolled forward •temporary solution •etc la de da, everything’s ﬁne Time Time change happens

70. • Conﬁrm stability, resolving steps Fixed the problem Figured out what the cause is Noticed there •rolled back was a problem •rolled forward •temporary solution •etc la de da, everything’s ﬁne Time Time change happens

71. • Conﬁrm stability, resolving steps • Communicate to support/community/execs Fixed the problem Figured out what the cause is Noticed there •rolled back was a problem •rolled forward •temporary solution •etc la de da, everything’s ﬁne Time Time change happens

72. Communications http://etsystatus.com twitter.com/etsystatus

73. Fixed the problem Figured out what the cause is Noticed there •rolled back was a problem •rolled forward •temporary solution •etc la de da, everything’s ﬁne Time Time change happens PostMortem

74. Time To Detect (TTD) Time To Resolve la de da, (TTR) la de da, everything’s ﬁne everything’s ﬁne Time change happens

75. Hypothetical Example: “We’re So Nimble!”

76. Nimble, But Stumbling?

77. Is There Any Pattern?

78. Nimble, But Stumbling? +

79. Nimble, But Stumbling? +

80. Maybe this is too Maybe you’re much suck? } changing too much at once? } Happening too often?

81. What percentage of incidents are related to change? http://www.ﬂickr.com/photos/78364563@N00/2467989781/

82. What percentage of change- related incidents are “off-hours”? http://www.ﬂickr.com/photos/jeffreyanthonyrafolpiano/3266123838

83. What percentage of change- related incidents are “off-hours”? Do they have higher or lower TTR? http://www.ﬂickr.com/photos/jeffreyanthonyrafolpiano/3266123838

84. What types of change have the worst success rates? http://www.ﬂickr.com/photos/lwr/2257949828/

85. What types of change have the worst success rates? Which ones have the best success rates? http://www.ﬂickr.com/photos/lwr/2257949828/

86. Does your TTD/TTR increase depending on the: - SIZE? - FREQUENCY? http://www.ﬂickr.com/photos/45409431@N00/2521827947/

87. Side effect is that you’re also tracking successful changes to production as well http://www.ﬂickr.com/photos/wwworks/2313927146

88. Q2 2010 Incident Success Type Successes Failures Minutes(Sev1 Rate /2) App code 420 5 98.81 8 Conﬁg 404 3 99.26 5 DB Schema 15 1 93.33 10 DNS 45 0 100 0 Network (misc) 5 0 100 0 Network (core) 1 0 100 0

89. Q2 2010 Incident Success Type Successes Failures Minutes(Se ! Rate v1/2) App code 420 5 98.81 8 Conﬁg 404 3 99.26 5 DB Schema 15 1 93.33 10 DNS 45 0 100 0 Network (misc) 5 0 100 0 Network (core) 1 0 100 0

90. Some Observations

91. Incident Observations Morale Length of Incident/Outage

92. Incident Observations Mistakes Length of Incident/Outage

93. Change Observations Change Size Change Frequency

94. Change Observations Huge changesets deployed rarely Change Size Change Frequency

95. Change Observations Huge changesets (high TTR) deployed rarely Change Size Change Frequency

96. Change Observations Huge changesets (high TTR) deployed rarely Change Size Tiny changesets deployed often Change Frequency

97. Change Observations Huge changesets (high TTR) deployed rarely Change Size Tiny changesets deployed often (low TTR) Change Frequency

98. Speciﬁcally.... la de da, What if this was only 5 } everything’s ﬁne lines of code that were changed? Does that feel safer? change happens (it should)

99. Pay attention to this stuff http://www.ﬂickr.com/photos/plasticbag/2461247090/

100. We’re Hiring Ops! SF & NYC In May: - $22.9M of goods were sold by the community - 1,895,943 new items listed - 239,340 members joined

101. The End

102. Bonus Time!!1!

103. Continuous Deployment Described in 6 graphs (Originally Cal Henderson’s idea)