In today’s world, a company must be a “Learning Organization” in order to be successful and innovative. Learning from both failure and success, in order to implement small incremental improvements is critical. But until you implement and apply new information, you haven’t truly “learned” anything and you certainly haven’t improved. According to the 2015 Monitoring Survey, most companies leverage metrics from monitoring and logging purely for performance analytics and trending. If high availability and reliability are important, they also leverage metrics to alert on fault and anomaly detection. Despite these “best practices”, the metrics are primarily only used as context to keep things “running” or return them back to “normal” if there’s a problem. Rarely is that data used as a method to identify areas of improvement once services have been restored. When an outage occurs to your system, you will absolutely repair and restore services as best you know how, but are you paying attention to the data from the recovery efforts? What were operators seeing during diagnosis and remediation? What were their actions? What was going on with everyone, including conversations? A step-by-step replay of exactly what took place during that outage. This “old-view” perspective on the purpose of monitoring, logging, and alerting leaves the full value of metrics unrealized. It fails to address what’s important to the overall business objective and it lacks any hope of seeking out innovation or disruption of the status quo. This talk will illustrate how to identify if your company is making the best use of metrics and ways to not only learn from failure, but to become a “Learning Company”.

1. The Unrealized Role of:
Monitoring & Alerting
@jasonhand | VictorOps | #AllDayDevOps

2. @jasonhand | VictorOps | #AllDayDevOps

3. @jasonhand | VictorOps | #AllDayDevOps

4. THE UNREALIZED
ROLE OF:
Monitoring
& Alerting
@jasonhand | VictorOps | #AllDayDevOps

5. JASON
HAND
DevOps Evangelist
VictorOps
@jasonhand | VictorOps | #AllDayDevOps

6. @jasonhand | VictorOps | #AllDayDevOps

7. 2015
MONITORING
SURVEY@jasonhand | VictorOps | #AllDayDevOps

8. @jasonhand | VictorOps | #AllDayDevOps

9. WHY ARE YOU COLLECTING THIS DATA?
NOTE: You may choose more than one
▸ Performance analysis and trending
▸ Fault and Anomaly detection
▸ Capacity Planning
▸ A/B Testing
@jasonhand | VictorOps | #AllDayDevOps

10. THE RESULTS
NOTE: Respondents may have chose more than one
▸ Performance analysis and trending - 63%
▸ Fault and Anomaly detection - 53%
▸ Capacity Planning - 45%
▸ A/B Testing - 11%
@jasonhand | VictorOps | #AllDayDevOps

11. Tyranny of the
S.L.A.(Service Level Agreement)
@jasonhand | VictorOps | #AllDayDevOps

12. HIGH
AVAILABILITYPrediction & Prevention
@jasonhand | VictorOps | #AllDayDevOps

13. @jasonhand | VictorOps | #AllDayDevOps

14. @jasonhand | VictorOps | #AllDayDevOps

15. THAT'S IMPORTANT
... BUT ...@jasonhand | VictorOps | #AllDayDevOps

16. @jasonhand | VictorOps | #AllDayDevOps

17. BUSINESS
OBJECTIVES?@jasonhand | VictorOps | #AllDayDevOps

18. HAPPY CAMPER
@jasonhand | VictorOps | #AllDayDevOps

19. CUSTOMERSwant more than just
99.999% UPTIME@jasonhand | VictorOps | #AllDayDevOps

20. @jasonhand | VictorOps | #AllDayDevOps

21. WHERE'S THE
INNOVATION?
@jasonhand | VictorOps | #AllDayDevOps

22. HOW IMPORTANT
IS
Learning & Innovation?
@jasonhand | VictorOps | #AllDayDevOps

23. @jasonhand | VictorOps | #AllDayDevOps

24. @jasonhand | VictorOps | #AllDayDevOps

25. @jasonhand | VictorOps | #AllDayDevOps

26. @jasonhand | VictorOps | #AllDayDevOps

27. The result of underutilizing monitoring & alerting
is that the IT department and the organization have
no chance to...
LEARN,
IMPROVE, OR
INNOVATE.@jasonhand | VictorOps | #AllDayDevOps

28. CONTINUALLY UNDERSTANDING & RESPONDING
TO THE FEEDBACK
from
monitoring, logging, & alerting
allows you to use information about events in the past to drive future
actions.
@jasonhand | VictorOps | #AllDayDevOps

29. @jasonhand | VictorOps | #AllDayDevOps

30. @jasonhand | VictorOps | #AllDayDevOps

31. It's not just about
PREDICTION
& PREVENTION
@jasonhand | VictorOps | #AllDayDevOps

32. RESPOND &
REPAIR
...QUICKLY
@jasonhand | VictorOps | #AllDayDevOps

33. NOPE
@jasonhand | VictorOps | #AllDayDevOps

34. MTTRRather Than
MTBF@jasonhand | VictorOps | #AllDayDevOps

35. FAILURE IS
INEVITABLE
@jasonhand | VictorOps | #AllDayDevOps

36. US·ER
/ˈYOOZƏR/
DISTRIBUTED FAULT INJECTION TEST SUITE FOR
PRODUCTION.
credit: Leon Fayer (@papa_fire)
@jasonhand | VictorOps | #AllDayDevOps

37. SUCCESS
is a result of
FAILURE@jasonhand | VictorOps | #AllDayDevOps

38. UNDERSTAND
LEARN
INNOVATE@jasonhand | VictorOps | #AllDayDevOps

39. RE·SIL·IENT/RƏˈZILYƏNT/
The ability to resist, absorb, recover from or successfully adapt to
adversity or a change in conditions
@jasonhand | VictorOps | #AllDayDevOps

40. CHANGE
can cause failure
but innovation requires
CHANGE
@jasonhand | VictorOps | #AllDayDevOps

41. CONFLICT
@jasonhand | VictorOps | #AllDayDevOps

42. CHANGEREQUIRED@jasonhand | VictorOps | #AllDayDevOps

43. Without deviation from the norm,
progress is not possible
— Frank Zappa
@jasonhand | VictorOps | #AllDayDevOps

44. What Did You
LEARNFrom the Recovery Efforts?
(including monitoring & alerting)
@jasonhand | VictorOps | #AllDayDevOps

45. POSTMORTEMS / LEARNING REVIEWS:
Stories of:
WHAT TOOK PLACE
leading up to & during
the disruption & recovery efforts
@jasonhand | VictorOps | #AllDayDevOps

46. WHO WAS
INVOLVED?@jasonhand | VictorOps | #AllDayDevOps

47. WHAT DID THEY
SEE?@jasonhand | VictorOps | #AllDayDevOps

48. WHAT WAS
SAID?@jasonhand | VictorOps | #AllDayDevOps

49. WHAT
ACTIONSWERE TAKEN?
jhand.co/chatopsbook
@jasonhand | VictorOps | #AllDayDevOps

50. HOW DO
events & actions
CORRELATE
OVER TIME?@jasonhand | VictorOps | #AllDayDevOps

51. 5 Why's@jasonhand | VictorOps | #AllDayDevOps

52. 5 Why's@jasonhand | VictorOps | #AllDayDevOps

53. WHAT IS THE "cause"
OF THE PROBLEM?
Root Cause is ...
@jasonhand | VictorOps | #AllDayDevOps

54. OUR...
obsession with
"Root Cause"
@jasonhand | VictorOps | #AllDayDevOps

55. ASKING "WHY"
.. leads to ..
BLAME@jasonhand | VictorOps | #AllDayDevOps

56. BLAMING
LEADS TO..
operators hiding relevant & important
information
@jasonhand | VictorOps | #AllDayDevOps

57. We must
BELIEVEthat our operators are doing their best given the
constraints of the "system"
@jasonhand | VictorOps | #AllDayDevOps

58. "We are here to"
LEARNFrom Failure
(and success)
@jasonhand | VictorOps | #AllDayDevOps

59. RATHER THAN ..
@jasonhand | VictorOps | #AllDayDevOps

60. AVOIDFAILURE@jasonhand | VictorOps | #AllDayDevOps

61. WHAT'S THE
STORY?@jasonhand | VictorOps | #AllDayDevOps

62. INNOVATE
Learning from both success & failure
to develop & implement
small incremental improvements
is critical.
@jasonhand | VictorOps | #AllDayDevOps

63. MONITORING &
ALERTINGHelps us understand the story in greater detail
@jasonhand | VictorOps | #AllDayDevOps

64. LEARNING
ORGANIZATION
@jasonhand | VictorOps | #AllDayDevOps

65. Learning does NOT come from
READING
&
LISTENING@jasonhand | VictorOps | #AllDayDevOps

66. Learning comes from
DOING@jasonhand | VictorOps | #AllDayDevOps

67. Real Learning comes from:
OBSERVING
ORIENTING
DECIDING
ACTING
John Boyd's OODA Loop
@jasonhand | VictorOps | #AllDayDevOps

68. Example:
LEARNING TO PLAY THE
DOBRO GUITAR@jasonhand | VictorOps | #AllDayDevOps

69. @jasonhand | VictorOps | #AllDayDevOps

70. LEARNING
@jasonhand | VictorOps | #AllDayDevOps

71. WHY?
Go from knowing...
to understanding...
to learning
NOTE:
(Requires making mistakes)
@jasonhand | VictorOps | #AllDayDevOps

72. We will trade some uptime in exchange for innovation
-Dave Hahn (Netflix)
DevOpsDays Boise 2016
@jasonhand | VictorOps | #AllDayDevOps

73. SHIFT OUR GAZE
from:
MAINTAINING
& PROTECTING
@jasonhand | VictorOps | #AllDayDevOps

74. LEARNING
Which leads to...
IMPROVING
& INNOVATING
@jasonhand | VictorOps | #AllDayDevOps

75. WE INCREASE VALUE OF:
- Monitoring & Alerting
- IT teams
- Products & Services
- Organization
@jasonhand | VictorOps | #AllDayDevOps

76. HYPOTHESIZE
EXPLORE
STRETCH
EXPERIMENT
FAIL
LEARN
Try Again
@jasonhand | VictorOps | #AllDayDevOps

77. @jasonhand | VictorOps | #AllDayDevOps

78. LEARNING & INNOVATING
leads to uncovering new ways of
BUILDING, DEPLOYING, AND MAINTAINING
SOFTWARE & INFRASTRUCTURE
Which leads to...
@jasonhand | VictorOps | #AllDayDevOps

79. RESILIENTSYSTEMS@jasonhand | VictorOps | #AllDayDevOps

80. The
By-product
of a highly
RESILIENT
system is ...
@jasonhand | VictorOps | #AllDayDevOps

81. @jasonhand | VictorOps | #AllDayDevOps

82. HIGHLY
AVAILABLE
SYSTEM@jasonhand | VictorOps | #AllDayDevOps

83. THE UNREALIZED
ROLE OF:
Monitoring
& Alerting is ....
@jasonhand | VictorOps | #AllDayDevOps

84. LEARNING
&
INNOVATION@jasonhand | VictorOps | #AllDayDevOps

85. THANK
YOUBe Victorious!
@jasonhand | VictorOps | #AllDayDevOps

86. @jasonhand | VictorOps | #AllDayDevOps

87. Monitoring Survey: https://kartar.net/2015/08/monitoring-
survey-2015---metrics/
Firefighter: https://www.learyfirefighters.org/wp-content/uploads/
2013/09/cover-slide-1.jpg
Mechanic: https://upload.wikimedia.org/wikipedia/commons/4/4b/
Flickr_-_Israel_Defense_Forces_-
_Airplane_Technician,_March_2010.jpg
Gnome Plan: http://www.nerdfitness.com/wp-content/uploads/
2012/04/Screen-Shot-2012-03-30-at-3.15.38-AM-1024x7591.jpg
NOC: https://upload.wikimedia.org/wikipedia/commons/0/03/@jasonhand | VictorOps | #AllDayDevOps

88. References:
Kodak: http://file.answcdn.com/answ-cld/image/upload/v1/tk/
brand_image/b59911fc/
91d6e71d30a0878dfe3cb30a22751cb874a3ea8c.jpeg
VW Camper: https://upload.wikimedia.org/wikipedia/commons/d/d7/
VW_Camper.jpg
Blockbuster: https://jordanandeddie.files.wordpress.com/2013/11/
blockbuster-feature.jpg
Borders: http://smashingtops.com/wp-content/uploads/2012/06/
borders_logo1.jpg@jasonhand | VictorOps | #AllDayDevOps

89. Chained Hands: https://www.google.com/url?
sa=i&rct=j&q=&esrc=s&source=images&cd=&ved=0ahUKEwjgrNCD
h5TMAhXJs4MKHaoZDssQjBwIBA&url=http%3A%2F
%2Fwww.publicdomainpictures.net%2Fdownload-picture.php
%3Fadresar%3D50000%26soubor%3Dhands-in-chains.jpg%26id
%3D40426&bvm=bv.119745492,d.amc&psig=AFQjCNFIdnDPzSqiLA-
znIW5SCTCUHhqEw&ust=1460926880336203
Inevitable: http://vignette4.wikia.nocookie.net/matrix/images/5/51/
SMITH.png/revision/latest?cb=20110214092002
Bulb: https://smhttp-ssl-37293.nexcesscdn.net/media/catalog/
@jasonhand | VictorOps | #AllDayDevOps

90. scoreboard/1000/Safety-Awareness-Sign-DSE-195271000.gif
Stewie:
http://chroniclesofredmark.com/wp-content/uploads/2014/01/
Stewie.gif
change: http://i.imgur.com/EQyC6N3.gif
Hard drive: https://i.imgur.com/pWsKSEf.gif
Change: https://farm6.staticflickr.com/
5208/5270199049df99b234e9od.jpg
Value: https://d13yacurqjgara.cloudfront.net/users/6437/
screenshots/1405551/value-cropped.gif
@jasonhand | VictorOps | #AllDayDevOps