SlideShare a Scribd company logo
1 of 67
Download to read offline
Skeptics in the Church of Data Getting Evangelical
AaronAldrich–CageData/VictorOps
@CrayZeigh#Galaxz17
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 2
Aaron Aldrich
@CrayZeigh
@cagedata
@victorops
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 3
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 4
incidents happen
and they suck
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 5
we monitor
systems religiously
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 6
The Trinity of System Monitoring
• Thou shalt
monitor CPU
Usage
• Thou shalt
track processes
•Thou shalt
monitor disk
space
•Thou shalt
monitor I/Os
• Thou shalt
monitor
memory
availability
• Thou shalt
track activity
Process Memory Disk
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 7
sacraments
baptism
confession
repentance
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 8
100%	uptime
Ein Minuten
Bitte
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 10
monitoring
resources makes
sense when
tracking physical
system health
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 11
The New Data Center
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 12
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 13
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 14
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 15
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 16
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 17
if you’re deciding
which alarms are
important, you have
too many
Empathetic Monitoring
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 19
start monitoring
business metrics
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 20
revenue
courtesey Leon Fayer of OmniTI (@papa_fire)
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 21
user latency
courtesey Leon Fayer of OmniTI (@papa_fire)
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 22
courtesey Leon Fayer of OmniTI (@papa_fire)
database performance
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 23
courtesey Leon Fayer of OmniTI (@papa_fire)
decline rate (%)
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 24
how can I be a superhero?
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 25
“She doesn’t care whether or
not [the data center is literally
on fire] … just as long as his
ship’s coming in…”
-Cake, Italian Leather Sofa
(lightly interpreted)
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 26
have a chat with
your business
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 27
Spoiler Alert
90% of the
time you’ll
be listening
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 28
What	happens	on	a	normal	day?	
What	happens	on	a	good	day?	
What	happens	on	a	bad	day?
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 29
If	you	could	have	insight	
into	one	piece	of	the	
organization,	what	would	
it	be?
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 30
listen	to	your	business
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 31
context is crucial
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 32
new allies become evangelists
New Partners
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 34
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 35
what about marketing?
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 36
(https://blogs.adobe.com/digitalmarketing/wp-content/uploads/2015/10/laxalt-fig1.jpg)
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 37
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 38
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 39
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 40
The Chasm
(Moore – Crossing the Chasm 1991)
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 41
Reformation
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 43
change conducted poorly will
break your organization
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 44
strict change enforcement
creates unrest
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 45
…harder	than	you	think	
…takes	longer	than	you	expect	
…self-initiated
change is…
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 46
1.Flow
2.Feedback
3.Continuous
Experimentation &
Learning
The Three Ways
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 47
1.Flow
2.Feedback
3.Continuous
Experimentation &
Learning
The Three Ways
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 48
-enables continuous improvement
Amplify Feedback Loops
You Them
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 49
retrospectives
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 50
invite the whole
business
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 51
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 52
Reframe the goal:
we don’t want to find out what
should have happened, we want to
learn how to improve.
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 53
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 54
blame creates subject
matter experts in
information
suppression
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 55
ritual is important
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 56
“Before we begin, I want to remind
everyone that postmortems are
blameless. We’re not here to discuss
what should have or could have
happened because we’re here to
learn and gain new information for
the future.”
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 57
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 58
• VictorOps Blameless Postmortem Guide 

http://bit.ly/2nBAvq2

(https://victorops.com/wp-content/uploads/2014/09/Blameless-Post-Mortems.pdf)
• Chef Postmortem Template - http://bit.ly/2noY4S1

(https://gist.github.com/btm/7cb421f5fe7d1003083a)
• Etsy’s Debriefing Facilitation Guide 

http://etsy.me/2iVP7Bj

(https://extfiles.etsy.com/DebriefingFacilitationGuide.pdf)
Retrospective Resources
The charge
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 60
stop thinking about your
systems’ health and start
thinking about your
business’s health
Be Skeptical of your Traditions
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 61
partner with data-driven
departments in your org
to proselytize the benefits
of business monitoring
Make new evangelists
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 62
build safe environments,
amplify feedback and
foster learning
Safely Reform
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 63
Change the
world
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 64
Change the
world(Or at least your monitoring)
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 65
Thanks!
@CrayZeigh
Aaron Aldrich
http://cageit.us/GalaxZ17
SlideShare:
©		2016	All	Rights	Reserved	CONFIDENTIAL@CrayZeigh 66
67
Boots

More Related Content

More from Aaron Aldrich

DevOpsDays 2018 - Operating Human Systems: MTBF v. MTTR
DevOpsDays 2018 - Operating Human Systems: MTBF v. MTTRDevOpsDays 2018 - Operating Human Systems: MTBF v. MTTR
DevOpsDays 2018 - Operating Human Systems: MTBF v. MTTRAaron Aldrich
 
2018-01 DevOpsDays NYC: Operating Human Systems: MTBF v. MTTR
2018-01 DevOpsDays NYC: Operating Human Systems: MTBF v. MTTR2018-01 DevOpsDays NYC: Operating Human Systems: MTBF v. MTTR
2018-01 DevOpsDays NYC: Operating Human Systems: MTBF v. MTTRAaron Aldrich
 
DevOpsDays DC 2017 - Better Living Through Storytelling
DevOpsDays DC 2017 -  Better Living Through StorytellingDevOpsDays DC 2017 -  Better Living Through Storytelling
DevOpsDays DC 2017 - Better Living Through StorytellingAaron Aldrich
 
DevOpsDays TO - Managing Fires
DevOpsDays TO - Managing FiresDevOpsDays TO - Managing Fires
DevOpsDays TO - Managing FiresAaron Aldrich
 
DevOpsDays SEA - Managing Fires: Leadership through Crisis
DevOpsDays SEA - Managing Fires: Leadership through CrisisDevOpsDays SEA - Managing Fires: Leadership through Crisis
DevOpsDays SEA - Managing Fires: Leadership through CrisisAaron Aldrich
 
Better Living Through Storytelling
Better Living Through StorytellingBetter Living Through Storytelling
Better Living Through StorytellingAaron Aldrich
 
DevOpsDays Madison 2016 - Humane Treatment of On-Call Engineers (Ignite)
DevOpsDays Madison 2016 - Humane Treatment of On-Call Engineers (Ignite)DevOpsDays Madison 2016 - Humane Treatment of On-Call Engineers (Ignite)
DevOpsDays Madison 2016 - Humane Treatment of On-Call Engineers (Ignite)Aaron Aldrich
 
DOD-OH - The Humane Treatment of Engineers
DOD-OH - The Humane Treatment of EngineersDOD-OH - The Humane Treatment of Engineers
DOD-OH - The Humane Treatment of EngineersAaron Aldrich
 

More from Aaron Aldrich (9)

DevOpsDays 2018 - Operating Human Systems: MTBF v. MTTR
DevOpsDays 2018 - Operating Human Systems: MTBF v. MTTRDevOpsDays 2018 - Operating Human Systems: MTBF v. MTTR
DevOpsDays 2018 - Operating Human Systems: MTBF v. MTTR
 
2018-01 DevOpsDays NYC: Operating Human Systems: MTBF v. MTTR
2018-01 DevOpsDays NYC: Operating Human Systems: MTBF v. MTTR2018-01 DevOpsDays NYC: Operating Human Systems: MTBF v. MTTR
2018-01 DevOpsDays NYC: Operating Human Systems: MTBF v. MTTR
 
DevOpsDays DC 2017 - Better Living Through Storytelling
DevOpsDays DC 2017 -  Better Living Through StorytellingDevOpsDays DC 2017 -  Better Living Through Storytelling
DevOpsDays DC 2017 - Better Living Through Storytelling
 
DevOpsDays TO - Managing Fires
DevOpsDays TO - Managing FiresDevOpsDays TO - Managing Fires
DevOpsDays TO - Managing Fires
 
DevOpsDays SEA - Managing Fires: Leadership through Crisis
DevOpsDays SEA - Managing Fires: Leadership through CrisisDevOpsDays SEA - Managing Fires: Leadership through Crisis
DevOpsDays SEA - Managing Fires: Leadership through Crisis
 
Better Living Through Storytelling
Better Living Through StorytellingBetter Living Through Storytelling
Better Living Through Storytelling
 
DevOpsDays Madison 2016 - Humane Treatment of On-Call Engineers (Ignite)
DevOpsDays Madison 2016 - Humane Treatment of On-Call Engineers (Ignite)DevOpsDays Madison 2016 - Humane Treatment of On-Call Engineers (Ignite)
DevOpsDays Madison 2016 - Humane Treatment of On-Call Engineers (Ignite)
 
DOD-OH - The Humane Treatment of Engineers
DOD-OH - The Humane Treatment of EngineersDOD-OH - The Humane Treatment of Engineers
DOD-OH - The Humane Treatment of Engineers
 
Managing fires
Managing firesManaging fires
Managing fires
 

Recently uploaded

Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxFIDO Alliance
 
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptxCyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptxMasterG
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfalexjohnson7307
 
How to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in PakistanHow to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in Pakistandanishmna97
 
Event-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingEvent-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingScyllaDB
 
How to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfHow to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfdanishmna97
 
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdfFrisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdfAnubhavMangla3
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctBrainSell Technologies
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Skynet Technologies
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGDSC PJATK
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...FIDO Alliance
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxFIDO Alliance
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Paige Cruz
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxFIDO Alliance
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024Lorenzo Miniero
 
Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMKumar Satyam
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform EngineeringMarcus Vechiato
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...panagenda
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewDianaGray10
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftshyamraj55
 

Recently uploaded (20)

Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptxCyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdf
 
How to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in PakistanHow to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in Pakistan
 
Event-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingEvent-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream Processing
 
How to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfHow to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cf
 
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdfFrisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptx
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDM
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overview
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
 

Skeptics in the Church of Data Getting Evangelical