BDA304 Data-driven Post Mortems

© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Jason Yee, Datadog
18 April 2017
Data-driven Postmortems

@GITBISECT
TECHNICAL
WRITER/EVANGELIST
“DOCS & TALKS”
TRAVEL HACKER &
WHISKEY HUNTER
about me:
@DATADOGHQ
SAAS BASED
MONITORING
TRILLION DATA POINTS
PER DAY
about Datadog:

“The problems we work on at
Datadog are hard and often don't
have obvious, clean-cut solutions,
so it's useful to cultivate your
troubleshooting skills, no matter
what role you work in.”
Internal Datadog Developer Guide
TW: @gitbisect @datadoghq

“THE ONLY
REAL
MISTAKE
IS THE
ONE FROM
WHICH WE
LEARN
- Henry Ford

“AN ANALYSIS OR
DISCUSSION OF AN EVENT
HELD SOON AFTER IT HAS
OCCURRED, ESPECIALLY
IN ORDER TO DETERMINE
WHY IT WAS A FAILURE.”
OXFORD ENGLISH DICTIONARY
Oxford English Dictionary
POSTMORTEM

DAMON EDWARDS & JOHN WILLIS - DEVOPSDAY LOS ANGELES
WHAT IS
DEVOPS?
▸ Culture
▸ Automation
▸ Metrics
▸ Sharing

DAMON EDWARDS & JOHN WILLIS - DEVOPSDAY LOS ANGELES
OUR FOCUS
AREA
▸ Culture
▸ Sharing

WHY DOES THIS MATTER
TO

You’re either building a
learning organization… or you
will be losing to someone who
is. Andrew Clay Shafer
WINNING OR LOSING?

CULTURE & SHARING ARE GREAT, BUT WHAT
ABOUT

Follow
@honest_update
on Twitter

COLLECTING DATA
IS CHEAP;
NOT HAVING IT
WHEN YOU NEED
IT CAN BE
EXPENSIVE
SO INSTRUMENT ALL THE THINGS!

4 QUALITIES OF
GOOD METRICS
NOT ALL METRICS ARE EQUAL

How granular?
• AWS CloudWatch – 1 minute
• Google Stackdriver – 1 minute
• MS Azure
• 1 minute up to 24 hours
• 1 hour up to 7 days
• 1 day up to 30 days
• Datadog – seconds

How granular?
1 second
5 minutes
1 minute

Tagged & filterable
us-east-1e
total
us-east-1b

Sequoias by Allie Caulfield. CC-BY2.0
4. Long-lived

How long of a retention?
• AWS CloudWatch: 15 months!
• 1 minute granularity up to 15 days
• 5 minute granularity up to 63 days
• 1 hour granularity up to 15 months
• Google Stackdriver: 6 weeks
• MS Azure: 90 days
• Datadog: 15 months, no aggregation

Donuts by Irina Slutsky CC BY 2.0

Texas-sized Donut by Roy Niswanger CC BY 2.0

RECURSE UNTIL YOU FIND THE
TECHNICAL CAUSES

THERE IS
NO
SINGULA
R
”ROOT
Roots by CRCHF CC BY SA 4.0

TECHNICAL ISSUES HAVE NON-TECHNICAL
CAUSES

IF YOU’RE
STILL
RESPONDI
NG TO THE
INCIDENT,
IT’S NOT
TIME FOR A

HUMAN DATA
DATA COLLECTION:
WHO?▸ Everyone!
▸ Responders
▸ Identifiers
▸ Affected Users

HUMAN DATA
DATA
COLLECTION:
WHAT?
▸Their perspective
▸What they did
▸What they thought
▸Why they thought/did it

“WRITING IS NATURE’S WAY OF
LETTING YOU KNOW HOW
SLOPPY YOUR THINKING IS.”
RICHARD
GUINDON

TELLING STORIES
“ONE
PICTURE IS
WORTH
TEN
THOUSAND
WORDS”
- ANCIENT

HUMAN DATA
DATA COLLECTION:
WHEN?▸ As soon as possible.
▸ Memory drops sharply within 20 minutes
▸ Susceptibility to “false memory” increases
▸ Get your project managers involved!

HUMAN DATA
DATA
SKEW/CORRUPTION▸ Stress
▸ Sleep deprivation
▸ Burnout

HUMAN DATA
DATA
SKEW/CORRUPTION▸ Blame
▸ Fear of punitive action

CULTURE & SHARING RESOURCES
BLAMELESS
POSTMORTEMS▸ Blameless Postmortems by John Allspaw
http://bit.ly/etsy-blameless
▸ The Human Side of Postmortems by Dave
Zwieback
http://bit.ly/human-postmortem

HUMAN DATA
DATA
SKEW/CORRUPTION▸ Bias
▸ Anchoring
▸ Hindsight
▸ Outcome
▸ Availability (Recency)
▸ Bandwagon Effect

HOW WE
DO
POSTMORT
EMS
AT

DATADOG POSTMORTEMS
A FEW NOTES
▸ Postmortems emailed to company wide
▸ Scheduled recurring postmortem meetings

DATADOG’S POSTMORTEM TEMPLATE (1/5)
SUMMARY: WHAT
HAPPENED?▸ Describe what happened here at a high-level --
think of it as an abstract in a scientific paper.
▸ What was the impact on customers?
▸ What was the severity of the outage?
▸ What components were affected?
▸ What ultimately resolved the outage?

HOW WAS THE
OUTAGE DETECTED?▸ We want to make sure we detected the issue early
and would catch the same issue if it were to
repeat.
▸ Did we have a metric that showed the outage?
▸ Was there a monitor on that metric?
▸ How long did it take for us to declare an outage?

*Names and links changed for privacy/security.

* Portion of dashboard linked from previous slide.

HOW DID WE
RESPOND?▸ Who was the incident owner & who else was
involved?
▸ Slack archive links and timeline of events!
▸ What went well?
▸ What didn’t go so well?

*Names and links changed for privacy/security.

CHAT
OPS
ARCHI
VES
FTW!*Names changed

*Names changed
TRACK LEARNINGS
AS YOU GO

WHY DID IT HAPPEN?
▸ Deep dive into the cause
▸ Examples from this incident:
▸ http://bit.ly/dd-statuspage
▸ http://bit.ly/alq-postmortem

HOW DO WE PREVENT IT
IN THE FUTURE?▸ Link to Github issues and Trello cards
▸ Now?
▸ Next?
▸ Later?
▸ Follow up notes

DATADOG’S POSTMORTEM TEMPLATE
RECAP:
▸ What happened (summary)?
▸ How did we detect it?
▸ How did we respond?
▸ Why did it happen (deep dive)?
▸ Actionable next steps!

KEEP LEARNING
MORE RESOURCES
▸ Postmortem Template
http://bit.ly/postmortem-template
▸ The Infinite Hows - John Allspaw
http://bit.ly/infinite-hows
▸ “Blameless” Postmortems don’t work - J Paul Reed
http://bit.ly/blameless-dont-work

LET’S TALK!@GITBISECT
@DATADOGHQ

Slides: bit.ly/aws-postmortems
Twitter: @gitbisect
Email: jason.yee@datadoghq.com

BDA304 Data-driven Post Mortems

Recommended

Recommended

More Related Content

What's hot

What's hot (6)

Similar to BDA304 Data-driven Post Mortems

Similar to BDA304 Data-driven Post Mortems (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Recently uploaded

Recently uploaded (20)

BDA304 Data-driven Post Mortems