Data-Driven Postmortems - SRV208 - Anaheim AWS Summit
•
1 like•254 views
The DevOps movement has influenced not only the tools we use in modern development and operations engineering, but also how we work. It has changed how we respond when systems stop working or don't work as expected. In this session, we share methods and techniques for gathering data and using it effectively to mitigate or avoid future failures. This session is brought to you by AWS Partner, Datadog.
Recommended
Recommended
More Related Content
Similar to Data-Driven Postmortems - SRV208 - Anaheim AWS Summit
Similar to Data-Driven Postmortems - SRV208 - Anaheim AWS Summit (20)
More from Amazon Web Services
More from Amazon Web Services (20)
Data-Driven Postmortems - SRV208 - Anaheim AWS Summit
- 2. @dbenamy Developer Efficiency @datadoghq SaaS-based monitoring Trillions of data points per day
- 3. “THE ONLY REAL MISTAKE IS THE ONE FROM WHICH WE LEARN NOTHING.” - Henry Ford @dbenamy @datadoghq
- 4. COLLECTING DATA IS CHEAP; NOT HAVING DATA WHEN YOU NEED IT CAN BE EXPENSIVE. @dbenamy @datadoghq
- 5. “The problems we work on at Datadog are hard and often don't have obvious, clear solutions, so it's useful to cultivate your troubleshooting skills, no matter what role you work in.” Internal Datadog Developer Guide @dbenamy @datadoghq
- 6. DAMON EDWARDS & JOHN WILLIS - DEVOPSDAY MOUNTAIN VIEW 2010 WHAT IS DEVOPS? ▸ Culture ▸ Automation ▸ Metrics ▸ Sharing @dbenamy @datadoghq
- 7. DAMON EDWARDS & JOHN WILLIS – DEVOPSDAY, MOUNTAIN VIEW 2010 TODAY’S FOCUS @dbenamy @datadoghq ▸ Culture ▸ Automation ▸ Metrics ▸ Sharing
- 9. CULTURE & SHARING RESOURCES BLAMELESS POSTMORTEMS ▸ Blameless Postmortems, by John Allspaw http://bit.ly/etsy-blameless ▸ The Human Side of Postmortems, by Dave Zwieback http://bit.ly/human-postmortem @dbenamy @datadoghq
- 10. CULTURE & SHARING ARE GREAT, BUT WHAT ABOUT @dbenamy @datadoghq
- 12. COLLECTING DATA IS CHEAP; NOT HAVING DATA WHEN YOU NEED IT CAN BE EXPENSIVE. SO INSTRUMENT ALL THE THINGS! @dbenamy @datadoghq
- 13. FOUR QUALITIES OF GOOD METRICS NOT ALL METRICS ARE EQUAL @dbenamy @datadoghq
- 14. 1. WELL UNDERSTOOD @dbenamy @datadoghq
- 15. 1. WELL UNDERSTOOD @dbenamy @datadoghq
- 16. 1. WELL UNDERSTOOD @dbenamy @datadoghq
- 17. @dbenamy @datadoghq 2. SUFFICIENT GRANULARITY
- 18. 1 second Peak 46% 1 minute Peak 36% 5 minutes Peak 12%
- 19. 1 second Peak 46% 1 minute Peak 36% 5 minutes Peak 12%
- 20. 1 second Peak 46% 1 minute Peak 36% 5 minutes Peak 12%
- 21. 3. TAGGED & FILTERABLE @dbenamy @datadoghq
- 25. Query-Based Monitoring “What’s the average throughput of application:nginx per version?” “How many requests per second is my role:accounting-app running application:postgresql hosted in region:us-west-1 compared to region:us- east-1?” @dbenamy @datadoghq
- 30. P.S. - June 1! Mark your calendar! @dbenamy @datadoghq
- 31. RECURSE UNTIL YOU FIND THE TECHNICAL CAUSES @dbenamy @datadoghq
- 32. -Future you
- 33. RECURSE UNTIL YOU FIND THE TECHNICAL CAUSES @dbenamy @datadoghq
- 34. There is no singular “root cause.”
- 35. HUMAN ELEMENT TECHNICAL ISSUES HAVE NON-TECHNICAL CAUSES @dbenamy @datadoghq
- 36. IF YOU’RE STILL RESPONDING TO THE INCIDENT, IT’S NOT TIME FOR A REVIEW @dbenamy @datadoghq
- 37. HUMAN DATA DATA COLLECTION: WHO? ▸ Everyone! ▸ Responders ▸ Identifiers ▸ Affected users @dbenamy @datadoghq
- 38. HUMAN DATA DATA COLLECTION: WHAT? ▸Their perspective ▸What they did ▸What they thought ▸Why they thought or did it @dbenamy @datadoghq
- 39. “WRITING IS NATURE’S WAY OF LETTING YOU KNOW HOW SLOPPY YOUR THINKING IS.” RICHARD GUINDON @dbenamy @datadoghq
- 40. TELLING STORIES “A PICTURE IS WORTH A THOUSAND WORDS” - ANCIENT PROVERB @dbenamy @datadoghq
- 41. HUMAN DATA DATA COLLECTION: WHEN? ▸ As soon as possible ▸ Memory drops sharply within 20 minutes ▸ Susceptibility to “false memory” increases ▸ Get managers involved! @dbenamy @datadoghq
- 42. HUMAN DATA DATA SKEW / CORRUPTION ▸ Stress ▸ Sleep deprivation ▸ Burnout @dbenamy @datadoghq
- 43. HUMAN DATA DATA SKEW / CORRUPTION ▸ Blame ▸ Fear of punitive action @dbenamy @datadoghq
- 44. HUMAN DATA DATA SKEW / CORRUPTION ▸ Bias ▸ Anchoring ▸ Hindsight ▸ Outcome ▸ Availability (recency) ▸ Bandwagon effect @dbenamy @datadoghq
- 45. HOW WE DO POSTMORTEMS AT DATADOG @dbenamy @datadoghq
- 46. DATADOG POSTMORTEMS A FEW NOTES ▸ Findings emailed company wide ▸ Scheduled recurring review meetings @dbenamy @datadoghq
- 47. DATADOG POSTMORTEMS A FEW NOTES ▸ Findings emailed company wide ▸ Scheduled recurring review meetings @dbenamy @datadoghq
- 48. DATADOG’S POSTMORTEM TEMPLATE (1/5) SUMMARY: WHAT HAPPENED? ▸ Describe what happened here at a high level. Think of it as an abstract in a scientific paper. ▸ What was the impact on customers? ▸ What was the severity of the outage? ▸ What components were affected? ▸ What ultimately resolved the outage? @dbenamy @datadoghq
- 49. DATADOG’S POSTMORTEM TEMPLATE (1/5) SUMMARY: WHAT HAPPENED? ▸ Describe what happened here at a high level. Think of it as an abstract in a scientific paper. ▸ What was the impact on customers? ▸ What was the severity of the outage? ▸ What components were affected? ▸ What ultimately resolved the outage? @dbenamy @datadoghq
- 52. DATADOG’S POSTMORTEM TEMPLATE (2/5) HOW WAS THE OUTAGE DETECTED? ▸ We want to make sure that we detected the issue early and that we would catch the same issue if it were to repeat. ▸ Did we have a metric that showed the outage? ▸ Was there a monitor on that metric? ▸ How long did it take for us to declare an outage? @dbenamy @datadoghq
- 60. DATADOG’S POSTMORTEM TEMPLATE (3/5) HOW DID WE RESPOND? ▸ Who was the incident owner, and who else was involved? ▸ Slack archive links and timeline of events! ▸ What went well? ▸ What didn’t go so well? @dbenamy @datadoghq
- 61. DATADOG’S POSTMORTEM TEMPLATE (3/5) HOW DID WE RESPOND? ▸ Who was the incident owner, and who else was involved? ▸ Slack archive links and timeline of events! ▸ What went well? ▸ What didn’t go so well? @dbenamy @datadoghq
- 63. CHATOPS ARCHIVES FTW! *Names changed @dbenamy @datadoghq
- 64. *Names changed TRACK LEARNINGS AS YOU GO @dbenamy @datadoghq
- 65. DATADOG’S POSTMORTEM TEMPLATE (4/5) WHY DID IT HAPPEN? ▸ Deep dive into the cause ▸ Examples from this incident ▸ http://bit.ly/dd-statuspage ▸ http://bit.ly/alq-postmortem @dbenamy @datadoghq
- 67. DATADOG’S POSTMORTEM TEMPLATE (5/5) HOW DO WE PREVENT IT IN THE FUTURE? ▸ Link to GitHub issues and Trello cards ▸ Now? ▸ Next? ▸ Later? ▸ Follow-up notes @dbenamy @datadoghq
- 68. DATADOG’S POSTMORTEM TEMPLATE (5/5) HOW DO WE PREVENT IT IN THE FUTURE? ▸ Link to GitHub issues and Trello cards ▸ Now? ▸ Next? ▸ Later? ▸ Follow-up notes @dbenamy @datadoghq
- 70. DATADOG’S POSTMORTEM TEMPLATE RECAP: ▸ What happened (summary)? ▸ How did we detect it? ▸ How did we respond? ▸ Why did it happen (deep dive)? ▸ Actionable next steps! @dbenamy @datadoghq
- 71. KEEP LEARNING MORE RESOURCES ▸ Our template http://bit.ly/postmortem-template ▸ Post-Incident Reviews, by Jason Hand http://bit.ly/post-incident-review ▸ The Infinite Hows, by John Allspaw http://bit.ly/infinite-hows ▸ “Blameless” Postmortems Don’t Work, by J. Paul Reed http://bit.ly/blameless-dont-work @dbenamy @datadoghq