Morgue
A Post Mortem Tool
Etsy
@BethanyMacri

Monday, October 21, 13
Daniel Schauenberg
@mrtazz
github.com/mrtazz

Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Shit happens.

Monday, October 21, 13
What I’m talking about today
• Post mortems
• Morgue

Monday, October 21, 13
What is a
post mortem?

Monday, October 21, 13
Types of events

Monday, October 21, 13
Types of events
• Site outage

Monday, October 21, 13
Types of events
• Site outage
• New users cannot register

Monday, October 21, 13
Types of events
• Site outage
• New users cannot register
• Buyers cannot checkout

Monday, October 21, 13
Types of events
• Site outage
• New users cannot register
• Buyers cannot checkout
• Sellers cannot list items

Monday, Oc...
Types of events
• Site outage
• New users cannot register
• Buyers cannot checkout
• Sellers cannot list items
• Data loss...
Types of events
• Site outage
• New users cannot register
• Buyers cannot checkout
• Sellers cannot list items
• Data loss...
Severity levels
P1 Complete outage
P2 Loss or degradation for subset
P3 Loss of minor functionality
P4 No member-visible i...
How is a post
mortem conducted?

Monday, October 21, 13
Who
• The person/people who:

Monday, October 21, 13
Who
• The person/people who:
• introduced the problem

Monday, October 21, 13
Who
• The person/people who:
• introduced the problem
• identified the problem

Monday, October 21, 13
Who
• The person/people who:
• introduced the problem
• identified the problem
• responded to the problem

Monday, October ...
Who
• The person/people who:
• introduced the problem
• identified the problem
• responded to the problem
• debugged the pr...
Who
• The person/people who:
• introduced the problem
• identified the problem
• responded to the problem
• debugged the pr...
What to discuss

Monday, October 21, 13
What to discuss
• what happened, and when

Monday, October 21, 13
What to discuss
• what happened, and when
• assumptions

Monday, October 21, 13
What to discuss
• what happened, and when
• assumptions
• solutions

Monday, October 21, 13
What to discuss
• what happened, and when
• assumptions
• solutions
• prevention

Monday, October 21, 13
Monday, October 21, 13
“It’s not your fault”

Monday, October 21, 13
Blameless

Monday, October 21, 13
What to record

Monday, October 21, 13
What to record
• Metrics

Monday, October 21, 13
What to record
• Metrics
• Possible investigations

Monday, October 21, 13
What to record
• Metrics
• Possible investigations
• Possible resolutions

Monday, October 21, 13
What to record
• Metrics
• Possible investigations
• Possible resolutions
• Assumptions

Monday, October 21, 13
What to record
• Metrics
• Possible investigations
• Possible resolutions
• Assumptions
• Remediation
Monday, October 21, ...
So what’s wrong
with a wiki?

Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
What is Morgue?

Monday, October 21, 13
Stack
• PHP
• Slim framework

• MySQL
• Apache
• Twitter bootstrap

Monday, October 21, 13
Current features
• General

Monday, October 21, 13

• Etsy
Current features
• General
•

Monday, October 21, 13

Time calculations

• Etsy
Current features
• General
•

Time calculations

•

Severity level

Monday, October 21, 13

• Etsy
Current features
• General
•

Time calculations

•

Severity level

•

Timezone support

Monday, October 21, 13

• Etsy
Current features
• General
•

Time calculations

•

Severity level

•

Timezone support

•

Markdown for
“What Happened”

...
Current features
• General
•

Time calculations

•

Severity level

•

Timezone support

•

Markdown for
“What Happened”

...
Current features
• General
•

Time calculations

•

Severity level

•

Timezone support

•

Markdown for
“What Happened”

...
Current features
• General
•

Time calculations

•

Severity level

•

Timezone support

•

Markdown for
“What Happened”

...
Current features
• General
•

Time calculations

•

Severity level

•

Timezone support

•

Markdown for
“What Happened”

...
Current features
• General
•

Time calculations

•

Severity level

•

Timezone support

•

Markdown for
“What Happened”

...
Current features
• General

• Etsy

•

Time calculations

•

Etsystatus time

•

Severity level

•

Contact/GCal

•

Timez...
Current features
• General

• Etsy

•

Time calculations

•

Etsystatus time

•

Severity level

•

Contact/GCal

•

Timez...
Current features
• General

• Etsy

•

Time calculations

•

Etsystatus time

•

Severity level

•

Contact/GCal

•

Timez...
Current features
• General

• Etsy

•

Time calculations

•

Etsystatus time

•

Severity level

•

Contact/GCal

•

Timez...
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
10/19/2013

Monday, October 21, 13
10/19/2013

Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
Monday, October 21, 13
How did Morgue
help?

Monday, October 21, 13
Severity levels recorded
Wiki
100
75
50
25
0
Monday, October 21, 13

P1

P2

P3

P4

P5

No data
Severity levels recorded
Wiki

Morgue

100
75
50
25
0
Monday, October 21, 13

P1

P2

P3

P4

P5

No data
Wiki
100
75
50
25
0

Monday, October 21, 13

Severity

Times

IRC

Remediation

Images
Wiki

Morgue

100
75
50
25
0

Monday, October 21, 13

Severity

Times

IRC

Remediation

Images
What’s the future of
Morgue?

Monday, October 21, 13
Feature ideas
• Graphs
• by severity
• by tags
• Remediation enforcement
• Copy to warn “should”s and “could”s

Monday, Oc...
github.com/Etsy/
morgue

Monday, October 21, 13
Pull requests
welcome

Monday, October 21, 13
Questions?
github.com/Etsy/morgue
@BethanyMacri
bmacri@etsy.com

Monday, October 21, 13
Source Credits
http://media.heavy.com/media/2013/03/morgue.jpg
http://authorstephanihecht.files.wordpress.com/2012/11/mista...
Monday, October 21, 13
Upcoming SlideShare
Loading in...5
×

Morgue , helping better understand events by building a post mortem tool - Bethany Macri

2,256

Published on

Published in: Education, Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,256
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
12
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Morgue , helping better understand events by building a post mortem tool - Bethany Macri

  1. 1. Morgue A Post Mortem Tool Etsy @BethanyMacri Monday, October 21, 13
  2. 2. Daniel Schauenberg @mrtazz github.com/mrtazz Monday, October 21, 13
  3. 3. Monday, October 21, 13
  4. 4. Monday, October 21, 13
  5. 5. Monday, October 21, 13
  6. 6. Monday, October 21, 13
  7. 7. Monday, October 21, 13
  8. 8. Monday, October 21, 13
  9. 9. Monday, October 21, 13
  10. 10. Monday, October 21, 13
  11. 11. Shit happens. Monday, October 21, 13
  12. 12. What I’m talking about today • Post mortems • Morgue Monday, October 21, 13
  13. 13. What is a post mortem? Monday, October 21, 13
  14. 14. Types of events Monday, October 21, 13
  15. 15. Types of events • Site outage Monday, October 21, 13
  16. 16. Types of events • Site outage • New users cannot register Monday, October 21, 13
  17. 17. Types of events • Site outage • New users cannot register • Buyers cannot checkout Monday, October 21, 13
  18. 18. Types of events • Site outage • New users cannot register • Buyers cannot checkout • Sellers cannot list items Monday, October 21, 13
  19. 19. Types of events • Site outage • New users cannot register • Buyers cannot checkout • Sellers cannot list items • Data loss Monday, October 21, 13
  20. 20. Types of events • Site outage • New users cannot register • Buyers cannot checkout • Sellers cannot list items • Data loss • Performance degradation Monday, October 21, 13
  21. 21. Severity levels P1 Complete outage P2 Loss or degradation for subset P3 Loss of minor functionality P4 No member-visible impact P5 Anything else Monday, October 21, 13
  22. 22. How is a post mortem conducted? Monday, October 21, 13
  23. 23. Who • The person/people who: Monday, October 21, 13
  24. 24. Who • The person/people who: • introduced the problem Monday, October 21, 13
  25. 25. Who • The person/people who: • introduced the problem • identified the problem Monday, October 21, 13
  26. 26. Who • The person/people who: • introduced the problem • identified the problem • responded to the problem Monday, October 21, 13
  27. 27. Who • The person/people who: • introduced the problem • identified the problem • responded to the problem • debugged the problem Monday, October 21, 13
  28. 28. Who • The person/people who: • introduced the problem • identified the problem • responded to the problem • debugged the problem • anyone interested Monday, October 21, 13
  29. 29. What to discuss Monday, October 21, 13
  30. 30. What to discuss • what happened, and when Monday, October 21, 13
  31. 31. What to discuss • what happened, and when • assumptions Monday, October 21, 13
  32. 32. What to discuss • what happened, and when • assumptions • solutions Monday, October 21, 13
  33. 33. What to discuss • what happened, and when • assumptions • solutions • prevention Monday, October 21, 13
  34. 34. Monday, October 21, 13
  35. 35. “It’s not your fault” Monday, October 21, 13
  36. 36. Blameless Monday, October 21, 13
  37. 37. What to record Monday, October 21, 13
  38. 38. What to record • Metrics Monday, October 21, 13
  39. 39. What to record • Metrics • Possible investigations Monday, October 21, 13
  40. 40. What to record • Metrics • Possible investigations • Possible resolutions Monday, October 21, 13
  41. 41. What to record • Metrics • Possible investigations • Possible resolutions • Assumptions Monday, October 21, 13
  42. 42. What to record • Metrics • Possible investigations • Possible resolutions • Assumptions • Remediation Monday, October 21, 13
  43. 43. So what’s wrong with a wiki? Monday, October 21, 13
  44. 44. Monday, October 21, 13
  45. 45. Monday, October 21, 13
  46. 46. Monday, October 21, 13
  47. 47. Monday, October 21, 13
  48. 48. Monday, October 21, 13
  49. 49. Monday, October 21, 13
  50. 50. Monday, October 21, 13
  51. 51. What is Morgue? Monday, October 21, 13
  52. 52. Stack • PHP • Slim framework • MySQL • Apache • Twitter bootstrap Monday, October 21, 13
  53. 53. Current features • General Monday, October 21, 13 • Etsy
  54. 54. Current features • General • Monday, October 21, 13 Time calculations • Etsy
  55. 55. Current features • General • Time calculations • Severity level Monday, October 21, 13 • Etsy
  56. 56. Current features • General • Time calculations • Severity level • Timezone support Monday, October 21, 13 • Etsy
  57. 57. Current features • General • Time calculations • Severity level • Timezone support • Markdown for “What Happened” Monday, October 21, 13 • Etsy
  58. 58. Current features • General • Time calculations • Severity level • Timezone support • Markdown for “What Happened” • Images Monday, October 21, 13 • Etsy
  59. 59. Current features • General • Time calculations • Severity level • Timezone support • Markdown for “What Happened” • Images • Tags Monday, October 21, 13 • Etsy
  60. 60. Current features • General • Time calculations • Severity level • Timezone support • Markdown for “What Happened” • Images • Tags • History Monday, October 21, 13 • Etsy
  61. 61. Current features • General • Time calculations • Severity level • Timezone support • Markdown for “What Happened” • Images • Tags • History Monday, October 21, 13 • Etsy
  62. 62. Current features • General • Time calculations • Severity level • Timezone support • Markdown for “What Happened” • Images • Tags • History Monday, October 21, 13 • Etsy • Etsystatus time
  63. 63. Current features • General • Etsy • Time calculations • Etsystatus time • Severity level • Contact/GCal • Timezone support • Markdown for “What Happened” • Images • Tags • History Monday, October 21, 13
  64. 64. Current features • General • Etsy • Time calculations • Etsystatus time • Severity level • Contact/GCal • Timezone support • IRC • Markdown for “What Happened” • Images • Tags • History Monday, October 21, 13
  65. 65. Current features • General • Etsy • Time calculations • Etsystatus time • Severity level • Contact/GCal • Timezone support • IRC • Markdown for “What Happened” • Jira • Images • Tags • History Monday, October 21, 13
  66. 66. Current features • General • Etsy • Time calculations • Etsystatus time • Severity level • Contact/GCal • Timezone support • IRC • Markdown for “What Happened” • Jira • Forums • Images • Tags • History Monday, October 21, 13
  67. 67. Monday, October 21, 13
  68. 68. Monday, October 21, 13
  69. 69. Monday, October 21, 13
  70. 70. Monday, October 21, 13
  71. 71. Monday, October 21, 13
  72. 72. Monday, October 21, 13
  73. 73. Monday, October 21, 13
  74. 74. Monday, October 21, 13
  75. 75. Monday, October 21, 13
  76. 76. Monday, October 21, 13
  77. 77. Monday, October 21, 13
  78. 78. Monday, October 21, 13
  79. 79. Monday, October 21, 13
  80. 80. Monday, October 21, 13
  81. 81. Monday, October 21, 13
  82. 82. Monday, October 21, 13
  83. 83. Monday, October 21, 13
  84. 84. Monday, October 21, 13
  85. 85. Monday, October 21, 13
  86. 86. Monday, October 21, 13
  87. 87. Monday, October 21, 13
  88. 88. 10/19/2013 Monday, October 21, 13
  89. 89. 10/19/2013 Monday, October 21, 13
  90. 90. Monday, October 21, 13
  91. 91. Monday, October 21, 13
  92. 92. Monday, October 21, 13
  93. 93. Monday, October 21, 13
  94. 94. Monday, October 21, 13
  95. 95. Monday, October 21, 13
  96. 96. Monday, October 21, 13
  97. 97. How did Morgue help? Monday, October 21, 13
  98. 98. Severity levels recorded Wiki 100 75 50 25 0 Monday, October 21, 13 P1 P2 P3 P4 P5 No data
  99. 99. Severity levels recorded Wiki Morgue 100 75 50 25 0 Monday, October 21, 13 P1 P2 P3 P4 P5 No data
  100. 100. Wiki 100 75 50 25 0 Monday, October 21, 13 Severity Times IRC Remediation Images
  101. 101. Wiki Morgue 100 75 50 25 0 Monday, October 21, 13 Severity Times IRC Remediation Images
  102. 102. What’s the future of Morgue? Monday, October 21, 13
  103. 103. Feature ideas • Graphs • by severity • by tags • Remediation enforcement • Copy to warn “should”s and “could”s Monday, October 21, 13
  104. 104. github.com/Etsy/ morgue Monday, October 21, 13
  105. 105. Pull requests welcome Monday, October 21, 13
  106. 106. Questions? github.com/Etsy/morgue @BethanyMacri bmacri@etsy.com Monday, October 21, 13
  107. 107. Source Credits http://media.heavy.com/media/2013/03/morgue.jpg http://authorstephanihecht.files.wordpress.com/2012/11/mistake-2.jpg?w=640 http://4.bp.blogspot.com/-A9OibuJMGlM/TfSyX6cMjXI/AAAAAAAAADc/cM4JwCkt1zg/s1600/ Good.Will.Hunting.1997.XviD.DTS.CD3-WAF.avi_001529070.jpg https://www.hackerschool.com/assets/slideshow_01-540531c0d5690f948a4d5d02c5400827.jpg http://twa2r.files.wordpress.com/2011/01/neo-bullets.jpg http://commons.wikimedia.org/wiki/File:University-of-Virginia-Rotunda.jpg http://abovethelaw.com/wp-content/uploads/2012/03/law-school-gavels.jpg http://nyogalleristny.files.wordpress.com/2012/05/8850-munch-the-scream-e1336013995741.jpg? w=300&h=233 http://smallbizelevator.com/wp-content/uploads/et_temp/stay-handmade-etsy-63270_200x200.jpg Monday, October 21, 13
  108. 108. Monday, October 21, 13
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×