Morgue
A Post Mortem Tool
Etsy
@BethanyMacri
Monday, October 21, 13

Daniel Schauenberg
@mrtazz
github.com/mrtazz
Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Shit happens.
Monday, October 21, 13

What I’m talking about today
• Post mortems
• Morgue
Monday, October 21, 13

What is a
post mortem?
Monday, October 21, 13

Types of events
Monday, October 21, 13

Types of events
• Site outage
Monday, October 21, 13

Types of events
• Site outage
• New users cannot register
Monday, October 21, 13

Types of events
• Site outage
• New users cannot register
• Buyers cannot checkout
Monday, October 21, 13

Types of events
• Site outage
• New users cannot register
• Buyers cannot checkout
• Sellers cannot list items
Monday, October 21, 13

Types of events
• Site outage
• New users cannot register
• Buyers cannot checkout
• Sellers cannot list items
• Data loss
Monday, October 21, 13

Types of events
• Site outage
• New users cannot register
• Buyers cannot checkout
• Sellers cannot list items
• Data loss
• Performance degradation
Monday, October 21, 13

Severity levels
P1 Complete outage
P2 Loss or degradation for subset
P3 Loss of minor functionality
P4 No member-visible impact
P5 Anything else
Monday, October 21, 13

How is a post
mortem conducted?
Monday, October 21, 13

Who
• The person/people who:
Monday, October 21, 13

Who
• The person/people who:
• introduced the problem
Monday, October 21, 13

Who
• The person/people who:
• introduced the problem
• identified the problem
Monday, October 21, 13

Who
• The person/people who:
• introduced the problem
• identified the problem
• responded to the problem
Monday, October 21, 13

Who
• The person/people who:
• introduced the problem
• identified the problem
• responded to the problem
• debugged the problem
Monday, October 21, 13

Who
• The person/people who:
• introduced the problem
• identified the problem
• responded to the problem
• debugged the problem
• anyone interested
Monday, October 21, 13

What to discuss
Monday, October 21, 13

What to discuss
• what happened, and when
Monday, October 21, 13

What to discuss
• what happened, and when
• assumptions
Monday, October 21, 13

What to discuss
• what happened, and when
• assumptions
• solutions
Monday, October 21, 13

What to discuss
• what happened, and when
• assumptions
• solutions
• prevention
Monday, October 21, 13

Monday, October 21, 13

“It’s not your fault”
Monday, October 21, 13

Blameless
Monday, October 21, 13

What to record
Monday, October 21, 13

What to record
• Metrics
Monday, October 21, 13

What to record
• Metrics
• Possible investigations
Monday, October 21, 13

What to record
• Metrics
• Possible investigations
• Possible resolutions
Monday, October 21, 13

What to record
• Metrics
• Possible investigations
• Possible resolutions
• Assumptions
Monday, October 21, 13

What to record
• Metrics
• Possible investigations
• Possible resolutions
• Assumptions
• Remediation
Monday, October 21, 13

So what’s wrong
with a wiki?
Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

What is Morgue?
Monday, October 21, 13

Stack
• PHP
• Slim framework
• MySQL
• Apache
• Twitter bootstrap
Monday, October 21, 13

Current features
• General
Monday, October 21, 13
• Etsy

Current features
• General
•
Monday, October 21, 13
Time calculations
• Etsy

Current features
• General
•
Time calculations
•
Severity level
Monday, October 21, 13
• Etsy

Current features
• General
•
Time calculations
•
Severity level
•
Timezone support
Monday, October 21, 13
• Etsy

Current features
• General
•
Time calculations
•
Severity level
•
Timezone support
•
Markdown for
“What Happened”
Monday, October 21, 13
• Etsy

Current features
• General
•
Time calculations
•
Severity level
•
Timezone support
•
Markdown for
“What Happened”
•
Images
Monday, October 21, 13
• Etsy

Current features
• General
•
Time calculations
•
Severity level
•
Timezone support
•
Markdown for
“What Happened”
•
Images
•
Tags
Monday, October 21, 13
• Etsy

Current features
• General
•
Time calculations
•
Severity level
•
Timezone support
•
Markdown for
“What Happened”
•
Images
•
Tags
•
History
Monday, October 21, 13
• Etsy

Current features
• General
•
Time calculations
•
Severity level
•
Timezone support
•
Markdown for
“What Happened”
•
Images
•
Tags
•
History
Monday, October 21, 13
• Etsy

Current features
• General
•
Time calculations
•
Severity level
•
Timezone support
•
Markdown for
“What Happened”
•
Images
•
Tags
•
History
Monday, October 21, 13
• Etsy
•
Etsystatus time

Current features
• General
• Etsy
•
Time calculations
•
Etsystatus time
•
Severity level
•
Contact/GCal
•
Timezone support
•
Markdown for
“What Happened”
•
Images
•
Tags
•
History
Monday, October 21, 13

Current features
• General
• Etsy
•
Time calculations
•
Etsystatus time
•
Severity level
•
Contact/GCal
•
Timezone support
•
IRC
•
Markdown for
“What Happened”
•
Images
•
Tags
•
History
Monday, October 21, 13

Current features
• General
• Etsy
•
Time calculations
•
Etsystatus time
•
Severity level
•
Contact/GCal
•
Timezone support
•
IRC
•
Markdown for
“What Happened”
•
Jira
•
Images
•
Tags
•
History
Monday, October 21, 13

Current features
• General
• Etsy
•
Time calculations
•
Etsystatus time
•
Severity level
•
Contact/GCal
•
Timezone support
•
IRC
•
Markdown for
“What Happened”
•
Jira
•
Forums
•
Images
•
Tags
•
History
Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

10/19/2013
Monday, October 21, 13

10/19/2013
Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

Monday, October 21, 13

How did Morgue
help?
Monday, October 21, 13

Severity levels recorded
Wiki
100
75
50
25
0
Monday, October 21, 13
P1
P2
P3
P4
P5
No data

Severity levels recorded
Wiki
Morgue
100
75
50
25
0
Monday, October 21, 13
P1
P2
P3
P4
P5
No data

Wiki
100
75
50
25
0
Monday, October 21, 13
Severity
Times
IRC
Remediation
Images

Wiki
Morgue
100
75
50
25
0
Monday, October 21, 13
Severity
Times
IRC
Remediation
Images

What’s the future of
Morgue?
Monday, October 21, 13

Feature ideas
• Graphs
• by severity
• by tags
• Remediation enforcement
• Copy to warn “should”s and “could”s
Monday, October 21, 13

github.com/Etsy/
morgue
Monday, October 21, 13

Pull requests
welcome
Monday, October 21, 13

Questions?
github.com/Etsy/morgue
@BethanyMacri
bmacri@etsy.com
Monday, October 21, 13

Source Credits
http://media.heavy.com/media/2013/03/morgue.jpg
http://authorstephanihecht.files.wordpress.com/2012/11/mistake-2.jpg?w=640
http://4.bp.blogspot.com/-A9OibuJMGlM/TfSyX6cMjXI/AAAAAAAAADc/cM4JwCkt1zg/s1600/
Good.Will.Hunting.1997.XviD.DTS.CD3-WAF.avi_001529070.jpg
https://www.hackerschool.com/assets/slideshow_01-540531c0d5690f948a4d5d02c5400827.jpg
http://twa2r.files.wordpress.com/2011/01/neo-bullets.jpg
http://commons.wikimedia.org/wiki/File:University-of-Virginia-Rotunda.jpg
http://abovethelaw.com/wp-content/uploads/2012/03/law-school-gavels.jpg
http://nyogalleristny.files.wordpress.com/2012/05/8850-munch-the-scream-e1336013995741.jpg?
w=300&h=233
http://smallbizelevator.com/wp-content/uploads/et_temp/stay-handmade-etsy-63270_200x200.jpg
Monday, October 21, 13

Monday, October 21, 13