Morgue , helping better understand events by building a post mortem tool - Bethany Macri

1. Morgue A Post Mortem Tool Etsy @BethanyMacri Monday, October 21, 13

2. Daniel Schauenberg @mrtazz github.com/mrtazz Monday, October 21, 13

3. Monday, October 21, 13

11. Shit happens. Monday, October 21, 13

12. What I’m talking about today • Post mortems • Morgue Monday, October 21, 13

13. What is a post mortem? Monday, October 21, 13

14. Types of events Monday, October 21, 13

15. Types of events • Site outage Monday, October 21, 13

16. Types of events • Site outage • New users cannot register Monday, October 21, 13

17. Types of events • Site outage • New users cannot register • Buyers cannot checkout Monday, October 21, 13

18. Types of events • Site outage • New users cannot register • Buyers cannot checkout • Sellers cannot list items Monday, October 21, 13

19. Types of events • Site outage • New users cannot register • Buyers cannot checkout • Sellers cannot list items • Data loss Monday, October 21, 13

20. Types of events • Site outage • New users cannot register • Buyers cannot checkout • Sellers cannot list items • Data loss • Performance degradation Monday, October 21, 13

21. Severity levels P1 Complete outage P2 Loss or degradation for subset P3 Loss of minor functionality P4 No member-visible impact P5 Anything else Monday, October 21, 13

22. How is a post mortem conducted? Monday, October 21, 13

23. Who • The person/people who: Monday, October 21, 13

24. Who • The person/people who: • introduced the problem Monday, October 21, 13

25. Who • The person/people who: • introduced the problem • identiﬁed the problem Monday, October 21, 13

26. Who • The person/people who: • introduced the problem • identiﬁed the problem • responded to the problem Monday, October 21, 13

27. Who • The person/people who: • introduced the problem • identiﬁed the problem • responded to the problem • debugged the problem Monday, October 21, 13

28. Who • The person/people who: • introduced the problem • identiﬁed the problem • responded to the problem • debugged the problem • anyone interested Monday, October 21, 13

29. What to discuss Monday, October 21, 13

30. What to discuss • what happened, and when Monday, October 21, 13

31. What to discuss • what happened, and when • assumptions Monday, October 21, 13

32. What to discuss • what happened, and when • assumptions • solutions Monday, October 21, 13

33. What to discuss • what happened, and when • assumptions • solutions • prevention Monday, October 21, 13

35. “It’s not your fault” Monday, October 21, 13

36. Blameless Monday, October 21, 13

37. What to record Monday, October 21, 13

38. What to record • Metrics Monday, October 21, 13

39. What to record • Metrics • Possible investigations Monday, October 21, 13

40. What to record • Metrics • Possible investigations • Possible resolutions Monday, October 21, 13

41. What to record • Metrics • Possible investigations • Possible resolutions • Assumptions Monday, October 21, 13

42. What to record • Metrics • Possible investigations • Possible resolutions • Assumptions • Remediation Monday, October 21, 13

43. So what’s wrong with a wiki? Monday, October 21, 13

51. What is Morgue? Monday, October 21, 13

52. Stack • PHP • Slim framework • MySQL • Apache • Twitter bootstrap Monday, October 21, 13

53. Current features • General Monday, October 21, 13 • Etsy

54. Current features • General • Monday, October 21, 13 Time calculations • Etsy

55. Current features • General • Time calculations • Severity level Monday, October 21, 13 • Etsy

56. Current features • General • Time calculations • Severity level • Timezone support Monday, October 21, 13 • Etsy

57. Current features • General • Time calculations • Severity level • Timezone support • Markdown for “What Happened” Monday, October 21, 13 • Etsy

58. Current features • General • Time calculations • Severity level • Timezone support • Markdown for “What Happened” • Images Monday, October 21, 13 • Etsy

59. Current features • General • Time calculations • Severity level • Timezone support • Markdown for “What Happened” • Images • Tags Monday, October 21, 13 • Etsy

60. Current features • General • Time calculations • Severity level • Timezone support • Markdown for “What Happened” • Images • Tags • History Monday, October 21, 13 • Etsy

61. Current features • General • Time calculations • Severity level • Timezone support • Markdown for “What Happened” • Images • Tags • History Monday, October 21, 13 • Etsy

62. Current features • General • Time calculations • Severity level • Timezone support • Markdown for “What Happened” • Images • Tags • History Monday, October 21, 13 • Etsy • Etsystatus time

63. Current features • General • Etsy • Time calculations • Etsystatus time • Severity level • Contact/GCal • Timezone support • Markdown for “What Happened” • Images • Tags • History Monday, October 21, 13

64. Current features • General • Etsy • Time calculations • Etsystatus time • Severity level • Contact/GCal • Timezone support • IRC • Markdown for “What Happened” • Images • Tags • History Monday, October 21, 13

65. Current features • General • Etsy • Time calculations • Etsystatus time • Severity level • Contact/GCal • Timezone support • IRC • Markdown for “What Happened” • Jira • Images • Tags • History Monday, October 21, 13

66. Current features • General • Etsy • Time calculations • Etsystatus time • Severity level • Contact/GCal • Timezone support • IRC • Markdown for “What Happened” • Jira • Forums • Images • Tags • History Monday, October 21, 13

88. 10/19/2013 Monday, October 21, 13

89. 10/19/2013 Monday, October 21, 13

97. How did Morgue help? Monday, October 21, 13

98. Severity levels recorded Wiki 100 75 50 25 0 Monday, October 21, 13 P1 P2 P3 P4 P5 No data

99. Severity levels recorded Wiki Morgue 100 75 50 25 0 Monday, October 21, 13 P1 P2 P3 P4 P5 No data

100. Wiki 100 75 50 25 0 Monday, October 21, 13 Severity Times IRC Remediation Images

101. Wiki Morgue 100 75 50 25 0 Monday, October 21, 13 Severity Times IRC Remediation Images

102. What’s the future of Morgue? Monday, October 21, 13

103. Feature ideas • Graphs • by severity • by tags • Remediation enforcement • Copy to warn “should”s and “could”s Monday, October 21, 13

104. github.com/Etsy/ morgue Monday, October 21, 13

105. Pull requests welcome Monday, October 21, 13

106. Questions? github.com/Etsy/morgue @BethanyMacri bmacri@etsy.com Monday, October 21, 13

107. Source Credits http://media.heavy.com/media/2013/03/morgue.jpg http://authorstephanihecht.files.wordpress.com/2012/11/mistake-2.jpg?w=640 http://4.bp.blogspot.com/-A9OibuJMGlM/TfSyX6cMjXI/AAAAAAAAADc/cM4JwCkt1zg/s1600/ Good.Will.Hunting.1997.XviD.DTS.CD3-WAF.avi_001529070.jpg https://www.hackerschool.com/assets/slideshow_01-540531c0d5690f948a4d5d02c5400827.jpg http://twa2r.files.wordpress.com/2011/01/neo-bullets.jpg http://commons.wikimedia.org/wiki/File:University-of-Virginia-Rotunda.jpg http://abovethelaw.com/wp-content/uploads/2012/03/law-school-gavels.jpg http://nyogalleristny.files.wordpress.com/2012/05/8850-munch-the-scream-e1336013995741.jpg? w=300&h=233 http://smallbizelevator.com/wp-content/uploads/et_temp/stay-handmade-etsy-63270_200x200.jpg Monday, October 21, 13

Morgue , helping better understand events by building a post mortem tool - Bethany Macri

Devopsdays

Morgue , helping better understand events by building a post mortem tool - Bethany Macri