Nov 2018 - QCon San Francisco - Service Ownership at Slack
•
1 like•74 views
Service Ownership at Slack - talk given at QCon SF November 2018.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Nov 2018 - QCon San Francisco - Service Ownership at Slack
Similar to Nov 2018 - QCon San Francisco - Service Ownership at Slack (20)
Recently uploaded
Recently uploaded (20)
Nov 2018 - QCon San Francisco - Service Ownership at Slack
- 1. Service Ownership Learn Faster Holly Allen Service Engineering @hollyjallen
- 2. Holly Allen Software development and leadership for 18 years
- 8. S L O W 😩 @hollyjallen,#QConSF Nov 2018
- 22. “” Executive dedication to learning @hollyjallen,#QConSF Nov 2018
- 23. “” High Trust Teams @hollyjallen,#QConSF Nov 2018
- 26. Slack launched February 2014 @hollyjallen,#QConSF Nov 2018 🚀
- 27. @hollyjallen,#QConSF Nov 2018 5 Years Grew to 13+ million weekly active users, with active sessions of 10+ hours a day
- 28. @hollyjallen,#QConSF Nov 2018 5 Years From 10 to 15,000 servers In 25 cloud data centers world-wide
- 29. @hollyjallen,#QConSF Nov 2018 5 Years From 8 to 1,200 people In 9 offices world-wide
- 32. “” ✅ Continuous Deployment ✅ Experiment Frameworks ✅ User Research @hollyjallen,#QConSF Nov 2018
- 33. Something didn't scale... @hollyjallen,#QConSF Nov 2018
- 35. “” Who should be responsible for the management, monitoring and operation of a production application? @hollyjallen,#QConSF Nov 2018
- 36. “” Centralized Operations Division of Labor @hollyjallen,#QConSF Nov 2018
- 37. Devs Features Scale Architecture @hollyjallen,#QConSF Nov 2018 Ops Cloud Infra Deployment Monitoring
- 38. “” Ops is getting the pages @hollyjallen,#QConSF Nov 2018
- 39. “” Product Development grew faster than Operations, A lot faster @hollyjallen,#QConSF Nov 2018
- 40. 20 Product Developers @hollyjallen,#QConSF Nov 2018 1 Ops Engineer
- 41. “” How can operations reliably reach the developers when there's a problem? @hollyjallen,#QConSF Nov 2018
- 42. “” "Call Maude, she knows how this works" @hollyjallen,#QConSF Nov 2018
- 43. Devs I've never been on-call before, this is scary! @hollyjallen,#QConSF Nov 2018 Ops Now I know I can find a developer when I need to.
- 44. “” Ops is getting the pages first pages Ultra-senior devs on-call @hollyjallen,#QConSF Nov 2018
- 46. “” How can operations reliably reach the developers when there's a problem? @hollyjallen,#QConSF Nov 2018
- 47. Most devs go on-call Fall 2017 @hollyjallen,#QConSF Nov 2018 📟
- 49. “” "Wait, I'm on-call now?" @hollyjallen,#QConSF Nov 2018
- 50. Devs I'm glad I'm only on call a few times a year @hollyjallen,#QConSF Nov 2018 Ops I'll be able to reach a search engineer if I need to.
- 51. “” Learn by Doing @hollyjallen,#QConSF Nov 2018
- 52. “” On-call 3 times a year 🤔 @hollyjallen,#QConSF Nov 2018
- 53. “” Ops is getting the pages first pages Ultra-senior devs on-call Seven One dev rotations @hollyjallen,#QConSF Nov 2018
- 54. “” Continuous Deployment 100+ prod deploys a day @hollyjallen,#QConSF Nov 2018
- 58. “” Page the dev @hollyjallen,#QConSF Nov 2018
- 59. Devs I don't understand this part of the code @hollyjallen,#QConSF Nov 2018 Ops These are the machine alerts I'm seeing
- 61. “” "Call Andy, he knows how this works" @hollyjallen,#QConSF Nov 2018
- 62. “” Postmortems weren't a great place for learning @hollyjallen,#QConSF Nov 2018
- 63. “” Can we catch problems earlier? @hollyjallen,#QConSF Nov 2018
- 67. “” Investing in tech to make detection and remediation faster @hollyjallen,#QConSF Nov 2018
- 68. Reorg! Fall 2017 @hollyjallen,#QConSF Nov 2018 Operations is out Service Engineering is in
- 69. “” How can Slack ensure that developers know when there's a problem? @hollyjallen,#QConSF Nov 2018
- 72. “” "We are the toolsmith and specialists. We empower Service Ownership" @hollyjallen,#QConSF Nov 2018
- 73. Devs Features Reliability Performance Postmortems @hollyjallen,#QConSF Nov 2018 Service Cloud Platform Observability tools Service Discovery Define best practice
- 74. I joined Slack in February 2018 @hollyjallen,#QConSF Nov 2018 👋
- 75. “” How to empower development teams to improve service reliability? @hollyjallen,#QConSF Nov 2018
- 76. Define service health and operational maturity @hollyjallen,#QConSF Nov 2018 • At least one alerting health metric, like latency or throughput
- 77. “” Send metrics to Prometheus Observability team is here to help! 🔮 @hollyjallen,#QConSF Nov 2018
- 78. Define service health and operational maturity @hollyjallen,#QConSF Nov 2018 • Team should be on-call ready • At least 4, preferably 6 engineers participating to make it sustainable • 24/7 or during the weekday, depending on the service
- 79. Define service health and operational maturity @hollyjallen,#QConSF Nov 2018 • Runbooks for standard actions and troubleshooting • Central location in our code repository • Up to date and useable by any engineer
- 80. Define service health and operational maturity @hollyjallen,#QConSF Nov 2018 • Paging alerts should link to the runbook • Make responding to an page easy • Practice incident response
- 81. “” Incident Lunch ⛑ @hollyjallen,#QConSF Nov 2018
- 82. Site Reliability Engineers @hollyjallen,#QConSF Nov 2018 • Devops generalists • Emotional intelligence • Mentoring • Ambassadors • Operational maturity
- 83. “” SRE embedded in dev teams @hollyjallen,#QConSF Nov 2018
- 84. “” @hollyjallen,#QConSF Nov 2018 Devs OpsSRE
- 85. Devs Um, where are the SREs? @hollyjallen,#QConSF Nov 2018 SREs I'm over here doing operational tasks
- 86. “” SRE Ops is still getting the first pages @hollyjallen,#QConSF Nov 2018
- 87. “” How do we lower operational burden on the SREs? @hollyjallen,#QConSF Nov 2018
- 88. “” Plan: Send paging alerts to the development teams @hollyjallen,#QConSF Nov 2018
- 89. Devs We need training @hollyjallen,#QConSF Nov 2018 SREs We're going to plan this out perfectly
- 91. “” Host level alerts Hundreds of them @hollyjallen,#QConSF Nov 2018
- 92. “” Test with the users @hollyjallen,#QConSF Nov 2018
- 96. “” @hollyjallen,#QConSF Nov 2018 Devs OpsSRE
- 97. “” How do we test our understanding of how Slack will fail? @hollyjallen,#QConSF Nov 2018
- 98. “” "Disasterpiece Theater is an ongoing series of exercises in which we will purposely cause a part of Slack to fail." @hollyjallen,#QConSF Nov 2018
- 100. Success Metrics @hollyjallen,#QConSF Nov 2018 • Increased engineer confidence • Validate reliability improvements • Learn something new • Practice incident response
- 102. “” How do we ensure the teams are being alerted, instead of skillsets? @hollyjallen,#QConSF Nov 2018
- 103. “” How do we make postmortems a place for learning again? @hollyjallen,#QConSF Nov 2018
- 104. “” How do we make sure a capable incident commander is available for all incidents? @hollyjallen,#QConSF Nov 2018
- 108. “” Copy the questions Not the answers @hollyjallen,#QConSF Nov 2018
- 109. “” Change is possible You don't have to be ready @hollyjallen,#QConSF Nov 2018
- 112. “” Learn Faster @hollyjallen,#QConSF Nov 2018