Service Ownership @Slack
•
3 likes•473 views
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2Ffp4js. Holly Allen talks about the bumps and scrapes, triumphs and pitfalls of Slack’s journey from a centralized ops team to development teams that own the full lifecycle of their systems. Filmed at qconsf.com. Holly Allen is a leader in Service Engineering at Slack, with SRE, Safety Engineering, and Storage in her portfolio. She is tireless in her efforts to make Slack the software reliable and scalable, and Slack the company a delightful place to work. Prior to Slack, she worked at startups, DreamWorks Animation, and was Director of Engineering at 18F, a civic tech startup in the US government.
Recommended
Recommended
More Related Content
What's hot
What's hot (14)
Similar to Service Ownership @Slack
Similar to Service Ownership @Slack (20)
More from C4Media
More from C4Media (20)
Recently uploaded
Recently uploaded (20)
Service Ownership @Slack
- 1. Service Ownership Learn Faster Holly Allen Service Engineering @hollyjallen
- 2. InfoQ.com: News & Community Site • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ slack-devops
- 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
- 4. Holly Allen Software development and leadership for 18 years
- 10. S L O W 😩 @hollyjallen,#QConSF Nov 2018
- 24. “” Executive dedication to learning @hollyjallen,#QConSF Nov 2018
- 25. “” High Trust Teams @hollyjallen,#QConSF Nov 2018
- 28. Slack launched February 2014 @hollyjallen,#QConSF Nov 2018 🚀
- 29. @hollyjallen,#QConSF Nov 2018 5 Years Grew to 13+ million weekly active users, with active sessions of 10+ hours a day
- 30. @hollyjallen,#QConSF Nov 2018 5 Years From 10 to 15,000 servers In 25 cloud data centers world-wide
- 31. @hollyjallen,#QConSF Nov 2018 5 Years From 8 to 1,200 people In 9 offices world-wide
- 34. “” ✅ Continuous Deployment ✅ Experiment Frameworks ✅ User Research @hollyjallen,#QConSF Nov 2018
- 35. Something didn't scale... @hollyjallen,#QConSF Nov 2018
- 37. “” Who should be responsible for the management, monitoring and operation of a production application? @hollyjallen,#QConSF Nov 2018
- 38. “” Centralized Operations Division of Labor @hollyjallen,#QConSF Nov 2018
- 39. Devs Features Scale Architecture @hollyjallen,#QConSF Nov 2018 Ops Cloud Infra Deployment Monitoring
- 40. “” Ops is getting the pages @hollyjallen,#QConSF Nov 2018
- 41. “” Product Development grew faster than Operations, A lot faster @hollyjallen,#QConSF Nov 2018
- 42. 20 Product Developers @hollyjallen,#QConSF Nov 2018 1 Ops Engineer
- 43. “” How can operations reliably reach the developers when there's a problem? @hollyjallen,#QConSF Nov 2018
- 44. “” "Call Maude, she knows how this works" @hollyjallen,#QConSF Nov 2018
- 45. Devs I've never been on-call before, this is scary! @hollyjallen,#QConSF Nov 2018 Ops Now I know I can find a developer when I need to.
- 46. “” Ops is getting the pages first pages Ultra-senior devs on-call @hollyjallen,#QConSF Nov 2018
- 48. “” How can operations reliably reach the developers when there's a problem? @hollyjallen,#QConSF Nov 2018
- 49. Most devs go on-call Fall 2017 @hollyjallen,#QConSF Nov 2018 📟
- 51. “” "Wait, I'm on-call now?" @hollyjallen,#QConSF Nov 2018
- 52. Devs I'm glad I'm only on call a few times a year @hollyjallen,#QConSF Nov 2018 Ops I'll be able to reach a search engineer if I need to.
- 53. “” Learn by Doing @hollyjallen,#QConSF Nov 2018
- 54. “” On-call 3 times a year 🤔 @hollyjallen,#QConSF Nov 2018
- 55. “” Ops is getting the pages first pages Ultra-senior devs on-call Seven One dev rotations @hollyjallen,#QConSF Nov 2018
- 56. “” Continuous Deployment 100+ prod deploys a day @hollyjallen,#QConSF Nov 2018
- 60. “” Page the dev @hollyjallen,#QConSF Nov 2018
- 61. Devs I don't understand this part of the code @hollyjallen,#QConSF Nov 2018 Ops These are the machine alerts I'm seeing
- 63. “” "Call Andy, he knows how this works" @hollyjallen,#QConSF Nov 2018
- 64. “” Postmortems weren't a great place for learning @hollyjallen,#QConSF Nov 2018
- 65. “” Can we catch problems earlier? @hollyjallen,#QConSF Nov 2018
- 69. “” Investing in tech to make detection and remediation faster @hollyjallen,#QConSF Nov 2018
- 70. Reorg! Fall 2017 @hollyjallen,#QConSF Nov 2018 Operations is out Service Engineering is in
- 71. “” How can Slack ensure that developers know when there's a problem? @hollyjallen,#QConSF Nov 2018
- 74. “” "We are the toolsmith and specialists. We empower Service Ownership" @hollyjallen,#QConSF Nov 2018
- 75. Devs Features Reliability Performance Postmortems @hollyjallen,#QConSF Nov 2018 Service Cloud Platform Observability tools Service Discovery Define best practice
- 76. I joined Slack in February 2018 @hollyjallen,#QConSF Nov 2018 👋
- 77. “” How to empower development teams to improve service reliability? @hollyjallen,#QConSF Nov 2018
- 78. Define service health and operational maturity @hollyjallen,#QConSF Nov 2018 • At least one alerting health metric, like latency or throughput
- 79. “” Send metrics to Prometheus Observability team is here to help! 🔮 @hollyjallen,#QConSF Nov 2018
- 80. Define service health and operational maturity @hollyjallen,#QConSF Nov 2018 • Team should be on-call ready • At least 4, preferably 6 engineers participating to make it sustainable • 24/7 or during the weekday, depending on the service
- 81. Define service health and operational maturity @hollyjallen,#QConSF Nov 2018 • Runbooks for standard actions and troubleshooting • Central location in our code repository • Up to date and useable by any engineer
- 82. Define service health and operational maturity @hollyjallen,#QConSF Nov 2018 • Paging alerts should link to the runbook • Make responding to an page easy • Practice incident response
- 83. “” Incident Lunch ⛑ @hollyjallen,#QConSF Nov 2018
- 84. Site Reliability Engineers @hollyjallen,#QConSF Nov 2018 • Devops generalists • Emotional intelligence • Mentoring • Ambassadors • Operational maturity
- 85. “” SRE embedded in dev teams @hollyjallen,#QConSF Nov 2018
- 86. “” @hollyjallen,#QConSF Nov 2018 Devs OpsSRE
- 87. Devs Um, where are the SREs? @hollyjallen,#QConSF Nov 2018 SREs I'm over here doing operational tasks
- 88. “” SRE Ops is still getting the first pages @hollyjallen,#QConSF Nov 2018
- 89. “” How do we lower operational burden on the SREs? @hollyjallen,#QConSF Nov 2018
- 90. “” Plan: Send paging alerts to the development teams @hollyjallen,#QConSF Nov 2018
- 91. Devs We need training @hollyjallen,#QConSF Nov 2018 SREs We're going to plan this out perfectly
- 93. “” Host level alerts Hundreds of them @hollyjallen,#QConSF Nov 2018
- 94. “” Test with the users @hollyjallen,#QConSF Nov 2018
- 98. “” @hollyjallen,#QConSF Nov 2018 Devs OpsSRE
- 99. “” How do we test our understanding of how Slack will fail? @hollyjallen,#QConSF Nov 2018
- 100. “” "Disasterpiece Theater is an ongoing series of exercises in which we will purposely cause a part of Slack to fail." @hollyjallen,#QConSF Nov 2018
- 102. Success Metrics @hollyjallen,#QConSF Nov 2018 • Increased engineer confidence • Validate reliability improvements • Learn something new • Practice incident response
- 104. “” How do we ensure the teams are being alerted, instead of skillsets? @hollyjallen,#QConSF Nov 2018
- 105. “” How do we make postmortems a place for learning again? @hollyjallen,#QConSF Nov 2018
- 106. “” How do we make sure a capable incident commander is available for all incidents? @hollyjallen,#QConSF Nov 2018
- 110. “” Copy the questions Not the answers @hollyjallen,#QConSF Nov 2018
- 111. “” Change is possible You don't have to be ready @hollyjallen,#QConSF Nov 2018
- 114. “” Learn Faster @hollyjallen,#QConSF Nov 2018
- 116. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ slack-devops