Service Ownership at Slack - a talk at DevOpsDays Chicago Aug 2019

1. Service
Ownership
Learn Faster
Holly Allen
she/her
@hollyjallen

2. Holly Allen
Software development and
leadership for 19 years

3. @hollyjallen #DevOpsDaysChi Aug 2019

4. @hollyjallen #DevOpsDaysChi Aug 2019

5. @hollyjallen #DevOpsDaysChi Aug 2019

6. Software! 😍

7. @hollyjallen #DevOpsDaysChi Aug 2019

8. S L O W 😩

9. @hollyjallen #DevOpsDaysChi Aug 2019

10. @hollyjallen #DevOpsDaysChi Aug 2019
MeasureDesign
Learn

11. Toyota
Production
System

12. @hollyjallen #DevOpsDaysChi Aug 2019

13. @hollyjallen #DevOpsDaysChi Aug 2019

14. @hollyjallen #DevOpsDaysChi Aug 2019

15. @hollyjallen #DevOpsDaysChi Aug 2019

16. @hollyjallen #DevOpsDaysChi Aug 2019

17. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Kaizen
Continuous Improvement

18. @hollyjallen #DevOpsDaysChi Aug 2019
MeasureDesign
Learn

19. @hollyjallen #DevOpsDaysChi Aug 2019

20. @hollyjallen #DevOpsDaysChi Aug 2019

21. @hollyjallen #DevOpsDaysChi Aug 2019

22. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Why do some teams deliver the
right things fast?

23. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Executive dedication to learning

24. “”
@hollyjallen #DevOpsDaysChi Aug 2019
High Trust Teams

25. @hollyjallen #DevOpsDaysChi Aug 2019
MeasureDesign
Learn

27. Slack launched
February 2014
🚀

28. @hollyjallen #DevOpsDaysChi Aug 2019
5 Years
Grew to 10+ million daily
active users in more than 150
countries
5 Years

29. @hollyjallen #DevOpsDaysChi Aug 2019
5 Years
From 100 to 15,000 servers
In 25 cloud data centers
world-wide
5 Years

30. @hollyjallen #DevOpsDaysChi Aug 2019
5 Years
From 8 to 1,600 people
In 10 offices world-wide
5 Years

31. @hollyjallen #DevOpsDaysChi Aug 2019
MeasureDesign
Learn

32. @hollyjallen #DevOpsDaysChi Aug 2019

33. “”
@hollyjallen #DevOpsDaysChi Aug 2019
✅ Continuous Deployment
✅ Experiment Frameworks
✅ User Research

34. Something didn't scale...

35. Centralized
Operations 😭

36. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Who should be responsible for the
management, monitoring and operation of a
production application?

37. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Centralized Operations

38. Devs
Features
Scale
Architecture
Ops
Cloud Infra
Deployment
Monitoring

39. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Ops is getting the pages

40. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Product Development grew faster
than Operations,
A lot faster

41. 20 Product
Developers
1 Ops
Engineer

42. “”
@hollyjallen #DevOpsDaysChi Aug 2019
How can operations reliably reach the
developers when there's a problem?

43. Devs
I've never been
on-call before,
this is scary!
Ops
Now I know I
can find a
developer when
I need to.

44. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Ops is getting the pages first pages
Ultra-senior devs on-call

45. “”
@hollyjallen #DevOpsDaysChi Aug 2019
"Call Maude, she knows how this works"

46. @hollyjallen #DevOpsDaysChi Aug 2019
MeasureDesign
Learn

47. “”
@hollyjallen #DevOpsDaysChi Aug 2019
How can operations reliably reach the
developers when there's a problem?

48. Most devs
go on-call
Fall 2017
📟

49. “”
@hollyjallen #DevOpsDaysChi Aug 2019
"Wait, I'm on-call now?"

50. Devs
I'm glad I'm only
on call a few
times a year
Ops
I'll be able to
reach a frontend
engineer if I
need to.

51. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Learn by Doing

52. “”
@hollyjallen #DevOpsDaysChi Aug 2019
On-call 3 times a year 🤔

53. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Ops is getting the pages first pages
Ultra-senior devs on-call
Seven One dev rotations

54. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Continuous Deployment
Dozens of prod deploys a day

55. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Human Routers

56. Devs
I don't
understand this
part of the code
Ops
These are the
machine alerts
I'm seeing

57. “”
@hollyjallen #DevOpsDaysChi Aug 2019
"Call Andy, he knows how this works"

58. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Postmortems weren't a great
place for learning

59. Reorg!
Fall 2017
Operations is out
Service Engineering is in

60. “”
@hollyjallen #DevOpsDaysChi Aug 2019
How can we ensure that developers know
when there's a problem?

61. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Centralized Operations
Service Ownership

62. “”
@hollyjallen #DevOpsDaysChi Aug 2019
High Trust
Learning Organization

63. Devs
Features
Reliability
Performance
Service Eng
Cloud Platform
Observability tools
Service Discovery
Define best practice

64. Site
Reliability
Engineers
• Devops generalists
• Emotional intelligence
• Mentoring
• Ambassadors
• Operational maturity

65. “”
@hollyjallen #DevOpsDaysChi Aug 2019
SRE embedded in dev teams

66. Devs
Um, where are
the SREs?
SREs
I'm over here
doing
operational
tasks

67. “”
@hollyjallen #DevOpsDaysChi Aug 2019
SRE Ops is still getting the first pages

68. “”
@hollyjallen #DevOpsDaysChi Aug 2019
How do we lower operational burden on the
SREs?

69. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Plan: Send these paging alerts to
the development teams

70. Define
service
health and
operational
maturity
• The teams have to know
what they own
• We had a lot of features and
services without formal
ownership

71. Define
service
health and
operational
maturity
• At least one alerting health
metric, like latency or
throughput

72. Define
service
health and
operational
maturity
• Teams needed to get on-
call ready
• At least 4, preferably 6+
engineers participating to
make it sustainable

73. Devs
We need
training
SREs
We're going to
plan this out
perfectly

74. @hollyjallen #DevOpsDaysChi Aug 2019

75. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Host level alerts
Hundreds of them

76. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Test with the users

77. @hollyjallen #DevOpsDaysChi Aug 2019

78. @hollyjallen #DevOpsDaysChi Aug 2019

79. Everything
was fine!
💪

80. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Empowered
Continuous Improvement

81. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Devs Ops are getting the first pages
Ultra-senior devs on-call
Dozens Seven One of dev rotations

82. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Centralized Operations
Service Ownership

83. “”
@hollyjallen #DevOpsDaysChi Aug 2019
How do we make postmortems
a source of learning?

84. “”
@hollyjallen #DevOpsDaysChi Aug 2019
How do we make sure that incident
command is a skillset throughout Slack?

85. “”
@hollyjallen #DevOpsDaysChi Aug 2019
How do we make it even easier
to operate a service?

86. @hollyjallen #DevOpsDaysChi Aug 2019
MeasureDesign
Learn

87. @hollyjallen #DevOpsDaysChi Aug 2019

88. @hollyjallen #DevOpsDaysChi Aug 2019

89. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Copy the questions
Not the answers

90. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Change is possible
You don't have to be ready

91. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Empowered
High Trust
Continuous Improvement

92. @hollyjallen #DevOpsDaysChi Aug 2019
MeasureDesign
Learn

93. “”
@hollyjallen #DevOpsDaysChi Aug 2019
Design Thoughtfully
Measure Ruthlessly
Learn Faster

94. Holly Allen
she/her
@hollyjallen
Thank You!