Having application developers responsible for their services in production reduces resolution times during incidents and leads to better products.

1. Proprietary & Confidential
The How and
Why of
Full-Service
Ownership
PagerDuty Community Team

2. Proprietary & Confidential
Hi!
Mandi Walls
DevOps Advocate
@LNXCHK
mwalls@pagerduty.com

3. Proprietary & Confidential

4. Proprietary & Confidential
Every Business is a Digital Business
Make payments
Shop online
Be entertained
Order food
Be connected
Get around
Do work
Buy anything
Stay healthy

5. Proprietary & Confidential
Culture: Ownership is Changing
Dev Insourced and Ops Outsourced
Full-Service Ownership
“Outsourced” to IT

6. Proprietary & Confidential
Digital Transformation Requires Major Change
DevOps has
become
mainstream
and strategic
Every company is
becoming a
software company
Cloud migration:
really powerful,
really hard
Imperative for
companies to meet
customer
expectations
(reliability, performance,
innovation)
Developer and
technology
headcount growing
rapidly

7. Proprietary & Confidential
Operational Efficiency Maturity Behavior

8. Proprietary & Confidential
A culture and operating model where
the people writing the code for a service
own it in production
Full-Service Ownership

9. Proprietary & Confidential
Knowing which services impact
which customers in which ways
Why is This Important?

10. Proprietary & Confidential
The goal of full service ownership
is to reduce the time to resolve
incidents by bringing in the Subject
Matter Expert as the first
responder.

11. Centralized Incident Process
L1 triage
15 mins
1 hour
10 mins
25 mins
Incident Commanders
Subject Matter Experts
Validates an action with
business metrics
Hands over report to the
new on-call
Manual remediation
Locates bridge call details
30 mins 1 hour 2 hours 2 days Collate
information for
a postmortem
Incident
response
Need Context
Additional Help
Identify Changes
Social
Containers
Microservices
Network
Security
Apps & Services
Database
CI/CD
Cloud
Overwhelmed by
services proliferation

12. Proprietary & Confidential
Organize Services to Owner Teams
Social
Containers
Microservices
Network
Security
Apps & Services
Database
Servers
Cloud
CI/CD
Routed to distinct services
owned by separate teams
Services
Database
Mobile
Online Catalog
Shopping Cart
Web App
Database Team
Web App Team
Shopping Cart Team
Catalog Team
Mobile Team

13. Proprietary & Confidential
1. Have greater impact on end-user experience
2. Be more accountable for production performance
3. Drive down MTTR
3 Key Benefits to Application Devs

14. Proprietary & Confidential
● Know what customers like and want
● Understand how changes impact customers
● Have greater autonomy to do what’s right for the
users
End-User Experience

15. Proprietary & Confidential
● Incentivize the most knowledgeable
● Shorten feedback loops for production
improvements
Production Performance

16. Proprietary & Confidential
● Contacted as a subject matter expert (SME) for the
systems your team owns
● Shared through an on-call rotation cycle
● Defined communications channels
● Acknowledge, Triage, Escalate or Resolve
Being On-Call

17. Proprietary & Confidential
● Mean time to Resolve (MTTR)
● Minimize user impact
● Improves customer satisfaction
Improve MTTR

18. Proprietary & Confidential

19. Proprietary & Confidential
Full-service ownership, in the
simplest form, empowers
engineers to have responsibility
for their code in production,
leading to high-quality software.

20. Proprietary & Confidential

21. Proprietary & Confidential
Defining a Service

22. Proprietary & Confidential
What is a Service
AA discrete piece of functionality that provides
value and that is wholly owned by a team

23. Proprietary & Confidential
A service can be a lot of things:
● Microservice
● Slice of a monolith
● Piece of functionality
● Internal Tool
● Component
● Shared Infrastructure
● Feature
If it provides
value to
other people,
it’s a service

24. Proprietary & Confidential
Shared Understanding

25. Proprietary & Confidential
Define the team that will own the
service

26. Proprietary & Confidential
Set up the on-call rotation for this
service

27. Proprietary & Confidential
Ensure the team is correctly sized

28. Proprietary & Confidential
Service definitions help with problem
resolution

29. Proprietary & Confidential
What about a monolith?

30. Proprietary & Confidential
How about dependencies?

31. Proprietary & Confidential
Lifecycle of a Service

32. Proprietary & Confidential
In the Beginning...
Where do your services come from?
• New projects or features built from new product
requirements
• Deconstruction of monoliths or other
improvements
• Externally developed services

33. Proprietary & Confidential
Pre-Production
• Local development
• Easy-to-use pipelines
• Testing
• Packaging
• Deploying

34. Proprietary & Confidential
Ship It
● Talk to other prod teams - SRE, Customer Service
● Define and test alerts
● Write runbook
● Reviews with marketing and PR

35. Proprietary & Confidential
Running in Production
• Meet the users!
• Work with Customer Support
• Maintenance at all layers
• Steady state

36. Proprietary & Confidential
Sunsetting
• Deprecation
• Help customers migrate
• Turn it off?

37. Proprietary & Confidential
Service Ownership Functions

38. Proprietary & Confidential
A Shared Responsibility

39. Proprietary & Confidential
Service Documentation
Service documentation is a shared responsibility among every
engineer contributing to the code base. You wrote it, you
document it.

40. Proprietary & Confidential
Your service should make sense to
others who will interact with it

41. Proprietary & Confidential
What’s in a Name?
(Service Names)

42. Proprietary & Confidential
Be Specific

43. Proprietary & Confidential
Names that are specific:
● “User authenticator”
● “Payment processor”
● “Shopping Cart”
● “Login”
● “Report generator”
● “Email tracking code”

44. Proprietary & Confidential
Interesting less ideal names:
● “PacMan”
● “Artemis”
● “Apollo”
● “BurgunDB”

45. Proprietary & Confidential
Include descriptions:
● What is the intent of this service, component, this slice of
functionality?
● How does this thing deliver value?
● What does it contribute to?
● How will this impact customers?

46. Proprietary & Confidential
Other considerations:
● Dependencies
● APIs
● Deployment Mechanisms

47. Proprietary & Confidential
Service Availability

48. Proprietary & Confidential
Service Tiers

49. Proprietary & Confidential
Tier 1 services example:
● 24/7 on-call
● Multiple levels of robustness
● Disaster recovery plan
● Clear and updated runbook

50. Proprietary & Confidential
Tier 2 & 3 services example:
● Monday-Friday support escalation
● Supporting functionality, not critical path
● New services that are not generally available

51. Proprietary & Confidential
Runbooks

52. Proprietary & Confidential
Production Operations

53. Proprietary & Confidential
Project Management

54. Proprietary & Confidential
Product Ownership
Customers are always asking for uptime, performance, and
security - they just don’t always use those words.

55. Proprietary & Confidential
Champions

56. Proprietary & Confidential
Best Practices

57. Proprietary & Confidential
● Communication
● Compromise
● Commitment
● It’s all about relationships
Making the Shift

58. Proprietary & Confidential
Focus on Customer Needs

59. Proprietary & Confidential
Agility Helps

60. Proprietary & Confidential
Start Small

61. Proprietary & Confidential
Greenfield vs. Brownfield

62. Proprietary & Confidential
Blameless Culture

63. Proprietary & Confidential
Practice Makes Perfect

64. Proprietary & Confidential
Application Techniques

65. Proprietary & Confidential
Engage with Necessary Teams

66. Proprietary & Confidential
Alerting

67. Proprietary & Confidential

68. Proprietary & Confidential
Escalation Policies

69. Proprietary & Confidential
First Level

70. Proprietary & Confidential
Second Level

71. Proprietary & Confidential
Third Level

72. Proprietary & Confidential
Tips for Scaling

73. Proprietary & Confidential
Onboarding

74. Proprietary & Confidential
Shadow On-call

75. Proprietary & Confidential
Practice Empathy

76. Proprietary & Confidential
https://ownership.pagerduty.com/
https://pagerduty.com/ops-guides