Site Reliability Engineering (SRE) is a set of principles, practices, and organizational constructs that seek to balance the reliability of a service with the need to continually deliver new features. An error budget is the primary construct used to help balance these seemingly competing goals. This is an introduction to error budgets and their components: service level indicators (SLIs) and service level objectives (SLOs). We will discuss the art of creating and implementing SLOs. Attendees will be able to: • Describe the key concepts, namely, Error Budget, Service Level Indicator (SLIs), and Service Level Objectives (SLOs) • Recommend actions to take when the error budget is over consumed • Recommend actions to take when excess error budget remains In the spirit of DevOps, Error Budgets and SLOs work best when they are agreed to in collaboration with many different constituents across the business. As such, this presentation is appropriate for: • Product Owners and Product Managers • Business decision makers • Developers • Operators • And anyone else interested in building and operating services that deliver business and customer value.

1. Using Error Budgets to
Prioritize Work
Nathen Harvey / @nathenharvey
He/Him

2. Incentives are not aligned
Developers
Agility
Operators
Stability

3. Incentives are not aligned

4. Incentives are not aligned
The Business

5. Incentives are not aligned
The Business IT

6. Just who is the business?
The Business IT

7. Just who is the business?
The Business
Business-y things
IT

8. Just who is the business?
The Business
Business-y things
IT
Engineer-y things

9. Why is the business?

10. Nathen Harvey
Developer Advocate, Google
@nathenharvey

11. DevOps

12. Why is the business?

13. Online
Retailer

14. What is "reliable"?
How do you know if the
application is working?

15. Browse
Catalog

16. Browse
Catalog
Select
Item

17. Browse
Catalog
Select
Item
Purchase

18. Browse
Catalog
Select
Item
Purchase
Browser
Browser

19. Browse
Catalog
Select
Item
Purchase
Browser
Browser

20. Server
Server
/catalog
Browse
Catalog
Select
Item
Purchase
Browser
Browser

21. /catalog
List of Items
Server
Server
Browse
Catalog
Select
Item
Purchase
Browser
Browser

22. Browser Server
Browse
Catalog
Select
Item
Purchase
Browser Server
/catalog
List of Items
/api/addToCart

23. Browser Server
Browse
Catalog
Select
Item
Purchase
Browser Server
/catalog
List of Items
/api/addToCart
Status

24. Browser Server
Browse
Catalog
Select
Item
Purchase
Browser Server
/catalog
List of Items
/api/addToCart
Status
/checkout

25. Browser Server
Browse
Catalog
Select
Item
Purchase
Browser Server
/catalog
List of Items
/api/addToCart
Status
/checkout
Payment Page

26. Browser Server
Browse
Catalog
Select
Item
Purchase
Browser Server
/catalog
List of Items
/api/addToCart
Status
/checkout
Payment Page
/api/pay

27. /charge
Browser Server
Browse
Catalog
Select
Item
Purchase
Browser Server
/catalog
List of Items
/api/addToCart
Status
/checkout
Payment Page
/api/pay
Payment
Processor
Payment
Processor

28. status
Browser Server
Browse
Catalog
Select
Item
Purchase
Browser Server
/catalog
List of Items
/api/addToCart
Status
/checkout
Payment Page
/api/pay
Payment
Processor
Payment
Processor
/charge

29. Browser Server
Browse
Catalog
Select
Item
Purchase
Browser Server
/catalog
List of Items
/api/addToCart
Status
/checkout
Payment Page
/api/pay
Status Page
Payment
Processor
Payment
Processor
/charge
status

30. Ops Review
Which pages…
● Were unactionable?
● Show a pattern?
● Can be eliminated with engineering work?
Reduce your maintenance costs,
get your return in future team bandwidth.

31. Browser Server
Browse
Catalog
Select
Item
Purchase
Browser Server
/catalog
List of Items
/api/addToCart
Status
/checkout
Payment Page
/api/pay
Status Page
status
Server
Payment
Processor
Payment
Processor
/charge
status

32. /charge
status
Browser Server
Browse
Catalog
Select
Item
Purchase
Browser Server
/catalog
List of Items
/api/addToCart
Status
/checkout
Payment Page
/api/pay
Status Page
Server
Payment
Processor
Payment
Processor

33. IT - heroic effort
Resolving the Latency Issue

34. IT - heroic effort
Product Management - kept pushing for features
Resolving the Latency Issue

35. IT - showing signs of burnout
Net Result

36. IT - showing signs of burnout
Product Management - frustrated with pace of
development
Net Result

37. Site Reliability Engineering
(SRE)
To protect, provide for, and progress
software and systems with an
ever-watchful eye on their availability,
latency, performance, and capacity.

39. ● Often defined on the “Golden Signals” of a service:
traffic, error rate, and latency
● May also use custom metrics or
“saturation”
● Istio gathers these automatically for each of your
services
Service level indicators
A carefully defined quantitative measure of some aspect of
the level of service that is provided.
SLIs | Metrics that you use to
define the SLO targets

40. status
Browser Server
Browse
Catalog
Select
Item
Purchase
Browser Server
/catalog
List of Items
/api/addToCart
Status
/checkout
Payment Page
/api/pay
Status Page
Payment
Processor
Payment
Processor
/charge

41. status
Browser Server
Browse
Catalog
Select
Item
Purchase
Browser Server
/catalog
List of Items
/api/addToCart
Status
/checkout
Payment Page
/api/pay
Status Page
Payment
Processor
Payment
Processor
/charge

42. /charge
status
Browser Server
Browse
Catalog
Select
Item
Purchase
Browser Server
/catalog
List of Items
/api/addToCart
Status
/checkout
Payment Page
/api/pay
Status Page
Payment
Processor
Payment
Processor

43. Browser Server
Payment
Processor
Browse
Catalog
Select
Item
Purchase
Browser Server
Payment
Processor
/catalog
List of Items
/api/addToCart
Status
/checkout
Payment Page
/api/pay
Status Page
/charge
status

44. The payment status page should load quickly
Latency

45. ● How do we define quickly?
● When does the timer start / stop?
The payment status page should load quickly
Latency

46. The proportion of HTTP POST requests for /api/pay
that send their entire response within X ms
measured at the load balancer.
● How do we define quickly?
● When does the timer start / stop?
The payment status page should load quickly
Latency

47. ● The Golden Signals make great
SLIs to set SLOs on
● Examples:
○ Return 99.9% of calls within 200ms
○ At least 99.5% of calls
return non-errors
● You may alert when out of compliance
Service level objective
A target value or range of values for a service level that is
measured by an SLI.
SLOs | Metrics that you use to
define the SLO targets

48. 😡😋
SLOs should capture the performance and availability levels that,
if barely met, would keep the typical customer of a service happy
“meets target SLO” ⇒ “happy customers”
“sad customers” ⇒ “misses target SLO”

49. SLOs
Service SLI Type Objective
Purchase Latency 99% < 500ms in last 28 days

50. SLOs
Service SLI Type Objective
Purchase Latency 99% < 500ms in last 28 days
Purchase Latency 75% < 200ms in last 28 days

51. Error Budgets

52. Error Budget
A principled way to define the
desired reliability of a service
An acceptable level of
unreliability. This is a budget that
can be allocated
An agreement that helps
prioritize engineering work

53. What are the consequences when we
exhaust or overspend
our error budget?

54. Consequences
/ freeze feature releases

55. Consequences may include
/ freeze feature releases
/ prioritize post mortem items
/ automate deployment pipelines *
/ improve monitoring and observability
/ require SRE consultation
/ return the pager

56. What should we spend
our error budget on?

57. Error budgets can accommodate
/ releasing new features
/ expected system changes
/ inevitable failure in hardware, networks, etc.
/ planned downtime
/ risky experiments

58. Error Budget
An agreement that helps
prioritize engineering work

59. Intentional
Practice

60. Both of these are now
available in HTML
format for free!
landing.google.com/sre/books

61. Nathen Harvey
Developer Advocate, Google
@nathenharvey

62. Using Error Budgets to
Prioritize Work
Nathen Harvey / @nathenharvey
He/Him