Practical implementation of error budgets

1. Error budgets
Practical implementation

2. Who am I
● SRE team lead @AnchorFree
● @onorua
● https://www.meetup.com/Kubernetes-Kyiv/
● Deal a lot with production
● Introduced Error Budgets @Anchorfree
Yaroslav Molochko

3. Anchorfree in numbers
● 650M customers
● Several thousands nodes
● 148 containers deal with user traffic
● 16M metrics

4. Let’s talk about
production

10. Possible options to prioritisation
1. Review documentation
2. Escalate
3. Round-Robin
4. Random

16. SLO
(Service Level Objective)
Binding target for collections of SLIs

17. SLO practice
1. Obliged part
2. Validity Period
3. Expression

18. SLI
(Service Level Indicator)
A measure of the service level provided by a service
provider

21. 99.9%
99%
95%
S
L
O

22. Error Budget

25. E
R
R
O
R
B
U
D
G
E
T
99.9%
99%
95%

26. 43.2m
432m
7.2h
2160m
36h
E
R
R
O
R
B
U
D
G
E
T

27. 100%
10%
100%
E
R
R
O
R
B
U
D
G
E
T
R
E
M
A
I
N
I
N

28. E
R
R
O
R
B
U
D
G
E
T
R
E
M
A
I
N
I
N
43.2m
43.2m
2160m
36h

29. Burn rate

30. 0.01%
min
0.1%
min
1%
min
B
U
R
N
R
A
T
E

31. 60m 120m
1%
min
0.432 * 0.01 * 60 sec = 0.26sec per second
36h
43.2m
43.2min
166 m
180m
0.432 * 0.1 * 60 sec = 2.592 sec per second
43.2min
16 m
166 m
16 m 100m
ErrorBudgetremaining

32. 99.9%
99%
95%
S
L
O

33. What happens when Error Budget is fully utilized
● Your team switch to maintenance mode for the service
● Your team stop to onboard new services
● Nothing but tests and hotfix are allowed to get into

34. Stages of Error Budgets
adoption

39. Get support from your peers

40. Extra responsibility?
Don’t forget, you have at least 2 sides for “contract”

41. This will not work in our
unique case
Tell me that when you become Google

42. Sell this to your boss
1. Acknowledge responsibility for subsystem
2. Focus on user needs
3. Agree on what happens when Error Budget is beyond budget?
4. Get a sign-off from neighbor teams and management

43. Measure what is important to your user

47. Measure symptoms

49. How to merge
multiple SLI
(symptoms) into
one SLO?

50. Obvious options
● AVG
● MAX
● MIN
Non Obvious options
● An Approach for QoS aware Service Composition based on Genetic
Algorithms. (link)

52. Logic AND over bool
1
0
1
0
1
0

53. Logic AND over bool
● 1 * 1 * ... = 1 All services are within SLO
● 1 * 0 * … = 0 At least one service is acting
up

54. Logic AND over bool
min(sli:gpr_edge_message_delivery:last30d >= bool 0.9999)
*
min(sli:gpr_edge_message_95le:last30d >= bool 0.95)
== bool 1

55. Burning Rate for
alerts?

56. Alert attributes
● Detection time
● Precision
● Reset time

57. Default alerting:
Error
rate
0
1000
5 10 15 20
time
ALERT!

58. Default alerting: FOR 10m
Error
rate
0
1000
5 10 15 20
time

59. Burn Rate alerting:
Error
rate
0
1000
5 10 15 20
time
ALERT! ALERT! ALERT!
ALERT!

60. Burn Rate alerting: AND over 2 windows
Error
rate
0
1000
5 10 15 20
time
ALERT! ALERT!

61. Burn rate calculation
BR =
W
P
* E
BR =
30 * 24
1
2% =
720
1
0.02 = 14.4

63. Severity Long
Window
Short
Window
Burn rate Error
budget
consumed
Page 1 hour 5 minutes 14.4 2%
Page 6 hours 30 minutes 6 5%
Ticket 3 days 6 hours 1 10%
Site Reliability Workbook | Ways to Alert on Significant Events | page 85

64. Main alerting takeaways
● Short vs Long windows ratio is 1/12 (magic ratio)
● Don’t use extra FOR in alert manager
● Burn rate is not a magic number
● 2% leave you with 50 alerts per month within budget

65. Error budgets
● Rules (you know how to play, you know how to score)
● Self-Escalation
● Over commitment protection
● Put your users first!

66. Q.A.