Download as PDF, PPTX
![95th percentile home page latency over 5
minutes < 500ms
Home page request response code != 5xx
Home page request served in < 100ms
Metric Identifier
[Metric Identifier] [Operator] [Metric Value]
EXAMPLE SLIS
#observabilitysummit @phredmoyer](https://image.slidesharecdn.com/reliableobservabilityatscale-200408165930/75/Reliable-observability-at-scale-Error-Budgets-for-1-000-21-2048.jpg)
![95th percentile home page latency over 5
minutes < 500ms
Home page request response code != 5xx
Homepage request served in < 100ms
Operator
[Metric Identifier] [Operator] [Metric Value]
EXAMPLE SLIS
#observabilitysummit @phredmoyer](https://image.slidesharecdn.com/reliableobservabilityatscale-200408165930/75/Reliable-observability-at-scale-Error-Budgets-for-1-000-22-2048.jpg)
![95th percentile home page latency over 5
minutes < 500ms
Home page request response code != 5xx
Home page request served in < 100ms
Metric Value
[Metric Identifier] [Operator] [Metric Value]
EXAMPLE SLIS
#observabilitysummit @phredmoyer](https://image.slidesharecdn.com/reliableobservabilityatscale-200408165930/75/Reliable-observability-at-scale-Error-Budgets-for-1-000-23-2048.jpg)
![95th percentile home page latency over 5
minutes < 500ms
Home page request response code != 5xx
Home page request served in < 100ms
[Metric Identifier] [Operator] [Metric Value]
EXAMPLE SLIS
#observabilitysummit @phredmoyer](https://image.slidesharecdn.com/reliableobservabilityatscale-200408165930/75/Reliable-observability-at-scale-Error-Budgets-for-1-000-24-2048.jpg)
![[Success Objective] [SLI] [Period]
Success Objective
99% of 95th percentile home page latency
over 5 minutes < 500ms over the trailing
month
99% of home page request response code
!= 5xx over last 7 days
95% of home page requests served in <
100ms over last 24 hours
EXAMPLE SLOS
#observabilitysummit @phredmoyer](https://image.slidesharecdn.com/reliableobservabilityatscale-200408165930/75/Reliable-observability-at-scale-Error-Budgets-for-1-000-28-2048.jpg)
![EXAMPLE SLOS
[Success Objective] [SLI] [Period]
SLI
99% of 95th percentile home page latency
over 5 minutes < 500ms over the trailing
month
99% of home page request response code
!= 5xx over last 7 days
95% of home page requests served in <
100ms over last 24 hours
#observabilitysummit @phredmoyer](https://image.slidesharecdn.com/reliableobservabilityatscale-200408165930/75/Reliable-observability-at-scale-Error-Budgets-for-1-000-29-2048.jpg)
![EXAMPLE SLOS
99% of 95th percentile home page latency
over 5 minutes < 500ms over the trailing
month
99% of home page request response code
!= 5xx over last 7 days
95% of home page requests served in <
100ms over last 24 hours
[Success Objective] [SLI] [Period]
Period
#observabilitysummit @phredmoyer](https://image.slidesharecdn.com/reliableobservabilityatscale-200408165930/75/Reliable-observability-at-scale-Error-Budgets-for-1-000-30-2048.jpg)
![EXAMPLE SLOS
99% of 95th percentile home page latency
over 5 minutes < 500ms over the trailing
month
99% of home page request response code
!= 5xx over last 7 days
95% of home page requests served in <
100ms over last 24 hours
[Success Objective] [SLI] [Period]
#observabilitysummit @phredmoyer](https://image.slidesharecdn.com/reliableobservabilityatscale-200408165930/75/Reliable-observability-at-scale-Error-Budgets-for-1-000-31-2048.jpg)
![EXAMPLE EBS
Allow 1% failure of 95th percentile home
page latency over 5 minutes < 500ms over
the trailing month
Allow 1% failure of home page request
response code != 5xx over last 7 days
Allow 5% failure of home page requests
served in < 100ms over last 24 hours
[Error Budget] [SLI] [Period]
Error Budget
#observabilitysummit @phredmoyer](https://image.slidesharecdn.com/reliableobservabilityatscale-200408165930/75/Reliable-observability-at-scale-Error-Budgets-for-1-000-35-2048.jpg)
![EXAMPLE EBS
Allow 1% failure of 95th percentile home
page latency over 5 minutes < 500ms over
the trailing month
Allow 1% failure of home page request
response code != 5xx over last 7 days
Allow 5% failure of home page requests
served in < 100ms over last 24 hours
[Error Budget] [SLI] [Period]
SLI
#observabilitysummit @phredmoyer](https://image.slidesharecdn.com/reliableobservabilityatscale-200408165930/75/Reliable-observability-at-scale-Error-Budgets-for-1-000-36-2048.jpg)
![EXAMPLE EBS
Allow 1% failure of 95th percentile home
page latency over 5 minutes < 500ms over
the trailing month
Allow 1% failure of home page request
response code != 5xx over last 7 days
Allow 5% failure of home page requests
served in < 100ms over last 24 hours
[Error Budget] [SLI] [Period]
Period
#observabilitysummit @phredmoyer](https://image.slidesharecdn.com/reliableobservabilityatscale-200408165930/75/Reliable-observability-at-scale-Error-Budgets-for-1-000-37-2048.jpg)
![EXAMPLE EBS
Allow 1% failure of 95th percentile home
page latency over 5 minutes < 500ms over
the trailing month
Allow 1% failure of home page request
response code != 5xx over last 7 days
Allow 5% failure of home page requests
served in < 100ms over last 24 hours
[Error Budget] [SLI] [Period]
#observabilitysummit @phredmoyer](https://image.slidesharecdn.com/reliableobservabilityatscale-200408165930/75/Reliable-observability-at-scale-Error-Budgets-for-1-000-38-2048.jpg)
![Lots of teams; lots of tools
Metrics: Prometheus / StatsD => Datadog
Logs: JSON => [ ELK, Datadog, AWS ]
APM: Datadog
Network: [ Datadog, ThousandEyes ]
Distributed Tracing: WIP
#observabilitysummit @phredmoyer](https://image.slidesharecdn.com/reliableobservabilityatscale-200408165930/75/Reliable-observability-at-scale-Error-Budgets-for-1-000-42-2048.jpg)
Uploaded byFred Moyer
Reliable observability at scale: Error Budgets for 1,000+
This document summarizes a presentation about implementing service level objectives (SLOs) and error budgets at scale. It discusses establishing service level indicators (SLIs) to define good and bad service, setting SLOs as targets for SLIs over time periods, and calculating error budgets as the complement of SLOs. The presentation provides examples of SLIs, SLOs, and error budgets for latency and availability. It also discusses challenges including variance from real users and different stakeholders' needs, and recommends approaches like flexible latency metrics and measuring as close to users as possible.
More Related Content
Similar to Reliable observability at scale: Error Budgets for 1,000+
Reliable observability at scale: Error Budgets for 1,000+
- 1. Reliable Observability atScale Error Budgets for 1,000+ Observability Practitioner’s Summit 11/18/2019 #observabilitysummit @phredmoyer
- 2.
- 3.
- 4. Hi, I’m Fred SLOgician Thinksabout SLOs, SLIs, Error Budgets Observability Hacker TSDBs, StatsD, Prometheus, Histograms Software Engineer (SRE) 15+ yrs C, Perl, Ruby, Go, Python, blabla Dad Two kids, needs more sleep/coffee #observabilitysummit @phredmoyer
- 5.
- 6. THE HARD THINGABOUT EBS & SLOS TOOLING, APPROACH, IMPLEMENTATION ERROR BUDGET DEMOCRATIZATION QUESTIONS ZENDESK ARCHITECTURE FLYOVER #observabilitysummit @phredmoyer
- 7.
- 8.
- 9.
- 10. ProxyCDN RoR μSVC #observabilitysummit@phredmoyer
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18. 2016 2018 2019 ABRIEF HISTORY OF SLOs/SLIs/EBs SRECON EUR: Developing Effective SLIs and SLOs BLOG: Latency SLOs Done RIght OPS K8SCON: Latency SLOs Done RIght @LIZTHE GREY TALK: Effective SLOs SRECON US: Latency SLOs Done Right BAYLISA: Practical SLOs with EBs SRECON APAC: Latency SLOs Done Right SRECON DUB: Several SLO/EB Talks SCALE17X: Latency SLOs Done Right #observabilitysummit @phredmoyer
- 19. SLIs Delineates ‘Good’ vs ‘Bad’Requests #observabilitysummit @phredmoyer
- 20. 95th percentile homepage latency over 5 minutes < 500ms Home page request response code != 5xx Home page request served in < 100ms EXAMPLE SLIS #observabilitysummit @phredmoyer
- 21. 95th percentile homepage latency over 5 minutes < 500ms Home page request response code != 5xx Home page request served in < 100ms Metric Identifier [Metric Identifier] [Operator] [Metric Value] EXAMPLE SLIS #observabilitysummit @phredmoyer
- 22. 95th percentile homepage latency over 5 minutes < 500ms Home page request response code != 5xx Homepage request served in < 100ms Operator [Metric Identifier] [Operator] [Metric Value] EXAMPLE SLIS #observabilitysummit @phredmoyer
- 23. 95th percentile homepage latency over 5 minutes < 500ms Home page request response code != 5xx Home page request served in < 100ms Metric Value [Metric Identifier] [Operator] [Metric Value] EXAMPLE SLIS #observabilitysummit @phredmoyer
- 24. 95th percentile homepage latency over 5 minutes < 500ms Home page request response code != 5xx Home page request served in < 100ms [Metric Identifier] [Operator] [Metric Value] EXAMPLE SLIS #observabilitysummit @phredmoyer
- 25. SLOs Binding target forSLIs #observabilitysummit @phredmoyer
- 26. SLO = #goodreqs / #totalreqs +Time range #observabilitysummit @phredmoyer
- 27. 99% of 95thpercentile home page latency over 5 minutes < 500ms over the trailing month 99% of home page request response code != 5xx over last 7 days 95% of home page requests served in < 100ms over last 24 hours EXAMPLE SLOS #observabilitysummit @phredmoyer
- 28. [Success Objective] [SLI][Period] Success Objective 99% of 95th percentile home page latency over 5 minutes < 500ms over the trailing month 99% of home page request response code != 5xx over last 7 days 95% of home page requests served in < 100ms over last 24 hours EXAMPLE SLOS #observabilitysummit @phredmoyer
- 29. EXAMPLE SLOS [Success Objective][SLI] [Period] SLI 99% of 95th percentile home page latency over 5 minutes < 500ms over the trailing month 99% of home page request response code != 5xx over last 7 days 95% of home page requests served in < 100ms over last 24 hours #observabilitysummit @phredmoyer
- 30. EXAMPLE SLOS 99% of95th percentile home page latency over 5 minutes < 500ms over the trailing month 99% of home page request response code != 5xx over last 7 days 95% of home page requests served in < 100ms over last 24 hours [Success Objective] [SLI] [Period] Period #observabilitysummit @phredmoyer
- 31. EXAMPLE SLOS 99% of95th percentile home page latency over 5 minutes < 500ms over the trailing month 99% of home page request response code != 5xx over last 7 days 95% of home page requests served in < 100ms over last 24 hours [Success Objective] [SLI] [Period] #observabilitysummit @phredmoyer
- 32. Nobody’s Perfect Error Budget= 1-SLO #observabilitysummit @phredmoyer
- 33. Success Objective ==99% Error Budget = 1-0.99 == 1% #observabilitysummit @phredmoyer
- 34. EXAMPLE EBS Allow 1%failure of 95th percentile home page latency over 5 minutes < 500ms over the trailing month Allow 1% failure of home page request response code != 5xx over last 7 days Allow 5% failure of home page requests served in < 100ms over last 24 hours #observabilitysummit @phredmoyer
- 35. EXAMPLE EBS Allow 1%failure of 95th percentile home page latency over 5 minutes < 500ms over the trailing month Allow 1% failure of home page request response code != 5xx over last 7 days Allow 5% failure of home page requests served in < 100ms over last 24 hours [Error Budget] [SLI] [Period] Error Budget #observabilitysummit @phredmoyer
- 36. EXAMPLE EBS Allow 1%failure of 95th percentile home page latency over 5 minutes < 500ms over the trailing month Allow 1% failure of home page request response code != 5xx over last 7 days Allow 5% failure of home page requests served in < 100ms over last 24 hours [Error Budget] [SLI] [Period] SLI #observabilitysummit @phredmoyer
- 37. EXAMPLE EBS Allow 1%failure of 95th percentile home page latency over 5 minutes < 500ms over the trailing month Allow 1% failure of home page request response code != 5xx over last 7 days Allow 5% failure of home page requests served in < 100ms over last 24 hours [Error Budget] [SLI] [Period] Period #observabilitysummit @phredmoyer
- 38. EXAMPLE EBS Allow 1%failure of 95th percentile home page latency over 5 minutes < 500ms over the trailing month Allow 1% failure of home page request response code != 5xx over last 7 days Allow 5% failure of home page requests served in < 100ms over last 24 hours [Error Budget] [SLI] [Period] #observabilitysummit @phredmoyer
- 39. Keys to ErrorBudget Democratization Real world examples that are easy to reference Formulas that can be parsed by humans and code Be explicit; small details make big differences #observabilitysummit @phredmoyer
- 40.
- 41.
- 42. Lots of teams;lots of tools Metrics: Prometheus / StatsD => Datadog Logs: JSON => [ ELK, Datadog, AWS ] APM: Datadog Network: [ Datadog, ThousandEyes ] Distributed Tracing: WIP #observabilitysummit @phredmoyer
- 43. StatsD - notjust for servers Measuring service performance is (mostly) easy Client apps are more difficult Disconnects Caching (CDN, Proxy) Large browser & device variance #observabilitysummit @phredmoyer
- 44. Logs, Traces, Metrics Conway’sLaw; experts for each ‘pillar’ Democratize Expertise #ask-sre Reliability Champions `Observability 101` `Hands On With Datadog` #observabilitysummit @phredmoyer
- 45.
- 46. Metrics for SLIs Lies,Darn Lies, and Percentiles Easy to get the math wrong Missing the X Factor - Sample Volume Many vendors have bugs in percentile tools Can’t aggregate them (well, most of them) #observabilitysummit @phredmoyer
- 47. Metrics for SLIs Counters Easyto understand Easy to implement Easy to aggregate Easy to get the math right #observabilitysummit @phredmoyer
- 48. Metrics for SLIs LatencySLIs via counters Request time < 500ms Count em’ up, divide by total reqs Add success objective and time range for SLO 99% of request times < 500ms over trailing week #observabilitysummit @phredmoyer
- 49.
- 50. Metrics for SLIs FlexibleLatency SLIs Histogram based # reqs 100-200ms, 200-300ms, etc One time series for each latency band zen.app.request.sli{path:/foo;bin:gt_500_le_600} #observabilitysummit @phredmoyer
- 51. Metrics for SLIs FlexibleLatency SLIs 10..20...100ms 100..200...1,000ms 1,000..1,500...10,000ms 10,000..15,000...60,000ms Latency == 547ms, metric tag `le_600`, `gt_500_le_600` #observabilitysummit @phredmoyer
- 52. Metrics for SLIs FlexibleLatency SLIs Low errors per latency band Not as precise as HDR Histograms Possible cardinality expansion issues Can implement on any monitoring vendor or TSDB #observabilitysummit @phredmoyer
- 53.
- 54.
- 55. The hard thingabout Error Budgets and SLOs
- 56.
- 57.
- 58.
- 59. Proxy CDN ROR μSVC SLI_1 SLI_2 SLI_3 SLI_4 Needfor low variance increases #observabilitysummit @phredmoyer
- 60. Real Users Varied usagepatterns iPads, phones, laptops LTE, Fiber, DSL, 3G #observabilitysummit @phredmoyer
- 61. Proxy CDN ROR μSVC SLI_1 SLI_2 SLI_3 SLI_4 Focuson the users! #observabilitysummit @phredmoyer
- 62. Different SLOs/EBs forDifferent Folks 99% of home page requests < 500ms over... 5 minutes - NOC / SRE 1 hour - Product Engineers 1 week - Product Managers 1 month - VPs 1 quarter - CXOs #observabilitysummit @phredmoyer
- 63. Keys to ErrorBudgets at Scale Give everyone a formula to follow for SLIs/SLOs/EBs Use simple tools that can deliver rich results Use latency bands (histograms) for duration data Measure SLIs as close to the client as possible Use EBs with appropriate time ranges for audiences #observabilitysummit @phredmoyer
- 64.