George Hantzaras discusses scaling service level objectives (SLOs) with Kubernetes and cloud-native observability. He covers defining SLOs, how they relate to customer experience, common issues implementing and scaling SLOs, and how cloud-native observability and SLO-as-code can help address these issues. Specifically, implementing SLOs as code with tools like OpenSLO and Sloth can help automate SLO monitoring and make it easier to scale across growing services and distributed systems. Observability as code using metrics, dashboards, alerting and GitOps also provides a standardized approach to scale monitoring infrastructure.

1. Scaling SLOs
With Kubernetes and cloud-native observability
Open Source Monitoring Conference – Nuremberg, Nov. 2022

2. Hello!
I am George Hantzaras
● Director, Cloud Platform Engineering @Citrix
● Organizer, Athens Cloud Computing Meetup Group
● Organizer, Athens Hashicorp User Group
● Creator of Ploigos (ploigos.co)
● Find me: @iamhantzo
● Based in Athens, Greece
2

3. Agenda
● SLO primer
● How SRE and Customer Experience are
related
● Issues with implementing and scaling SLOs
● Cloud-native observability and SLO-as-code
3

4. 1.
Defining SL{A,O,I}s
Why do reliability metrics matter

5. “
You can’t manage what you
don’t measure.
5

6. How did we come up with this
6

7. How do we measure reliability
7

8. How SLOs
work
How do we define, measure,
and analyze our SLI, SLO, and
SLA
8

9. Defining SLOs
● Define what’s important for the business.
That’s our (SLI)
● Calculate what our target for this metric
should be (SLO)
● Make a promise to our customers about the
service we’ll be offering in relation to this
metric (SLA)
● These can be time-based or event-based
9

10. But what happens with this information
● Product development teams get an error
budget
● Allowed threshold of bad behavior of our
service related to a specific SLO
● Error Budget = 1 – SLO
● Budget Burn = 1 – SLI
● We want Budget > Burn
10

11. 2.
How are these related
to Customer Experience

12. What really is reliability
● Reliability is the probability of failure-free
operation of a computer program for a
specified period in a specified environment
● A customer facing view of quality of our cloud
delivery
12

13. Reliability is CX
13
SLOs should focus on representing user impact
and experience
Reliability engineering is a part of CX
engineering in SaaS
Reliability drives cloud adoption

14. Defining
SLOs
Is not a purely technical task
14

15. Customer-centric approach to SLO
● Customers are different
● Availability should focus on what users
perceive
● It’s not about uptime anymore but availability
of functionality
● Combine SLO data with business and CRM
data
15

16. What does that mean for the SLI/SLO
process
● Define the critical user journeys in your
service and prioritize
● Determine which metrics are important for
each journey
● For those metrics, look at historical data and
set targets
● Operationalize those metrics and insights
16

17. 3.
Issues with implementing
and scaling SLOs

18. Common implementation issues
● SLIs are not customer-focused or not aligned
with the business
● No clear stakeholders and ownership over
SLOs
● Reactive usage of error budgets
● Unrealistic SLOs (targets)
● Manual SLI/SLO evaluation process
18

19. Common scaling issues
Different standards in monitoring.
● A large number of engineers and a
growing number of teams brings many
different approached to observability
● Different standard around metric and logs,
can bring different results
19

20. Common scaling issues
Scaling services and distributed systems create
complexity
20

21. Common scaling issues
● A common understanding of service metrics
● Scaling and automating the evaluation
process of SLI/SLOs
● Tools agnosticism for monitoring
infrastructure
21

22. 4.
Observability and SLOs as
code

23. Cloud-native observability
23
Real-time insights
Provide real-time observability
and insights for cloud operations,
product teams, customer
success, and leadership.
Self-service
Serve elastic infrastructure and
applications, in such a way, that
no operational overhead is added
to onboard a new service.
Alerting
Configurable alerting for multiple
platforms, using different
communication channels.
Integrations
Open observability infrastructure
which easily integrates with
customer data sources, reporting
tools, CI/CD, and more
SLO Framework
A new company-wide SLO
framework, adapted to latest
architectures and focused on
customer experience.
Operationalizing error budgets
for product insights

24. Powered by
24
Metrics Events Logs Traces

25. Observability as code
25
Metrics
as code
Dashboards
as code
Alerting
as code
GitOps Zero Trust

26. Observability as code
26
Commit + CI
Observability
configuration is added
via pull request. Build is
triggered.
Build
Build is executed and
final TF module and
variables are created.
Testing
IaC modules are tested
and notification for test
outcomes is sent
Deploy
Assets are published to
artifactory and ready to
be deployed in
production. Optionally,
deployment is triggered

27. Observability as code
27

28. Operationalizing
28

29. How about SLOs as code
● OpenSLO specification
● Open specification for defining SLOs to
enable a common, vendor–agnostic to SLOs
● Yaml-based format
● Sloth for Prometheus
29

30. SLOs a code – OpenSLO
30

31. SLOs a code – Sloth
31

32. SLOs a code – Sloth
32

33. Sloth example
33

34. Sloth example
34

35. Sloth example
35

36. Putting it all
together
36

37. Observability Architecture
37

38. SLO monitoring
38

39. Lessons Learnt
39
On-call ≠ Reporting
Monitoring and
alerting for usage by
SRE and on call
teams, have totally
different
requirements that
monitoring and
alerting for uptime
reporting, leadership
visibility etc
Customer focused
SLI/SLO reporting
needs to reflect
customer impact.
Observability should
focus on measuring
what the customers
are experiencing
Start somewhere
Observability is a
journey of continuous
improvement. You
should find the low
hanging fruit and
start from there. And
have in mind that you
should always keep on
working on it

40. 5.
Wrapping up

41. Key takeaways
● Monitoring has evolved
● Observability is not just about measuring
● Reliability has also adapted to the technology
and delivery advancements
● Observability is a really important aspect of
reliability
● How we can scale Observability and Reliability
measurement with cloud-native tools
41

42. Thank you!
Stay in touch
● Find me: @iamhantzo
● LinkedIn: George Hantzaras
42