The document is an introductory discussion by Bram Vogelaar on observability (o11y) in systems, highlighting its importance in monitoring internal states through external outputs. It covers essential concepts, pitfalls of traditional monitoring, and emphasizes the need for a cultural shift towards embracing failure for better observability. Key metrics are defined, including utilization, concurrency, and error rates, alongside practical advice for effective monitoring in modern engineering environments.

1. Observability (o11y):
a gentle introdcution
Bram Vogelaar
@attachmentgenie

2. We have all been there…x

3. And that's when we show Bill…

4. Amongst other things…

5. $ whoami
• Used to be a Molecular Biologist
• Then became a Dev
• Now an Ops
• Currently Cloud Engineer @ The Factory

6. SNMP pulls hundreds of checks, logs live in databases
Push thousands of time-series to Graphite + statsd , Logstash for
logs
Prometheus scrapes millions of metrics, Logs are everywhere,
Traces are a thing now
How it started -> How is it going

7. Monitoring Symptoms

8. The monitoring everything age
The act of observing changes the system being observed
Base CPU load significantly visible
aka HDDs and CPUs killed by monitoring love

9. Aim becomes 100% uptime

10. Monitoring solutions were not catching up
While Applications were deployed 10x a day, we were doing
manual configuration of Single Source of Truth
Eventual consistency with huge latency
Causing Chaos and Distrust in the Observer

11. Pager overload

12. “Observability is defined as a measure of how
well internal states of a system can be inferred
from knowledge of its external outputs.”
Engineering Definition

13. R.E.D & U.S.E
U = Utilization, as canonically defined
S = Concurrency
E = Error Rate, as a throughput metric
R = Request Throughput, in requests per second
E = Request Error Rate, as either a throughput metric or a
fraction of overall throughput
D = Latency, Residence Time, or Response Time; all three are
widely used

14. Logs are insurance

15. Large complex systems will always be in a
(somewhat) degraded state

16. 3 Pillars Definition

17. Get engineers to be comfortable facing failure

18. If it hurts do it more often…

19. Don’t stare at dashboards

20. monitoring – luck == Observability
Cultural Definition

21. SL{O,I,A}
Service-Level Agreement (SLA) => Contract
Service-Level Objective (SLO) => Promise
Service-Level Indicator (SLI) => Observe

22. Only alert on SLO violations
And definitely on HDD space useage
Don’t alert on symptoms
Do alert on the rate you are using the error budget

23. What we need to visualize
$/€ revenue
# of sales
# of signups
# of Api calls
“(Ab)Normal” application use

24. Instrumentation

25. OpenTelemetry => exiting times

26. monitoring – luck == Observability
monitoring + devops == Observability
Bram’s Definition

27. Contact
bram@attachmentgenie.com
@attachmentgenie
https://www.slideshare.net/attachmentgenie

28. Questions ?
The Floor is yours…