As engineers we spend much of our time getting stuff to production and making sure our infrastructure doesn’t burn down out right. We however spend very little time learning to understand and respond to outages. Does our platform degrade in a graceful way or what does a high cpu load really mean? What can we learn from level 1 outages to be able to run our platforms more reliably. Plenty of people are jumping on the new hype, Observability, lots of them are replacing their “legacy” monitoring stack. Not all of them achieve the goals they set. But observability is not a tool — it is a property of a system. Moving from many small black boxes to a more holistic view of your system. In this talk we ll talk about how to prepare teams to tweak their testing and monitoring setup and work instructions to quickly observe, react to and resolve problems. We look at improving your monitoring by adapting your culture and then maybe your tooling. Where we as engineers not only write, maintain and operate our software platforms but actively pursue ways to learn and predict its (non-functional) behavior. Furthermore we ll discuss the need for and the options of not only monitoring our platforms and it's envitable outages, but also their (potential) length and impact. We ll look at tools like at using Service Level Objects for ways to prepare teams to tweak their testing and monitoring setup and runbooks to quickly observe, react to and resolve problems.

1. Observability (o11y):
a gentle introdcution
Bram Vogelaar
@attachmentgenie

2. We have all been there…x

3. And that's when we show Bill…

4. Amongst other things…

5. $ whoami
• Used to be a Molecular Biologist
• Then became a Dev
• Now an Ops
• Currently Cloud Engineer @ The Factory

6. SNMP pulls hundreds of checks, logs live in databases
Push thousands of time-series to Graphite + statsd , Logstash for
logs
Prometheus scrapes millions of metrics, Logs are everywhere,
Traces are a thing now
How it started -> How is it going

7. Monitoring Symptoms

8. The monitoring everything age
The act of observing changes the system being observed
Base CPU load significantly visible
aka HDDs and CPUs killed by monitoring love

9. Aim becomes 100% uptime

10. Monitoring solutions were not catching up
While Applications were deployed 10x a day, we were doing
manual configuration of Single Source of Truth
Eventual consistency with huge latency
Causing Chaos and Distrust in the Observer

11. Pager overload

12. “Observability is defined as a measure of how
well internal states of a system can be inferred
from knowledge of its external outputs.”
Engineering Definition

13. R.E.D & U.S.E
U = Utilization, as canonically defined
S = Concurrency
E = Error Rate, as a throughput metric
R = Request Throughput, in requests per second
E = Request Error Rate, as either a throughput metric or a
fraction of overall throughput
D = Latency, Residence Time, or Response Time; all three are
widely used

14. Logs are insurance

15. Large complex systems will always be in a
(somewhat) degraded state

16. 3 Pillars Definition

17. Get engineers to be comfortable facing failure

18. If it hurts do it more often…

19. Don’t stare at dashboards

20. monitoring – luck == Observability
Cultural Definition

21. SL{O,I,A}
Service-Level Agreement (SLA) => Contract
Service-Level Objective (SLO) => Promise
Service-Level Indicator (SLI) => Observe

22. Only alert on SLO violations
And definitely on HDD space useage
Don’t alert on symptoms
Do alert on the rate you are using the error budget

23. What we need to visualize
$/€ revenue
# of sales
# of signups
# of Api calls
“(Ab)Normal” application use

24. Instrumentation

25. OpenTelemetry => exiting times

26. monitoring – luck == Observability
monitoring + devops == Observability
Bram’s Definition

27. Contact
bram@attachmentgenie.com
@attachmentgenie
https://www.slideshare.net/attachmentgenie

28. Questions ?
The Floor is yours…