We all know the wall of illegible wall of small graphs, that we like to present to (senior) management and auditors as proof that we do observability. It doesnt matter that the person ( and the associated knowledge) has long since left the company, nor that the dashboard doesnt autorefresh and still show the same data from when we turned on the monitoring PC. In an ever changing IT landscape we deserve better than that. We shouldn't have to rely on the gut instinct of the senior engineer on deck about the general shape of the dashboard to know where to start fixing whatever it is that needs fixing. We should aim to collect and present such information both from the it and business side that let a relatively inexperienced oncall engineer differentiate between a P1 incident and a major client/customer/continet going to bed. We should be telling beautiful stories with data and dashboards in such a way that (even) management can pull up global dashboard and can determine business relevant information like "is our advertisement campaign having any impact". We should also not be afraid to have multiple dashboards that show different relevant information to stack holders (e.g sales figures for management and links to runbooks for engineers)

1. Scraping metrics for fun and profit
Bram Vogelaar
@attachmentgenie

2. Confidential and Proprietary
~ ❯ whoami
• Used to be a Molecular Biologist
• Then became a Dev, now an Ops
• Currently Cloud Engineer @ Seaplane.io
• Amsterdam HUG organizer
• Author of soon to be released LGTM book

3. Confidential and Proprietary

4. Confidential and Proprietary
Can you honestly answer this question?
Can you figure out if your
platform is in an error
state?

5. Confidential and Proprietary
Large complex systems will always be in a (somewhat)
degraded state

6. Confidential and Proprietary
3 pillars of observability

7. Confidential and Proprietary
3 pillars of observability
Metrics
Logs
Traces

8. Confidential and Proprietary
Metrics
● Open-Source tool to do metrics collection and storage
● Since 2012/2015, by SoundCloud
● CNCF graduated project
● HTTP pull model
● PromQL DSL
https://prometheus.io/

9. Confidential and Proprietary
Metrics Ecosystem

10. Confidential and Proprietary
Viewing (Node Exporter) Metrics

11. Confidential and Proprietary
BYOD exporter / Beg, borrow, steal
https://prometheus.io/docs/instrumenting/exporters/

12. Confidential and Proprietary
Prometheus Configuration
global:
scrape_interval: 15s
scrape_configs:
- job_name: o11y
static_configs:
- targets:
- localhost:9100
- localhost:3100
- localhost:3200

13. Confidential and Proprietary
Prometheus/Metrics HA,
Dynamic Scraping
global:
scrape_interval: 10s
scrape_configs:
- job_name: o11y
static_configs:
- targets:
- localhost:9100
- localhost:3100
- localhost:3200
- job_name: o11y
scrape_interval: 10s
consul_sd_configs:
- server: localhost:8500
tags:
- metrics

14. Confidential and Proprietary
Write your own exporter
package main
import (
"net/http"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
func main() {
http.Handle("/metrics", promhttp.Handler())
http.ListenAndServe(":2112", nil)
}

15. Confidential and Proprietary
Write your own exporter
var pingCounter = prometheus.NewCounter(
prometheus.CounterOpts{
Name: "ping_request_count",
Help: "No of request handled by Ping handler",
},
)
func ping(w http.ResponseWriter, req *http.Request) {
pingCounter.Inc()
fmt.Fprintf(w, "pong")
}
func main() {
prometheus.MustRegister(pingCounter)
…

16. Confidential and Proprietary
Write your own exporter

17. Confidential and Proprietary
Forcing labels / Decorating metrics
global:
scrape_interval: 10s
external_labels:
datacenter: "%{::trusted.extensions.pp_datacenter}"
region: "%{::trusted.extensions.pp_region}" <- use ${IATA_CODES}
env: "%{::trusted.extensions.pp_environment}"

18. Confidential and Proprietary
PromQL
http_requests_total{job=”nginx", handler="/grafana"}[5m]
sum by (job) ( rate(http_requests_total[5m]) )
sum by (region) (bird_protocol_up{ip_version="ipv4",proto="BGP",env="$environment"})
https://prometheus.io/docs/prometheus/latest/querying/basics/

19. Confidential and Proprietary
Tensegrity of observability
Metrics
Logs
Traces
Alerts
Dashboards

20. Confidential and Proprietary
Dashboards
● Grafana is a multi-platform open source analytics and
interactive visualization web application
● Since 2014, by Torkel Ödegaard
https://grafana.com/oss
/grafana/

21. Confidential and Proprietary

22. Confidential and Proprietary
Zooming to what you need to see

23. Confidential and Proprietary
What is going on here?

24. Confidential and Proprietary
Annotate Graphs

25. Confidential and Proprietary
Correlate metrics with events

26. Confidential and Proprietary
Data should tell beautiful stories
https://www.youtube.com/watch?v=hVimVzgtD6w
https://www.youtube.com/watch?v=jbkSRLYSojo

27. Confidential and Proprietary
Data should tell beautiful stories

28. Confidential and Proprietary
Canvas Plugin

29. Confidential and Proprietary
Correlate events with metrics

30. Confidential and Proprietary
Instrument data flows

31. Confidential and Proprietary
Correlate events with metrics
Marketing Campaign
ON

32. Confidential and Proprietary
Don’t stare at dashboards

33. Confidential and Proprietary
Alerts are PromQL queries too
- alert: PrometheusJobMissing
expr: absent(up{job="prometheus"})
for: 0m
labels:
severity: warning
annotations:
summary: Prometheus job missing (instance {{ $labels.instance }})
description: "A Prometheus job has disappearedn VALUE = {{ $value }}n LABELS = {{ $labels
}}"
https://awesome-prometheus-alerts.grep.to/

34. Confidential and Proprietary
R.E.D & U.S.E
U = Utilization, as canonically defined
S = Concurrency
E = Error Rate, as a throughput metric
R = Request Throughput, in requests per second
E = Request Error Rate, as either a throughput metric or a
fraction of overall throughput
D = Latency, Residence Time, or Response Time; all three are
widely used

35. Confidential and Proprietary
Instrumentation

36. Confidential and Proprietary
What we need to visualize
$/€ revenue
# of sales
# of signups
# of Api calls
“(Ab)Normal” application use

37. Confidential and Proprietary
SL{O,I,A}
Service-Level Agreement (SLA) => Contract
Service-Level Objective (SLO) => Wish/Hope
Service-Level Indicator (SLI) => Observe

38. Confidential and Proprietary
Only alert on SLO violation
And definitely on HDD space usage
Don’t alert on symptoms
Do alert on the rate you are using the error budget

39. Questions Before Takeoff?
bram@attachmentgenie.com
@attachmentgenie
https://www.slideshare.net/attachmentgenie