1. Prometheus
Julien Pivotto (@roidelapluie)
FLOSS UK Spring Conference
April 26th, 2018

2. user{name="roidelapluie"} 1
I like Open Source
I like monitoring
I like automation
... and all of that is my daily job at inuits

3. Traditional Monitoring
It's up
No it's down
It is degraded
I don't know

4. Questions that come after:
It's up but it is performant?
It's down but for everyone?
Its is degraded but are the users impacted?
Is it even relevant?

5. Metrics Monitoring
Gather fine grained data at frequent interval
Make them useful by labelling them ; store
them
Analyze them to understand what is going on

6. We are in the cloud era.
Here are some buzzwords for you
cloud, API, openstack, devops, docker, bimodal,
stateless, kubernetes, orchestration, automation,
serverless, docker, humanops, ansible, continuous
deployment, cri-o, jenkins, agile, docker, red hat,
containers, virtualization, provisionning,
monitoring, observability...

7. What is the cloud
Scale
Velocity
Change

8. On Premise looks like the cloud
Nowadays you have no choice.
Scale
Velocity
Change

9. What are the needs
Automation
Scalability

10. Bye bye
all-in-one tools
tools that don't scale
tools you can not automate

11. We need deserve better tools
Our customers ask us to respond fast, in
seconds
We make hundreds of operations per second
What is your monitoring frequency... 5
minutes?

12. Time to get better tools /
protocols

13. Prometheus
https://prometheus.io/

14. Cloud Native
Easy to configure, deploy, maintain
Designed in multiple services
Container ready
Orchestration ready (dynamic config)
Fuzziness

15. Data Centric
A Metric in Prometheus has metadata:
myql_global_status_handlers_total{handler="tmp_write"}
1122
And lots of function to filter, change, remove...
those metadata while fetching them.

16. Open Source
Apache 2.0
Go
Support for multiple OS
Many "exporters":
https://github.com/prometheus/prometheus/wiki/Defa
ult-port-allocations

17. Efficient
Prometheus is designed to fetch data in an
interval measured in seconds
Millions of datapoints
Big improvements in prometheus 2.0

18. Storage
Prometheus 2.x uses prometheus/tsdb
2 hours blocks ; later compacted in up to 10
days blocks
15000 metrics/s ends up ad 800 MiB/day

19. How does it work?

20. How does it work?

21. How does it work?

22. How does it work?

23. How does it work?

24. Exporters
Exporters expose metrics with an HTTP API
Bindings available for many languages
Exporters do not save data ; they are not
"proxies" and don't "cache" anything

25. Exploring Metrics

26. Exploring Metrics

27. Exploring Metrics

28. Exploring Metrics

29. PromQL
mysql_global_status_commands_total

30. PromQL
mysql_global_status_commands_total{command="select"}

31. PromQL
mysql_global_status_commands_total
{command=~"select|set_options"}

32. PromQL
mysql_global_status_commands_total{command=~"select|se
t_options"}

33. PromQL
deriv(mysql_global_status_connections[5m])

34. PromQL
{__name__=~".+innodb.+cache.*"}
predict_linear(mysql_heartbeat_lag_seconds[5m], 60*2)
sum(rate(mysql_global_status_commands_total{command=~"
(commit|rollback)"}[5m])) without (command)

35. Demo
node_exporter
grok_exporter
blackbox_exporter
snmp_exporter
service discovery

36. Security
Prometheus supports TLS client (also with
authentication)
Exporter side is your business
We use it with traefik (reverse proxy in go
with native metrics)
We manage certs with ansible

37. Alerting
Prometheus has recording rules
Record frequent queries or alerting
Alert base on any PromQL queries.

38. One tool does one job...
Prometheus collects data
Exporters expose data
Grafana graphes data
Alertmanager dispatches alerts

39. Alertmanager
Open Source
Same developers as Prometheus
Second service

40. What is the Alertmanager
doing?
Receives alerts
Group them
Inhibits them
Dispatches them
Deals with HA

41. How to alerts?
Email
Some vendors: Slack, Hipchat, VictorOps,
pagerduty, ...
Generic Wehbook -> Plugin anything you want

42. High Availability
2 prometheus servers do exact the same job
They send alerts to Alertmanagers
Alertmanagers are clustered not to send the
same notification twice

44. Grafana
Open Source (Apache 2.0)
Web app
Specialized in visualization
Pluggable
Multiple datasources: prometheus, graphite,
influxdb...
Has an API!

45. History of Grafana
Grafana is a fork of Kibana 3 ; used to be JS-
Driven.
Now fully featured, requires a database, multi-
projects/users support, etc...

46. Grafana and Prometheus
Prometheus shipped its own consoles
Now it recommends Grafana and deprecated
its own consoles

47. Grafana Dashboards

48. Grafana Dashboards

49. Time Picker

50. Configure Prometheus in
Grafana

51. Configure Prometheus in
Grafana

52. Prometheus Dashboard

53. Creating Grafana Dashboards
Takes time
Requires deep knowledge of the tools
Improved over time
Easy to share (json + online library)

54. Conclusion
Open Source
Flexible and dynamic
Multiple small pieces
Low on resources
Rich ecosystem
For Cloud and On Premise

55. Julien Pivotto
roidelapluie
roidelapluie@inuits.eu
Inuits
https://inuits.eu
info@inuits.eu
Contact
I will give a
prometheus
workshop
at OSMC.de!