This document discusses hardware-level data center monitoring using Prometheus. It outlines the speaker's data center which contains over 2,000 servers and 200 network devices. It then provides a brief introduction to Prometheus, highlighting its reliability, scalability, flexibility and ease of integration. Several Prometheus exporters are described that monitor nodes, network devices, and other systems, replacing tools like Nagios, Ganglia and Cacti. Methods for merging data from different sources are demonstrated. The transition to Prometheus monitoring is deemed successful due to the many available integrations and ease of developing new ones.

1. Hardware-level data-center
monitoring with Prometheus
Conrad Hoffmann

2. Outline
I. Our data-center
II. Brief intro to Prometheus
III. All my exporters
IV. TL;DR & Soon™

3. Outline
I. Our data-center
II. Brief intro to Prometheus
III. All my exporters
IV. TL;DR & Soon™
AMS5

6. 2118 servers
56 racks

7. 2118 servers
56 racks
200 network devices

8. 2118 servers
56 racks
200 network devices
2 * 2 generic uplinks
3 AWS Direct Connect
3 Google X-Connect

9. Where we started...
& NRPE
Cloud Watch
Cacti

10. What’s paging you at night?
Collection Visualization Alerting
Cacti ✔ ✔ ✔
CloudWatch ✔ ✔ ✔
Ganglia ✔
Graphite ✔ ✔
Icinga/Nagios ✔ ✔ ✔
Smokeping ✔ ✔ ✔
Statsd ✔

11. https://xkcd.com/927/

12. Outline
I. Our data-center
II. Brief intro to Prometheus
III. All my exporters
IV. TL;DR & Soon™
prometheus.io

13. The Promise of Prometheus
Prometheus is a reliable, scalable, flexible monitoring and
alerting system that is easy to integrate and focused on real
time metrics.

14. Prometheus: reliability
● Pull-based (“scrape”)
● List of known targets
○ Can be dynamic, e.g. DNS or service discovery
● Built-in meta-monitoring
● Redundancy is easy

15. Prometheus: scalability
● Performant, efficient storage
● Scales well to available resources
● Easy to scale horizontally
● Federation

16. Prometheus: flexibility
● Multi-dimensional, label-based data model
● Each data point is defined by
○ A metric name
○ An arbitrary number of key-value pairs (labels)
○ A value
○ A timestamp (added by Prometheus)
● Data points with identical metric names and labels form a time series
● Powerful query language allows for easy aggregation based on labels

17. Prometheus: flexibility
Target exposes:
http_responses_total{backend="foo",code="2xx"} 804
http_responses_total{backend="foo",code="4xx"} 3170
http_responses_total{backend="bar",code="2xx"} 6637
http_responses_total{backend="bar",code="4xx"} 26
Possible query:
sum(http_responses_total{backend="foo"})

18. Prometheus: ease of integration
● Data format is text based
● Scrapes are HTTP requests
● Many integrations exist already
● Excellent tooling/libraries to write new ones

19. Application
Prometheus: ease of integration

20. Host node
exporter
Prometheus: ease of integration

21. Host SNMP
exporter
Router B
Router A
Prometheus: ease of integration
Network

22. Host SNMP
exporter
Router B
Router A
Prometheus: ease of integration
Network

23. Nomen est omen...
● Alerting
● Silencing
● Alert grouping & routing
● High availability
Alertmanager

24. Displays data from many sources:
● Prometheus
● Graphite
● Influx
● OpenTSDB
● Elasticsearch
● MySQL/Postgres
● CloudWatch
● ...
Grafana
grafana.com

25. Outline
I. Our data-center
II. Brief intro to Prometheus
III. All my exporters
IV. TL;DR & Soon™
Now withProtips!

26. Node exporter
● Exports: OS- and hardware-level metrics for running systems
● Replaces: Ganglia, some Icinga/NRPE checks
● Noteworthy:
○ Comes with many collectors built-in
○ Use WMI exporter on Windows

27. Protip I
Use the node exporter’s text file collector as an easy integration point for
custom metrics!
Examples: Chef data, RAID controller data, SMART data, cron jobs, ...
node
exporter
script
Text
file
Host

28. Blackbox exporter
● Exports: data about probes against endpoints that don’t support
Prometheus natively (DNS, HTTP(S), ICMP, TCP)
● Replaces: Smokeping, some Icinga checks
● Noteworthy:
○ Monitor TLS certificate expiry :)

29. Blackbox exporter - Smokeping replacement
1. Send ICMP probe every five seconds

30. Blackbox exporter - Smokeping replacement
2. Alert on target down and packet loss
ALERT SmokepingTargetDown
IF probe_success{job="smokeping"} == 0
FOR 2m
ALERT SmokepingTargetPacketLoss
IF 100*(1-avg_over_time(probe_success{job="smokeping"}[2m]))> 20

31. Blackbox exporter - Smokeping replacement
3. Use Prometheus aggregation functions in Grafana

32. Blackbox exporter - Smokeping replacement

33. Protip II
Scrape more, scrape faster!
● ~ 1M metrics
● > 5000 targets
● Mostly 10s scrape interval, some 5s, some longer
● 50 days retention time
● 250 GB storage ¯_(ツ)_/¯

34. SNMP exporter
● Exports: SNMP data from network devices
● Replaces: Cacti
● Noteworthy:
○ a pain to configure

35. SNMP exporter - Cacti replacement
Once you have got the right SNMP config, alerts and nice graphs are easy!

36. SNMP exporter - Cacti replacement
Cacti’s killer feature: the weathermap plugin!
https://network-weathermap.com/

37. SNMP exporter - Cacti replacement
There is a diagram panel type in Grafana, but…
… we’re not quite there yet ¯_(ツ)_/¯

39. Protip III
Build a dedicated long-term Prometheus server:
● Scrape only a few selected metrics
● Yank retention time way up
● Make backups (hot backups possible in Prometheus >2.1)
Very useful data for estimating e.g. future bandwidth needs!

40. Collins exporter - Collins?
● https://tumblr.github.io/collins
● Infrastructure management / IPAM
● Server inventory, classification and lifecycle management

41. Collins exporter
● Exports: asset inventory data from Collins
● Replaces: a bunch of scripts
● Noteworthy:
○ https://github.com/soundcloud/collins_exporter

42. Collins exporter

43. Collins exporter
● Another candidate for long-term storage
● Valuable data for capacity planning

44. Protip IV
Build your own integrations!
Collins exporter:
● Written in Go
● 1 source file
● 264 lines total ¯_(ツ)_/¯

45. IPMI exporter
● Exports: IPMI data retrieved from BMCs
● Replaces: many Nagios/NRPE checks
● Noteworthy:
○ https://github.com/soundcloud/ipmi_exporter
○ Works regardless of hosts power state

46. IPMI exporter
● Mostly sensor data: temperature, fans, power consumption
● Mostly used for alerting:
○ Fans
○ Power supplies
○ Batteries

47. Protip V
Make use of techniques to ingest non-numeric data!*
● Use labels to expose (semi-)static data of interest
*...but do it with some caution!
ipmi_bmc_info{firmware_revision="2.52",manufacturer_id="Dell_Inc"} 1

48. Protip V
Make use of techniques to ingest non-numeric data!*
● Use labels and binary values to represent state
*...but do it with some caution!
collins_asset_state{tag="ABCD1234",state="Allocated"}
collins_asset_state{tag="ABCD1234",state="Maintenance"}
collins_asset_state{tag="ABCD1234",state="Unallocated"}
1
0
0

49. And now: merging data sources
Example: BMC Firmware revisions of certain server types

50. And now: merging data sources
Query: ipmi_bmc_info{firmware_revision!="2.52"}
Result: ipmi_bmc_info{firmware_revision="2.41",instance="10.1.2.3",...}

51. And now: merging data sources
Query: ipmi_bmc_info{firmware_revision!="2.52"}
Result: ipmi_bmc_info{firmware_revision="2.41",instance="10.1.2.3",...}
Query: collins_asset_details{nodeclass="app-2"}
Result: collins_asset_details{ipmi_address="10.1.2.3",...}

52. And now: merging data sources
Query: ipmi_bmc_info{firmware_revision!="2.52"}
Result: ipmi_bmc_info{firmware_revision="2.41",instance="10.1.2.3",...}
Query: collins_asset_details{nodeclass="app-2"}
Result: collins_asset_details{ipmi_address="10.1.2.3",...}
Query: label_replace(ipmi_bmc_info, "ipmi_address", "$1", "instance", "(.*)")
Result: ipmi_bmc_info{firmware_revision="2.41",ipmi_address="10.1.2.3",...}

53. And now: merging data sources
Query: collins_asset_details{nodeclass="app-2"} *
on (ipmi_address)
group_left(firmware_revision)
label_replace(ipmi_bmc_info{firmware_revision!="2.52"}, "ipmi_address", "$1", "instance", "(.*)")
Result: {firmware_revision="2.41",ipmi_address="10.1.2.3",
nodeclass="app-2",primary_address="10.10.20.30",tag="ABCD1234"}

54. And now: merging data sources
Query: collins_asset_details{nodeclass="app-2"} *
on (ipmi_address)
group_left(firmware_revision)
label_replace(ipmi_bmc_info{firmware_revision!="2.52"}, "ipmi_address", "$1", "instance", "(.*)")
* on (tag) group_left(status) (collins_asset_status == 1)
Result: {firmware_revision="2.41",ipmi_address="10.1.2.3",
nodeclass="app-2",primary_address="10.10.20.30",tag="ABCD1234",status="Allocated"}

55. Where we are now...
& NRPE
Cloud Watch
Cacti
✘
✘ ✘ ✘
✘ ✘
✘

56. Collection Visualization Alerting
CloudWatch ✔ ✔ ✔
Graphite (✔)
Prometheus ✔
Grafana ✔
Alertmanager ✔
What’s paging you at night?

57. What’s up with this CloudWatch thing?
● There is a CloudWatch exporter
● However, CloudWatch internal architecture is fundamentally
incompatible with Prometheus
● Using CloudWatch as Grafana data source can incur costs

58. Outline
I. Our data-center
II. Brief intro to Prometheus
III. All my exporters
IV. TL;DR & Soon™

59. So, is it working?
● Yes

60. Was it worth it?
● Yes

61. Why was it worth it?
● Many integrations readily available
● New ones are easy to write
● Quality and quantity of monitoring has
increased
● Monitoring and alerting has become much
more consistent
● Easy to merge data sources for alerting or
graphing
This is true across the entire organization, not just infrastructure!

62. Soon: long term storage
● Not a primary concern for Prometheus
● Simple solution as explained
● Remote (read/)write interface
● Some features in Prometheus 2.0 to allow external solutions
○ Check out e.g. Thanos: https://github.com/improbable-eng/thanos

63. Soon: forging a standard?
OpenMetrics working group
● https://github.com/RichiH/OpenMetrics

64. This is the end...
Thank you!