Over the past 6 months at Fastly we've migrated away from our legacy monitoring systems and have deployed Prometheus as our primary system for infrastructure and application monitoring. The Prometheus approach posed some unique challenges over traditional monitoring systems, whilst at the same time enabling us to easily scale our monitoring infrastructure alongside our global network growth. It hasn't been completely smooth sailing and deploying Prometheus across a globe spanning network serving over 10% of the world's internet traffic has raised its fair share of challenges both technical and cultural. In this presentation you will learn how we addressed these issues in ways that deviate slightly from conventional wisdom, the mistakes we made along the way, and how the new system has been received by our teams. We hope that our experiences can help you succeed in deploying Prometheus within your organization.

1. Monitoring at Scale
Migrating to Prometheus at Fastly
PROMCON 2018 | Marcus Barczak
@ickymettle

5. How were we
monitoring Fastly?

6. +

7. ๏ Operational overhead.
๏ Limited graphing functions.
๏ No alerting support,
๏ No real API for consuming metric data.
Growing pains with Ganglia

8. aaS
+
+

9. ๏ Now supporting two systems.
๏ Where do I put my metrics?
๏ Still writing external plugins and agents.
๏ Monitoring treated as a "post-release" phase.
Growing pains doubled

10. Scaling our infrastructure
horizontally
Required scaling our monitoring
vertically

11. Third time lucky

12. ๏ Scale with our infrastructure growth,
๏ Be easy to deploy and operate.
๏ Engineer friendly instrumentation libraries.
๏ First class API support for data access.
๏ To reinvigorate our monitoring culture.
See: https://peter.bourgon.org/observability-the-hard-parts/
Project goals

13. ?

15. ๏ Build a proof of concept.
๏ Pair with pilot team to instrument their services.
๏ Iterate through the rest.
๏ Run both systems in parallel.
๏ Decommission SaaS system and Ganglia.
Getting started

16. Infrastructure
build

17. prometheus A prometheus B
scrapes
targets
SJC
scrapes
targets

18. prometheus A prometheus B
scrapes
targets
SJC
scrapes
targets
prometheus A prometheus B
scrapes
targets
JFK
scrapes
targets
prometheus A prometheus B
scrapes
targets
ATL
scrapes
targets

19. prometheus A prometheus B
scrapes
targets
SJC
scrapes
targets
prometheus A prometheus B
scrapes
targets
JFK
scrapes
targets
prometheus A prometheus B
scrapes
targets
ATL
scrapes
targets
GCP
federator A federator B
frontend stack

20. prometheus A prometheus B
scrapes
targets
SJC
scrapes
targets
prometheus A prometheus B
scrapes
targets
JFK
scrapes
targets
prometheus A prometheus B
scrapes
targets
ATL
scrapes
targets
GCP
federator A federator B
frontend stack
Query Traffic (TLS)

21. Prometheus Server
Software Stack
Ghost Tunnel
TLS termination and auth.
Service Discovery Sidecar
Target configuration
Rules Loader
Recording and Alert rules
Prometheus

22. Prometheus Server
Software Stack
Ghost Tunnel
TLS termination and auth.
Service Discovery Sidecar
Target configuration
Rules Loader
Recording and Alert rules
Prometheus
Typical Server
Software Stack
Service Discovery Proxy
Service discovery and
TLS exporter proxy
Exporters
Built into services or sidecar

23. Build your own
service discovery?

24. Fastly's infrastructure
is bare metal hardware
no cloud conveniences

25. ๏ Automatic discovery of targets.
๏ Self-service registration of exporter endpoints,
๏ TLS encryption for all exporter traffic.
๏ Minimal exposure of exporter TCP ports.
Service discovery requirements

26. Prometheus Server
Software Stack
Ghost Tunnel
TLS termination and auth.
PromSD Sidecar
Target configuration
Prometheus
Typical Server
Software Stack
PromSD Proxy
Service discovery and
TLS exporter proxy
Exporters
Built into services or sidecar
generates config
for prometheus
scrapes proxied targets over TLS
queries for
available targets

27. promsd sidecar
"exporter_hosts": [
"10.0.0.1",
"10.0.0.2",
"10.0.0.3",
"10.0.0.4"
]
configly
fetch list of hosts
in a datacenter
1
promsd proxy
request /targets endpoint
for each host to get list
of available scrape targets
32
3
output all targets as a
file service discovery
JSON file
4
Prometheus reads
the file and scrapes
the configured
targets.
{
"targets": [
“10.0.0.1:9702”,
“10.0.0.2:9702”
],
"labels": {
"__metrics_path__": “/node_exporter_9100/metrics",
"job": “node_exporter”
}
},
{
"targets": [
“10.0.0.1:9702”,
“10.0.0.2:9702”
],
"labels": {
"__metrics_path__": "/varnishstat_exporter_19102/metrics",
"job": "varnishstat_exporter"
}
}
PromSD sidecar

28. promsd proxy
fetch list of installed
systemd services
node_exporter
process_exporter
systemd
"node_exporter": {
"prometheus_properties": {
"target": "127.0.0.1:9100"
}
},
…
"varnishstat_exporter": {
"prometheus_properties": {
"target": "127.0.0.1:19102"
}
}
for each corresponding
systemd service fetch the
local exporter target address
varnishstat_exporter
1
32
3
configly
exposes an API
used by prometheus
and promsd sidecar
/node_exporter_9100/metrics
/varnish_exporter_19102/metrics
/targetssidecar
PromSD proxy

29. ๏ Really easy to leverage the file SD mechanism.
๏ New targets can be added with one line of config.
๏ TLS and authentication everywhere.
๏ Single exporter port open per host.
It worked!

30. Prometheus Adoption

31. Prometheus at Scale at Fastly
114 Prometheus servers globally
28.4M time series
2.2M million samples/second

32. ... a few hours later

33. ๏ Engineers love it.
๏ Dashboard and alert quality have increased.
๏ PromQL enables some deep insights.
๏ Scaling linearly with our infrastructure growth.
Prometheus wins

34. ๏ Metrics exploration without prior knowledge.
๏ Alertmanager's flexibility.
๏ Federation and global views.
๏ Long term storage still an open question.
Still some rough edges.

35. 😍

36. Thanks!
@ickymettle fastly.com