Slides from the Utah Software Architecture Meetup April 6, 2017 https://www.meetup.com/Utah-Software-Architecture-Meetup/events/238365117/

1. Prometheus
Multi-dimensional metrics
Clint Checketts

2. Prometheus - That Fire Guy

5. Baseball

6. Stats
Times at bat
Batting Average
Pitching speed
AssistsBases run
Home runs
Wins and Losses

7. Application Stats
Deploy speed
Endpoint latency
Database
performance
Error countsCache hits
Garbage
collection
Outages

8. Home Run Stats

10. Scouts vs Stat-heads

12. Multi-dimensional Metrics

14. #gifee

15. #gifee - Google Infrastructure for
Everyone Else
• Kubernetes (borg)
• Prometheus (borgmon)
• Open Tracing (Dapper)

16. Why Prometheus? Ownership
• To ensure that engineers
• have confidence in where the metrics are coming from,
• can minimize friction on creating dashboards they need,
• and improve alerts that affect them
• Types of Metrics
• Infrastructure
• Application Performance
• Feature Usage

17. Prometheus Pedigree
• Open Source
(Apache 2)
• Cloud Native
Computing
Foundation
• Inspired by Google
metrics ‘borgmon’
• Created Nov 2012

18. Prometheus Feature Set
• Metrics gathering
• Infrastructure exporters
• Application instrumentation
• Query language
• Alerting
• Graphing

19. Technical Summary
• Self contained, very easy to run
• Doesn’t use external DB
• Can run even if everything else is on fire
• Very efficient memory/disk usage
• Pull model for monitoring
• Service discovery model to determine what to
monitor (AWS, DNS, Kubernetes, etc)
• Keeps active series and queries in memory
• Memory usage dictates scaling model

20. Metrics Collection
Grafana Prometheus AWS
Kubernetes
Java Application
Graph Data
Metric Data
Service Discovery

21. Alerting
Prometheus
Alerting
Alert Manager
Routing

22. Exporters
Allows Prometheus to
scrape services that aren’t
Prometheus aware
Examples
Node Exporter
SNMP Exporter
MySQL Exporter
Jenkins Exporter
RabbitMQ Exporter

23. Pull Architecture
Application
Prometheus
Grafana
Kubernete
s
Graphs from
Collects from Discovers from

24. Grafana
 Graphing/Dashboarding service
 Templates
 Multiple query overlays (offset queries)

25. Metrics Types:
 counter - example total requests
 inc()
 guage - measure a value at a given time
 Inc() dec() set()
 histogram - quartiles with sample data
 Observe()
 summary - counts and totals
 startTimer() Observe()

26. How do I get this
goodness?
 Client Libraries
 Exporters
 Baked into tools
 Docker
 Kubernetes

27. Metric names and labels
 Names
 Explain what is being measured
 Include the unit (or ‘count’)
 Labels (Examples)
 Application name
 Quartile
 Endpoint
http_request_duration_sec{app=”api”, method=“get”,
quartile=“0.5”, handler=“/users”, statusCode=“200”} 0.2

28. PromQL examples
 All current request duration
 http_request_duration{app=“apiContetnt”}
 How many envs in a pool are available?
 env_pool_count{envtype=“domo/brief/master”} –
env_initing_count{envtype=“domo/brief/master”}
 What are my non-success request rates?
 sum(irate(http_request_duration_milliseconds_count{app=”a
piContent”, statusCode!=“200”}[1m])) by (statusCode) * 60

29. Summary
 Multi-dimensional
metrics are powerful
 Libraries support is
ready
 Go!