1.
Better Monitoring for Python
Inclusive Monitoring with Prometheus

2.
Monitoring is often a Problem, not a Solution
Themes common among companies I talk to:
● Monitoring tools are limited, both technically and conceptually
● Tools don’t scale well and are unwieldy to manage
● Operational practices don’t align with the business
Result: Engineers continuously woken up for non-issues, get fatigued

3.
Fundamental Challenge is Limited Visibility

4.
Prometheus
Inspired by Google’s Borgmon monitoring system.
Started in 2012 by ex-Googlers working in Soundcloud as an open source project.
Mainly written in Go. Publically launched in early 2015.
100+ companies using it including Digital Ocean, GoPro, Apple, Red Hat and
Google.

5.
Services have Internals

6.
Monitor the Internals

7.
Monitor as a Service, not as Machines

8.
Inclusive Monitoring
Don’t monitor just at the edges:
● Instrument client libraries
● Instrument server libraries (e.g. HTTP/RPC)
● Instrument business logic
Library authors get information about usage.
Application developers get monitoring of common components for free.
Dashboards and alerting can be provided out of the box, customised for your
organisation!

9.
Let’s Talk Code
pip install prometheus_client
from prometheus_client import Summary, start_http_server
REQUEST_DURATION = Summary('request_duration_seconds',
'Request duration in seconds')
@REQUEST_DURATION.time()
def my_handler(request):
pass // Your code here
start_http_server(8000)

10.
Multiple Dimensions (No Evil Twins Please)
from prometheus_client import Counter
REQUESTS = Summary('requests_total',
'Total requests', ['method'])
def my_handler(request):
REQUESTS.labels(request.method).inc()
pass // Your code here

11.
Exceptional Circumstances In Progress
from prometheus_client import Counter, Gauge
EXCEPTIONS = Counter('exceptions_total', 'Total exceptions')
IN_PROGRESS = Gauge('inprogress_requests', 'In progress')
@EXCEPTIONS.count_exceptions()
@IN_PROGRESS.track_inprogress()
def my_handler(request):
pass // Your code here

12.
Getting Data Out
from prometheus_client import start_http_server
if __name__ == '__main__':
start_http_server(8080)
Also possible with Django, Twisted etc.

13.
Oh Noes, My Vendor Lock In!
from prometheus_client.bridge.graphite import GraphiteBridge
gb = GraphiteBridge(('graphite.your.org', 2003))
gb.start(10.0) # Push every 10 seconds
Exposition to Prometheus doesn’t use a special API, can be hooked into whatever
monitoring system you have.
Instrument once, work with everything!

14.
Running Prometheus
Grab binary from http://www.robustperception.io/prometheus-nightly-binaries/
Put config in prometheus.yml:
scrape_configs:
- job_name: python
target_groups:
targets: [localhost:8000]
Run it:
./prometheus

15.
Dashboards

16.
Prometheus is as Powerful as Python
Can slide, dice, aggregate and do math to produce your alerts
If it’s computable, Prometheus can compute it*
Alert on what matters, not on what your monitoring limit you to!
* Halting, tractability and sanity not guaranteed

17.
Links!
Blog Posts: www.robustperception.io/tag/prometheus
Demo: demo.robustperception.io
Python Client: https://github.com/prometheus/client_python
Project Website: prometheus.io