SlideShare a Scribd company logo
1 of 17
Download to read offline
Better Monitoring for Python
Inclusive Monitoring with Prometheus
Monitoring is often a Problem, not a Solution
Themes common among companies I talk to:
● Monitoring tools are limited, both technically and conceptually
● Tools don’t scale well and are unwieldy to manage
● Operational practices don’t align with the business
Result: Engineers continuously woken up for non-issues, get fatigued
Fundamental Challenge is Limited Visibility
Prometheus
Inspired by Google’s Borgmon monitoring system.
Started in 2012 by ex-Googlers working in Soundcloud as an open source project.
Mainly written in Go. Publically launched in early 2015.
100+ companies using it including Digital Ocean, GoPro, Apple, Red Hat and
Google.
Services have Internals
Monitor the Internals
Monitor as a Service, not as Machines
Inclusive Monitoring
Don’t monitor just at the edges:
● Instrument client libraries
● Instrument server libraries (e.g. HTTP/RPC)
● Instrument business logic
Library authors get information about usage.
Application developers get monitoring of common components for free.
Dashboards and alerting can be provided out of the box, customised for your
organisation!
Let’s Talk Code
pip install prometheus_client
from prometheus_client import Summary, start_http_server
REQUEST_DURATION = Summary('request_duration_seconds',
'Request duration in seconds')
@REQUEST_DURATION.time()
def my_handler(request):
pass // Your code here
start_http_server(8000)
Multiple Dimensions (No Evil Twins Please)
from prometheus_client import Counter
REQUESTS = Summary('requests_total',
'Total requests', ['method'])
def my_handler(request):
REQUESTS.labels(request.method).inc()
pass // Your code here
Exceptional Circumstances In Progress
from prometheus_client import Counter, Gauge
EXCEPTIONS = Counter('exceptions_total', 'Total exceptions')
IN_PROGRESS = Gauge('inprogress_requests', 'In progress')
@EXCEPTIONS.count_exceptions()
@IN_PROGRESS.track_inprogress()
def my_handler(request):
pass // Your code here
Getting Data Out
from prometheus_client import start_http_server
if __name__ == '__main__':
start_http_server(8080)
Also possible with Django, Twisted etc.
Oh Noes, My Vendor Lock In!
from prometheus_client.bridge.graphite import GraphiteBridge
gb = GraphiteBridge(('graphite.your.org', 2003))
gb.start(10.0) # Push every 10 seconds
Exposition to Prometheus doesn’t use a special API, can be hooked into whatever
monitoring system you have.
Instrument once, work with everything!
Running Prometheus
Grab binary from http://www.robustperception.io/prometheus-nightly-binaries/
Put config in prometheus.yml:
scrape_configs:
- job_name: python
target_groups:
targets: [localhost:8000]
Run it:
./prometheus
Dashboards
Prometheus is as Powerful as Python
Can slide, dice, aggregate and do math to produce your alerts
If it’s computable, Prometheus can compute it*
Alert on what matters, not on what your monitoring limit you to!
* Halting, tractability and sanity not guaranteed
Links!
Blog Posts: www.robustperception.io/tag/prometheus
Demo: demo.robustperception.io
Python Client: https://github.com/prometheus/client_python
Project Website: prometheus.io

More Related Content

What's hot

Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)Brian Brazil
 
Monitoring Kubernetes with Prometheus
Monitoring Kubernetes with PrometheusMonitoring Kubernetes with Prometheus
Monitoring Kubernetes with PrometheusGrafana Labs
 
Fall in Love with Graphs and Metrics using Grafana
Fall in Love with Graphs and Metrics using GrafanaFall in Love with Graphs and Metrics using Grafana
Fall in Love with Graphs and Metrics using Grafanatorkelo
 
End to-end monitoring with the prometheus operator - Max Inden
End to-end monitoring with the prometheus operator - Max IndenEnd to-end monitoring with the prometheus operator - Max Inden
End to-end monitoring with the prometheus operator - Max IndenParis Container Day
 
Cloud Monitoring with Prometheus
Cloud Monitoring with PrometheusCloud Monitoring with Prometheus
Cloud Monitoring with PrometheusQAware GmbH
 
Infrastructure & System Monitoring using Prometheus
Infrastructure & System Monitoring using PrometheusInfrastructure & System Monitoring using Prometheus
Infrastructure & System Monitoring using PrometheusMarco Pas
 
OpenTelemetry For Architects
OpenTelemetry For ArchitectsOpenTelemetry For Architects
OpenTelemetry For ArchitectsKevin Brockhoff
 
How to monitor your micro-service with Prometheus?
How to monitor your micro-service with Prometheus?How to monitor your micro-service with Prometheus?
How to monitor your micro-service with Prometheus?Wojciech Barczyński
 
Grafana introduction
Grafana introductionGrafana introduction
Grafana introductionRico Chen
 
Getting Started Monitoring with Prometheus and Grafana
Getting Started Monitoring with Prometheus and GrafanaGetting Started Monitoring with Prometheus and Grafana
Getting Started Monitoring with Prometheus and GrafanaSyah Dwi Prihatmoko
 
Monitoring With Prometheus
Monitoring With PrometheusMonitoring With Prometheus
Monitoring With PrometheusKnoldus Inc.
 
Monitoring microservices with Prometheus
Monitoring microservices with PrometheusMonitoring microservices with Prometheus
Monitoring microservices with PrometheusTobias Schmidt
 
Prometheus - Intro, CNCF, TSDB,PromQL,Grafana
Prometheus - Intro, CNCF, TSDB,PromQL,GrafanaPrometheus - Intro, CNCF, TSDB,PromQL,Grafana
Prometheus - Intro, CNCF, TSDB,PromQL,GrafanaSridhar Kumar N
 
Airflow Best Practises & Roadmap to Airflow 2.0
Airflow Best Practises & Roadmap to Airflow 2.0Airflow Best Practises & Roadmap to Airflow 2.0
Airflow Best Practises & Roadmap to Airflow 2.0Kaxil Naik
 
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)Brian Brazil
 
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEOClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEOAltinity Ltd
 

What's hot (20)

Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
 
Monitoring Kubernetes with Prometheus
Monitoring Kubernetes with PrometheusMonitoring Kubernetes with Prometheus
Monitoring Kubernetes with Prometheus
 
Fall in Love with Graphs and Metrics using Grafana
Fall in Love with Graphs and Metrics using GrafanaFall in Love with Graphs and Metrics using Grafana
Fall in Love with Graphs and Metrics using Grafana
 
End to-end monitoring with the prometheus operator - Max Inden
End to-end monitoring with the prometheus operator - Max IndenEnd to-end monitoring with the prometheus operator - Max Inden
End to-end monitoring with the prometheus operator - Max Inden
 
Cloud Monitoring with Prometheus
Cloud Monitoring with PrometheusCloud Monitoring with Prometheus
Cloud Monitoring with Prometheus
 
Prometheus and Grafana
Prometheus and GrafanaPrometheus and Grafana
Prometheus and Grafana
 
Infrastructure & System Monitoring using Prometheus
Infrastructure & System Monitoring using PrometheusInfrastructure & System Monitoring using Prometheus
Infrastructure & System Monitoring using Prometheus
 
Monitoring With Prometheus
Monitoring With PrometheusMonitoring With Prometheus
Monitoring With Prometheus
 
OpenTelemetry For Architects
OpenTelemetry For ArchitectsOpenTelemetry For Architects
OpenTelemetry For Architects
 
How to monitor your micro-service with Prometheus?
How to monitor your micro-service with Prometheus?How to monitor your micro-service with Prometheus?
How to monitor your micro-service with Prometheus?
 
Grafana introduction
Grafana introductionGrafana introduction
Grafana introduction
 
Getting Started Monitoring with Prometheus and Grafana
Getting Started Monitoring with Prometheus and GrafanaGetting Started Monitoring with Prometheus and Grafana
Getting Started Monitoring with Prometheus and Grafana
 
Monitoring With Prometheus
Monitoring With PrometheusMonitoring With Prometheus
Monitoring With Prometheus
 
Cloud Monitoring tool Grafana
Cloud Monitoring  tool Grafana Cloud Monitoring  tool Grafana
Cloud Monitoring tool Grafana
 
Grafana.pptx
Grafana.pptxGrafana.pptx
Grafana.pptx
 
Monitoring microservices with Prometheus
Monitoring microservices with PrometheusMonitoring microservices with Prometheus
Monitoring microservices with Prometheus
 
Prometheus - Intro, CNCF, TSDB,PromQL,Grafana
Prometheus - Intro, CNCF, TSDB,PromQL,GrafanaPrometheus - Intro, CNCF, TSDB,PromQL,Grafana
Prometheus - Intro, CNCF, TSDB,PromQL,Grafana
 
Airflow Best Practises & Roadmap to Airflow 2.0
Airflow Best Practises & Roadmap to Airflow 2.0Airflow Best Practises & Roadmap to Airflow 2.0
Airflow Best Practises & Roadmap to Airflow 2.0
 
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)
 
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEOClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
ClickHouse Query Performance Tips and Tricks, by Robert Hodges, Altinity CEO
 

Similar to Better Monitoring for Python: Inclusive Monitoring with Prometheus (Pycon Ireland 2015, Lightning Talk)

Prometheus (Microsoft, 2016)
Prometheus (Microsoft, 2016)Prometheus (Microsoft, 2016)
Prometheus (Microsoft, 2016)Brian Brazil
 
Microservices and Prometheus (Microservices NYC 2016)
Microservices and Prometheus (Microservices NYC 2016)Microservices and Prometheus (Microservices NYC 2016)
Microservices and Prometheus (Microservices NYC 2016)Brian Brazil
 
Prometheus: A Next Generation Monitoring System (FOSDEM 2016)
Prometheus: A Next Generation Monitoring System (FOSDEM 2016)Prometheus: A Next Generation Monitoring System (FOSDEM 2016)
Prometheus: A Next Generation Monitoring System (FOSDEM 2016)Brian Brazil
 
Open Source ERP Technologies for Java Developers
Open Source ERP Technologies for Java DevelopersOpen Source ERP Technologies for Java Developers
Open Source ERP Technologies for Java Developerscboecking
 
PyCon AU 2012 - Debugging Live Python Web Applications
PyCon AU 2012 - Debugging Live Python Web ApplicationsPyCon AU 2012 - Debugging Live Python Web Applications
PyCon AU 2012 - Debugging Live Python Web ApplicationsGraham Dumpleton
 
Debugging Planning Issues Using Calcite's Built-in Loggers
Debugging Planning Issues Using Calcite's Built-in LoggersDebugging Planning Issues Using Calcite's Built-in Loggers
Debugging Planning Issues Using Calcite's Built-in LoggersStamatis Zampetakis
 
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)Brian Brazil
 
Performance eng prakash.sahu
Performance eng prakash.sahuPerformance eng prakash.sahu
Performance eng prakash.sahuDr. Prakash Sahu
 
performancetestingjmeter-121109061704-phpapp02
performancetestingjmeter-121109061704-phpapp02performancetestingjmeter-121109061704-phpapp02
performancetestingjmeter-121109061704-phpapp02Gopi Raghavendra
 
performancetestingjmeter-121109061704-phpapp02 (1)
performancetestingjmeter-121109061704-phpapp02 (1)performancetestingjmeter-121109061704-phpapp02 (1)
performancetestingjmeter-121109061704-phpapp02 (1)QA Programmer
 
Performance and Scalability Testing with Python and Multi-Mechanize
Performance and Scalability Testing with Python and Multi-MechanizePerformance and Scalability Testing with Python and Multi-Mechanize
Performance and Scalability Testing with Python and Multi-Mechanizecoreygoldberg
 
TDC 2015 - POA - Trilha PHP - Shit Happens
TDC 2015 - POA - Trilha PHP - Shit HappensTDC 2015 - POA - Trilha PHP - Shit Happens
TDC 2015 - POA - Trilha PHP - Shit HappensJackson F. de A. Mafra
 
Iphone client-server app with Rails backend (v3)
Iphone client-server app with Rails backend (v3)Iphone client-server app with Rails backend (v3)
Iphone client-server app with Rails backend (v3)Sujee Maniyam
 
Dev ops presentation
Dev ops presentationDev ops presentation
Dev ops presentationAhmed Kamel
 
Monitor your Java application with Prometheus Stack
Monitor your Java application with Prometheus StackMonitor your Java application with Prometheus Stack
Monitor your Java application with Prometheus StackWojciech Barczyński
 
Monitoring und Metriken im Wunderland
Monitoring und Metriken im WunderlandMonitoring und Metriken im Wunderland
Monitoring und Metriken im WunderlandD
 
HFM API Deep Dive – Making a Better Financial Management Client
HFM API Deep Dive – Making a Better Financial Management ClientHFM API Deep Dive – Making a Better Financial Management Client
HFM API Deep Dive – Making a Better Financial Management ClientCharles Beyer
 
Performance Instrumentation for PL/SQL: When, Why, How
Performance Instrumentation for PL/SQL: When, Why, HowPerformance Instrumentation for PL/SQL: When, Why, How
Performance Instrumentation for PL/SQL: When, Why, HowKaren Morton
 
'BIG Testing' with Hans Buwalda
'BIG Testing' with Hans Buwalda 'BIG Testing' with Hans Buwalda
'BIG Testing' with Hans Buwalda TEST Huddle
 

Similar to Better Monitoring for Python: Inclusive Monitoring with Prometheus (Pycon Ireland 2015, Lightning Talk) (20)

Prometheus (Microsoft, 2016)
Prometheus (Microsoft, 2016)Prometheus (Microsoft, 2016)
Prometheus (Microsoft, 2016)
 
Microservices and Prometheus (Microservices NYC 2016)
Microservices and Prometheus (Microservices NYC 2016)Microservices and Prometheus (Microservices NYC 2016)
Microservices and Prometheus (Microservices NYC 2016)
 
Prometheus: A Next Generation Monitoring System (FOSDEM 2016)
Prometheus: A Next Generation Monitoring System (FOSDEM 2016)Prometheus: A Next Generation Monitoring System (FOSDEM 2016)
Prometheus: A Next Generation Monitoring System (FOSDEM 2016)
 
Open Source ERP Technologies for Java Developers
Open Source ERP Technologies for Java DevelopersOpen Source ERP Technologies for Java Developers
Open Source ERP Technologies for Java Developers
 
PyCon AU 2012 - Debugging Live Python Web Applications
PyCon AU 2012 - Debugging Live Python Web ApplicationsPyCon AU 2012 - Debugging Live Python Web Applications
PyCon AU 2012 - Debugging Live Python Web Applications
 
Debugging Planning Issues Using Calcite's Built-in Loggers
Debugging Planning Issues Using Calcite's Built-in LoggersDebugging Planning Issues Using Calcite's Built-in Loggers
Debugging Planning Issues Using Calcite's Built-in Loggers
 
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
 
Performance eng prakash.sahu
Performance eng prakash.sahuPerformance eng prakash.sahu
Performance eng prakash.sahu
 
Kscope presentation 2013
Kscope presentation 2013Kscope presentation 2013
Kscope presentation 2013
 
performancetestingjmeter-121109061704-phpapp02
performancetestingjmeter-121109061704-phpapp02performancetestingjmeter-121109061704-phpapp02
performancetestingjmeter-121109061704-phpapp02
 
performancetestingjmeter-121109061704-phpapp02 (1)
performancetestingjmeter-121109061704-phpapp02 (1)performancetestingjmeter-121109061704-phpapp02 (1)
performancetestingjmeter-121109061704-phpapp02 (1)
 
Performance and Scalability Testing with Python and Multi-Mechanize
Performance and Scalability Testing with Python and Multi-MechanizePerformance and Scalability Testing with Python and Multi-Mechanize
Performance and Scalability Testing with Python and Multi-Mechanize
 
TDC 2015 - POA - Trilha PHP - Shit Happens
TDC 2015 - POA - Trilha PHP - Shit HappensTDC 2015 - POA - Trilha PHP - Shit Happens
TDC 2015 - POA - Trilha PHP - Shit Happens
 
Iphone client-server app with Rails backend (v3)
Iphone client-server app with Rails backend (v3)Iphone client-server app with Rails backend (v3)
Iphone client-server app with Rails backend (v3)
 
Dev ops presentation
Dev ops presentationDev ops presentation
Dev ops presentation
 
Monitor your Java application with Prometheus Stack
Monitor your Java application with Prometheus StackMonitor your Java application with Prometheus Stack
Monitor your Java application with Prometheus Stack
 
Monitoring und Metriken im Wunderland
Monitoring und Metriken im WunderlandMonitoring und Metriken im Wunderland
Monitoring und Metriken im Wunderland
 
HFM API Deep Dive – Making a Better Financial Management Client
HFM API Deep Dive – Making a Better Financial Management ClientHFM API Deep Dive – Making a Better Financial Management Client
HFM API Deep Dive – Making a Better Financial Management Client
 
Performance Instrumentation for PL/SQL: When, Why, How
Performance Instrumentation for PL/SQL: When, Why, HowPerformance Instrumentation for PL/SQL: When, Why, How
Performance Instrumentation for PL/SQL: When, Why, How
 
'BIG Testing' with Hans Buwalda
'BIG Testing' with Hans Buwalda 'BIG Testing' with Hans Buwalda
'BIG Testing' with Hans Buwalda
 

More from Brian Brazil

OpenMetrics: What Does It Mean for You (PromCon 2019, Munich)
OpenMetrics: What Does It Mean for You (PromCon 2019, Munich)OpenMetrics: What Does It Mean for You (PromCon 2019, Munich)
OpenMetrics: What Does It Mean for You (PromCon 2019, Munich)Brian Brazil
 
Evolution of Monitoring and Prometheus (Dublin 2018)
Evolution of Monitoring and Prometheus (Dublin 2018)Evolution of Monitoring and Prometheus (Dublin 2018)
Evolution of Monitoring and Prometheus (Dublin 2018)Brian Brazil
 
Evaluating Prometheus Knowledge in Interviews (PromCon 2018)
Evaluating Prometheus Knowledge in Interviews (PromCon 2018)Evaluating Prometheus Knowledge in Interviews (PromCon 2018)
Evaluating Prometheus Knowledge in Interviews (PromCon 2018)Brian Brazil
 
Anatomy of a Prometheus Client Library (PromCon 2018)
Anatomy of a Prometheus Client Library (PromCon 2018)Anatomy of a Prometheus Client Library (PromCon 2018)
Anatomy of a Prometheus Client Library (PromCon 2018)Brian Brazil
 
Prometheus for Monitoring Metrics (Fermilab 2018)
Prometheus for Monitoring Metrics (Fermilab 2018)Prometheus for Monitoring Metrics (Fermilab 2018)
Prometheus for Monitoring Metrics (Fermilab 2018)Brian Brazil
 
Evolving Prometheus for the Cloud Native World (FOSDEM 2018)
Evolving Prometheus for the Cloud Native World (FOSDEM 2018)Evolving Prometheus for the Cloud Native World (FOSDEM 2018)
Evolving Prometheus for the Cloud Native World (FOSDEM 2018)Brian Brazil
 
Prometheus for Monitoring Metrics (Percona Live Europe 2017)
Prometheus for Monitoring Metrics (Percona Live Europe 2017)Prometheus for Monitoring Metrics (Percona Live Europe 2017)
Prometheus for Monitoring Metrics (Percona Live Europe 2017)Brian Brazil
 
Evolution of the Prometheus TSDB (Percona Live Europe 2017)
Evolution of the Prometheus TSDB  (Percona Live Europe 2017)Evolution of the Prometheus TSDB  (Percona Live Europe 2017)
Evolution of the Prometheus TSDB (Percona Live Europe 2017)Brian Brazil
 
Staleness and Isolation in Prometheus 2.0 (PromCon 2017)
Staleness and Isolation in Prometheus 2.0 (PromCon 2017)Staleness and Isolation in Prometheus 2.0 (PromCon 2017)
Staleness and Isolation in Prometheus 2.0 (PromCon 2017)Brian Brazil
 
Rule 110 for Prometheus (PromCon 2017)
Rule 110 for Prometheus (PromCon 2017)Rule 110 for Prometheus (PromCon 2017)
Rule 110 for Prometheus (PromCon 2017)Brian Brazil
 
Counting with Prometheus (CloudNativeCon+Kubecon Europe 2017)
Counting with Prometheus (CloudNativeCon+Kubecon Europe 2017)Counting with Prometheus (CloudNativeCon+Kubecon Europe 2017)
Counting with Prometheus (CloudNativeCon+Kubecon Europe 2017)Brian Brazil
 
Prometheus: From Berlin to Bonanza (Keynote CloudNativeCon+Kubecon Europe 2017)
Prometheus:  From Berlin to Bonanza (Keynote CloudNativeCon+Kubecon Europe 2017)Prometheus:  From Berlin to Bonanza (Keynote CloudNativeCon+Kubecon Europe 2017)
Prometheus: From Berlin to Bonanza (Keynote CloudNativeCon+Kubecon Europe 2017)Brian Brazil
 
What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)Brian Brazil
 
An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)Brian Brazil
 
Provisioning and Capacity Planning (Travel Meets Big Data)
Provisioning and Capacity Planning (Travel Meets Big Data)Provisioning and Capacity Planning (Travel Meets Big Data)
Provisioning and Capacity Planning (Travel Meets Big Data)Brian Brazil
 
Prometheus - Open Source Forum Japan
Prometheus  - Open Source Forum JapanPrometheus  - Open Source Forum Japan
Prometheus - Open Source Forum JapanBrian Brazil
 
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...Brian Brazil
 
So You Want to Write an Exporter
So You Want to Write an ExporterSo You Want to Write an Exporter
So You Want to Write an ExporterBrian Brazil
 
An Exploration of the Formal Properties of PromQL
An Exploration of the Formal Properties of PromQLAn Exploration of the Formal Properties of PromQL
An Exploration of the Formal Properties of PromQLBrian Brazil
 
Life of a Label (PromCon2016, Berlin)
Life of a Label (PromCon2016, Berlin)Life of a Label (PromCon2016, Berlin)
Life of a Label (PromCon2016, Berlin)Brian Brazil
 

More from Brian Brazil (20)

OpenMetrics: What Does It Mean for You (PromCon 2019, Munich)
OpenMetrics: What Does It Mean for You (PromCon 2019, Munich)OpenMetrics: What Does It Mean for You (PromCon 2019, Munich)
OpenMetrics: What Does It Mean for You (PromCon 2019, Munich)
 
Evolution of Monitoring and Prometheus (Dublin 2018)
Evolution of Monitoring and Prometheus (Dublin 2018)Evolution of Monitoring and Prometheus (Dublin 2018)
Evolution of Monitoring and Prometheus (Dublin 2018)
 
Evaluating Prometheus Knowledge in Interviews (PromCon 2018)
Evaluating Prometheus Knowledge in Interviews (PromCon 2018)Evaluating Prometheus Knowledge in Interviews (PromCon 2018)
Evaluating Prometheus Knowledge in Interviews (PromCon 2018)
 
Anatomy of a Prometheus Client Library (PromCon 2018)
Anatomy of a Prometheus Client Library (PromCon 2018)Anatomy of a Prometheus Client Library (PromCon 2018)
Anatomy of a Prometheus Client Library (PromCon 2018)
 
Prometheus for Monitoring Metrics (Fermilab 2018)
Prometheus for Monitoring Metrics (Fermilab 2018)Prometheus for Monitoring Metrics (Fermilab 2018)
Prometheus for Monitoring Metrics (Fermilab 2018)
 
Evolving Prometheus for the Cloud Native World (FOSDEM 2018)
Evolving Prometheus for the Cloud Native World (FOSDEM 2018)Evolving Prometheus for the Cloud Native World (FOSDEM 2018)
Evolving Prometheus for the Cloud Native World (FOSDEM 2018)
 
Prometheus for Monitoring Metrics (Percona Live Europe 2017)
Prometheus for Monitoring Metrics (Percona Live Europe 2017)Prometheus for Monitoring Metrics (Percona Live Europe 2017)
Prometheus for Monitoring Metrics (Percona Live Europe 2017)
 
Evolution of the Prometheus TSDB (Percona Live Europe 2017)
Evolution of the Prometheus TSDB  (Percona Live Europe 2017)Evolution of the Prometheus TSDB  (Percona Live Europe 2017)
Evolution of the Prometheus TSDB (Percona Live Europe 2017)
 
Staleness and Isolation in Prometheus 2.0 (PromCon 2017)
Staleness and Isolation in Prometheus 2.0 (PromCon 2017)Staleness and Isolation in Prometheus 2.0 (PromCon 2017)
Staleness and Isolation in Prometheus 2.0 (PromCon 2017)
 
Rule 110 for Prometheus (PromCon 2017)
Rule 110 for Prometheus (PromCon 2017)Rule 110 for Prometheus (PromCon 2017)
Rule 110 for Prometheus (PromCon 2017)
 
Counting with Prometheus (CloudNativeCon+Kubecon Europe 2017)
Counting with Prometheus (CloudNativeCon+Kubecon Europe 2017)Counting with Prometheus (CloudNativeCon+Kubecon Europe 2017)
Counting with Prometheus (CloudNativeCon+Kubecon Europe 2017)
 
Prometheus: From Berlin to Bonanza (Keynote CloudNativeCon+Kubecon Europe 2017)
Prometheus:  From Berlin to Bonanza (Keynote CloudNativeCon+Kubecon Europe 2017)Prometheus:  From Berlin to Bonanza (Keynote CloudNativeCon+Kubecon Europe 2017)
Prometheus: From Berlin to Bonanza (Keynote CloudNativeCon+Kubecon Europe 2017)
 
What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)
 
An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)
 
Provisioning and Capacity Planning (Travel Meets Big Data)
Provisioning and Capacity Planning (Travel Meets Big Data)Provisioning and Capacity Planning (Travel Meets Big Data)
Provisioning and Capacity Planning (Travel Meets Big Data)
 
Prometheus - Open Source Forum Japan
Prometheus  - Open Source Forum JapanPrometheus  - Open Source Forum Japan
Prometheus - Open Source Forum Japan
 
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...
 
So You Want to Write an Exporter
So You Want to Write an ExporterSo You Want to Write an Exporter
So You Want to Write an Exporter
 
An Exploration of the Formal Properties of PromQL
An Exploration of the Formal Properties of PromQLAn Exploration of the Formal Properties of PromQL
An Exploration of the Formal Properties of PromQL
 
Life of a Label (PromCon2016, Berlin)
Life of a Label (PromCon2016, Berlin)Life of a Label (PromCon2016, Berlin)
Life of a Label (PromCon2016, Berlin)
 

Recently uploaded

Summary IGF 2013 Bali - English (tata kelola internet / internet governance)
Summary  IGF 2013 Bali - English (tata kelola internet / internet governance)Summary  IGF 2013 Bali - English (tata kelola internet / internet governance)
Summary IGF 2013 Bali - English (tata kelola internet / internet governance)ICT Watch - Indonesia
 
Power of Social Media for E-commerce.pdf
Power of Social Media for E-commerce.pdfPower of Social Media for E-commerce.pdf
Power of Social Media for E-commerce.pdfrajats19920
 
Summary ID-IGF 2016 National Dialogue - English (tata kelola internet / int...
Summary  ID-IGF 2016 National Dialogue  - English (tata kelola internet / int...Summary  ID-IGF 2016 National Dialogue  - English (tata kelola internet / int...
Summary ID-IGF 2016 National Dialogue - English (tata kelola internet / int...ICT Watch - Indonesia
 
IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119APNIC
 
Tari Eason Warriors Come Out To Play T Shirts
Tari Eason Warriors Come Out To Play T ShirtsTari Eason Warriors Come Out To Play T Shirts
Tari Eason Warriors Come Out To Play T Shirtsrahman018755
 
办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...
办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...
办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...vmzoxnx5
 

Recently uploaded (6)

Summary IGF 2013 Bali - English (tata kelola internet / internet governance)
Summary  IGF 2013 Bali - English (tata kelola internet / internet governance)Summary  IGF 2013 Bali - English (tata kelola internet / internet governance)
Summary IGF 2013 Bali - English (tata kelola internet / internet governance)
 
Power of Social Media for E-commerce.pdf
Power of Social Media for E-commerce.pdfPower of Social Media for E-commerce.pdf
Power of Social Media for E-commerce.pdf
 
Summary ID-IGF 2016 National Dialogue - English (tata kelola internet / int...
Summary  ID-IGF 2016 National Dialogue  - English (tata kelola internet / int...Summary  ID-IGF 2016 National Dialogue  - English (tata kelola internet / int...
Summary ID-IGF 2016 National Dialogue - English (tata kelola internet / int...
 
IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119IP addressing and IPv6, presented by Paul Wilson at IETF 119
IP addressing and IPv6, presented by Paul Wilson at IETF 119
 
Tari Eason Warriors Come Out To Play T Shirts
Tari Eason Warriors Come Out To Play T ShirtsTari Eason Warriors Come Out To Play T Shirts
Tari Eason Warriors Come Out To Play T Shirts
 
办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...
办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...
办理澳洲USYD文凭证书学历认证【Q微/1954292140】办理悉尼大学毕业证书真实成绩单GPA修改/办理澳洲大学文凭证书Offer录取通知书/在读证明...
 

Better Monitoring for Python: Inclusive Monitoring with Prometheus (Pycon Ireland 2015, Lightning Talk)

  • 1. Better Monitoring for Python Inclusive Monitoring with Prometheus
  • 2. Monitoring is often a Problem, not a Solution Themes common among companies I talk to: ● Monitoring tools are limited, both technically and conceptually ● Tools don’t scale well and are unwieldy to manage ● Operational practices don’t align with the business Result: Engineers continuously woken up for non-issues, get fatigued
  • 3. Fundamental Challenge is Limited Visibility
  • 4. Prometheus Inspired by Google’s Borgmon monitoring system. Started in 2012 by ex-Googlers working in Soundcloud as an open source project. Mainly written in Go. Publically launched in early 2015. 100+ companies using it including Digital Ocean, GoPro, Apple, Red Hat and Google.
  • 7. Monitor as a Service, not as Machines
  • 8. Inclusive Monitoring Don’t monitor just at the edges: ● Instrument client libraries ● Instrument server libraries (e.g. HTTP/RPC) ● Instrument business logic Library authors get information about usage. Application developers get monitoring of common components for free. Dashboards and alerting can be provided out of the box, customised for your organisation!
  • 9. Let’s Talk Code pip install prometheus_client from prometheus_client import Summary, start_http_server REQUEST_DURATION = Summary('request_duration_seconds', 'Request duration in seconds') @REQUEST_DURATION.time() def my_handler(request): pass // Your code here start_http_server(8000)
  • 10. Multiple Dimensions (No Evil Twins Please) from prometheus_client import Counter REQUESTS = Summary('requests_total', 'Total requests', ['method']) def my_handler(request): REQUESTS.labels(request.method).inc() pass // Your code here
  • 11. Exceptional Circumstances In Progress from prometheus_client import Counter, Gauge EXCEPTIONS = Counter('exceptions_total', 'Total exceptions') IN_PROGRESS = Gauge('inprogress_requests', 'In progress') @EXCEPTIONS.count_exceptions() @IN_PROGRESS.track_inprogress() def my_handler(request): pass // Your code here
  • 12. Getting Data Out from prometheus_client import start_http_server if __name__ == '__main__': start_http_server(8080) Also possible with Django, Twisted etc.
  • 13. Oh Noes, My Vendor Lock In! from prometheus_client.bridge.graphite import GraphiteBridge gb = GraphiteBridge(('graphite.your.org', 2003)) gb.start(10.0) # Push every 10 seconds Exposition to Prometheus doesn’t use a special API, can be hooked into whatever monitoring system you have. Instrument once, work with everything!
  • 14. Running Prometheus Grab binary from http://www.robustperception.io/prometheus-nightly-binaries/ Put config in prometheus.yml: scrape_configs: - job_name: python target_groups: targets: [localhost:8000] Run it: ./prometheus
  • 16. Prometheus is as Powerful as Python Can slide, dice, aggregate and do math to produce your alerts If it’s computable, Prometheus can compute it* Alert on what matters, not on what your monitoring limit you to! * Halting, tractability and sanity not guaranteed
  • 17. Links! Blog Posts: www.robustperception.io/tag/prometheus Demo: demo.robustperception.io Python Client: https://github.com/prometheus/client_python Project Website: prometheus.io