Prometheus is a Cloud-Native Data-Centric Open-
Source Performant Simple metrics collection,
analysis and alerting tool.
Very easy to configure, deploy, maintain
Designed in multiple services
Orchestration ready (dynamic config)
Support for multiple OS
Prometheus is designed to fetch data in an
interval measured in SECONDS
Designed to handle lots of metrics
New storage engine in 2.0 to scale even better
and support usecases like kubernetes
A Metric in Prometheus has metadata:
And lots of function to filter, change, remove...
those metadata while fetching them.
Counters (always go up)
Gauge (go up and down)
Histograms (aggregate by buckets)
Summary (percentiles) (most of the time
A word about
Prometheus vs Graphite
Prometheus does not see a metric as an "event".
Metrics are current value until they are replaced.
You can not see when a metric has been included
For Events, Prometheus refers to Elasticsearch.
Prometheus uses HTTP and
http(s) is supported
basic auth / token also
(exposing https is not Prometheus job)
$ curl http://127.0.0.1:9090/metrics
# HELP prometheus_notifications_queue_length
# The number of alert notifications in the queue.
# TYPE prometheus_notifications_queue_length gauge
# HELP prometheus_notifications_sent_total
# Total number of alerts successfully sent.
# TYPE prometheus_notifications_sent_total counter
Exporters expose metrics with an HTTP API
They connect to the real target
Bindings available for many languages
Exporters do not save data ; they are not
"proxies" and don't "cache" anything
MySQL Master <-> MySQL Master
MySQL Master -> MySQL Slave
MySQL Master -> MySQL Slave -> MySQL
MySQL Masters -> MySQL Slaves -> MySQL
Slaves -> MySQL Slaves
MySQL Master -> MySQL Slaves
pt-heartbeart is a daemon that updates an entry
with current timestamp on a mysql server every
On the replica, you can check the timestamp and
do �NOW���timestamp�to get the real lag.
When prometheus has an alert, it sends it.
Every minute by default, as long as the alert is
Alertmanager, a separated daemon, do the
rest of the work
What is alertmanager doing?
5 nodes are down. Do you want 5 email?
Datacenter is on fire. Do you want to know that
switches, hosts, services are down?
I upgrade kernel and reboot the service. Do I want
mails during that time?
Alertmanager can notify you with:
3rd party services
SMS (sachet: 3rd party integration using
You can send alerts to defined people based
Everything to logs mailbox
Network to net team SMS
Svc to app team SMS
Network to net team mail
Svc to app team mail
High Availability: Prometheus
Have different prometheis servers with the
They do not talk to each other
All of them fetch the same data
They monitor each other
High availability: Alertmanager
Have multiple alertmanagers with the same
Prometheis send alerts to all alertmanagers
Alert manager talk to each other not to send
the same notification
One tool does one job...
Prometheus will collect data
Alertmanager will send notifications
Exporters will expose data
Grafana will graph data
Open Source (Apache 2.0)
Specialized in visualization
Multiple datasources: prometheus, graphite,
Has an API!
History of Grafana
Grafana is a fork of Kibana 3 ; used to be JS-
Now fully featured, requires a database, multi-
projects/users support, etc...
Grafana and Prometheus
Prometheus shipped its own consoles
Now it recommends Grafana and deprecated
its own consoles
Multiple Prometheus instances
You can add multiple prometheus instances
You can add dropdown on the top to select
which one you want to use
Use case: prometheus HA, local prometheus
(with access mode=direct)
Creating Grafana Dashboards
Requires deep knowledge of the tools
Improved over time
Easy to share (json + online library)
Percona Grafana Dashboard
Percona Open Sourced Grafana Dashboards
Covering MySQL, Mongo and Linux monitoring
Part of a bigger picture, PMM, but usable
Open Source (AGPL!)
Installing Percona Graphes
Enable File dashboards in Grafana
Clone grafana-dashboards to the configured
location (or make a package)
Use the Grafana API to upload the JSON's.
You'll need mysqld_exporter, with a user
Performance Schema for full set of metrics
Other grafana features
Folders for dashboards (comming soon)
Jsonnet library to create dashboard
Brand new, still WIP
Prometheus and Grafana are first-class
Totally different approach than other tools
Embeddable into your apps
Percona Dashboards gets your graphes ready
in no-time with minimal efforts