SlideShare a Scribd company logo
1 of 36
Download to read offline
Monitoring With Prometheus
Richard Langlois P. Eng. and Gervais Naoussi, Sept. 2018
Agenda
2
• Monitoring Concepts
• Micrometer
• Prometheus
• Grafana
• Demo
Monitoring
3
Monitoring is the tools and processes by which you measure your technology systems.
A monitoring system has two customers:
• Technology (Engineering, Operations, DevOps)
• The business (measure the value that technology delivers to business)
If you’re building a specification or user stories for your application:
include metrics and monitoring for each component of your application.
Don’t wait until the end of a project or just before deployment.
Monitoring
Approach to Monitoring
4
A good approach to your monitoring is to design a top-down monitoring plan based on
value.
Identify the parts of the application that deliver value and monitor those first, working
your way down the stack.
Monitoring for the correctness of a service first
 e.g. monitor the content or rates of a business transaction rather than the uptime
of the web server it runs on.
Monitoring
Monitoring Approaches
5
2 major approaches:
• Probing monitoring probes the outside of an application (black-box monitoring).
e.g. Nagios
• Introspection monitoring looks at what’s inside the application (white-box monitoring)
application is instrumented and returns measurements of its state
Monitoring
Pull vs Push
6
Two approaches to how monitoring checks are executed:
• Pull-based: systems scrape or check a remote application—for example, an endpoint
containing metrics.
• Push-based: applications emit events that are received by the monitoring system.
Prometheus is primarily a pull-based system, but it also supports receiving events pushed
into a gateway.
Monitoring
Metric
7
Metrics are measures of properties of components of software or hardware.
To make a metric useful we keep track of its state, generally recording data points over time (called
observations).
An observation consists of:
value,
a timestamp,
and sometimes a series of properties that describe the observation such as a source or tags.
A collection of observations is called a time series.
Time series data is a chronologically ordered list of these observations.
Time series metrics are often visualized as a two-dimensional plot with data values on the y-axis and
time on the x-axis.
Monitoring
Types of monitoring data
8
Monitoring tools can collect 2 types of data:
• Metrics are stored as time series data that record the state of measures of your
applications.
• Logs are (usually textual) events emitted from an application.
Prometheus is primarily focused on collecting time series data.
Monitoring
Type of Metrics
9
Variety of different types of metrics:
Gauges: are numbers that are expected to go up or down. A snapshot of a specific measurement.
 e.g. Disk usage, number of customers present on a site.
Counters: are numbers that increase over time and never decrease.
 e.g. system uptime, number of sales in a month
Histograms: is a metric that samples observations. Each observation is counted and placed into
buckets.
Metric Summaries: mathematical transformations applied to metrics
• Average
• Median
• Standard Deviation
• Percentile
Agenda
10
• Monitoring Concepts
• Micrometer
• Prometheus
• Grafana
• Demo
Micrometer
Intro
11
Micrometer is a metrics instrumentation library allowing to instrument JVM-based
application code without vendor lock-in.
It provides a simple façade over the instrumentation clients for the most popular monitoring
systems.
Think SLF4J, but for application metrics.
As of Spring Boot 2.0.0.M5, Micrometer is the instrumentation library used by Spring.
Some supported monitoring systems:
• Datadog
• Graphite
• Influx
• JMX
• New Relic
• Prometheus
• SignalFX
• StatsD
Micrometer
API
12
Meter is the interface for collecting a set of measurements (called metrics).
MeterRegistry: eters are created from and held in a Meter Registry
Each supported monitoring system has an implementation of MeterRegistry.
SimpleMeterRegistry: Automatically autowired in Spring-based apps.
MeterRegistry registry = new SimpleMeterRegistry.
Set of meter primitives:
Timer, Counter, Gauge, DistributionSummary, LongTaskTimer, FunctionCounter, FunctionTimer
and TimeGauge.
Dimensions allow a particular named metric to be sliced to drill down.
E.g. Registry.counter(“http.server.requests”, “uri”, “/api/users”)
Fluent builder:
Counter counter = Counter .builder("counter")
.baseUnit(“ms")
.description("a description of what this counter does")
.tags("region", "test")
.register(registry);
Agenda
13
• Monitoring Concepts
• Micrometer
• Prometheus
• Grafana
• Demo
Prometheus
Intro
14
Prometheus is a simple, effective open-source monitoring system.
Promoted from incubation to graduation (in August 2018), in Cloud Native Computing Foundation (CNCF)
Prometheus works by scraping (pulling) time series data exposed from applications.
The time series data is exposed by the applications themselves often via client libraries or via proxies
called exporters, as HTTP endpoints.
Prometheus
Concepts
15
Prometheus calls the source of metrics it can scrape endpoints.
An endpoint usually corresponds to a single process, host, service, or application.
The resulting time series data is collected
and stored locally on the Prometheus server (15 days retention)
and can be sent from the server to external storage or to another time series database.
Prometheus can also define Rules for alerting.
Prometheus
PromQL – inbuilt querying language
16
The Prometheus server also comes with an inbuilt querying language, PromQL, allowing to
query and aggregate metrics.
Use this query language in the query input box in the Expression Browser.
e.g: Query all metrics with a label of quantile=“0.5”:
Prometheus
Expression Browser
17
http://localhost:9090/graph
Prometheus
Scalability
18
Designed to scale to millions of time series from many thousands of hosts.
Its data storage format is designed to keep disk use down and provide fast retrieval of time series
during queries and aggregations.
SSD disks are recommended for Prometheus servers, for speed and reliability.
Redundant Prometheus Architecture:
Prometheus
Data Model
19
Prometheus collects time series data.
Format:
<time series name>{<label name>=<label value>, ...}
Each time series is uniquely identified by the combination of names and key/value pairs
called labels (provide the dimensions).
Name usually describes the general nature of the time series data being collected
 e.g. total_website_visits as the total number of website visits.
Labels enable the Prometheus dimensional data model, they add context to a specific
time series.
 e.g. the name of the website, IP of the requester
Prometheus
Time Series Notation
20
Example
total_website_visits{site=“alithya.com", location="NJ", instance="webserver“, job="web"}
All time series generally have
• an instance label, which identifies the source host or application
• a job label, which contains the name of the job that scraped the specific time series.
Actual value of the time series is called a sample.
Consists of:
• A float64 value.
• A millisecond-precision timestamp.
Prometheus configuration
prometheus.yml
21
Prometheus is configured via YAML configuration files.
Default Configuration file has the following 4 YAML blocks defined:
Global: contains global settings for controlling the Prometheus server’s behavior.
Alerting: configures Prometheus’ alerting.
rule_files: specifies a list of files that can contain recording or alerting rules.
scrape_configs: specifies all of the targets that Prometheus will scrape.
Prometheus and Spring Boot
22
Spring Boot auto-configures a composite MeterRegistry and adds a registry to the composite for each of the supported
implementations that it finds on the classpath.
pom.xml:
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_spring_boot</artifactId>
<version>0.1.0</version>
</dependency>
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>simpleclient_hotspot</artifactId>
<version>0.1.0</version>
</dependency
The simpleclient_spring_boot dependency provides the @EnablePrometheusEndpoint annotation.
Adding it to a @configuration class, creates a HTTP endpoint accessible via /actuator/prometheus that exposes all registered
(actuator) metrics in a Prometheus data format.
Prometheus configuration
Scrape Config for Spring Boot application
23
Prometheus scrapes the following 2 endpoints
• /prometheus endpoint: contains Spring boot metrics
• /metrics endpoint: Prometheus own metrics
scrape_configs:
# The job name is added as a label `job=<job_name>` to any time series scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['PROM_IP:9090']
- job_name: 'spring-boot'
metrics_path: '/prometheus'
scrape_interval: 5s
static_configs:
- targets: ['App_IP:8080']
Prometheus
Dashboard
24
Example of metrics on Prometheus dashboard:
Prometheus
Alerting
25
Alerting is provided by a tool called Alertmanager
Alerting rules are defined on the Prometheus server.
When the threshold or criteria is met, an alert will be generated and pushed to Alertmanager.
The alerts are received on an HTTP endpoint on the Alertmanager.
Alertmanager handles deduplicating, grouping, and routing alerts to receivers (e.g. email, SMS, PagerDuty)
Prometheus
Alerting Configuration
26
A simple alertmanager.yml configuration file, sending alerts by email:
Prometheus
Alerting Web Interface
27
Web interface to:
• view current alerts
• manage maintenance window alert suppression (silences)
Prometheus
Pushgateway
28
Metrics can be pushed to Pushgateway when there isn’t a target from which to scrape metrics because:
• can’t reach the target resources because of security
• target resource has too short a lifespan (e.g. container starting, executing, and stopping).
• target resource doesn’t have an endpoint, (e.g. batch job).
Pushgateway sits between an application sending metrics and the Prometheus server.
Pushgateway is scraped as a target to deliver the metrics to the Prometheus server.
Agenda
29
• Monitoring Concepts
• Micrometer
• Prometheus
• Grafana
• Demo
Grafana
30
Prometheus UI is not really nice.
Alternative: Grafana is open source metrics Dashboard platform.
It supports multiple backend time-series databases including:
Prometheus , InfluxDB, Elasticsearch, Cloudwatch …
Example of Grafana dashboard:
Grafana
Prometheus as Data source
31
Grafana
Prometheus as Datasource
32
Before you create your first dashboard you need to add your data source.
Grafana
Prometheus as Datasource
33
Name: your choice
Default: Check to tell Grafana to search for data in this source by default
Type: Prometheus
URL: URL of the Prometheus server to query.
Agenda
34
• Monitoring Concepts
• Micrometer
• Prometheus
• Grafana
• Demo
References
35
Micrometer:
• https://micrometer.io
Prometheus
• Book: Monitoring with Prometheus, James Turnbull, 2018.
• https://prometheus.io
Graphana
• https://grafana.com/
Thank You
36

More Related Content

What's hot

Monitoring Kubernetes with Prometheus
Monitoring Kubernetes with PrometheusMonitoring Kubernetes with Prometheus
Monitoring Kubernetes with PrometheusGrafana Labs
 
Kubernetes and Prometheus
Kubernetes and PrometheusKubernetes and Prometheus
Kubernetes and PrometheusWeaveworks
 
Introduction to Prometheus
Introduction to PrometheusIntroduction to Prometheus
Introduction to PrometheusJulien Pivotto
 
MeetUp Monitoring with Prometheus and Grafana (September 2018)
MeetUp Monitoring with Prometheus and Grafana (September 2018)MeetUp Monitoring with Prometheus and Grafana (September 2018)
MeetUp Monitoring with Prometheus and Grafana (September 2018)Lucas Jellema
 
Prometheus – a next-gen Monitoring System
Prometheus – a next-gen Monitoring SystemPrometheus – a next-gen Monitoring System
Prometheus – a next-gen Monitoring SystemFabian Reinartz
 
Infrastructure & System Monitoring using Prometheus
Infrastructure & System Monitoring using PrometheusInfrastructure & System Monitoring using Prometheus
Infrastructure & System Monitoring using PrometheusMarco Pas
 
MySQL Monitoring using Prometheus & Grafana
MySQL Monitoring using Prometheus & GrafanaMySQL Monitoring using Prometheus & Grafana
MySQL Monitoring using Prometheus & GrafanaYoungHeon (Roy) Kim
 
Prometheus in Practice: High Availability with Thanos (DevOpsDays Edinburgh 2...
Prometheus in Practice: High Availability with Thanos (DevOpsDays Edinburgh 2...Prometheus in Practice: High Availability with Thanos (DevOpsDays Edinburgh 2...
Prometheus in Practice: High Availability with Thanos (DevOpsDays Edinburgh 2...Thomas Riley
 
Kubernetes Observability with Prometheus by Example
Kubernetes Observability with Prometheus by ExampleKubernetes Observability with Prometheus by Example
Kubernetes Observability with Prometheus by ExampleThomas Riley
 
How to monitor your micro-service with Prometheus?
How to monitor your micro-service with Prometheus?How to monitor your micro-service with Prometheus?
How to monitor your micro-service with Prometheus?Wojciech Barczyński
 
Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안
Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안
Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안SANG WON PARK
 
Introduction To Terraform
Introduction To TerraformIntroduction To Terraform
Introduction To TerraformSasitha Iresh
 
Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...
Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...
Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...Amazon Web Services
 
Prometheus
PrometheusPrometheus
Prometheuswyukawa
 
End to-end monitoring with the prometheus operator - Max Inden
End to-end monitoring with the prometheus operator - Max IndenEnd to-end monitoring with the prometheus operator - Max Inden
End to-end monitoring with the prometheus operator - Max IndenParis Container Day
 

What's hot (20)

Monitoring With Prometheus
Monitoring With PrometheusMonitoring With Prometheus
Monitoring With Prometheus
 
Monitoring Kubernetes with Prometheus
Monitoring Kubernetes with PrometheusMonitoring Kubernetes with Prometheus
Monitoring Kubernetes with Prometheus
 
Kubernetes and Prometheus
Kubernetes and PrometheusKubernetes and Prometheus
Kubernetes and Prometheus
 
Introduction to Prometheus
Introduction to PrometheusIntroduction to Prometheus
Introduction to Prometheus
 
Prometheus monitoring
Prometheus monitoringPrometheus monitoring
Prometheus monitoring
 
Prometheus and Grafana
Prometheus and GrafanaPrometheus and Grafana
Prometheus and Grafana
 
MeetUp Monitoring with Prometheus and Grafana (September 2018)
MeetUp Monitoring with Prometheus and Grafana (September 2018)MeetUp Monitoring with Prometheus and Grafana (September 2018)
MeetUp Monitoring with Prometheus and Grafana (September 2018)
 
Prometheus – a next-gen Monitoring System
Prometheus – a next-gen Monitoring SystemPrometheus – a next-gen Monitoring System
Prometheus – a next-gen Monitoring System
 
Infrastructure & System Monitoring using Prometheus
Infrastructure & System Monitoring using PrometheusInfrastructure & System Monitoring using Prometheus
Infrastructure & System Monitoring using Prometheus
 
MySQL Monitoring using Prometheus & Grafana
MySQL Monitoring using Prometheus & GrafanaMySQL Monitoring using Prometheus & Grafana
MySQL Monitoring using Prometheus & Grafana
 
Prometheus in Practice: High Availability with Thanos (DevOpsDays Edinburgh 2...
Prometheus in Practice: High Availability with Thanos (DevOpsDays Edinburgh 2...Prometheus in Practice: High Availability with Thanos (DevOpsDays Edinburgh 2...
Prometheus in Practice: High Availability with Thanos (DevOpsDays Edinburgh 2...
 
Kubernetes Observability with Prometheus by Example
Kubernetes Observability with Prometheus by ExampleKubernetes Observability with Prometheus by Example
Kubernetes Observability with Prometheus by Example
 
Prometheus 101
Prometheus 101Prometheus 101
Prometheus 101
 
How to monitor your micro-service with Prometheus?
How to monitor your micro-service with Prometheus?How to monitor your micro-service with Prometheus?
How to monitor your micro-service with Prometheus?
 
Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안
Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안
Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안
 
Introduction To Terraform
Introduction To TerraformIntroduction To Terraform
Introduction To Terraform
 
Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...
Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...
Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...
 
Prometheus + Grafana = Awesome Monitoring
Prometheus + Grafana = Awesome MonitoringPrometheus + Grafana = Awesome Monitoring
Prometheus + Grafana = Awesome Monitoring
 
Prometheus
PrometheusPrometheus
Prometheus
 
End to-end monitoring with the prometheus operator - Max Inden
End to-end monitoring with the prometheus operator - Max IndenEnd to-end monitoring with the prometheus operator - Max Inden
End to-end monitoring with the prometheus operator - Max Inden
 

Similar to Monitoring with Prometheus

How to Improve the Observability of Apache Cassandra and Kafka applications...
How to Improve the Observability of Apache Cassandra and Kafka applications...How to Improve the Observability of Apache Cassandra and Kafka applications...
How to Improve the Observability of Apache Cassandra and Kafka applications...Paul Brebner
 
Monitoring in Big Data Platform - Albert Lewandowski, GetInData
Monitoring in Big Data Platform - Albert Lewandowski, GetInDataMonitoring in Big Data Platform - Albert Lewandowski, GetInData
Monitoring in Big Data Platform - Albert Lewandowski, GetInDataGetInData
 
Google Cloud Platform monitoring with Zabbix
Google Cloud Platform monitoring with ZabbixGoogle Cloud Platform monitoring with Zabbix
Google Cloud Platform monitoring with ZabbixMax Kuzkin
 
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic System
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic SystemTimely Year Two: Lessons Learned Building a Scalable Metrics Analytic System
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic SystemAccumulo Summit
 
Monitoring as Software Validation
Monitoring as Software ValidationMonitoring as Software Validation
Monitoring as Software ValidationBioDec
 
Performance eng prakash.sahu
Performance eng prakash.sahuPerformance eng prakash.sahu
Performance eng prakash.sahuDr. Prakash Sahu
 
Dot Net performance monitoring
 Dot Net performance monitoring Dot Net performance monitoring
Dot Net performance monitoringKranthi Paidi
 
Sql server lesson12
Sql server lesson12Sql server lesson12
Sql server lesson12Ala Qunaibi
 
Sql server lesson12
Sql server lesson12Sql server lesson12
Sql server lesson12Ala Qunaibi
 
Webinar - Building Custom Extensions With AppDynamics
Webinar - Building Custom Extensions With AppDynamicsWebinar - Building Custom Extensions With AppDynamics
Webinar - Building Custom Extensions With AppDynamicsTodd Radel
 
ApacheCon2019 Talk: Improving the Observability of Cassandra, Kafka and Kuber...
ApacheCon2019 Talk: Improving the Observability of Cassandra, Kafka and Kuber...ApacheCon2019 Talk: Improving the Observability of Cassandra, Kafka and Kuber...
ApacheCon2019 Talk: Improving the Observability of Cassandra, Kafka and Kuber...Paul Brebner
 
Observability and its application
Observability and its applicationObservability and its application
Observability and its applicationThao Huynh Quang
 
How to Monitor Application Performance in a Container-Based World
How to Monitor Application Performance in a Container-Based WorldHow to Monitor Application Performance in a Container-Based World
How to Monitor Application Performance in a Container-Based WorldKen Owens
 
Microservices and Prometheus (Microservices NYC 2016)
Microservices and Prometheus (Microservices NYC 2016)Microservices and Prometheus (Microservices NYC 2016)
Microservices and Prometheus (Microservices NYC 2016)Brian Brazil
 
Azure Monitoring Overview
Azure Monitoring OverviewAzure Monitoring Overview
Azure Monitoring Overviewgjuljo
 
Apache Eagle at Hadoop Summit 2016 San Jose
Apache Eagle at Hadoop Summit 2016 San JoseApache Eagle at Hadoop Summit 2016 San Jose
Apache Eagle at Hadoop Summit 2016 San JoseHao Chen
 
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)Brian Brazil
 
Monitoring Node.js Microservices on CloudFoundry with Open Source Tools and a...
Monitoring Node.js Microservices on CloudFoundry with Open Source Tools and a...Monitoring Node.js Microservices on CloudFoundry with Open Source Tools and a...
Monitoring Node.js Microservices on CloudFoundry with Open Source Tools and a...Tony Erwin
 

Similar to Monitoring with Prometheus (20)

Prometheus workshop
Prometheus workshopPrometheus workshop
Prometheus workshop
 
How to Improve the Observability of Apache Cassandra and Kafka applications...
How to Improve the Observability of Apache Cassandra and Kafka applications...How to Improve the Observability of Apache Cassandra and Kafka applications...
How to Improve the Observability of Apache Cassandra and Kafka applications...
 
Monitoring in Big Data Platform - Albert Lewandowski, GetInData
Monitoring in Big Data Platform - Albert Lewandowski, GetInDataMonitoring in Big Data Platform - Albert Lewandowski, GetInData
Monitoring in Big Data Platform - Albert Lewandowski, GetInData
 
Google Cloud Platform monitoring with Zabbix
Google Cloud Platform monitoring with ZabbixGoogle Cloud Platform monitoring with Zabbix
Google Cloud Platform monitoring with Zabbix
 
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic System
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic SystemTimely Year Two: Lessons Learned Building a Scalable Metrics Analytic System
Timely Year Two: Lessons Learned Building a Scalable Metrics Analytic System
 
Monitoring as Software Validation
Monitoring as Software ValidationMonitoring as Software Validation
Monitoring as Software Validation
 
Performance eng prakash.sahu
Performance eng prakash.sahuPerformance eng prakash.sahu
Performance eng prakash.sahu
 
Dot Net performance monitoring
 Dot Net performance monitoring Dot Net performance monitoring
Dot Net performance monitoring
 
Sql server lesson12
Sql server lesson12Sql server lesson12
Sql server lesson12
 
Sql server lesson12
Sql server lesson12Sql server lesson12
Sql server lesson12
 
Webinar - Building Custom Extensions With AppDynamics
Webinar - Building Custom Extensions With AppDynamicsWebinar - Building Custom Extensions With AppDynamics
Webinar - Building Custom Extensions With AppDynamics
 
ApacheCon2019 Talk: Improving the Observability of Cassandra, Kafka and Kuber...
ApacheCon2019 Talk: Improving the Observability of Cassandra, Kafka and Kuber...ApacheCon2019 Talk: Improving the Observability of Cassandra, Kafka and Kuber...
ApacheCon2019 Talk: Improving the Observability of Cassandra, Kafka and Kuber...
 
Observability and its application
Observability and its applicationObservability and its application
Observability and its application
 
How to Monitor Application Performance in a Container-Based World
How to Monitor Application Performance in a Container-Based WorldHow to Monitor Application Performance in a Container-Based World
How to Monitor Application Performance in a Container-Based World
 
Microservices and Prometheus (Microservices NYC 2016)
Microservices and Prometheus (Microservices NYC 2016)Microservices and Prometheus (Microservices NYC 2016)
Microservices and Prometheus (Microservices NYC 2016)
 
Azure Monitoring Overview
Azure Monitoring OverviewAzure Monitoring Overview
Azure Monitoring Overview
 
Apache Eagle: Secure Hadoop in Real Time
Apache Eagle: Secure Hadoop in Real TimeApache Eagle: Secure Hadoop in Real Time
Apache Eagle: Secure Hadoop in Real Time
 
Apache Eagle at Hadoop Summit 2016 San Jose
Apache Eagle at Hadoop Summit 2016 San JoseApache Eagle at Hadoop Summit 2016 San Jose
Apache Eagle at Hadoop Summit 2016 San Jose
 
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)
 
Monitoring Node.js Microservices on CloudFoundry with Open Source Tools and a...
Monitoring Node.js Microservices on CloudFoundry with Open Source Tools and a...Monitoring Node.js Microservices on CloudFoundry with Open Source Tools and a...
Monitoring Node.js Microservices on CloudFoundry with Open Source Tools and a...
 

More from Richard Langlois P. Eng.

Continuous Test Automation, by Richard Langlois P. Eng. and Yuri Pechenko.
Continuous Test Automation, by Richard Langlois P. Eng. and Yuri Pechenko.Continuous Test Automation, by Richard Langlois P. Eng. and Yuri Pechenko.
Continuous Test Automation, by Richard Langlois P. Eng. and Yuri Pechenko.Richard Langlois P. Eng.
 
Microservice Architecture Patterns, by Richard Langlois P. Eng.
Microservice Architecture Patterns, by Richard Langlois P. Eng.Microservice Architecture Patterns, by Richard Langlois P. Eng.
Microservice Architecture Patterns, by Richard Langlois P. Eng.Richard Langlois P. Eng.
 
Reactive Programming in Java and Spring Framework 5
Reactive Programming in Java and Spring Framework 5Reactive Programming in Java and Spring Framework 5
Reactive Programming in Java and Spring Framework 5Richard Langlois P. Eng.
 
Introduction to Reactive Microservices Architecture.
Introduction to Reactive Microservices Architecture.Introduction to Reactive Microservices Architecture.
Introduction to Reactive Microservices Architecture.Richard Langlois P. Eng.
 

More from Richard Langlois P. Eng. (7)

Continuous Test Automation, by Richard Langlois P. Eng. and Yuri Pechenko.
Continuous Test Automation, by Richard Langlois P. Eng. and Yuri Pechenko.Continuous Test Automation, by Richard Langlois P. Eng. and Yuri Pechenko.
Continuous Test Automation, by Richard Langlois P. Eng. and Yuri Pechenko.
 
Microservice Architecture Patterns, by Richard Langlois P. Eng.
Microservice Architecture Patterns, by Richard Langlois P. Eng.Microservice Architecture Patterns, by Richard Langlois P. Eng.
Microservice Architecture Patterns, by Richard Langlois P. Eng.
 
Reactive Programming in Java and Spring Framework 5
Reactive Programming in Java and Spring Framework 5Reactive Programming in Java and Spring Framework 5
Reactive Programming in Java and Spring Framework 5
 
What's New in Java 9
What's New in Java 9What's New in Java 9
What's New in Java 9
 
DevOps, Yet Another IT Revolution
DevOps, Yet Another IT RevolutionDevOps, Yet Another IT Revolution
DevOps, Yet Another IT Revolution
 
What is new in JUnit5
What is new in JUnit5What is new in JUnit5
What is new in JUnit5
 
Introduction to Reactive Microservices Architecture.
Introduction to Reactive Microservices Architecture.Introduction to Reactive Microservices Architecture.
Introduction to Reactive Microservices Architecture.
 

Recently uploaded

Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...
Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...
Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...soniya singh
 
Engage Usergroup 2024 - The Good The Bad_The Ugly
Engage Usergroup 2024 - The Good The Bad_The UglyEngage Usergroup 2024 - The Good The Bad_The Ugly
Engage Usergroup 2024 - The Good The Bad_The UglyFrank van der Linden
 
why an Opensea Clone Script might be your perfect match.pdf
why an Opensea Clone Script might be your perfect match.pdfwhy an Opensea Clone Script might be your perfect match.pdf
why an Opensea Clone Script might be your perfect match.pdfjoe51371421
 
5 Signs You Need a Fashion PLM Software.pdf
5 Signs You Need a Fashion PLM Software.pdf5 Signs You Need a Fashion PLM Software.pdf
5 Signs You Need a Fashion PLM Software.pdfWave PLM
 
Unit 1.1 Excite Part 1, class 9, cbse...
Unit 1.1 Excite Part 1, class 9, cbse...Unit 1.1 Excite Part 1, class 9, cbse...
Unit 1.1 Excite Part 1, class 9, cbse...aditisharan08
 
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...stazi3110
 
Introduction to Decentralized Applications (dApps)
Introduction to Decentralized Applications (dApps)Introduction to Decentralized Applications (dApps)
Introduction to Decentralized Applications (dApps)Intelisync
 
Cloud Management Software Platforms: OpenStack
Cloud Management Software Platforms: OpenStackCloud Management Software Platforms: OpenStack
Cloud Management Software Platforms: OpenStackVICTOR MAESTRE RAMIREZ
 
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...ICS
 
Salesforce Certified Field Service Consultant
Salesforce Certified Field Service ConsultantSalesforce Certified Field Service Consultant
Salesforce Certified Field Service ConsultantAxelRicardoTrocheRiq
 
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...kellynguyen01
 
DNT_Corporate presentation know about us
DNT_Corporate presentation know about usDNT_Corporate presentation know about us
DNT_Corporate presentation know about usDynamic Netsoft
 
EY_Graph Database Powered Sustainability
EY_Graph Database Powered SustainabilityEY_Graph Database Powered Sustainability
EY_Graph Database Powered SustainabilityNeo4j
 
Asset Management Software - Infographic
Asset Management Software - InfographicAsset Management Software - Infographic
Asset Management Software - InfographicHr365.us smith
 
The Essentials of Digital Experience Monitoring_ A Comprehensive Guide.pdf
The Essentials of Digital Experience Monitoring_ A Comprehensive Guide.pdfThe Essentials of Digital Experience Monitoring_ A Comprehensive Guide.pdf
The Essentials of Digital Experience Monitoring_ A Comprehensive Guide.pdfkalichargn70th171
 
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASEBATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASEOrtus Solutions, Corp
 
The Ultimate Test Automation Guide_ Best Practices and Tips.pdf
The Ultimate Test Automation Guide_ Best Practices and Tips.pdfThe Ultimate Test Automation Guide_ Best Practices and Tips.pdf
The Ultimate Test Automation Guide_ Best Practices and Tips.pdfkalichargn70th171
 
Der Spagat zwischen BIAS und FAIRNESS (2024)
Der Spagat zwischen BIAS und FAIRNESS (2024)Der Spagat zwischen BIAS und FAIRNESS (2024)
Der Spagat zwischen BIAS und FAIRNESS (2024)OPEN KNOWLEDGE GmbH
 
Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdf
Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdfLearn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdf
Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdfkalichargn70th171
 

Recently uploaded (20)

Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...
Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...
Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...
 
Engage Usergroup 2024 - The Good The Bad_The Ugly
Engage Usergroup 2024 - The Good The Bad_The UglyEngage Usergroup 2024 - The Good The Bad_The Ugly
Engage Usergroup 2024 - The Good The Bad_The Ugly
 
why an Opensea Clone Script might be your perfect match.pdf
why an Opensea Clone Script might be your perfect match.pdfwhy an Opensea Clone Script might be your perfect match.pdf
why an Opensea Clone Script might be your perfect match.pdf
 
5 Signs You Need a Fashion PLM Software.pdf
5 Signs You Need a Fashion PLM Software.pdf5 Signs You Need a Fashion PLM Software.pdf
5 Signs You Need a Fashion PLM Software.pdf
 
Unit 1.1 Excite Part 1, class 9, cbse...
Unit 1.1 Excite Part 1, class 9, cbse...Unit 1.1 Excite Part 1, class 9, cbse...
Unit 1.1 Excite Part 1, class 9, cbse...
 
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
Building a General PDE Solving Framework with Symbolic-Numeric Scientific Mac...
 
Introduction to Decentralized Applications (dApps)
Introduction to Decentralized Applications (dApps)Introduction to Decentralized Applications (dApps)
Introduction to Decentralized Applications (dApps)
 
Cloud Management Software Platforms: OpenStack
Cloud Management Software Platforms: OpenStackCloud Management Software Platforms: OpenStack
Cloud Management Software Platforms: OpenStack
 
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
 
Exploring iOS App Development: Simplifying the Process
Exploring iOS App Development: Simplifying the ProcessExploring iOS App Development: Simplifying the Process
Exploring iOS App Development: Simplifying the Process
 
Salesforce Certified Field Service Consultant
Salesforce Certified Field Service ConsultantSalesforce Certified Field Service Consultant
Salesforce Certified Field Service Consultant
 
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
 
DNT_Corporate presentation know about us
DNT_Corporate presentation know about usDNT_Corporate presentation know about us
DNT_Corporate presentation know about us
 
EY_Graph Database Powered Sustainability
EY_Graph Database Powered SustainabilityEY_Graph Database Powered Sustainability
EY_Graph Database Powered Sustainability
 
Asset Management Software - Infographic
Asset Management Software - InfographicAsset Management Software - Infographic
Asset Management Software - Infographic
 
The Essentials of Digital Experience Monitoring_ A Comprehensive Guide.pdf
The Essentials of Digital Experience Monitoring_ A Comprehensive Guide.pdfThe Essentials of Digital Experience Monitoring_ A Comprehensive Guide.pdf
The Essentials of Digital Experience Monitoring_ A Comprehensive Guide.pdf
 
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASEBATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
 
The Ultimate Test Automation Guide_ Best Practices and Tips.pdf
The Ultimate Test Automation Guide_ Best Practices and Tips.pdfThe Ultimate Test Automation Guide_ Best Practices and Tips.pdf
The Ultimate Test Automation Guide_ Best Practices and Tips.pdf
 
Der Spagat zwischen BIAS und FAIRNESS (2024)
Der Spagat zwischen BIAS und FAIRNESS (2024)Der Spagat zwischen BIAS und FAIRNESS (2024)
Der Spagat zwischen BIAS und FAIRNESS (2024)
 
Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdf
Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdfLearn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdf
Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdf
 

Monitoring with Prometheus

  • 1. Monitoring With Prometheus Richard Langlois P. Eng. and Gervais Naoussi, Sept. 2018
  • 2. Agenda 2 • Monitoring Concepts • Micrometer • Prometheus • Grafana • Demo
  • 3. Monitoring 3 Monitoring is the tools and processes by which you measure your technology systems. A monitoring system has two customers: • Technology (Engineering, Operations, DevOps) • The business (measure the value that technology delivers to business) If you’re building a specification or user stories for your application: include metrics and monitoring for each component of your application. Don’t wait until the end of a project or just before deployment.
  • 4. Monitoring Approach to Monitoring 4 A good approach to your monitoring is to design a top-down monitoring plan based on value. Identify the parts of the application that deliver value and monitor those first, working your way down the stack. Monitoring for the correctness of a service first  e.g. monitor the content or rates of a business transaction rather than the uptime of the web server it runs on.
  • 5. Monitoring Monitoring Approaches 5 2 major approaches: • Probing monitoring probes the outside of an application (black-box monitoring). e.g. Nagios • Introspection monitoring looks at what’s inside the application (white-box monitoring) application is instrumented and returns measurements of its state
  • 6. Monitoring Pull vs Push 6 Two approaches to how monitoring checks are executed: • Pull-based: systems scrape or check a remote application—for example, an endpoint containing metrics. • Push-based: applications emit events that are received by the monitoring system. Prometheus is primarily a pull-based system, but it also supports receiving events pushed into a gateway.
  • 7. Monitoring Metric 7 Metrics are measures of properties of components of software or hardware. To make a metric useful we keep track of its state, generally recording data points over time (called observations). An observation consists of: value, a timestamp, and sometimes a series of properties that describe the observation such as a source or tags. A collection of observations is called a time series. Time series data is a chronologically ordered list of these observations. Time series metrics are often visualized as a two-dimensional plot with data values on the y-axis and time on the x-axis.
  • 8. Monitoring Types of monitoring data 8 Monitoring tools can collect 2 types of data: • Metrics are stored as time series data that record the state of measures of your applications. • Logs are (usually textual) events emitted from an application. Prometheus is primarily focused on collecting time series data.
  • 9. Monitoring Type of Metrics 9 Variety of different types of metrics: Gauges: are numbers that are expected to go up or down. A snapshot of a specific measurement.  e.g. Disk usage, number of customers present on a site. Counters: are numbers that increase over time and never decrease.  e.g. system uptime, number of sales in a month Histograms: is a metric that samples observations. Each observation is counted and placed into buckets. Metric Summaries: mathematical transformations applied to metrics • Average • Median • Standard Deviation • Percentile
  • 10. Agenda 10 • Monitoring Concepts • Micrometer • Prometheus • Grafana • Demo
  • 11. Micrometer Intro 11 Micrometer is a metrics instrumentation library allowing to instrument JVM-based application code without vendor lock-in. It provides a simple façade over the instrumentation clients for the most popular monitoring systems. Think SLF4J, but for application metrics. As of Spring Boot 2.0.0.M5, Micrometer is the instrumentation library used by Spring. Some supported monitoring systems: • Datadog • Graphite • Influx • JMX • New Relic • Prometheus • SignalFX • StatsD
  • 12. Micrometer API 12 Meter is the interface for collecting a set of measurements (called metrics). MeterRegistry: eters are created from and held in a Meter Registry Each supported monitoring system has an implementation of MeterRegistry. SimpleMeterRegistry: Automatically autowired in Spring-based apps. MeterRegistry registry = new SimpleMeterRegistry. Set of meter primitives: Timer, Counter, Gauge, DistributionSummary, LongTaskTimer, FunctionCounter, FunctionTimer and TimeGauge. Dimensions allow a particular named metric to be sliced to drill down. E.g. Registry.counter(“http.server.requests”, “uri”, “/api/users”) Fluent builder: Counter counter = Counter .builder("counter") .baseUnit(“ms") .description("a description of what this counter does") .tags("region", "test") .register(registry);
  • 13. Agenda 13 • Monitoring Concepts • Micrometer • Prometheus • Grafana • Demo
  • 14. Prometheus Intro 14 Prometheus is a simple, effective open-source monitoring system. Promoted from incubation to graduation (in August 2018), in Cloud Native Computing Foundation (CNCF) Prometheus works by scraping (pulling) time series data exposed from applications. The time series data is exposed by the applications themselves often via client libraries or via proxies called exporters, as HTTP endpoints.
  • 15. Prometheus Concepts 15 Prometheus calls the source of metrics it can scrape endpoints. An endpoint usually corresponds to a single process, host, service, or application. The resulting time series data is collected and stored locally on the Prometheus server (15 days retention) and can be sent from the server to external storage or to another time series database. Prometheus can also define Rules for alerting.
  • 16. Prometheus PromQL – inbuilt querying language 16 The Prometheus server also comes with an inbuilt querying language, PromQL, allowing to query and aggregate metrics. Use this query language in the query input box in the Expression Browser. e.g: Query all metrics with a label of quantile=“0.5”:
  • 18. Prometheus Scalability 18 Designed to scale to millions of time series from many thousands of hosts. Its data storage format is designed to keep disk use down and provide fast retrieval of time series during queries and aggregations. SSD disks are recommended for Prometheus servers, for speed and reliability. Redundant Prometheus Architecture:
  • 19. Prometheus Data Model 19 Prometheus collects time series data. Format: <time series name>{<label name>=<label value>, ...} Each time series is uniquely identified by the combination of names and key/value pairs called labels (provide the dimensions). Name usually describes the general nature of the time series data being collected  e.g. total_website_visits as the total number of website visits. Labels enable the Prometheus dimensional data model, they add context to a specific time series.  e.g. the name of the website, IP of the requester
  • 20. Prometheus Time Series Notation 20 Example total_website_visits{site=“alithya.com", location="NJ", instance="webserver“, job="web"} All time series generally have • an instance label, which identifies the source host or application • a job label, which contains the name of the job that scraped the specific time series. Actual value of the time series is called a sample. Consists of: • A float64 value. • A millisecond-precision timestamp.
  • 21. Prometheus configuration prometheus.yml 21 Prometheus is configured via YAML configuration files. Default Configuration file has the following 4 YAML blocks defined: Global: contains global settings for controlling the Prometheus server’s behavior. Alerting: configures Prometheus’ alerting. rule_files: specifies a list of files that can contain recording or alerting rules. scrape_configs: specifies all of the targets that Prometheus will scrape.
  • 22. Prometheus and Spring Boot 22 Spring Boot auto-configures a composite MeterRegistry and adds a registry to the composite for each of the supported implementations that it finds on the classpath. pom.xml: <dependency> <groupId>io.prometheus</groupId> <artifactId>simpleclient_spring_boot</artifactId> <version>0.1.0</version> </dependency> <dependency> <groupId>io.prometheus</groupId> <artifactId>simpleclient_hotspot</artifactId> <version>0.1.0</version> </dependency The simpleclient_spring_boot dependency provides the @EnablePrometheusEndpoint annotation. Adding it to a @configuration class, creates a HTTP endpoint accessible via /actuator/prometheus that exposes all registered (actuator) metrics in a Prometheus data format.
  • 23. Prometheus configuration Scrape Config for Spring Boot application 23 Prometheus scrapes the following 2 endpoints • /prometheus endpoint: contains Spring boot metrics • /metrics endpoint: Prometheus own metrics scrape_configs: # The job name is added as a label `job=<job_name>` to any time series scraped from this config. - job_name: 'prometheus' # metrics_path defaults to '/metrics' # scheme defaults to 'http'. static_configs: - targets: ['PROM_IP:9090'] - job_name: 'spring-boot' metrics_path: '/prometheus' scrape_interval: 5s static_configs: - targets: ['App_IP:8080']
  • 24. Prometheus Dashboard 24 Example of metrics on Prometheus dashboard:
  • 25. Prometheus Alerting 25 Alerting is provided by a tool called Alertmanager Alerting rules are defined on the Prometheus server. When the threshold or criteria is met, an alert will be generated and pushed to Alertmanager. The alerts are received on an HTTP endpoint on the Alertmanager. Alertmanager handles deduplicating, grouping, and routing alerts to receivers (e.g. email, SMS, PagerDuty)
  • 26. Prometheus Alerting Configuration 26 A simple alertmanager.yml configuration file, sending alerts by email:
  • 27. Prometheus Alerting Web Interface 27 Web interface to: • view current alerts • manage maintenance window alert suppression (silences)
  • 28. Prometheus Pushgateway 28 Metrics can be pushed to Pushgateway when there isn’t a target from which to scrape metrics because: • can’t reach the target resources because of security • target resource has too short a lifespan (e.g. container starting, executing, and stopping). • target resource doesn’t have an endpoint, (e.g. batch job). Pushgateway sits between an application sending metrics and the Prometheus server. Pushgateway is scraped as a target to deliver the metrics to the Prometheus server.
  • 29. Agenda 29 • Monitoring Concepts • Micrometer • Prometheus • Grafana • Demo
  • 30. Grafana 30 Prometheus UI is not really nice. Alternative: Grafana is open source metrics Dashboard platform. It supports multiple backend time-series databases including: Prometheus , InfluxDB, Elasticsearch, Cloudwatch … Example of Grafana dashboard:
  • 32. Grafana Prometheus as Datasource 32 Before you create your first dashboard you need to add your data source.
  • 33. Grafana Prometheus as Datasource 33 Name: your choice Default: Check to tell Grafana to search for data in this source by default Type: Prometheus URL: URL of the Prometheus server to query.
  • 34. Agenda 34 • Monitoring Concepts • Micrometer • Prometheus • Grafana • Demo
  • 35. References 35 Micrometer: • https://micrometer.io Prometheus • Book: Monitoring with Prometheus, James Turnbull, 2018. • https://prometheus.io Graphana • https://grafana.com/