This document discusses monitoring Elasticsearch and Nginx with Prometheus and Grafana. It introduces the speaker and their work at LINE developing a browser log collection system using Nginx and Elasticsearch. It then covers setting up Prometheus to monitor Nginx using the nginx_exporter plugin, which scrapes Nginx status pages. Metrics on request rates, connections and response times are shown. Elasticsearch is monitored using the elasticsearch_exporter. Example Grafana dashboards visualizing metrics from these exporters like request rates, Java heap usage and connection counts are also presented. The speaker expresses that Prometheus and Grafana work well together for time-series monitoring and the exporters provide a convenient way to export metrics.

1. Elasticsearch, Nginx monitoring
with Prometheus and Grafana
2016/08/18
Wataru Yukawa(@wyukawa)
Prometheus London meetup

2. Who am I?
• Data Engineer at LINE
• LINE makes a messaging application of the
same name, in addition to other related
services
• It is the most popular messaging platform in
Japan
• I like Premier League

3. Agenda
• browser log collection system
• monitor Elasticsearch/Nginx

4. browser log collection system
• Frontend engineer develops JavaScript but it’s
difficult to debug because there are many kind
of browsers/devices.
• So we create browser log collection system
like Google Analytics by using web beacon
• Nginx logs are sent to ES by fluentd
• Fluentd is an OSS log collector like logstash,
flume written in ruby.

5. What middleware should be
monitored?
• Nginx
• Elasticsearch
• Fluentd
– I will talk in PromCon 2016

6. Nginx monitoring
• Use ngx_http_stub_status_module
# monitoring
location /___nginx_status {
stub_status on;
allow 127.0.0.1;
deny all;
}
$ curl -s http://localhost/___nginx_status
Active connections: 2
server accepts handled requests
301042 301042 4242850
Reading: 0 Writing: 1 Waiting: 1

7. lifecycle of connection
https://www.datadoghq.com/blog/how-to-monitor-nginx/

8. Nginx monitoring
• Use nginx_exporter
– nginx_exporter uses
ngx_http_stub_status_module result
# HELP nginx_connections_current Number of connections currently processed by nginx
# TYPE nginx_connections_current gauge
nginx_connections_current{state="active"} 2
nginx_connections_current{state="reading"} 0
nginx_connections_current{state="waiting"} 1
nginx_connections_current{state="writing"} 1
# HELP nginx_connections_processed_total Number of connections processed by nginx
# TYPE nginx_connections_processed_total counter
nginx_connections_processed_total{stage="accepted"} 300966
nginx_connections_processed_total{stage="any"} 4.241939e+06
nginx_connections_processed_total{stage="handled"} 300966

9. nginx dashboard

10. request connection
irate(nginx_connections_processed_total{...,stage="any"}[1m])

11. Elasticsearch monitoring
• Use elasticsearch_exporter
• Use Marvel

12. Elasticsearch dashboard

13. java heap
https://github.com/elastic/elasticsearch/issues/18635

14. My feeling
• Prometheus’s query is really powerful
especially when metric type is counter
• We can use rate/irate function.
• nginx_exporter and elasticsearch_exporter are
convienient
– easy to setup
• Prometheus and Grafana are a perfect
combination

15. References
• https://github.com/discordianfish/nginx_expo
rter
• https://github.com/ewr/elasticsearch_exporte
r