promgen - prometheus
management tool
tokuhirom
Who are you?
Web Application Engineer
• Server side engineer
• Java
• Perl
• Server Admin
www1.example.com
node_exporter
(httpd)
jmx_exporter
(httpd)
Periodic GET request
Why Prometheus?
It’s good for lazy web programmers
Get data → Get stats
Async and lazy
Easy to integrate with
grafana
But there’s no server
list management tool
official method to manage
servers, is writing YAML.
With consul or something
YAML file contains:
exporter port number
alerting rules
Server admins need to
write BIG YAML FILE.
Editing YAML via
github pull-request
We want to manage it
via web app.
We wrote web application
to manage server list.
We are managing the
server list in our
deployment tool
promgen can integrate
with our deployment tool
(Of course, it’s pluggable)
Note: We haven’t use
consul or similar
tool(now).
Promgen
• Simple prometheus manager app
• exporter port management
• alert management
Screenshot
ProjectService
Project
Project
Host
Host
Host
Data structure
Managed by Deploy tool
Farm
Farm
Farm
Alerting
Alert
Manager
Hipchatpromgen-alerting
Web hook
Mail
MySQL
Architecture
• Ruby
• Sequel
• MySQL
• erubis
• sinatra
Why ruby?
I want to run it on
prometheus host.
We shouldn’t use much
memory
Part 2.
Monitoring Java
processes via Prometheus
There are 3 patterns
1. Collect data from MBean
in target JVM
App JVM
(Runs jmx_exporter jetty)
prom
HTTP
• Pros
• Easy to run
• Cons
• Hard to maintain YAML file
- pattern: '^com.zaxxer.hikari<type=Pool ((w+)
)><>ThreadsAwaitingConnection'
name: hikaricp_pool_threads_awaiting_connection
labels:
source: "$1"
help: "HikariCP threads awaiting connection(pending
threads) $1"
type: GAUGE
2. Collect data from JMX
outside from target JVM
App
JVM
jmx_exporter
JVM
prom
HTTPJMX
• Pros
• Easy to deploy
• Safety
• Cons
• Memory overhead
• Hard to maintain YAML file
3. Collect data by
simpleclient_java
App JVM
(Run simpleclient_java jetty)
prom
HTTP
• Pros
• Clean
• lightweight
• no YAML required
• Cons
• Requires coding
simpleclient_java
simpleclient_dropwizard
Drop wizard
metrics
MetricRegistry
client_java
simpleclient_hotspot
JVM core client_java
Default
simpleclient_log4j/logback
log4j/logback client_java
Error/Warn rate
simpleclient_springboot
Spring Boot
Metrics
client_java
NEW!
simpleclient_jetty
Jetty
requests
status
etc.
client_java
NEW!
simpleclient_hikaricp
HikariCP client_java
REJECTED!
Questions?

promgen - prometheus managemnet tool / simpleclient_java hacks @ Prometheus casual