How to design a system to visualize critical business metrics and KPI’s for proprietary database applications using Docker, Grafana, Prometheus and custom Java EE ThinWAR agent. Presentation for Docker Athens meetup group, given Jul 5 2017 by Vasilis Stergioulis of InsurIT Advisors

1. Business metrics visualization with Grafana and Prometheus
Vasilis Stergioulis @vas_stergioulis
2017-07-05 Docker Athens meetup

2. InsurIT AdvisorsInsurIT Advisors 22
A system architect, senior software engineer that really enjoys writing code, with many years of expertise in
the private insurance sector, deep knowledge of the private insurance system, intermediaries and distribution
networks and enterprise-level challenges.
With experience in the operations of insurance and reinsurance companies operating in the European Union
and the European Economic Area under the directive of Solvency II framework.
Participates in the Java Community Process (JCP) Program as Associate Member and focusing on Java EE
applications.
Vasilis Stergioulis, @vas_stergioulis
Founder / Software Engineer at InsurIT Advisors
linkedin.com/in/vassterg
github.com/vassterg
gitlab.com/vassterg
facebook.com/vstergioulis
vassterg@insuritadvisors.com

3. InsurIT AdvisorsInsurIT Advisors 3
Agenda
• Big datasets and analytics
• Real time business intelligence
• Time series metrics, monitoring
• Prometheus, Grafana
• Data instrumentation and collector agent
• Docker use
3

4. InsurIT AdvisorsInsurIT Advisors 4
Big datasets and analytics
• OLTP, OLAP, BI Analytics, Replicas etc
• Big data volumes, hard to digest
• Need data scientists, not for business users
• Time constraints limit real time performance
• Data usually need transformations
4

5. InsurIT AdvisorsInsurIT Advisors 5
Real time business intelligence
• Need a way to look inside systems
• Measure:
– critical data counters and execution times
– business values and components
– rates of business changes and events
• In real time with minimal impact on inspected systems
5

6. InsurIT AdvisorsInsurIT Advisors 6
Time series metrics, monitoring
• Instrument in business boundaries
• Profile execution of services
• Gather data point metrics
• Data points as rows of key-values in one table
• Collect table data as snapshot of system state
• Convert snapshot to time series
• Process and display data
6

7. InsurIT AdvisorsInsurIT Advisors 7
Prometheus
• Time series metrics, monitoring
• Multidimensional (with labels)
• Self contained, self tuned
• Graphs, rules, alerts
• Grafana support
7

8. InsurIT AdvisorsInsurIT Advisors 8
Grafana
• Beautiful analytics and monitoring
• Open source platform
• Multiple datasources, panels, dashboards
• Query, visualize, alert on and understand metrics
8

9. InsurIT AdvisorsInsurIT Advisors 9
Instrumentation code
• A table to hold current values of metrics
• Simple operations, increment and set
• Autonomous transactions
• Minimum overhead
• Counters, gauges and summaries
• Measure values, time and errors
9

10. InsurIT AdvisorsInsurIT Advisors 10
CREATE TABLE instrumentation_metrics
(
im_id NUMBER PRIMARY KEY,
im_metric VARCHAR2 (200 BYTE) NOT NULL,
im_value NUMBER DEFAULT 0,
im_type VARCHAR2 (250 BYTE),
im_help VARCHAR2 (350 BYTE)
);
Table definition

11. InsurIT AdvisorsInsurIT Advisors 11
procedure some_proc(some_param varchar2) is
start timestamp := now();
begin
// … do some business things
metrics.incr_metric(‘someproc_time_total’, timer(start, systimestamp));
metrics.incr_metric(‘someproc_total’);
metrics.incr_metric(‘otherservice_invocations’);
exception
when others then
metrics.incr_metric(‘someproc_time_total’, timer(start, systimestamp));
metrics.incr_metric(‘someproc_failures’);
metrics.incr_metric(‘someproc_total’);
// … exception handling code
end;
Example code in plsql

12. InsurIT AdvisorsInsurIT Advisors 12
Collector agent
• Simple REST API
• Load and transform metrics
• Exposes as TEXT or JSON via GET
• Async with pooled db connections
• Very simple service, typical 4 KB WAR file
• One service on one application server
• Deployed as autonomous container
12

13. InsurIT AdvisorsInsurIT Advisors 13
Collector agent (cont.)
• No uber jar creation
• UltraThin war to docker image
• Pushed to remote registry in seconds
• Fast build automation pipelines
mvn clean package && docker build … && docker push
13

14. InsurIT AdvisorsInsurIT Advisors 14
DBCollector
Grafana
Prometheus
App App
Schematic topology

15. InsurIT AdvisorsInsurIT Advisors 15
Docker use
●
Every service layer is built and tested in each own container
as a component
●
Docker-compose is used to bundle components into
service(s)
●
Prometheus and Grafana can be in separate data centers
●
Collector agent can be near databases exposing only their
interface to outside world and protecting, through pooling, the
access to internal database
15

16. InsurIT AdvisorsInsurIT Advisors 16
References
Docker registry for images and source files
https://hub.docker.com/u/insurit/ and https://github.com/insuritadvisors/dockerdom
Source code for collector agent and database tooling
https://github.com/insuritadvisors/sql_exporter
Prometheus - Monitoring system and time series database
https://prometheus.io/ and https://github.com/prometheus
Grafana - The open platform for analytics and monitoring
https://grafana.com/ and https://github.com/grafana/grafana
Java EE - Java Enterprise Edition
http://oracle.com/javaee and https://github.com/javaee and https://jcp.org

17. Any questions?
17
Thank you!