Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
From logging to monitoring to reactive
insights
Christian Schneider
Christian Schneider
» For the last 6 years:
Open Source Architect at Talend
» Since October:
Computer scientist at Adobe.
...
What you will learn
» Apache Karaf Decanter + Elastic Search + Kibana
» Approaches for central monitoring and logging
» Cu...
Basic logging in OSGi
» Pax logging
» Logback
» Log4j2
Files
Monitoring
Hawtio
JVisualVM
● Great tools
● Developer focused
● Production?
Production logging
Central History
Drill down Custom processing
ELK Stack
Karaf
Elastic Search
+
Kibana
Logstash
Server
File Beat
Metric Beat
Apache Decanter
Karaf
+
Decanter
Elastic Search
+
Kibana
Decanter architecture
Collector
Event
Admin
Appender
Apache Decanter
Collectors
● Pax-Logging
● JMX
● EventAdmin
● Camel
● ActiveMQ
● ...
Events
Appenders
● File
● DB
● Elasti...
Decanter setup for single machine
JMX
Collector
Log
Collector
Camel
Collector
Event
Admin
Elastic
Search
Appender
Decanter for multiple servers
Server 1
+
Decanter
Server 2
+
Decanter
Kafka
Cluster
Kafka
Cluster
Elastic Search
+
Kibana
...
Decanter + logstash
Server 1
Karaf
Decanter
Server 2
Spring boot
Beats
Kafka
Cluster
Kafka
Cluster
Elastic Search
+
Kibana...
CXF Message logging
Client Server
Request
Out
Request
In
Response
Out
Response
Inslf4j slf4j
● Meta data as MDC properties...
Extending Decanter
» Collectors
» Appenders
» Filter, Enrich Events
» “Complex” Event Processing using Reactive Components
Event Processing use case
» Compute “up time” of bundles by noting start and stop
events and sending an event with the bun...
Demo time
Decanter in action
Thanks for your attention
Questions?
Further reading
Demo code
Apache Karaf decanter
CXF message logging
Reactive Components
Upcoming SlideShare
Loading in …5
×

From logging to monitoring to reactive insights

794 views

Published on

In a highly distributed world it is crucial to have the best possible insights into your application.
In the old days we achieved this using plain logging and jmx. This does not scale well to highly distributed processing like in microservices though.

This talk shows how to pimp you OSGi based systems with top notch monitoring and tracing capabilities. Using Apache Karaf Decanter we let karaf instances log into a common elastic search instance with almost zero effort and get insights using kibana. The Apache CXF REST and SOAP integrations allow to log and correlate service requests and replies.

Building on top of this we use the OSGi based reactive components framework and reactor.io streams to provide complex event processing capabilities. A typical use case for this is to describe and monitor service level agreements.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

From logging to monitoring to reactive insights

  1. 1. From logging to monitoring to reactive insights Christian Schneider
  2. 2. Christian Schneider » For the last 6 years: Open Source Architect at Talend » Since October: Computer scientist at Adobe. » Apache member and committer » Twitter @schneider_chris » Website liquid-reality.de
  3. 3. What you will learn » Apache Karaf Decanter + Elastic Search + Kibana » Approaches for central monitoring and logging » Custom reactive processing to produce higher level informations
  4. 4. Basic logging in OSGi » Pax logging » Logback » Log4j2 Files
  5. 5. Monitoring Hawtio JVisualVM ● Great tools ● Developer focused ● Production?
  6. 6. Production logging Central History Drill down Custom processing
  7. 7. ELK Stack Karaf Elastic Search + Kibana Logstash Server File Beat Metric Beat
  8. 8. Apache Decanter Karaf + Decanter Elastic Search + Kibana
  9. 9. Decanter architecture Collector Event Admin Appender
  10. 10. Apache Decanter Collectors ● Pax-Logging ● JMX ● EventAdmin ● Camel ● ActiveMQ ● ... Events Appenders ● File ● DB ● Elastic Search ● Kafka ● JMS ● ...
  11. 11. Decanter setup for single machine JMX Collector Log Collector Camel Collector Event Admin Elastic Search Appender
  12. 12. Decanter for multiple servers Server 1 + Decanter Server 2 + Decanter Kafka Cluster Kafka Cluster Elastic Search + Kibana Decanter
  13. 13. Decanter + logstash Server 1 Karaf Decanter Server 2 Spring boot Beats Kafka Cluster Kafka Cluster Elastic Search + Kibana Logstash Server Server 3 Tomcat Beats
  14. 14. CXF Message logging Client Server Request Out Request In Response Out Response Inslf4j slf4j ● Meta data as MDC properties ● Message id and correlation id ● Service name as logger name
  15. 15. Extending Decanter » Collectors » Appenders » Filter, Enrich Events » “Complex” Event Processing using Reactive Components
  16. 16. Event Processing use case » Compute “up time” of bundles by noting start and stop events and sending an event with the bundle runtime on stop » Alert when we we have more than x percent errors in logged CXF messages » Correlate request and response into one message sent through decanter » Only send aggregated events when your source produces Events very fast
  17. 17. Demo time Decanter in action
  18. 18. Thanks for your attention Questions?
  19. 19. Further reading Demo code Apache Karaf decanter CXF message logging Reactive Components

×