This document discusses using the ELK stack (Elasticsearch, Logstash, Kibana) along with Filebeat to solve challenges with centralized logging for microservices. It provides an overview of what the ELK stack is and how each component is used. Filebeat is used to collect and ship logs from services to Logstash for parsing and enrichment before being indexed into Elasticsearch. Spring Cloud Sleuth and Zipkin are discussed for enhancing logs with request tracing across services. Logback and Mapped Diagnostic Context are explained for adding structured fields like response time. Alerting can be done using Elastalert to send notifications from Elasticsearch data. Configuration and an example microservices app are presented at the end.

1. Log Analysis and Visualization using ELK Stack
(Elasticsearch, Logstash, Kibana) and Filebeat
By
Vineet K Sabharwal
https://www.linkedin.com/in/vineetkanwal/

2. Agenda
Challenges in logging for Microservices
What is ELK stack or Elastic Stack?
Using Filebeat (Need and Advantages)
Spring Cloud Sleuth and Zipkin
Logback and Mapped Diagnostic Context (MDC)
Using Spring AOP to add Response time
Alerting and Notifications using Elastalert
Configuration demo and Example Microservices

3. Challenges in logging for Microservices
Microservices are all about breaking things down to individual components. As a side effect, ops
procedures and monitoring are also breaking down per service and lose their power for the
system as a whole. The challenge here is to centralize the Application Logs which will come from
several different Microservices from docker containers running on multiple hosts.
Traditional logging is ineffective because microservices are stateless, distributed and
independent — you would produce too many logs to easily locate a problem. Logging must be
able to correlate events across several platforms.
As the system becomes highly fragmented with more and more microservices added for
performing specific tasks, there will be stronger need for centralized monitoring and logging, to
have a fair shot at understanding what’s going on.

4. What is ELK stack or Elastic Stack?
The ELK stack consists of Elasticsearch, Logstash, and Kibana.
Main advantages with Elastic Stack
◦ Open source, no license cost
◦ A vital component for building scalable search driven solutions
◦ Not only a search tool, but a full fletched Document database, perfect for your database offloading needs
◦ Flexible expert support options thanks to different type of Subscriptions
◦ Can be used as Business Intelligence tool

5. Using Filebeat (Need and Advantages)
Filebeat acts as a lightweight agent
deployed on the edge host, pumping
data into Logstash for aggregation,
filtering and enrichment.
Feeding logs directly to logstash using
appender introduces performance
overhead.
Filebeat is lightweight, supports SSL
and TLS encryption, supports back
pressure with a good built-in recovery
mechanism, and is extremely reliable.
Filebeat cannot turn logs into easy-
to-analyze structured log messages
using filters for log enhancements.
That’s the role played by Logstash.

6. Spring Cloud Sleuth and ZipkinSpring Cloud Sleuth is a powerful tool for enhancing logs in any application, but especially in a system built up of multiple
services.
It introduces unique IDs to your logging which are consistent between microservice calls which makes it possible to find
how a single request travels from one microservice to the next.
Spring Cloud Sleuth adds two types of IDs to your logging, one called a trace ID and the other called a span ID. The span ID
represents a basic unit of work, for example sending an HTTP request. The trace ID contains a set of span IDs, forming a
tree-like structure. The trace ID will remain the same as one microservice calls the next.
Zipkin shows how long a request took from one microservice to the next.
Spring Cloud Sleuth will send tracing information to any Zipkin server you point it to when you include the dependency
spring-cloud-sleuth-zipkin in your project.

7. Logback and Mapped Diagnostic Context
(MDC)
• Logback (https://logback.qos.ch/) is successor to the popular log4j project.
• Logback brings a very large number of improvements over log4j like logback-
classic implements the SLF4J API natively reducing the work involved in switching
logging frameworks, Graceful recovery from I/O failures, Automatic compression
of archived log files, filters, etc.
• Mapped Diagnostic Context (MDC) is a feature which lets the developer place
information in a diagnostic context that can be subsequently retrieved. For
instance, it can be used to record response time for each API request in micro
services.

8. Using Spring AOP to add Response time
• Measuring and analysing the response time that APIs take is very important part of
monitoring performance.
• Spring AOP can be used to add response time around APIs as aspects with minimum
performance overhead.
• First, you need to include the spring-aop, aspectj and cglib libraries as dependencies.
• Next, identify the APIs that need monitoring and put the AOP hooks in place.
• Add the response time as MDC (Mapped Diagnostic Context) variable for analysing in
Kibana.

9. Alerting and Notifications using Elastalert
ELK stack does not natively have an alerting system.
ElastAlert (https://elastalert.readthedocs.io/) is open source library from Yelp built using python, which
can be used to create alerts on top of Elasticsearch. These alerts can be email, JIRA , slack, hipchat and
many more.
ElastAlert has a global configuration file, config.yaml, which defines several aspects of its operation.
Rules are defined in the rules folder set in the config file.
Every file that ends in .yaml in the rules_folder will be run by default.

10. Configuration demo and Example Microservices

11. Questions?