The document provides an overview of logging and the Elastic Stack (Elasticsearch, Logstash, Kibana) for log analysis. It discusses what logs are, popular logging processors like Loggly and Splunk, and the key components of the Elastic Stack - Elasticsearch for storage, Logstash for processing, and Kibana for visualization. It also gives a live demo example of using Logstash to ingest application logs and explore them in Kibana.

1. Kibana + ElasticSearch + LogStash
By Dmitriy Mustafin
JavaMeetup
handle Log messages on Prod servers

2. My experience
- I started with machine code and punched cards. I used to write on
Assembler, then on C and Pascal, С++, Delphi, C#, Javascript, and
some other scripting languages. Now I mostly write on Java, and I’m
pretty much happy with that.
- DOS (it was a wonderful to have 21h interrupt), OS/2, QNX,
Windows (my favorite so far), Linux (and Yocto as well), MacOS/iOS.
- I studied at university but I still haven't defended my thesis (and I'm
not planning on doing this anytime soon). I keep learning new things
during my work.
- I was writing, designing, engineering, managing and solving
problems.
- Married, children, cat.

3. HYS Enterprise is a Dutch software
development company with more than
200 talented engineers
from all over the world
hys-enterprise.com

4. Agenda
● What is log and what’s this for?
● How can we do log totally senseless?
● Logging levels or how not to drown in gigs of data
● Most popular logging processors
● Elastic Stack as a result of smoking in the Netherlands
● This beautiful Kibana
● This nimble and gluttonous ElasticSearch
● This terrifying LogStash
● LogStash: real-life example (or even live demo)
● How can we avoid pitfalls of LogStash?
● ElasticBeats or “What else do you want from me?”

5. What is log and what’s
this for?
In computing, a log file is a file that records either events
that occur in an operating system or other software runs,
or messages between different users of a communication
software.
Logging is the act of keeping a log. In the simplest case,
messages are written to a single log file.
Wikipedia

6. How can we do log totally senseless?
Harmful advices:
● Do not use log-files!
● Never put datetime, log level, process id, thread id, specific entity grouping, message
itself into log message
● Never make file rolling!
● 15 GB log-file is ok
● 150+ GB log-file is perfect!
● Never zip old files
● Keep log files no more than 1 day (max 2 days)
● Nobody should to know about log-files!
● Never analyze log-files!
● Never put live data on big screen in developers room!

7. Logging levels or
how not to drown in gigs of data
+ custom log levels
Standard log levels built-in to Log4J 2 ™ intLevel
OFF 0
FATAL 100
ERROR 200
WARN 300
INFO 400
DEBUG 500
TRACE 600
ALL Int max

8. Most popular logging processors
Top 10 (?) Log Analysis Tools by KeyCDN:
1. Loggly
2. Logentries
3. GoAccess
4. logz.io
5. Graylog
6. Splunk
7. Logmatic.io
8. Logstash
9. Sumo Logic
10. Papertrail
11. Fluentd

9. Elastic Stack as a result of smoking in
the Netherlands
● Original author: Shay Banon
● Stable release: 6.4.2 / October 2, 2018 *
● Repository: github.com/elastic/elasticsearch
● Written in Java
● Operating system: Cross-platform
● Type: Search and index
● License: Apache License 2.0
● Website: www.elastic.co/products/elasticsearch

10. Elastic Stack as a result of smoking in
the Netherlands - cont.
● Shay Banon (Compass) 2004 - Downloads: 0
● June 2012 - Downloads: <16,000
● July 2015 - Downloads: 36,431,145
● October 2015 - Downloads: 44,378,846
● October 2016 - Downloads: 91,183,928
● May 2017 - Downloads: 137,715,884
● October 2017 - Downloads: 192,865,831
● February 2018 - Downloads: 267,972,265
18 offices in Europe, America, Asia

11. This beautiful Kibana
Kibana lets you visualize your Elasticsearch data and navigate the Elastic Stack, so you can
do anything from learning why you're getting paged at 2:00 a.m. to understanding the
impact rain might have on your quarterly numbers.
https://www.elastic.co/products/kibana
● Web UI
● Useful search and filtering
● Visualisations
● Dashboards
● Compute fields

14. This nimble and gluttonous
ElasticSearch
ElasticSearch is a distributed, RESTful search and analytics engine capable of solving a growing
number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover
the expected and uncover the unexpected.
https://www.elastic.co/products/elasticsearc
h
● ElasticSearch Is Fast. Really, Really Fast.
● Run It on Your Laptop or Hundreds of Servers with Petabytes of Data.
● Interact with ElasticSearch in the Programming Language You Choose.
● Extend ElasticSearch.

15. This terrifying LogStash
Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of
sources simultaneously, transforms it, and then sends it to your favorite “stash”.
https://www.elastic.co/products/logstas
h
● Takes 1GB or RAM (Java!)
● Parsing and computing @ onboard machine
● Network consumption is tens of MBps
● Looooong start (Java!)
● GrOk and RegExp
● Not easy to debug the script

16. Classic solution:
Prod server(s) Kibana server(s)
Our Service
Log file(s)
LogStash
ElasticSearch
Indexes
Kibana
How it works

17. LogStash: real-life example (live demo)
● Log file of Spring Boot application
● Log4j2 with MDC for entity grouping
● Sample GrOk config file (input, filter, output)
○ Discussing config file
○ Example of RegExing
○ Example of debugging GrOk matcher
● Start Elastic Stack on Windows machine
● Scan sample log-files
● Magic of Kibana
○ Discovery
○ Visualisation
○ Dashboard

18. How can we avoid
pitfalls of LogStash?
● Takes 1GB or RAM (Java!) ⇒ Rewrite it on Go
● Parsing and computing @ onboard machine ⇒ Move to other machine
● Network consumption is tens of MBps ⇒ Zip data (?)
● Looooong start (Java!) ⇒ Rewrite it on Go
● GrOk and RegExp ⇒ Life is pain...
● Not easy to debug the script ⇒ Keep calm and heavy breathing...

19. Prod server(s) Kibana server(s)
Our Service
Log file(s)
LogStash
ElasticSearch
Indexes
Kibana
FileBeat
Beat solution:
How it works now

20. ElasticBeats or
“What else do you want from me?”
Beats (Lightweight Data Shippers) is the platform for single-purpose data shippers. They install as
lightweight agents and send data from hundreds or thousands of machines to Logstash or
Elasticsearch.
https://www.elastic.co/products/beats
● Filebeat, Metricbeat, Packetbeat, Winlogbeat, Auditbeat, Heartbeat, etc…
● Tens of community beats...
● … and custom Beat constructor

21. Useful links
● About https://en.wikipedia.org/wiki/Kibana
● https://www.elastic.co/products/kibana
● https://www.elastic.co/products/logstash
● https://www.elastic.co/products/elasticsearch
● Easy install-config manual: http://knes1.github.io/blog/2015/2015-08-16-manage-spring-boot-logs-with-
elasticsearch-kibana-and-logstash.html
● Default patterns for Grok parser: https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns
● Web-site to test regular expressions: https://regex101.com/
● Web-site to test Grok expression: http://grokconstructor.appspot.com
● Search how-to: https://www.elastic.co/guide/en/elasticsearch/reference/6.x/query-dsl-query-string-
query.html#query-string-syntax

22. Thank you for your
attention!
Any Questions?