Log aggregation and analysis using Elastic Stack.

1. Log Aggregation and Analysis
Using Elastic Stack

3. 1
Logging - What ?
●
Human readable and machine parseable format
●
Record now, analyze later
●
Various sources of events
– Webservers
– User activity on a website
– Application logs
– Node metrics
– Other operational data
– Mobile / IoT devices

4. 2
Logging - Why ?
●
Record keeping
●
Operational Insights
– Monitor and optimize resource consumption / utilization
– Early detection – find out before it goes wrong
●
Incident forensics
– Where did it go wrong ?
– Consistency related bugs – Failing system is better than an incosistent system
●
Answer Questions in (near)real-time
– Functional metrics – How many users logged in within last hour?, Which location is most active right now?,
What's the average response time for X page?

5. 3
Available tools
Splunk Graylog Elastic stack
License Paid Free Freemium
Setup
complexity
Easy Medium Medium
Hosting On premise / Hosted On premise / Hosted On-premise / Hosted
Capabilities Ingestion / Storage / Analytics /
Alerts
Ingestion / Storage / Analytics /
Alerts
Ingestion / Storage /
Analytics / Alerts
Scalable ? Yes Yes Yes
Architecture Monolith Monolith Divided into different
components – Each can
be used separately
And many more - Grafana, Logsearch, MS Azure Log Analytics, Loggly

6. 4
Elastic Stack
Explore and visualize your data.
Search, Dashboards and many
more.
Lightweight
data shippers
Parse,
Enrich &
Transport
Data
Store, search, and analyze your
data.

7. 5
Step 1 - Gathering logs
Lightweight Data Shippers
Beats is the platform for single-purpose data
shippers. They install as lightweight agents and
send data from hundreds or thousands of
machines to Logstash or Elasticsearch.
●
Using log appenders / handlers
●
Beats

8. 6
Beats (by Dre Elastic)
●
Filebeat
– Reads from file
– Non-intrusive
●
Metricbeat
– Collects metrics from systems and services
– Modules available for Apache, nginx, Docker, Kafka, PostgreSQL and more
●
Packetbeat
– Lightweight network packet analyzer
– Modules available for HTTP, DNS, AMQP and more
●
Winlogbeat
– Collects windows event logs
●
Add your own
– Dozens of community developed beats available
– Extensible architecture – Easy to create on our own
– Written in Go

9. 7
Filebeat
filebeat:
prospectors:
­
paths:
­ /var/log/*.log
input_type: log
­
paths:
­ /var/log/apache2/*
input_type: log
output:
elasticsearch:
hosts: ["localhost:9200"]
Sample confiuration :
{
"@timestamp": "2016­01­06T00:00:00Z",
"type": "log",
"message": "<line from file>"
}
Sample published event:

10. 8
Metricbeat
metricbeat.modules:
­ module: redis
metricsets:
­ info
hosts: ["127.0.0.1:6379"]
output.elasticsearch:
hosts: ["localhost:9200"]
Sample configuration:
{
"@timestamp": "2016­01­06T00:00:00Z",
"type": "metricsets",
"redis": {
...
}
}
Sample published event:

11. 9
Step 2 - Processing logs using Logstash
●
Ingest-process-output pipeline
●
Ingest Data of All Shapes, Sizes, and Sources
– Beats, log4j, redis, tcp/udp, HTTP
●
Process
– Transform unstructured data to structured data using grok filter
– Filter out unnecessary data
– Mutate data (calculate fields, add extra context, get geo co-ordinates from IP address, etc)
●
Stash it away
– Data stores (elasticsearch, files, mongoDB, redis), other services (email, pagerduty, redmine,
irc, jira), brokers (kafka, rabbitMQ) and many more
●
Scalable, Durable

12. 10
Logstash – contd.
GeoIP Lookup /
Data Mutation /
Structuring data
input {
beats {
port => 5044
}
}
filter{
if[type] == "nginx­logs" {
grok {
match => {"message" => "%{PATTERN}"}
}
geoip {
source => "remote_addr"
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash­%{+YYYY.MM.dd}"
}
}
Sample confiuration :

13. 11
Step 3 - Storing logs in Elasticsearch
●
Distributed RESTful search and analytics engine (JSON/HTTP)
●
Fast – get your answers instantly
●
Scalable – Run on your laptop or hundreds of servers
●
Resilient and Highly Available – Clustering, Failure detection
●
Full text search, Aggregation, Geo filtering (within x mile radius),
Suggestions (show more like this), Fuzzy search, Scripting

14. 12
Step 4 - Explore and Visualize using Kibana
●
Works seamlessly with Elasticsearch
●
Easy yet powerful search interface
●
Supports histograms, line graphs, pie charts and many more
●
Visualize geospatial data
●
Extensible – Create your own visualization
●
Create and share dashboards

15. 13
Elastic Stack in Action

16. 14
Demo 1 – Parsing nginx logs
Nginx
Access
Logs
Filebeat
conf
Filebeat
Read file(s)
push to Logstash
Logstash
Parse logs
GeoIP lookup
User agent parsing
Push to Elasticsearch
Elasticsearch
Index and store
Kibana
Search and Analyze

17. 15
Demo 2 – Logs from a Django application
Elasticsearch
Index and store
Kibana
Search and Analyze
Logstash
Collect logs
Push to Elasticsearch
Django App
Logstash handler

18. 16
Demo 3 – Capture and monitor node metrics
Metricbeat
conf
Metricbeat
Read metrics
from nodes
Elasticsearch
Index and store
Kibana
Search and Analyze

19. 17
Q & A

20. 18
Thank You
Ahmedabad Java Meetup Group
https://www.meetup.com/Ahmedabad-Java-Meetup-Group/
@JavaMeetup AhmedabadJavaMeetup
Dhaval Mehta
dmehta17@gmail.com
@mehtadhaval07