This document summarizes an open source scalable log analytics solution. The solution uses Lumberjack for log collection, Logstash for indexing and filtering logs, Redis for buffering, Elasticsearch for indexing and searching logs, MongoDB for document storage, and Kibana with D3.js for visualization. Logs are collected from servers by Lumberjack, sent to Logstash for processing, indexed by Elasticsearch for searching, stored in MongoDB for retrieval, and visualized through dashboards and reports in Kibana. The solution allows for real-time log analysis, flexible searching and filtering, and scales horizontally as needs grow.

1. Open source Scalable Log Analytics
Presented by Vinod Nayal

2. Log Analytics Overview
Collection search and analysis of log collected from
various app servers
Ability to search by attributes within a timeframe and ability
to export related log files
Real time Reports/dash-board like specific events per
hour

3. Solution Architecture
Redis
Broker
Logstash
-Indexer
Elastic Search
Redis
Broker
Mongodb
writer
Indexer
weserver
lumberjack
webserver
lumberjack
webserver
lumberjack
Elastic Search
Elastic Search
mongodb
mongodb
mongodb
KibanaUI+D3.js
Agent (Web
browser)
 Lumberjack ,Log stash , Redis Log collection
 Elastic Search Indexing
 Mongodb Document Storage for 1 week
 Kibana,D3.js UI

4. Dashboard
 Ability to
search
and filter
by any
attribute
 Customiz
able Time
series
graphs
 Various
aggregati
on across
time
geographi
es host
etc
H I G H L I G H T S

5. Solution Highlights
 Log indexing in Elastic search distributed cluster.
 Log collection via lumberjack( logstash-forwarder) on various client
nodes . It has a very low memory footprint . It support compression
and encryption in log transmission .
 Collected logs are sent to logstash –servers which saves to elastic
search for indexing . log file are also sent to mongodb for keeping
original data for export and future integrated view . Documents in
mongodb will have a retention period of 5 -7 days
 Redis is used for buffering log events at server side , it make system
able to take peak loads without failure . It also provides pub sub
architecture for sending logs to multiple processing concurrently
 Log enrichment and filtering capability with logstash filters and
pluggable architecture
 Kibana Integration for Spunk like UI for log searching and analysis
 All technologies used are open source ,scalable ,distributed and
customisable

6. Solution Details – Why Elastic Search
 Distributed
Elastic search allows you to start small, but will grow with your
business. It is built to scale horizontally out of the box. As you
need more capacity, just add more nodes, and l et the cluster
reorganize itself to take advantage of the extra hardware.
 Multi-tenancy
A cluster can host multiple indices which can be queried
independently or as a group. Index aliases allow you to add
indexes on the fly, while being transparent to your application.
 Schema free
Elastic search allows you to get started easily. Toss it a JSON
document and it will try to detect the data structure, index the
data and make it searchable. Later, apply your domain specific
knowledge of your data to customize how your data is indexed.

7. Solution Details – Why LogStash
 Configurable and
customizable log
collection that can be
scaled by adding more
nodes at server side
 Inputs specifies where
to watch for logs .
 Filter and grok gives
filtering and regular
expression capability
 Output can be directed
to elastic search /
mongodb Redis/
logstash servers etc

8. Solution Details – Why Kibana
 Elasticsearch works seamlessly with kibana and gives ability to
interact with your data for visualizing logs and time-stamped data
 Highly scalable and Real-time analysis of streaming data
 Customisable splunk like UI and can integrate with D3.js for
augmenting capability

9. Vinod Nayal
Thank You