Open source log analytics


Published on

Open source log analytics

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Open source log analytics

  1. 1. Open source Scalable Log Analytics Presented by Vinod Nayal
  2. 2. Log Analytics Overview Collection search and analysis of log collected from various app servers Ability to search by attributes within a timeframe and ability to export related log files Real time Reports/dash-board like specific events per hour
  3. 3. Solution Architecture Redis Broker Logstash -Indexer Elastic Search Redis Broker Mongodb writer Indexer weserver lumberjack webserver lumberjack webserver lumberjack Elastic Search Elastic Search mongodb mongodb mongodb KibanaUI+D3.js Agent (Web browser)  Lumberjack ,Log stash , Redis Log collection  Elastic Search Indexing  Mongodb Document Storage for 1 week  Kibana,D3.js UI
  4. 4. Dashboard  Ability to search and filter by any attribute  Customiz able Time series graphs  Various aggregati on across time geographi es host etc H I G H L I G H T S
  5. 5. Solution Highlights  Log indexing in Elastic search distributed cluster.  Log collection via lumberjack( logstash-forwarder) on various client nodes . It has a very low memory footprint . It support compression and encryption in log transmission .  Collected logs are sent to logstash –servers which saves to elastic search for indexing . log file are also sent to mongodb for keeping original data for export and future integrated view . Documents in mongodb will have a retention period of 5 -7 days  Redis is used for buffering log events at server side , it make system able to take peak loads without failure . It also provides pub sub architecture for sending logs to multiple processing concurrently  Log enrichment and filtering capability with logstash filters and pluggable architecture  Kibana Integration for Spunk like UI for log searching and analysis  All technologies used are open source ,scalable ,distributed and customisable
  6. 6. Solution Details – Why Elastic Search  Distributed Elastic search allows you to start small, but will grow with your business. It is built to scale horizontally out of the box. As you need more capacity, just add more nodes, and l et the cluster reorganize itself to take advantage of the extra hardware.  Multi-tenancy A cluster can host multiple indices which can be queried independently or as a group. Index aliases allow you to add indexes on the fly, while being transparent to your application.  Schema free Elastic search allows you to get started easily. Toss it a JSON document and it will try to detect the data structure, index the data and make it searchable. Later, apply your domain specific knowledge of your data to customize how your data is indexed.
  7. 7. Solution Details – Why LogStash  Configurable and customizable log collection that can be scaled by adding more nodes at server side  Inputs specifies where to watch for logs .  Filter and grok gives filtering and regular expression capability  Output can be directed to elastic search / mongodb Redis/ logstash servers etc
  8. 8. Solution Details – Why Kibana  Elasticsearch works seamlessly with kibana and gives ability to interact with your data for visualizing logs and time-stamped data  Highly scalable and Real-time analysis of streaming data  Customisable splunk like UI and can integrate with D3.js for augmenting capability
  9. 9. Vinod Nayal Thank You