Your SlideShare is downloading. ×
Open source log analytics
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Open source log analytics

301

Published on

Open source log analytics

Open source log analytics

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
301
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Open source Scalable Log Analytics Presented by Vinod Nayal
  • 2. Log Analytics Overview Collection search and analysis of log collected from various app servers Ability to search by attributes within a timeframe and ability to export related log files Real time Reports/dash-board like specific events per hour
  • 3. Solution Architecture Redis Broker Logstash -Indexer Elastic Search Redis Broker Mongodb writer Indexer weserver lumberjack webserver lumberjack webserver lumberjack Elastic Search Elastic Search mongodb mongodb mongodb KibanaUI+D3.js Agent (Web browser)  Lumberjack ,Log stash , Redis Log collection  Elastic Search Indexing  Mongodb Document Storage for 1 week  Kibana,D3.js UI
  • 4. Dashboard  Ability to search and filter by any attribute  Customiz able Time series graphs  Various aggregati on across time geographi es host etc H I G H L I G H T S
  • 5. Solution Highlights  Log indexing in Elastic search distributed cluster.  Log collection via lumberjack( logstash-forwarder) on various client nodes . It has a very low memory footprint . It support compression and encryption in log transmission .  Collected logs are sent to logstash –servers which saves to elastic search for indexing . log file are also sent to mongodb for keeping original data for export and future integrated view . Documents in mongodb will have a retention period of 5 -7 days  Redis is used for buffering log events at server side , it make system able to take peak loads without failure . It also provides pub sub architecture for sending logs to multiple processing concurrently  Log enrichment and filtering capability with logstash filters and pluggable architecture  Kibana Integration for Spunk like UI for log searching and analysis  All technologies used are open source ,scalable ,distributed and customisable
  • 6. Solution Details – Why Elastic Search  Distributed Elastic search allows you to start small, but will grow with your business. It is built to scale horizontally out of the box. As you need more capacity, just add more nodes, and l et the cluster reorganize itself to take advantage of the extra hardware.  Multi-tenancy A cluster can host multiple indices which can be queried independently or as a group. Index aliases allow you to add indexes on the fly, while being transparent to your application.  Schema free Elastic search allows you to get started easily. Toss it a JSON document and it will try to detect the data structure, index the data and make it searchable. Later, apply your domain specific knowledge of your data to customize how your data is indexed.
  • 7. Solution Details – Why LogStash  Configurable and customizable log collection that can be scaled by adding more nodes at server side  Inputs specifies where to watch for logs .  Filter and grok gives filtering and regular expression capability  Output can be directed to elastic search / mongodb Redis/ logstash servers etc
  • 8. Solution Details – Why Kibana  Elasticsearch works seamlessly with kibana and gives ability to interact with your data for visualizing logs and time-stamped data  Highly scalable and Real-time analysis of streaming data  Customisable splunk like UI and can integrate with D3.js for augmenting capability
  • 9. Vinod Nayal Thank You

×