Downloaded 70 times
![Non-consistent log format
TOMCAT LOGS
A typical tomcat server startup log entry will look like this:
May 24, 2015 3:56:26 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deployment of web application archive softapache-tomcat-7.0.62webappssample.war has
finished in 253 ms
APACHE ACCESS LOGS – COMBINED LOG FORMAT
A typical Apache access log entry will look like this:
127.0.0.1 - - [24/May/2015:15:54:59 +0530] "GET /favicon.ico HTTP/1.1" 200 21630
IIS LOGS
A typical IIS log entry will look like this:
2012-05-02 17:42:15 172.24.255.255 - 172.20.255.255 80 GET /images/favicon.ico - 200 Mozilla/
4.0+(compatible;MSIE+5.5;+Windows+2000+Server)](https://image.slidesharecdn.com/technology-behind-real-time-log-analytics1-160528095346/85/Technology-behind-real-time-log-analytics-6-320.jpg)
Uploaded byData Science Thailand
Technology behind-real-time-log-analytics
This document discusses Real Time Log Analytics using the ELK (Elasticsearch, Logstash, Kibana) stack. It provides an overview of each component, including Elasticsearch for indexing and searching logs, Logstash for collecting, parsing, and enriching logs, and Kibana for visualizing and analyzing logs. It describes common use cases for log analytics like issue debugging and security analysis. It also covers challenges like non-consistent log formats and decentralized logs. The document includes examples of log entries from different systems and how ELK addresses issues like scalability and making logs easily searchable and reportable.
More Related Content
Similar to Technology behind-real-time-log-analytics
Technology behind-real-time-log-analytics
- 1. Technology behind RealTime Log Analytics ELK- Elasticsearch, Logstash and Kibana By Supaket Wongkampoo @ Predictive Analytics and Data Science Conference 28 May 2016
- 2. SUPAKET WONGKAMPOO Software Engineer@ Agoda *DevOps in passion* - Full Stack Developer - Virtualisation and Infrastruction as code (Puppet/Ansible) - Release Management and continuous development - Real time Log Analytics
- 3. State of theArt, Logging Terminology in Large Scale Data processing
- 4. Common use cases •*Issuedebugging •*Performance analysis •Security analysis •*Predictive analysis •Internet of things (IoT) and logging
- 5. Challenges in loganalysis •*Non-consistent log format •*Decentralized logs •Expert knowledge requirement
- 6. Non-consistent log format TOMCATLOGS A typical tomcat server startup log entry will look like this: May 24, 2015 3:56:26 PM org.apache.catalina.startup.HostConfig deployWAR INFO: Deployment of web application archive softapache-tomcat-7.0.62webappssample.war has finished in 253 ms APACHE ACCESS LOGS – COMBINED LOG FORMAT A typical Apache access log entry will look like this: 127.0.0.1 - - [24/May/2015:15:54:59 +0530] "GET /favicon.ico HTTP/1.1" 200 21630 IIS LOGS A typical IIS log entry will look like this: 2012-05-02 17:42:15 172.24.255.255 - 172.20.255.255 80 GET /images/favicon.ico - 200 Mozilla/ 4.0+(compatible;MSIE+5.5;+Windows+2000+Server)
- 7. DECENTRALIZED LOGS For oneor two servers' setup, finding out some information from logs involves running cat or tail commands or piping these results to grep command.
- 8.
- 9. Elasticsearch - Keyfeature •• Schema-free, REST & JSON based document store •• Distributed and horizontally scalable •• Open Source: Apache License 2.0 •• Zero configuration •• Written in Java, extensible
- 10. Elasticsearch - Term •Index - Logical collection of data; might be time based Analogous to a database • Replications - Read scalability, Removing SPOF • Sharding - Split logical data over several machines Write scalability, Control data flows
- 11. Elasticsearch - Distributedand scalable
- 12. Elasticsearch - Distributedand scalable
- 13. Elasticsearch - usecases • Product search engine, Products grouped, Allowing to filter • Scoring ✴ Possible influential factors, Age of the product, been ordered in last 24h In Stock?, No shipping costs, Special offer, Rating • Analytics ✴ Aggregation, multidimensional (Average revenue per category id per day)
- 14.
- 15. Logstash • Managing eventsand logs • Collect, parse, enrich, store data • Modular: many, many inputs and outputs • Open Source: Apache License 2.0 • Ruby app • Part of Elasticsearch family
- 16. Why collect ¢ralize logs? •Access log files without system access •Shell scripting: Too limited or slow •Using unique ids for errors, aggregate it across your stack •Reporting (everyone can create his/her own report) •Bonus points: Unify your data to make it easily •Searchable
- 17.
- 18. Logstash-Inputs • Monitoring: collectd,graphite, ganglia, snmptrap, zenoss • Datastores: elasticsearch, redis, sqlite, s3 • Queues: rabbitmq, zeromq, kafka • Logging: eventlog, lumberjack, gelf, log4j, relp, syslog, varnish log
- 19. Logstash-Filters •alter, anonymize, checksum,csv, drop, multiline •dns, date, extractnumbers, geoip, i18n, kv, noop, ruby, range •json, urldecode, useragent
- 20. Logstash-Outputs • Store: elasticsearch,gemfire, mongodb, redis, riak, rabbitmq • Monitoring: ganglia, graphite, graphtastic, nagios, opentsdb, statsd, zabb • Notification: email, hipchat, irc, pagerduty, sns • Protocol: http, lumberjack, metriccatcher, stomp,
- 21. Kibana •Flexible analytics anddata visualization platform
- 22.
- 23.
- 24. Hands on -ELK Web Web Web Web Web Web KafKa
- 25.