OSDC 2014: Lennart Koopmann - Log Analysis with Graylog2
Upcoming SlideShare
Loading in...5
×
 

OSDC 2014: Lennart Koopmann - Log Analysis with Graylog2

on

  • 583 views

Graylog2 is a free and open source log analysis tool that allows you to perform searches, analyse the data, build dashboards and set alarms using the streams system. Typical use cases range from ...

Graylog2 is a free and open source log analysis tool that allows you to perform searches, analyse the data, build dashboards and set alarms using the streams system. Typical use cases range from debugging platform problems & monitoring exception counts to displaying average pizza delivery time per state on a dashboard.
In this talk I will go through the architecture of Graylog2, what you can do with it and how to get your data into it.

Statistics

Views

Total Views
583
Views on SlideShare
537
Embed Views
46

Actions

Likes
1
Downloads
12
Comments
0

3 Embeds 46

http://www.netways.de 34
http://www.slideee.com 9
https://twitter.com 3

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

OSDC 2014: Lennart Koopmann - Log Analysis with Graylog2 OSDC 2014: Lennart Koopmann - Log Analysis with Graylog2 Presentation Transcript

  • Graylog2 Lennart Koopmann, OSDC 2014 @_lennart / www.graylog2.org
  • About me • 25 years old • Living in Hamburg, Germany • @_lennart on Twitter • Co-Founder of TORCH - The Graylog2 company.
  • Graylog2 history • Started as open source project by Lennart Koopmann in 2010 and was developed entirely in free time. • TORCH founded as company behind it in late 2012 after seeing massive growth and worldwide distribution in large scale setups. • Now team of 6 working full-time on it, three more people joining this summer. (and still hiring) • www.graylog2.org
  • Graylog2 history • Big rewrite of Graylog2 started in 2012 and finished with releasing a final v0.20.0 in February 2014 that addresses what we learnt from our first customers and all users. • Web Interface now (like the server) written in Java and easy to install. Prior versions used Ruby On Rails and were hard to install. • New web Interface focussing on powerful analytics. • Unified REST API communication for easy extending and integrating with other products, tools and scripts.
  • Free and open source analysis of any machine data written in your datacenter. ! Running on the JVM in your own environment. Not limited by licenses.
  • Basic architecture message sources graylog2-server graylog2-server ElasticSearch Cluster MongoDB graylog2-web-interface Your own reporting scripts Your own subscribers RESTREST Inputs
  • Architecture considerations • Use graylog2-radio for HA and high level buffering • Put load balancers in front and scale out horizontally
  • Architecture considerations • graylog2-server / graylog2-radio: Focus on CPU • ElasticSearch: Focus on RAM and IO • MongoDB: Replication set for failover, not much load • graylog2-web-interface: Not much load at all
  • Architecture considerations • http://support.torch.sh/help/kb/general/graylog2-architecture-high-level-overview
  • No message left behind 2014-04-04 14:05:43,147 INFO : org.graylog2.Core - SIGNAL received. Shutting down. 2014-04-04 14:05:43,150 INFO : org.graylog2.system.shutdown.GracefulShutdown - Graceful shutdown initiated. 2014-04-04 14:05:43,150 INFO : org.graylog2.system.shutdown.GracefulShutdown - Node status: [Halting [LB:DEAD]]. Waiting <5sec> for possible load balancers to recognize state change. ! … ! 2014-04-04 14:05:49,156 INFO : org.graylog2.system.shutdown.GracefulShutdown - Attempting to close input <org.graylog2.inputs.raw.udp.RawUDPInput.531f89283004f7b66a87e163> [Raw/Plaintext UDP]. 2014-04-04 14:05:49,157 INFO : org.graylog2.system.shutdown.GracefulShutdown - Input [org.graylog2.inputs.raw.udp.RawUDPInput.531f89283004f7b66a87e163] closed. Took [1ms] … ! 2014-04-04 14:05:49,158 INFO : org.graylog2.caches.Caches - Waiting until all caches are empty. 2014-04-04 14:05:49,158 INFO : org.graylog2.caches.Caches - All caches are empty. Continuing. 2014-04-04 14:05:49,159 INFO : org.graylog2.buffers.Buffers - Waiting until all buffers are empty. 2014-04-04 14:05:49,159 INFO : org.graylog2.buffers.Buffers - All buffers are empty. Continuing. ! … ! 2014-04-04 14:05:49,176 INFO : org.graylog2.system.shutdown.GracefulShutdown - Goodbye.
  • No message left behind
  • GELF http://graylog2.org/gelf • The “Graylog2 Extended Log Format”. Structured and compressed, based on JSON. • Optional UDP chunking allows sending a lot of data without having to care about connection management in your application (timeouts, …) if you don’t need transport security. • Already over 30 libraries from the community and integrated into the first products.
  • Streams • Performant realtime routing of messages based on rules. Matching applied when the message is received and processed. • Create streams like “SSH logins” or “Exceptions in application X” for quick access in the web interface (like saved searches) or alerts. • Be alerted based on message count thresholds or results of statistical computation of given relative time windows. “Send me an alert when the standard deviation of the response time in application X was higher than 100 in the last 10 minutes.” • Forward to other systems based on matched streams. “Forward all business intelligence related logs to another system. (to save license costs)”