OSDC 2014: Lennart Koopmann - Log Analysis with Graylog2


Published on

Graylog2 is a free and open source log analysis tool that allows you to perform searches, analyse the data, build dashboards and set alarms using the streams system. Typical use cases range from debugging platform problems & monitoring exception counts to displaying average pizza delivery time per state on a dashboard.
In this talk I will go through the architecture of Graylog2, what you can do with it and how to get your data into it.

Published in: Software, Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

OSDC 2014: Lennart Koopmann - Log Analysis with Graylog2

  1. 1. Graylog2 Lennart Koopmann, OSDC 2014 @_lennart / www.graylog2.org
  2. 2. About me • 25 years old • Living in Hamburg, Germany • @_lennart on Twitter • Co-Founder of TORCH - The Graylog2 company.
  3. 3. Graylog2 history • Started as open source project by Lennart Koopmann in 2010 and was developed entirely in free time. • TORCH founded as company behind it in late 2012 after seeing massive growth and worldwide distribution in large scale setups. • Now team of 6 working full-time on it, three more people joining this summer. (and still hiring) • www.graylog2.org
  4. 4. Graylog2 history • Big rewrite of Graylog2 started in 2012 and finished with releasing a final v0.20.0 in February 2014 that addresses what we learnt from our first customers and all users. • Web Interface now (like the server) written in Java and easy to install. Prior versions used Ruby On Rails and were hard to install. • New web Interface focussing on powerful analytics. • Unified REST API communication for easy extending and integrating with other products, tools and scripts.
  5. 5. Free and open source analysis of any machine data written in your datacenter. ! Running on the JVM in your own environment. Not limited by licenses.
  6. 6. Basic architecture message sources graylog2-server graylog2-server ElasticSearch Cluster MongoDB graylog2-web-interface Your own reporting scripts Your own subscribers RESTREST Inputs
  7. 7. Architecture considerations • Use graylog2-radio for HA and high level buffering • Put load balancers in front and scale out horizontally
  8. 8. Architecture considerations • graylog2-server / graylog2-radio: Focus on CPU • ElasticSearch: Focus on RAM and IO • MongoDB: Replication set for failover, not much load • graylog2-web-interface: Not much load at all
  9. 9. Architecture considerations • http://support.torch.sh/help/kb/general/graylog2-architecture-high-level-overview
  10. 10. No message left behind 2014-04-04 14:05:43,147 INFO : org.graylog2.Core - SIGNAL received. Shutting down. 2014-04-04 14:05:43,150 INFO : org.graylog2.system.shutdown.GracefulShutdown - Graceful shutdown initiated. 2014-04-04 14:05:43,150 INFO : org.graylog2.system.shutdown.GracefulShutdown - Node status: [Halting [LB:DEAD]]. Waiting <5sec> for possible load balancers to recognize state change. ! … ! 2014-04-04 14:05:49,156 INFO : org.graylog2.system.shutdown.GracefulShutdown - Attempting to close input <org.graylog2.inputs.raw.udp.RawUDPInput.531f89283004f7b66a87e163> [Raw/Plaintext UDP]. 2014-04-04 14:05:49,157 INFO : org.graylog2.system.shutdown.GracefulShutdown - Input [org.graylog2.inputs.raw.udp.RawUDPInput.531f89283004f7b66a87e163] closed. Took [1ms] … ! 2014-04-04 14:05:49,158 INFO : org.graylog2.caches.Caches - Waiting until all caches are empty. 2014-04-04 14:05:49,158 INFO : org.graylog2.caches.Caches - All caches are empty. Continuing. 2014-04-04 14:05:49,159 INFO : org.graylog2.buffers.Buffers - Waiting until all buffers are empty. 2014-04-04 14:05:49,159 INFO : org.graylog2.buffers.Buffers - All buffers are empty. Continuing. ! … ! 2014-04-04 14:05:49,176 INFO : org.graylog2.system.shutdown.GracefulShutdown - Goodbye.
  11. 11. No message left behind
  12. 12. GELF http://graylog2.org/gelf • The “Graylog2 Extended Log Format”. Structured and compressed, based on JSON. • Optional UDP chunking allows sending a lot of data without having to care about connection management in your application (timeouts, …) if you don’t need transport security. • Already over 30 libraries from the community and integrated into the first products.
  13. 13. Streams • Performant realtime routing of messages based on rules. Matching applied when the message is received and processed. • Create streams like “SSH logins” or “Exceptions in application X” for quick access in the web interface (like saved searches) or alerts. • Be alerted based on message count thresholds or results of statistical computation of given relative time windows. “Send me an alert when the standard deviation of the response time in application X was higher than 100 in the last 10 minutes.” • Forward to other systems based on matched streams. “Forward all business intelligence related logs to another system. (to save license costs)”