Successfully reported this slideshow.

Logstash: Get to know your logs

14

Share

Loading in …3
×
1 of 26
1 of 26

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Logstash: Get to know your logs

  1. 1. Logstash! Get to know your logs Dan Ivovich BMore on Rails 4/9/13
  2. 2. Dan Ivovich SmartLogic Solutions http://smartlogicsolutions.com Twitter - @danivovich
  3. 3. What is the goal? ● Collect, Parse, and Store your log events ● Make log events searchable ● Analyze log events
  4. 4. Why bother? ● Got logs? ○ syslog ○ nginx access log ○ application logs ○ database logs Are they all formatted the same?
  5. 5. 3 Parts ● Inputs ● Filters ● Outputs
  6. 6. Inputs ● Files ● TCP/UDP ● Redis ● AMQP ● rsyslog ● xmpp http://logstash.net/docs/1.1.9/ - Full list
  7. 7. Filters ● grep ● mutate ● anonymize ● date ● grok http://logstash.net/docs/1.1.9/ - Full list
  8. 8. Outputs ● Files ● TCP/UDP ● Redis ● AMQP ● elasticsearch http://logstash.net/docs/1.1.9/ - Full list
  9. 9. Getting Started input { stdin { type => "stdin-type"} } output { stdout { debug => true debug_format => "json"} } java -jar logstash-1.1.9-monolithic.jar agent -f logstash-simple.conf Type something!
  10. 10. See our message!
  11. 11. Parse something! input { stdin { type => "stdin-type"} } filter { grok { type => "stdin-type" pattern => "Hello %{DATA:message}!" } } output { stdout { debug => true debug_format => "json"} } java -jar logstash-1.1.9-monolithic.jar agent -f logstash-simple.conf Say Hello!
  12. 12. See our message in a field!
  13. 13. Life is better with search input { stdin { type => "stdin-type" } } output { stdout { debug => true debug_format => "json" } elasticsearch { embedded => true } } java -jar logstash-1.1.9-monolithic.jar agent -f logstash-search.conf cURL for it!
  14. 14. Search for the data
  15. 15. Well that isn't pretty Enter Kibana
  16. 16. Kibana is a friendly interface for your logs
  17. 17. Kibana Connects to Elasticsearch ● Logstash parses and structures data into Elasticsearch ● Kibana makes that data available ● Apache Lucene Query Syntax (from elasticsearch) ● Field statistics ● Range searches How do we put it together?
  18. 18. It Was Simple to Start input { stdin { type => "stdin-type" } } output { stdout { debug => true debug_format => "json" } elasticsearch { embedded => true } } java -jar logstash-1.1.9-monolithic.jar agent -f logstash-search.conf But Let's Get Real
  19. 19. On a server with logs
  20. 20. Logstash/Elasticsearch
  21. 21. Demo
  22. 22. Thoughts.... ● Easy to try out, but for anything real, you'll want a much more complicated configuration ● The variety of inputs is great ● Easy to build up a nice stack of filters
  23. 23. More Thoughts.... ● Slow to boot monolithic jar file can be frustrating ○ Flatjar? ● Hard to track down why logs aren't flowing ● Elasticsearch node discovery can be difficult ○ If your cluster doesn't have a node added to it when your client starts, your client isn't connected
  24. 24. More Information ● logstash.net ● grokdebug.herokuapp.com ● www.elasticsearch.org
  25. 25. Questions? http://smartlogicsolutions.com http://twitter.com/smartlogic http://github.com/smartlogic   http://fb.me/smartlogic

×