Advertisement

More Related Content

Similar to Logstash: Get to know your logs(20)

Advertisement
Advertisement

Logstash: Get to know your logs

  1. Logstash! Get to know your logs Dan Ivovich BMore on Rails 4/9/13
  2. Dan Ivovich SmartLogic Solutions http://smartlogicsolutions.com Twitter - @danivovich
  3. What is the goal? ● Collect, Parse, and Store your log events ● Make log events searchable ● Analyze log events
  4. Why bother? ● Got logs? ○ syslog ○ nginx access log ○ application logs ○ database logs Are they all formatted the same?
  5. 3 Parts ● Inputs ● Filters ● Outputs
  6. Inputs ● Files ● TCP/UDP ● Redis ● AMQP ● rsyslog ● xmpp http://logstash.net/docs/1.1.9/ - Full list
  7. Filters ● grep ● mutate ● anonymize ● date ● grok http://logstash.net/docs/1.1.9/ - Full list
  8. Outputs ● Files ● TCP/UDP ● Redis ● AMQP ● elasticsearch http://logstash.net/docs/1.1.9/ - Full list
  9. Getting Started input { stdin { type => "stdin-type"} } output { stdout { debug => true debug_format => "json"} } java -jar logstash-1.1.9-monolithic.jar agent -f logstash-simple.conf Type something!
  10. See our message!
  11. Parse something! input { stdin { type => "stdin-type"} } filter { grok { type => "stdin-type" pattern => "Hello %{DATA:message}!" } } output { stdout { debug => true debug_format => "json"} } java -jar logstash-1.1.9-monolithic.jar agent -f logstash-simple.conf Say Hello!
  12. See our message in a field!
  13. Life is better with search input { stdin { type => "stdin-type" } } output { stdout { debug => true debug_format => "json" } elasticsearch { embedded => true } } java -jar logstash-1.1.9-monolithic.jar agent -f logstash-search.conf cURL for it!
  14. Search for the data
  15. Well that isn't pretty Enter Kibana
  16. Kibana is a friendly interface for your logs
  17. Kibana Connects to Elasticsearch ● Logstash parses and structures data into Elasticsearch ● Kibana makes that data available ● Apache Lucene Query Syntax (from elasticsearch) ● Field statistics ● Range searches How do we put it together?
  18. It Was Simple to Start input { stdin { type => "stdin-type" } } output { stdout { debug => true debug_format => "json" } elasticsearch { embedded => true } } java -jar logstash-1.1.9-monolithic.jar agent -f logstash-search.conf But Let's Get Real
  19. On a server with logs
  20. Logstash/Elasticsearch
  21. Demo
  22. Thoughts.... ● Easy to try out, but for anything real, you'll want a much more complicated configuration ● The variety of inputs is great ● Easy to build up a nice stack of filters
  23. More Thoughts.... ● Slow to boot monolithic jar file can be frustrating ○ Flatjar? ● Hard to track down why logs aren't flowing ● Elasticsearch node discovery can be difficult ○ If your cluster doesn't have a node added to it when your client starts, your client isn't connected
  24. More Information ● logstash.net ● grokdebug.herokuapp.com ● www.elasticsearch.org
  25. Questions? http://smartlogicsolutions.com http://twitter.com/smartlogic http://github.com/smartlogic   http://fb.me/smartlogic
Advertisement