Logstash: Get to know your logs

4,412 views
4,099 views

Published on

Dan Ivovich walks through getting started with Logstash

Published in: Technology
1 Comment
13 Likes
Statistics
Notes
No Downloads
Views
Total views
4,412
On SlideShare
0
From Embeds
0
Number of Embeds
104
Actions
Shares
0
Downloads
140
Comments
1
Likes
13
Embeds 0
No embeds

No notes for slide

Logstash: Get to know your logs

  1. 1. Logstash! Get to know your logsDan IvovichBMore on Rails4/9/13
  2. 2. Dan Ivovich SmartLogic Solutionshttp://smartlogicsolutions.comTwitter - @danivovich
  3. 3. What is the goal?● Collect, Parse, and Store your log events● Make log events searchable● Analyze log events
  4. 4. Why bother?● Got logs? ○ syslog ○ nginx access log ○ application logs ○ database logs Are they all formatted the same?
  5. 5. 3 Parts● Inputs● Filters● Outputs
  6. 6. Inputs● Files● TCP/UDP● Redis● AMQP● rsyslog● xmpphttp://logstash.net/docs/1.1.9/ - Full list
  7. 7. Filters● grep● mutate● anonymize● date● grokhttp://logstash.net/docs/1.1.9/ - Full list
  8. 8. Outputs● Files● TCP/UDP● Redis● AMQP● elasticsearchhttp://logstash.net/docs/1.1.9/ - Full list
  9. 9. Getting Startedinput { stdin { type => "stdin-type"} }output { stdout { debug => true debug_format =>"json"} }java -jar logstash-1.1.9-monolithic.jar agent -flogstash-simple.conf Type something!
  10. 10. See our message!
  11. 11. Parse something!input { stdin { type => "stdin-type"} }filter { grok { type => "stdin-type" pattern =>"Hello %{DATA:message}!" } }output { stdout { debug => true debug_format =>"json"} }java -jar logstash-1.1.9-monolithic.jar agent -flogstash-simple.conf Say Hello!
  12. 12. See our message in a field!
  13. 13. Life is better with searchinput { stdin { type => "stdin-type" } }output { stdout { debug => true debug_format => "json" } elasticsearch { embedded => true }}java -jar logstash-1.1.9-monolithic.jar agent -flogstash-search.conf cURL for it!
  14. 14. Search for the data
  15. 15. Well that isnt pretty Enter Kibana
  16. 16. Kibana is a friendlyinterface for your logs
  17. 17. Kibana Connects to Elasticsearch● Logstash parses and structures data into Elasticsearch● Kibana makes that data available● Apache Lucene Query Syntax (from elasticsearch)● Field statistics● Range searchesHow do we put it together?
  18. 18. It Was Simple to Startinput { stdin { type => "stdin-type" } }output { stdout { debug => true debug_format => "json" } elasticsearch { embedded => true }}java -jar logstash-1.1.9-monolithic.jar agent -flogstash-search.conf But Lets Get Real
  19. 19. On a server with logs
  20. 20. Logstash/Elasticsearch
  21. 21. Demo
  22. 22. Thoughts....● Easy to try out, but for anything real, youll want a much more complicated configuration● The variety of inputs is great● Easy to build up a nice stack of filters
  23. 23. More Thoughts....● Slow to boot monolithic jar file can be frustrating ○ Flatjar?● Hard to track down why logs arent flowing● Elasticsearch node discovery can be difficult ○ If your cluster doesnt have a node added to it when your client starts, your client isnt connected
  24. 24. More Information● logstash.net● grokdebug.herokuapp.com● www.elasticsearch.org
  25. 25. Questions?http://smartlogicsolutions.comhttp://twitter.com/smartlogichttp://github.com/smartlogic http://fb.me/smartlogic

×