Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Logstash!                 Get to know your logsDan IvovichBMore on Rails4/9/13
Dan Ivovich          SmartLogic Solutionshttp://smartlogicsolutions.comTwitter - @danivovich
What is the goal?● Collect, Parse, and Store your log events● Make log events searchable● Analyze log events
Why bother?● Got logs?  ○ syslog  ○ nginx access log  ○ application logs  ○ database logs Are they all formatted the      ...
3 Parts● Inputs● Filters● Outputs
Inputs●   Files●   TCP/UDP●   Redis●   AMQP●   rsyslog●   xmpphttp://logstash.net/docs/1.1.9/ - Full list
Filters●   grep●   mutate●   anonymize●   date●   grokhttp://logstash.net/docs/1.1.9/ - Full list
Outputs●   Files●   TCP/UDP●   Redis●   AMQP●   elasticsearchhttp://logstash.net/docs/1.1.9/ - Full list
Getting Startedinput { stdin { type => "stdin-type"} }output { stdout { debug => true debug_format =>"json"} }java -jar lo...
See our message!
Parse something!input { stdin { type => "stdin-type"} }filter { grok { type => "stdin-type" pattern =>"Hello %{DATA:messag...
See our message in a field!
Life is better with searchinput { stdin { type => "stdin-type" } }output {  stdout { debug => true debug_format => "json" ...
Search for the data
Well that isnt pretty    Enter Kibana
Kibana is a friendlyinterface for your logs
Kibana Connects to Elasticsearch● Logstash parses and structures data into    Elasticsearch●   Kibana makes that data avai...
It Was Simple to Startinput { stdin { type => "stdin-type" } }output {  stdout { debug => true debug_format => "json" }  e...
On a server with logs
Logstash/Elasticsearch
Demo
Thoughts....● Easy to try out, but for anything real, youll want a much   more complicated configuration● The variety of i...
More Thoughts....● Slow to boot monolithic jar file can be frustrating   ○ Flatjar?● Hard to track down why logs arent flo...
More Information● logstash.net● grokdebug.herokuapp.com● www.elasticsearch.org
Questions?http://smartlogicsolutions.comhttp://twitter.com/smartlogichttp://github.com/smartlogic http://fb.me/smartlogic
Logstash: Get to know your logs
Upcoming SlideShare
Loading in …5
×

Logstash: Get to know your logs

4,893 views

Published on

Dan Ivovich walks through getting started with Logstash

Published in: Technology

Logstash: Get to know your logs

  1. 1. Logstash! Get to know your logsDan IvovichBMore on Rails4/9/13
  2. 2. Dan Ivovich SmartLogic Solutionshttp://smartlogicsolutions.comTwitter - @danivovich
  3. 3. What is the goal?● Collect, Parse, and Store your log events● Make log events searchable● Analyze log events
  4. 4. Why bother?● Got logs? ○ syslog ○ nginx access log ○ application logs ○ database logs Are they all formatted the same?
  5. 5. 3 Parts● Inputs● Filters● Outputs
  6. 6. Inputs● Files● TCP/UDP● Redis● AMQP● rsyslog● xmpphttp://logstash.net/docs/1.1.9/ - Full list
  7. 7. Filters● grep● mutate● anonymize● date● grokhttp://logstash.net/docs/1.1.9/ - Full list
  8. 8. Outputs● Files● TCP/UDP● Redis● AMQP● elasticsearchhttp://logstash.net/docs/1.1.9/ - Full list
  9. 9. Getting Startedinput { stdin { type => "stdin-type"} }output { stdout { debug => true debug_format =>"json"} }java -jar logstash-1.1.9-monolithic.jar agent -flogstash-simple.conf Type something!
  10. 10. See our message!
  11. 11. Parse something!input { stdin { type => "stdin-type"} }filter { grok { type => "stdin-type" pattern =>"Hello %{DATA:message}!" } }output { stdout { debug => true debug_format =>"json"} }java -jar logstash-1.1.9-monolithic.jar agent -flogstash-simple.conf Say Hello!
  12. 12. See our message in a field!
  13. 13. Life is better with searchinput { stdin { type => "stdin-type" } }output { stdout { debug => true debug_format => "json" } elasticsearch { embedded => true }}java -jar logstash-1.1.9-monolithic.jar agent -flogstash-search.conf cURL for it!
  14. 14. Search for the data
  15. 15. Well that isnt pretty Enter Kibana
  16. 16. Kibana is a friendlyinterface for your logs
  17. 17. Kibana Connects to Elasticsearch● Logstash parses and structures data into Elasticsearch● Kibana makes that data available● Apache Lucene Query Syntax (from elasticsearch)● Field statistics● Range searchesHow do we put it together?
  18. 18. It Was Simple to Startinput { stdin { type => "stdin-type" } }output { stdout { debug => true debug_format => "json" } elasticsearch { embedded => true }}java -jar logstash-1.1.9-monolithic.jar agent -flogstash-search.conf But Lets Get Real
  19. 19. On a server with logs
  20. 20. Logstash/Elasticsearch
  21. 21. Demo
  22. 22. Thoughts....● Easy to try out, but for anything real, youll want a much more complicated configuration● The variety of inputs is great● Easy to build up a nice stack of filters
  23. 23. More Thoughts....● Slow to boot monolithic jar file can be frustrating ○ Flatjar?● Hard to track down why logs arent flowing● Elasticsearch node discovery can be difficult ○ If your cluster doesnt have a node added to it when your client starts, your client isnt connected
  24. 24. More Information● logstash.net● grokdebug.herokuapp.com● www.elasticsearch.org
  25. 25. Questions?http://smartlogicsolutions.comhttp://twitter.com/smartlogichttp://github.com/smartlogic http://fb.me/smartlogic

×