Logstash: Get to know your logs

  • 3,065 views
Uploaded on

Dan Ivovich walks through getting started with Logstash

Dan Ivovich walks through getting started with Logstash

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
3,065
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
97
Comments
1
Likes
13

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Logstash! Get to know your logsDan IvovichBMore on Rails4/9/13
  • 2. Dan Ivovich SmartLogic Solutionshttp://smartlogicsolutions.comTwitter - @danivovich
  • 3. What is the goal?● Collect, Parse, and Store your log events● Make log events searchable● Analyze log events
  • 4. Why bother?● Got logs? ○ syslog ○ nginx access log ○ application logs ○ database logs Are they all formatted the same?
  • 5. 3 Parts● Inputs● Filters● Outputs
  • 6. Inputs● Files● TCP/UDP● Redis● AMQP● rsyslog● xmpphttp://logstash.net/docs/1.1.9/ - Full list
  • 7. Filters● grep● mutate● anonymize● date● grokhttp://logstash.net/docs/1.1.9/ - Full list
  • 8. Outputs● Files● TCP/UDP● Redis● AMQP● elasticsearchhttp://logstash.net/docs/1.1.9/ - Full list
  • 9. Getting Startedinput { stdin { type => "stdin-type"} }output { stdout { debug => true debug_format =>"json"} }java -jar logstash-1.1.9-monolithic.jar agent -flogstash-simple.conf Type something!
  • 10. See our message!
  • 11. Parse something!input { stdin { type => "stdin-type"} }filter { grok { type => "stdin-type" pattern =>"Hello %{DATA:message}!" } }output { stdout { debug => true debug_format =>"json"} }java -jar logstash-1.1.9-monolithic.jar agent -flogstash-simple.conf Say Hello!
  • 12. See our message in a field!
  • 13. Life is better with searchinput { stdin { type => "stdin-type" } }output { stdout { debug => true debug_format => "json" } elasticsearch { embedded => true }}java -jar logstash-1.1.9-monolithic.jar agent -flogstash-search.conf cURL for it!
  • 14. Search for the data
  • 15. Well that isnt pretty Enter Kibana
  • 16. Kibana is a friendlyinterface for your logs
  • 17. Kibana Connects to Elasticsearch● Logstash parses and structures data into Elasticsearch● Kibana makes that data available● Apache Lucene Query Syntax (from elasticsearch)● Field statistics● Range searchesHow do we put it together?
  • 18. It Was Simple to Startinput { stdin { type => "stdin-type" } }output { stdout { debug => true debug_format => "json" } elasticsearch { embedded => true }}java -jar logstash-1.1.9-monolithic.jar agent -flogstash-search.conf But Lets Get Real
  • 19. On a server with logs
  • 20. Logstash/Elasticsearch
  • 21. Demo
  • 22. Thoughts....● Easy to try out, but for anything real, youll want a much more complicated configuration● The variety of inputs is great● Easy to build up a nice stack of filters
  • 23. More Thoughts....● Slow to boot monolithic jar file can be frustrating ○ Flatjar?● Hard to track down why logs arent flowing● Elasticsearch node discovery can be difficult ○ If your cluster doesnt have a node added to it when your client starts, your client isnt connected
  • 24. More Information● logstash.net● grokdebug.herokuapp.com● www.elasticsearch.org
  • 25. Questions?http://smartlogicsolutions.comhttp://twitter.com/smartlogichttp://github.com/smartlogic http://fb.me/smartlogic