This document discusses Logstash, an open source tool for collecting, parsing, and storing log files. It can ingest logs from various sources using inputs, apply filters to parse and transform log events, and output the structured data to destinations like Elasticsearch for search and analysis. The document provides an overview of Logstash's core functionality and components, demonstrates simple usage examples, and discusses integrating it with Kibana for visualizing and exploring log data. It also shares some lessons learned in production usage and points to additional resources.

1. Logstash!
Get to know your logs
Dan Ivovich
BMore on Rails
4/9/13

2. Dan Ivovich
SmartLogic Solutions
http://smartlogicsolutions.com
Twitter - @danivovich

3. What is the goal?
● Collect, Parse, and Store your log events
● Make log events searchable
● Analyze log events

4. Why bother?
● Got logs?
○ syslog
○ nginx access log
○ application logs
○ database logs
Are they all formatted the
same?

6. 3 Parts
● Inputs
● Filters
● Outputs

7. Inputs
● Files
● TCP/UDP
● Redis
● AMQP
● rsyslog
● xmpp
http://logstash.net/docs/1.1.9/ - Full list

8. Filters
● grep
● mutate
● anonymize
● date
● grok
http://logstash.net/docs/1.1.9/ - Full list

9. Outputs
● Files
● TCP/UDP
● Redis
● AMQP
● elasticsearch
http://logstash.net/docs/1.1.9/ - Full list

10. Getting Started
input { stdin { type => "stdin-type"} }
output { stdout { debug => true debug_format =>
"json"} }
java -jar logstash-1.1.9-monolithic.jar agent -f
logstash-simple.conf
Type something!

11. See our message!

12. Parse something!
input { stdin { type => "stdin-type"} }
filter { grok { type => "stdin-type" pattern =>
"Hello %{DATA:message}!" } }
output { stdout { debug => true debug_format =>
"json"} }
java -jar logstash-1.1.9-monolithic.jar agent -f
logstash-simple.conf
Say Hello!

13. See our message in a field!

14. Life is better with search
input { stdin { type => "stdin-type" } }
output {
stdout { debug => true debug_format => "json" }
elasticsearch { embedded => true }
}
java -jar logstash-1.1.9-monolithic.jar agent -f
logstash-search.conf
cURL for it!

15. Search for the data

16. Well that isn't pretty
Enter Kibana

17. Kibana is a friendly
interface for your logs

18. Kibana Connects to Elasticsearch
● Logstash parses and structures data into
Elasticsearch
● Kibana makes that data available
● Apache Lucene Query Syntax (from elasticsearch)
● Field statistics
● Range searches
How do we put it together?

19. It Was Simple to Start
input { stdin { type => "stdin-type" } }
output {
stdout { debug => true debug_format => "json" }
elasticsearch { embedded => true }
}
java -jar logstash-1.1.9-monolithic.jar agent -f
logstash-search.conf
But Let's Get Real

20. On a server with logs

21. Logstash/Elasticsearch

22. Demo

23. Thoughts....
● Easy to try out, but for anything real, you'll want a much
more complicated configuration
● The variety of inputs is great
● Easy to build up a nice stack of filters

24. More Thoughts....
● Slow to boot monolithic jar file can be frustrating
○ Flatjar?
● Hard to track down why logs aren't flowing
● Elasticsearch node discovery can be difficult
○ If your cluster doesn't have a node added to it when
your client starts, your client isn't connected

25. More Information
● logstash.net
● grokdebug.herokuapp.com
● www.elasticsearch.org

26. Questions?
http://smartlogicsolutions.com
http://twitter.com/smartlogic
http://github.com/smartlogic
http://fb.me/smartlogic