1.
Logstash!
Get to know your logs
Dan Ivovich
BMore on Rails
4/9/13

2.
Dan Ivovich
SmartLogic Solutions
http://smartlogicsolutions.com
Twitter - @danivovich

3.
What is the goal?
● Collect, Parse, and Store your log events
● Make log events searchable
● Analyze log events

4.
Why bother?
● Got logs?
○ syslog
○ nginx access log
○ application logs
○ database logs
Are they all formatted the
same?

5.
3 Parts
● Inputs
● Filters
● Outputs

6.
Inputs
● Files
● TCP/UDP
● Redis
● AMQP
● rsyslog
● xmpp
http://logstash.net/docs/1.1.9/ - Full list

7.
Filters
● grep
● mutate
● anonymize
● date
● grok
http://logstash.net/docs/1.1.9/ - Full list

8.
Outputs
● Files
● TCP/UDP
● Redis
● AMQP
● elasticsearch
http://logstash.net/docs/1.1.9/ - Full list

9.
Getting Started
input { stdin { type => "stdin-type"} }
output { stdout { debug => true debug_format =>
"json"} }
java -jar logstash-1.1.9-monolithic.jar agent -f
logstash-simple.conf
Type something!

10.
See our message!

11.
Parse something!
input { stdin { type => "stdin-type"} }
filter { grok { type => "stdin-type" pattern =>
"Hello %{DATA:message}!" } }
output { stdout { debug => true debug_format =>
"json"} }
java -jar logstash-1.1.9-monolithic.jar agent -f
logstash-simple.conf
Say Hello!

12.
See our message in a field!

13.
Life is better with search
input { stdin { type => "stdin-type" } }
output {
stdout { debug => true debug_format => "json" }
elasticsearch { embedded => true }
}
java -jar logstash-1.1.9-monolithic.jar agent -f
logstash-search.conf
cURL for it!

14.
Search for the data

15.
Well that isn't pretty
Enter Kibana

16.
Kibana is a friendly
interface for your logs

17.
Kibana Connects to Elasticsearch
● Logstash parses and structures data into
Elasticsearch
● Kibana makes that data available
● Apache Lucene Query Syntax (from elasticsearch)
● Field statistics
● Range searches
How do we put it together?

18.
It Was Simple to Start
input { stdin { type => "stdin-type" } }
output {
stdout { debug => true debug_format => "json" }
elasticsearch { embedded => true }
}
java -jar logstash-1.1.9-monolithic.jar agent -f
logstash-search.conf
But Let's Get Real

19.
On a server with logs

20.
Logstash/Elasticsearch

21.
Demo

22.
Thoughts....
● Easy to try out, but for anything real, you'll want a much
more complicated configuration
● The variety of inputs is great
● Easy to build up a nice stack of filters

23.
More Thoughts....
● Slow to boot monolithic jar file can be frustrating
○ Flatjar?
● Hard to track down why logs aren't flowing
● Elasticsearch node discovery can be difficult
○ If your cluster doesn't have a node added to it when
your client starts, your client isn't connected

24.
More Information
● logstash.net
● grokdebug.herokuapp.com
● www.elasticsearch.org

25.
Questions?
http://smartlogicsolutions.com
http://twitter.com/smartlogic
http://github.com/smartlogic
http://fb.me/smartlogic