Logstash and friends

;
Logstash and friendsLogstash and friends
Julien PivottoJulien Pivotto
Techies Teach TechiesTechies Teach Techies
September 2, 2013September 2, 2013

;
Introduction
Logstash
Kibana
Conclusion
1 Introduction
2 Logstash
Missions
Inputs
Filters
Output
3 Kibana
4 Conclusion
Julien Pivotto Logstash and friends

;
Introduction
Logstash
Kibana
Conclusion
Logging
• Recording of eventsRecording of events
• Voice of your systems and applicationsVoice of your systems and applications
• It tells you almost everythingIt tells you almost everything
• It is a source of knowledgeIt is a source of knowledge

;
Introduction
Logstash
Kibana
Conclusion
Logging is useful
• Understanding outagesUnderstanding outages

;
Introduction
Logstash
Kibana
Conclusion
Logging is useful
• Understanding outagesUnderstanding outages
• not only when it’s wrongnot only when it’s wrong
• you can extract metricsyou can extract metrics
• no logs means somethingno logs means something
• it tells you what, why, who, whenit tells you what, why, who, when

;
Introduction
Logstash
Kibana
Conclusion
Logging in the wild
• SyslogSyslog
• |tee /var/log/myapp.log|tee /var/log/myapp.log
• Cron + MAILTO=Cron + MAILTO=
• &>/dev/null&>/dev/null

;
Introduction
Logstash
Kibana
Conclusion
Logging in the past
• Logging to ﬁles on each serverLogging to ﬁles on each server
• Using syslog protocolUsing syslog protocol
• DecentralizedDecentralized
• Reading requires SSH accessReading requires SSH access
• Not developer friendlyNot developer friendly

;
Introduction
Logstash
Kibana
Conclusion
The tools nowadays
• Jenkins, Icinga, Graphite, ForemanJenkins, Icinga, Graphite, Foreman
• Nice web interfacesNice web interfaces
• CentralizedCentralized
• Easy to useEasy to use

;
Introduction
Logstash
Kibana
Conclusion
Requirements
• Scalable toolsScalable tools
• Configured by text filesConfigured by text files
• Playing with existing toolsPlaying with existing tools
• ScalableScalable
• Following the Unix philosophyFollowing the Unix philosophy

;
Introduction
Logstash
Kibana
Conclusion
3 separate tools
• Elasticsearch, distributed search & analytics engineElasticsearch, distributed search & analytics engine
• Logstash, logs managementLogstash, logs management
• Kibana, very nice webui to ES and LogstashKibana, very nice webui to ES and Logstash

;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Logstash

;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Shipping the logs
• Some applications can only write to ﬁlesSome applications can only write to ﬁles
• But you need them on the main logstash serverBut you need them on the main logstash server
• Logstash can act as a daemon to ship the logsLogstash can act as a daemon to ship the logs
• Destinations can be syslog, redis,. . .Destinations can be syslog, redis,. . .
• Then you can act on your logsThen you can act on your logs

;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Collecting the logs
• You can plug logstash to a lot of data sourcesYou can plug logstash to a lot of data sources
• It can be passive or activeIt can be passive or active
• Listening on a UDP port vs checking mailsListening on a UDP port vs checking mails
• All your logs are managed by one applicationAll your logs are managed by one application
• It creates ﬁelds from the logsIt creates ﬁelds from the logs

;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Filtering the logs
• Making sense of a log messageMaking sense of a log message
• Finding what is importantFinding what is important
• Adding and removing ﬁeldsAdding and removing ﬁelds

;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Storing the logs
• Output to ElasticsearchOutput to Elasticsearch
• Sending information to statsdSending information to statsd
• Sending to your inbox, to icinga or ﬁlesSending to your inbox, to icinga or ﬁles

;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
What is an "event"?
• Several fieldsSeveral fields
• Several tagsSeveral tags
• A type (syslog message, irc message,. . . )A type (syslog message, irc message,. . . )
• A @message fieldA @message field
• A timestampA timestamp

;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Event

;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
http://www.ﬂickr.com/photos/quinnanya/7237788632/

;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
UDP and TCP input
• Compatible with rsyslog protocolCompatible with rsyslog protocol
• Each syslog talks with logstash directlyEach syslog talks with logstash directly
• Allow you to use the syslog toolchains: logger, rsyslogAllow you to use the syslog toolchains: logger, rsyslog
• UDP is shoot and forgetUDP is shoot and forget

;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
UDP and TCP input
Logstash conﬁguration
input {
udp {
type => syslog
port => 5544
}
tcp {
type => syslog
port => 5544
}
}

;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
UDP and TCP input
Rsyslog conﬁguration
*.* @logstash.example.com:5544
• IIn /etc/rsyslog.conf
• TThat line will forward all the logs to logstash
• LLogstash will make useful ﬁelds out of it: priority, severity,
program. . .

;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
File
• Enable you to use logstash with every applicationEnable you to use logstash with every application
• Useful to ship the logsUseful to ship the logs
• Acts as a tail -n 0 -FActs as a tail -n 0 -F
• It works even if you use logrotateIt works even if you use logrotate

;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
File
input {
file {
path => "/var/log/legacyapp.log"
type => "legacylog"
}
}

;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Grok
• Extract fields from textExtract fields from text
• Useful to read messagesUseful to read messages
• A lot of pre-existing patternsA lot of pre-existing patterns
• Uses Regex to find out fieldsUses Regex to find out fields

;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Grok
Input text
Invalid user oracle from 85.249.144.18
Grok pattern
Invalid user %{USERNAME:login} from %{IP:ip}
Result
{
"login": [
[ "oracle" ]
],
"ip": [
[ "85.249.144.18" ]
]
}

;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Grok
filter {
grok {
type => "syslog"
pattern => ["(?m)<%{POSINT:syslog_pri}>..."
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{@source_host}" ]
add_tag => "syslog-%{syslog_program}"
}
}

;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Grep
• Allows you to grep interresting messagesAllows you to grep interresting messages
• Useful to countUseful to count

;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Grep
filter {
grep {
add_field => ["outputirc", "A puppet package
has been deployed"]
add_tag => "outputirc"
drop => false
match => [ "syslog_program", "yum" ]
match => [ "@source_host", "puppetmaster" ]
match => [ "@message", "puppet-tree" ]
}
}

;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Geoip
filter{
geoip {
tags => ["syslog-httpd"]
source => ["client"]
}
}
• Transform ip address into geo dataTransform ip address into geo data
• Useful to ﬁlter by country/map the dataUseful to ﬁlter by country/map the data

;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
Elasticsearch
• Version of elasticsearch <=> version of logstashVersion of elasticsearch <=> version of logstash
• Unless you use the elasticsearch_http outputUnless you use the elasticsearch_http output
output {
elasticsearch {
}
}

;
Introduction
Logstash
Kibana
Conclusion
Missions
Inputs
Filters
Output
IRC
output {
irc {
channels => ["#example"]
host => "chat.freenode.net"
nick => "loggy"
port => 6667
tags => "outputirc"
user => "loggy"
format => "%{outputirc}"
}
}

;
Introduction
Logstash
Kibana
Conclusion
statsd
output {
statsd {
host => ’127.0.0.1’
sender => "logstash"
increment => [ "httpd.%{http_host}.r.%{response}",
"httpd.response.%{response}"]
count => ["apache.%{http_host}.bytes", "%{bytes}" ]
timing => ["apache.%{http_host}", "%{duration_msec}"]
tags => ’grokked-apache’
}
}

;
Introduction
Logstash
Kibana
Conclusion
Kibana
• Kibana is a web interface for Logstash/ESKibana is a web interface for Logstash/ES
• Kibana 1 was written in PHPKibana 1 was written in PHP
• Kibana 2 was written in RubyKibana 2 was written in Ruby
• Kibana 3 is written in AngularJSKibana 3 is written in AngularJS

;
Introduction
Logstash
Kibana
Conclusion
Kibana 3
• Everything happens in the browserEverything happens in the browser
• The browser is connected to ElasticsearchThe browser is connected to Elasticsearch
• You can save dashboards into ESYou can save dashboards into ES
• You can write/template dashboards to ﬁlesYou can write/template dashboards to ﬁles

;
Introduction
Logstash
Kibana
Conclusion
Installing kibana3
git clone https://github.com/elasticsearch/kibana.git
ssh -NL 9200:127.0.0.1:9200 elasticsearch &
python -m SimpleHTTPServer

;
Introduction
Logstash
Kibana
Conclusion
Kibana queries
Example of a kibana query
@fields.syslog_program:"httpd" AND @fields.http_host:"test.example.com" AND @fields.response:"404"
• LLucene query syntax
• SSimple and eﬀective
• PPoint & click web interface

;
Introduction
Logstash
Kibana
Conclusion
Kibana

;
Introduction
Logstash
Kibana
Conclusion
Conclusion
• Logstash is a small daemonLogstash is a small daemon
• Simple to package & deploy (jar ﬁle)Simple to package & deploy (jar ﬁle)
• Scalable thanks to ElasticsearchScalable thanks to Elasticsearch
• Developer friendly thanks to KibanaDeveloper friendly thanks to Kibana

Logstash and friends

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (6)

Similar to Logstash and friends

Similar to Logstash and friends (20)

More from Julien Pivotto

More from Julien Pivotto (20)

Recently uploaded

Recently uploaded (20)

Logstash and friends