Infrastructure coders logstash


Published on

A short talk at Infrastructure Coders Melbourne April 2013 meetup. Covers my first impressions of logstash.

Published in: Technology

Infrastructure coders logstash

  1. 1. logstashInfrastructure CodersMelbourneApril 2013David Lutz@dlutzy
  2. 2. What does logstash do?It does "stuff" with log files.
  3. 3. Typical day (or night) in the life of asysadmin...Somethings wrong.Check the log files.How?
  4. 4. grep
  5. 5. catgrepsedawktailsort
  6. 6. and pipeslots of pipes
  7. 7. Fine if you have one server. Butwhat if you have 10 or 100 or1000for i in `seq 1 10` ; do ssh server$i blah blah; donecluster sshSplunk perhaps?Problems with Splunk...
  8. 8. 1.eats log files2. digests data3. spits it outinto other apps
  9. 9. inputs● amqp● drupal_dblog● eventlog● exec● file● ganglia● gelf● gemfire● generator● heroku● irc● log4j● lumberjack● pipe● redis● relp● sqs● stdin● stomp● syslog● tcp● twitter● udp● xmpp● zenoss● zeromq
  10. 10. filters● alter● anonymize● checksum● csv● date● dns● environment● gelfify● geoip● grep● grok● grokdiscovery● grokdiscovery● json● kv● metrics● multiline● mutate● noop● split● syslog_pri● urldecode● xml● zeromq
  11. 11. outputs● amqp● boundary● circonus● cloudwatch● datadog● elasticsearch● elasticsearch_http● elasticsearch_river● email● exec● file● ganglia● gelf● gemfire● graphite● graphtastic● http● internal● irc● juggernaut● librato● loggly● lumberjack● metriccatcher● mongodb● nagios● nagios_nsca● null● opentsdb● pagerduty● pipe● redis● riak● riemann● sns● sqs● statsd● stdout● stomp● syslog● tcp● websocket● xmpp● zabbix● zeromq
  12. 12. How to: install logstashwget!
  13. 13. How to: run logstashjava -jar logstash-1.1.9-monolithic.jar agent -f logstash.conf -- webeasy!
  14. 14. How to: get some apache logs ininput {tcp {type => "apache"port => 3333}}
  15. 15. How to: get some apache logs intail -f /var/log/apache2/access.log |nc localhost 3333
  16. 16. How to: digest the logsfilter {grok {type => "apache"pattern => "%{COMBINEDAPACHELOG}"}date {type => "apache"timestamp => "dd/MMM/yyyy:HH:mm:ss Z"}}
  17. 17. How to: output to elasticsearchoutput {elasticsearch {embedded => false}}
  18. 18. How to: output to elasticsearchand graphite via statsdoutput {elasticsearch {embedded => false}statsd {increment => "apache.response.%{response}"}}