BRISBANE DEVOPS MEETUP!!! LOGSTASH !!!30TH MAY 2013Paul Czarkowski / paul@paulcz.net /@pczarkowski
WHO AM I ?MORE OPS THAN DEVStarted out as an SA at several ISPsMoved into Games ... IT Manager @Pandemic (Brisbane)Moved t...
WHO AM I ?STUFF I LIKE TO DO ...Automation ( and I dont just mean Puppetand/or Chef )Agile ( Scrum / Kanban )Cloud ( OpenS...
WHAT IS A LOG?A log is a human readable, machine parsablerepresentation of an event.LOG = TIMESTAMP + DATAJan 19 13:01:13 ...
THERES AN ISO FOR THAT!ISO 8601
A LOG IS HUMAN READABLE...“A human readable, machine parsablerepresentation of an event.”208.115.111.74 - - [13/Jan/2013:0...
 
...BUT LOGS ARE NOT !But theyre machine parsable right?
LOGS ARE MACHINE PARSEABLE208.115.111.74 - - [13/Jan/2013:04:28:55 -0500] "GET /robots.txt HTTP/1.1"301 303 "-" "Mozilla/5...
LOGS ARE MACHINE PARSEABLEActual Regex to parse apache logs.
LOGS ARE MACHINE PARSEABLEActual Regex to parse apache logs.
LOGS ARE MACHINE PARSEABLEUsers will now call PERL Ninja to solveevery problem they have ­ Hero Syndrome.Does it work for ...
SO WE AGREE ... THIS IS BAD.208.115.111.74 - - [13/Jan/2013:04:28:55 -0500] "GET /robots.txt HTTP/1.1"301 303 "-" "Mozilla...
LOGSTASH TURNS THISINTO THAT208.115.111.74 - - [13/Jan/2013:04:28:55 -0500] "GET /robots.txt HTTP/1.1"301 303 "-" "Mozilla...
LOGSTASH PLUGINS...cat chain.plugins | grep together | seds/like/unix/ > pipeline
LOGSTASH PLUGINS...~ 25+ INPUT PLUGINSfiletcp,udpamqp, zeromq, redis, sqstwitter, irclumberjack...
LOGSTASH PLUGINS...~ 20+ FILTER PLUGINSdategrokgeoipmultilinemetricssplit...
LOGSTASH PLUGINS...~ 40+ OUTPUT PLUGINSElasticsearchStatsD, Graphite, Nagios, PagerDutyamqp, zeromq, redis, sqsboundary, c...
TWO VERY IMPORTANT FILTERSLets talk briefly about two filters that arevery important to making our logs useful
FILTER - DATEtakes a timestamp and makes it ISO 8601CompliantTurns this:Into this:13/Jan/2013:04:28:55 -05002013-01-13T04:...
FILTER - GROKGrok parses arbitrary text and structures it.Makes complex regex patterns simple.USERNAME [a-zA-Z0-9_-]+USER ...
Remember our apache log from earlier?
Define Inputs and Filters.input {file {type => "apache"path => ["/var/log/httpd/httpd.log"]}}filter {grok {type => "apache...
Define some outputs.output {statsd {type => "apache"increment => "apache.response.%{response}"# Count one hit every event ...
INSTANT GRAPHIFICATION !
KIBANA
KIBANA
ANALYZE TWITTER STREAMSMarketingCustomer feedbackgood for load testing ­ bieber
LOGSTASH - TWITTER INPUTinput {twitter {type => "twitter"keywords => ["bieber","beiber"]user => "username"password => "***...
4% Bieber Fans cant spell his name.10% Tweets from Blackberry ( 50ishbusiness execs? )~ 200 Bieber tweets per minute.
ALREADY HAVE CENTRALRSYSLOG/SYSLOGNG SERVER?input {file {type => "syslog"path => ["/data/rsyslog/**/*.log"]}}filter {### a...
ACT AS A CENTRAL SYSLOG SERVERGOOD FOR APPLIANCES / SWITCHESinput {tcp {type => "syslog"port => "514"}udp {type => "syslog...
IN CLOUD? DONT OWNNETWORK?Use an encrypted TransportLogstash AgentLogstash Indexerinput { file { ... } }output {lumberjack...
SYSTEM METRICS ?input {exec {type => "system-loadavg"command => "cat /proc/loadavg | awk {print $1,$2,$3}"interval => 30}}...
UNIQUE PROBLEM TO SOLVE ?write a logstash module!Input ­ SnmptrapFilter ­ Translate Can do powerful things with [ boilerpl...
SCALING LOGSTASHUSE QUEUES ( RABBITMQ, REDIS) TO HELPSCALE HORIZONTALLY.Local log dir on clients = cheap queue
PERFORMANCE... EVENTS/SECHP BL460, 48Gb, 24 Cores.Several Filters ­ Groks, Mutates, etcjump in performance from setting 8 ...
MISC CONSIDERATIONSfile limits ( lol 1024 )Java ( use JRE7 for Performance )Limit Memory UsageElasticsearch ­ ~50% RAM for...
ELASTICSEARCHPrimary Storage/Search engine for logstashClusterable/Scalable Search EngineRESTful APILucene SyntaxVery larg...
FURTHER READINGhttp://www.logstash.net/http://www.logstashbook.com/ [JamesTurnbull]https://github.com/paulczar/vagrant­log...
OTHER TOOLS YOU SHOULD ALL BE USING...VagrantChef / Puppet ( obviously! )FPMOmnibusLXC containers for lightweight VMsOpenS...
QUESTIONS?Paul Czarkowski / paul@paulcz.net /@pczarkowski
Brisbane DevOps Meetup -  Logstash
Brisbane DevOps Meetup -  Logstash
Brisbane DevOps Meetup -  Logstash
Brisbane DevOps Meetup -  Logstash
Upcoming SlideShare
Loading in …5
×

Brisbane DevOps Meetup - Logstash

1,797 views
1,614 views

Published on

Logstash Presentation for Brisbane DevOps Meetup on 29th May 2013.

Published in: Technology, Design

Brisbane DevOps Meetup - Logstash

  1. 1. BRISBANE DEVOPS MEETUP!!! LOGSTASH !!!30TH MAY 2013Paul Czarkowski / paul@paulcz.net /@pczarkowski
  2. 2. WHO AM I ?MORE OPS THAN DEVStarted out as an SA at several ISPsMoved into Games ... IT Manager @Pandemic (Brisbane)Moved to BioWare to launch Star Wars:The Old RepublicTransferred to EA, Manage Cloud Team forInfrastruture OperationsThrough M&As have 9 years tenure at EA.
  3. 3. WHO AM I ?STUFF I LIKE TO DO ...Automation ( and I dont just mean Puppetand/or Chef )Agile ( Scrum / Kanban )Cloud ( OpenStack, Amazon )Monitoring / LoggingSolve problemsCook ( I would be a chef if they didnt workso hard for such little pay)
  4. 4. WHAT IS A LOG?A log is a human readable, machine parsablerepresentation of an event.LOG = TIMESTAMP + DATAJan 19 13:01:13 paulcz­laptop anacron[7712]:Normal exit (0 jobs run)120607 14:07:00 InnoDB: Starting an applybatch of log records to the database...[1225306053] SERVICE ALERT:FTPSERVER;FTPSERVICE;OK;SOFT;2;FTP OK ­ 0.029second response time on port 21 [220ProFTPD 1.3.1 Server ready.][Sat Jan 19 01:04:25 2013] [error] [client78.30.200.81] File does not exist:/opt/www/vhosts/crappywebsite/html/robots.txt
  5. 5. THERES AN ISO FOR THAT!ISO 8601
  6. 6. A LOG IS HUMAN READABLE...“A human readable, machine parsablerepresentation of an event.”208.115.111.74 - - [13/Jan/2013:04:28:55 -0500] "GET /robots.txt HTTP/1.1"301 303 "-" "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)"
  7. 7.  
  8. 8. ...BUT LOGS ARE NOT !But theyre machine parsable right?
  9. 9. LOGS ARE MACHINE PARSEABLE208.115.111.74 - - [13/Jan/2013:04:28:55 -0500] "GET /robots.txt HTTP/1.1"301 303 "-" "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)"
  10. 10. LOGS ARE MACHINE PARSEABLEActual Regex to parse apache logs.
  11. 11. LOGS ARE MACHINE PARSEABLEActual Regex to parse apache logs.
  12. 12. LOGS ARE MACHINE PARSEABLEUsers will now call PERL Ninja to solveevery problem they have ­ Hero Syndrome.Does it work for every possible log line ?Whos going to maintain that shit ?Is it even useful without being surroundedby [bad] sysadmin scripts ?
  13. 13. SO WE AGREE ... THIS IS BAD.208.115.111.74 - - [13/Jan/2013:04:28:55 -0500] "GET /robots.txt HTTP/1.1"301 303 "-" "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)"
  14. 14. LOGSTASH TURNS THISINTO THAT208.115.111.74 - - [13/Jan/2013:04:28:55 -0500] "GET /robots.txt HTTP/1.1"301 303 "-" "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)"{"client address": "208.115.111.74","user": null,"timestamp": "2013-01-13T04:28:55-0500","verb": "GET","path": "/robots.txt","query": null,"http version": 1.1,"response code": 301,"bytes": 303,"referrer": null"user agent": "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)"}
  15. 15. LOGSTASH PLUGINS...cat chain.plugins | grep together | seds/like/unix/ > pipeline
  16. 16. LOGSTASH PLUGINS...~ 25+ INPUT PLUGINSfiletcp,udpamqp, zeromq, redis, sqstwitter, irclumberjack...
  17. 17. LOGSTASH PLUGINS...~ 20+ FILTER PLUGINSdategrokgeoipmultilinemetricssplit...
  18. 18. LOGSTASH PLUGINS...~ 40+ OUTPUT PLUGINSElasticsearchStatsD, Graphite, Nagios, PagerDutyamqp, zeromq, redis, sqsboundary, cloudwatch, zabbixlumberjack...
  19. 19. TWO VERY IMPORTANT FILTERSLets talk briefly about two filters that arevery important to making our logs useful
  20. 20. FILTER - DATEtakes a timestamp and makes it ISO 8601CompliantTurns this:Into this:13/Jan/2013:04:28:55 -05002013-01-13T04:28:55-0500
  21. 21. FILTER - GROKGrok parses arbitrary text and structures it.Makes complex regex patterns simple.USERNAME [a-zA-Z0-9_-]+USER %{USERNAME}INT (?:[+-]?(?:[0-9]+))MONTH b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)bDAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)COMBINEDAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth}[%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|-)" %{NUMBER:response}(?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}
  22. 22. Remember our apache log from earlier?
  23. 23. Define Inputs and Filters.input {file {type => "apache"path => ["/var/log/httpd/httpd.log"]}}filter {grok {type => "apache"pattern => "%{COMBINEDAPACHELOG}"}date {type => "apache"}geoip {type => "apache"}}
  24. 24. Define some outputs.output {statsd {type => "apache"increment => "apache.response.%{response}"# Count one hit every event by response}elasticsearch {type => "apache"}}
  25. 25. INSTANT GRAPHIFICATION !
  26. 26. KIBANA
  27. 27. KIBANA
  28. 28. ANALYZE TWITTER STREAMSMarketingCustomer feedbackgood for load testing ­ bieber
  29. 29. LOGSTASH - TWITTER INPUTinput {twitter {type => "twitter"keywords => ["bieber","beiber"]user => "username"password => "*******"}}output {elasticsearch {type => "twitter"}}
  30. 30. 4% Bieber Fans cant spell his name.10% Tweets from Blackberry ( 50ishbusiness execs? )~ 200 Bieber tweets per minute.
  31. 31. ALREADY HAVE CENTRALRSYSLOG/SYSLOGNG SERVER?input {file {type => "syslog"path => ["/data/rsyslog/**/*.log"]}}filter {### a bunch of groks, a date, and other filters}output {type => "elasticsearch"}
  32. 32. ACT AS A CENTRAL SYSLOG SERVERGOOD FOR APPLIANCES / SWITCHESinput {tcp {type => "syslog"port => "514"}udp {type => "syslog"port => "514"}}filter {### a bunch of groks, a date, and other filters}output {type => "elasticsearch"}
  33. 33. IN CLOUD? DONT OWNNETWORK?Use an encrypted TransportLogstash AgentLogstash Indexerinput { file { ... } }output {lumberjack {hosts => ["logstash-indexer1", "logstash-indexer2"]ssl_certificate => "/etc/ssl/logstash.crt"}}input {lumberjack {ssl_certificate => "/etc/ssl/logstash.crt"ssl_key => "/etc/ssl/logstash.key"}}output { elasticsearch {} }
  34. 34. SYSTEM METRICS ?input {exec {type => "system-loadavg"command => "cat /proc/loadavg | awk {print $1,$2,$3}"interval => 30}}filter {grok {type => "system-loadavg"pattern => "%{NUMBER:load_avg_1m} %{NUMBER:load_avg_5m}%{NUMBER:load_avg_15m}"named_captures_only => true}}output {graphite {host => "10.10.10.10"port => 2003type => "system-loadavg"metrics => [ "hosts.%{@source_host}.load_avg.1m", "%{load_avg_1m}","hosts.%{@source_host}.load_avg.5m", "%{load_avg_5m}","hosts.%{@source_host}.load_avg.15m", "%{load_avg_15m}" ]}}
  35. 35. UNIQUE PROBLEM TO SOLVE ?write a logstash module!Input ­ SnmptrapFilter ­ Translate Can do powerful things with [ boilerplate + ] afew lines of ruby
  36. 36. SCALING LOGSTASHUSE QUEUES ( RABBITMQ, REDIS) TO HELPSCALE HORIZONTALLY.Local log dir on clients = cheap queue
  37. 37. PERFORMANCE... EVENTS/SECHP BL460, 48Gb, 24 Cores.Several Filters ­ Groks, Mutates, etcjump in performance from setting 8 filterworkers (­w 8)
  38. 38. MISC CONSIDERATIONSfile limits ( lol 1024 )Java ( use JRE7 for Performance )Limit Memory UsageElasticsearch ­ ~50% RAM for JavaHeapRedis / Rabbit ­ 1GbLogstash ­ 256Mb ­> 512Mb for JavaHeapBe safe ­ avoid deadly OOM killer.Kibana / Elasticsearch Security ­ BehindReverse Proxy!
  39. 39. ELASTICSEARCHPrimary Storage/Search engine for logstashClusterable/Scalable Search EngineRESTful APILucene SyntaxVery large topic on its own...
  40. 40. FURTHER READINGhttp://www.logstash.net/http://www.logstashbook.com/ [JamesTurnbull]https://github.com/paulczar/vagrant­logstashhttp://jujucharms.com/charms/precise/logstash­indexerLogstash Puppet (github/electrical)Logstash Chef (github/lusis)
  41. 41. OTHER TOOLS YOU SHOULD ALL BE USING...VagrantChef / Puppet ( obviously! )FPMOmnibusLXC containers for lightweight VMsOpenStack ( run a cloud locally for dev )
  42. 42. QUESTIONS?Paul Czarkowski / paul@paulcz.net /@pczarkowski

×