• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Brisbane DevOps Meetup -  Logstash
 

Brisbane DevOps Meetup - Logstash

on

  • 1,525 views

Logstash Presentation for Brisbane DevOps Meetup on 29th May 2013.

Logstash Presentation for Brisbane DevOps Meetup on 29th May 2013.

Statistics

Views

Total Views
1,525
Views on SlideShare
1,525
Embed Views
0

Actions

Likes
6
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Brisbane DevOps Meetup -  Logstash Brisbane DevOps Meetup - Logstash Document Transcript

    • BRISBANE DEVOPS MEETUP!!! LOGSTASH !!!30TH MAY 2013Paul Czarkowski / paul@paulcz.net /@pczarkowski
    • WHO AM I ?MORE OPS THAN DEVStarted out as an SA at several ISPsMoved into Games ... IT Manager @Pandemic (Brisbane)Moved to BioWare to launch Star Wars:The Old RepublicTransferred to EA, Manage Cloud Team forInfrastruture OperationsThrough M&As have 9 years tenure at EA.
    • WHO AM I ?STUFF I LIKE TO DO ...Automation ( and I dont just mean Puppetand/or Chef )Agile ( Scrum / Kanban )Cloud ( OpenStack, Amazon )Monitoring / LoggingSolve problemsCook ( I would be a chef if they didnt workso hard for such little pay)
    • WHAT IS A LOG?A log is a human readable, machine parsablerepresentation of an event.LOG = TIMESTAMP + DATAJan 19 13:01:13 paulcz­laptop anacron[7712]:Normal exit (0 jobs run)120607 14:07:00 InnoDB: Starting an applybatch of log records to the database...[1225306053] SERVICE ALERT:FTPSERVER;FTPSERVICE;OK;SOFT;2;FTP OK ­ 0.029second response time on port 21 [220ProFTPD 1.3.1 Server ready.][Sat Jan 19 01:04:25 2013] [error] [client78.30.200.81] File does not exist:/opt/www/vhosts/crappywebsite/html/robots.txt
    • THERES AN ISO FOR THAT!ISO 8601
    • A LOG IS HUMAN READABLE...“A human readable, machine parsablerepresentation of an event.”208.115.111.74 - - [13/Jan/2013:04:28:55 -0500] "GET /robots.txt HTTP/1.1"301 303 "-" "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)"
    •  
    • ...BUT LOGS ARE NOT !But theyre machine parsable right?
    • LOGS ARE MACHINE PARSEABLE208.115.111.74 - - [13/Jan/2013:04:28:55 -0500] "GET /robots.txt HTTP/1.1"301 303 "-" "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)"
    • LOGS ARE MACHINE PARSEABLEActual Regex to parse apache logs.
    • LOGS ARE MACHINE PARSEABLEActual Regex to parse apache logs.
    • LOGS ARE MACHINE PARSEABLEUsers will now call PERL Ninja to solveevery problem they have ­ Hero Syndrome.Does it work for every possible log line ?Whos going to maintain that shit ?Is it even useful without being surroundedby [bad] sysadmin scripts ?
    • SO WE AGREE ... THIS IS BAD.208.115.111.74 - - [13/Jan/2013:04:28:55 -0500] "GET /robots.txt HTTP/1.1"301 303 "-" "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)"
    • LOGSTASH TURNS THISINTO THAT208.115.111.74 - - [13/Jan/2013:04:28:55 -0500] "GET /robots.txt HTTP/1.1"301 303 "-" "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)"{"client address": "208.115.111.74","user": null,"timestamp": "2013-01-13T04:28:55-0500","verb": "GET","path": "/robots.txt","query": null,"http version": 1.1,"response code": 301,"bytes": 303,"referrer": null"user agent": "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)"}
    • LOGSTASH PLUGINS...cat chain.plugins | grep together | seds/like/unix/ > pipeline
    • LOGSTASH PLUGINS...~ 25+ INPUT PLUGINSfiletcp,udpamqp, zeromq, redis, sqstwitter, irclumberjack...
    • LOGSTASH PLUGINS...~ 20+ FILTER PLUGINSdategrokgeoipmultilinemetricssplit...
    • LOGSTASH PLUGINS...~ 40+ OUTPUT PLUGINSElasticsearchStatsD, Graphite, Nagios, PagerDutyamqp, zeromq, redis, sqsboundary, cloudwatch, zabbixlumberjack...
    • TWO VERY IMPORTANT FILTERSLets talk briefly about two filters that arevery important to making our logs useful
    • FILTER - DATEtakes a timestamp and makes it ISO 8601CompliantTurns this:Into this:13/Jan/2013:04:28:55 -05002013-01-13T04:28:55-0500
    • FILTER - GROKGrok parses arbitrary text and structures it.Makes complex regex patterns simple.USERNAME [a-zA-Z0-9_-]+USER %{USERNAME}INT (?:[+-]?(?:[0-9]+))MONTH b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)bDAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)COMBINEDAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth}[%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|-)" %{NUMBER:response}(?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}
    • Remember our apache log from earlier?
    • Define Inputs and Filters.input {file {type => "apache"path => ["/var/log/httpd/httpd.log"]}}filter {grok {type => "apache"pattern => "%{COMBINEDAPACHELOG}"}date {type => "apache"}geoip {type => "apache"}}
    • Define some outputs.output {statsd {type => "apache"increment => "apache.response.%{response}"# Count one hit every event by response}elasticsearch {type => "apache"}}
    • INSTANT GRAPHIFICATION !
    • KIBANA
    • KIBANA
    • ANALYZE TWITTER STREAMSMarketingCustomer feedbackgood for load testing ­ bieber
    • LOGSTASH - TWITTER INPUTinput {twitter {type => "twitter"keywords => ["bieber","beiber"]user => "username"password => "*******"}}output {elasticsearch {type => "twitter"}}
    • 4% Bieber Fans cant spell his name.10% Tweets from Blackberry ( 50ishbusiness execs? )~ 200 Bieber tweets per minute.
    • ALREADY HAVE CENTRALRSYSLOG/SYSLOGNG SERVER?input {file {type => "syslog"path => ["/data/rsyslog/**/*.log"]}}filter {### a bunch of groks, a date, and other filters}output {type => "elasticsearch"}
    • ACT AS A CENTRAL SYSLOG SERVERGOOD FOR APPLIANCES / SWITCHESinput {tcp {type => "syslog"port => "514"}udp {type => "syslog"port => "514"}}filter {### a bunch of groks, a date, and other filters}output {type => "elasticsearch"}
    • IN CLOUD? DONT OWNNETWORK?Use an encrypted TransportLogstash AgentLogstash Indexerinput { file { ... } }output {lumberjack {hosts => ["logstash-indexer1", "logstash-indexer2"]ssl_certificate => "/etc/ssl/logstash.crt"}}input {lumberjack {ssl_certificate => "/etc/ssl/logstash.crt"ssl_key => "/etc/ssl/logstash.key"}}output { elasticsearch {} }
    • SYSTEM METRICS ?input {exec {type => "system-loadavg"command => "cat /proc/loadavg | awk {print $1,$2,$3}"interval => 30}}filter {grok {type => "system-loadavg"pattern => "%{NUMBER:load_avg_1m} %{NUMBER:load_avg_5m}%{NUMBER:load_avg_15m}"named_captures_only => true}}output {graphite {host => "10.10.10.10"port => 2003type => "system-loadavg"metrics => [ "hosts.%{@source_host}.load_avg.1m", "%{load_avg_1m}","hosts.%{@source_host}.load_avg.5m", "%{load_avg_5m}","hosts.%{@source_host}.load_avg.15m", "%{load_avg_15m}" ]}}
    • UNIQUE PROBLEM TO SOLVE ?write a logstash module!Input ­ SnmptrapFilter ­ Translate Can do powerful things with [ boilerplate + ] afew lines of ruby
    • SCALING LOGSTASHUSE QUEUES ( RABBITMQ, REDIS) TO HELPSCALE HORIZONTALLY.Local log dir on clients = cheap queue
    • PERFORMANCE... EVENTS/SECHP BL460, 48Gb, 24 Cores.Several Filters ­ Groks, Mutates, etcjump in performance from setting 8 filterworkers (­w 8)
    • MISC CONSIDERATIONSfile limits ( lol 1024 )Java ( use JRE7 for Performance )Limit Memory UsageElasticsearch ­ ~50% RAM for JavaHeapRedis / Rabbit ­ 1GbLogstash ­ 256Mb ­> 512Mb for JavaHeapBe safe ­ avoid deadly OOM killer.Kibana / Elasticsearch Security ­ BehindReverse Proxy!
    • ELASTICSEARCHPrimary Storage/Search engine for logstashClusterable/Scalable Search EngineRESTful APILucene SyntaxVery large topic on its own...
    • FURTHER READINGhttp://www.logstash.net/http://www.logstashbook.com/ [JamesTurnbull]https://github.com/paulczar/vagrant­logstashhttp://jujucharms.com/charms/precise/logstash­indexerLogstash Puppet (github/electrical)Logstash Chef (github/lusis)
    • OTHER TOOLS YOU SHOULD ALL BE USING...VagrantChef / Puppet ( obviously! )FPMOmnibusLXC containers for lightweight VMsOpenStack ( run a cloud locally for dev )
    • QUESTIONS?Paul Czarkowski / paul@paulcz.net /@pczarkowski