Fluentd meetup in japan
Upcoming SlideShare
Loading in...5
×
 

Fluentd meetup in japan

on

  • 73,691 views

 

Statistics

Views

Total Views
73,691
Views on SlideShare
9,640
Embed Views
64,051

Actions

Likes
63
Downloads
180
Comments
1

39 Embeds 64,051

http://docs.fluentd.org 62083
http://www.fluentd.org 583
http://www.scoop.it 377
http://localhost 281
http://www.smallake.kr 207
http://wiki.dev.qops.net 121
http://play.daumcorp.com 95
http://fluentd-website-unify.herokuapp.com 88
http://dev.gluesys.com 86
http://rg443blog.wordpress.com 37
http://blogbigdatav5.blogspot.com.es 20
http://127.0.0.1 17
http://translate.googleusercontent.com 9
https://twitter.com 9
http://www-staging.fluentd.org 5
http://mighty-inlet-4687.herokuapp.com 4
http://us-w1.rockmelt.com 4
http://webcache.googleusercontent.com 2
http://polar-inlet-1645.herokuapp.com 2
http://live.swiftype.com 2
http://blogbigdatav5.blogspot.sg 1
http://fluentd-docs.herokuapp.com 1
http://blogbigdatav5.blogspot.com 1
http://news.google.com 1
http://morning-badlands-4779.herokuapp.com 1
http://secure-refuge-5547.herokuapp.com 1
http://protected-ridge-4086.herokuapp.com 1
http://10.160.1.106 1
http://tweetedtimes.com 1
http://www.twylah.com 1
http://www.googleadservices.com 1
http://www.adafruit.com 1
http://www.raspberrypithai.com 1
http://plus.url.google.com 1
http://vk-test.net.eximb.com 1
https://www.chatwork.com 1
http://m.facebook.com 1
http://www.google.co.in 1
http://blogbigdatav5.blogspot.co.il 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Fluentd meetup in japan Fluentd meetup in japan Presentation Transcript

  • Fluentd Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki
  • • Sadayuki Furuhashi > twitter: @frsyuki• Treasure Data, Inc. > Software Engineer; founder• Author of MessagePack• Author of Fluentd
  • What’s Fluentd?Its like syslogd, but uses JSON for log messages
  • What’s Fluentd? Application 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage }
  • What’s Fluentd? time Application tag 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage } record
  • What’s Fluentd? Application Fluentd lter / bu er / routing Storage
  • What’s Fluentd? Application Fluentd lter / bu er / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
  • What’s Fluentd? syslogd Scribe Application File Plug-in tailPlug-in Plug-in Fluentd lter / bu er / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
  • What’s Fluentd?• Client libraries > Ruby > Perl Application > PHP > Python > Java Fluentd > ...Fluent.open(“myapp”)Fluent.event(“login”, {“user”=>38})#=> 2012-02-04 04:56:01 myapp.login {“user”:38}
  • Fluentd & Event logsBefore: App server App server App server Application Application Application File File File ... File File File ... File File File ... File High latency must wait for a day Log server Hard to analyze complex text parsers
  • Fluentd & Event logsAfter: App server App server App server Application Application Application Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd
  • Fluentd & Event logs Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd Hadoop Amazon Ready to MongoDB / Hive S3 / EMR Analyze!
  • # receive events via HTTP # save alerts to a file<source> <match alert.**> type http type file port 8888 path /var/log/fluent/alerts</source> </match># read logs from a file # forward other logs to servers<source> # (load-balancing + fail-over) type tail <match **> path /var/log/httpd.log type forward format apache <server> tag apache.access host 192.168.0.11</source> weight 20 </server># save access logs to MongoDB <server><match apache.access> host 192.168.0.12 type mongo weight 60 host 127.0.0.1 </server></match> </match>
  • Fluentd vs Scribe• Deals with structured logs• Easy to install > “gem install fluentd” > apt-get and yum http://packages.treasure-data.com/• Easy to customize• add/modify plugins without re-compiling > “gem search -rd fluent-plugin”
  • Fluentd vs Flume• Easy to setup > “sudo fluentd --setup && fluentd”• Very small footprint > small engine (3,000 lines) + plugins• JVM-free• Easy to configure
  • Architecture of Fluentd
  • Architecture :: Input
  • Architecture :: Bu er
  • Architecture :: Output
  • Plugins :: out_forwardforward event logs Fluentd Heartbeat out_forward in_forward Fluentd Fluentd ✓ load balancing
  • Plugins :: out_forwardforward event logs Fluentd Heartbeat out_forwardφ accrual failure detector in_forward Fluentd Fluentd ✓ load balancing
  • Plugins :: out_copyduplicate event logs Fluentd out_copy out_mongo out_forward out_ le MongoDB File Fluentd
  • Plugins :: buf_ lereliable bu ering buf_ le Fluentd le le✓ Automatic retry le✓ 2^N retry interval le ✓ Persistent bu er
  • Plugins :: out_execexecute external programs Fluentd out_exec TSV → stdin ✓ Python external ✓ Perl program ✓ C++
  • Plugins :: out_exec_ lterexecute external programs out_exec_ lter stdin external Fluentd stdout program out_exec TSV → stdin ✓ Python external ✓ Perl program ✓ C++
  • Plugins :: in_execexecute external programs in_exec out_exec_ lter external stdout stdin external Fluentd program stdout program out_exec TSV → stdin ✓ Python external ✓ Perl program ✓ C++
  • Plugins :: in_tailRead event logs from a le Application File /var/log/access.log ✓ Apache log parser in_tail ✓ Syslog parser ✓ Custom parser Fluentd
  • Plugins :: in_tailApache log parser87.12.1.87 - - [04/Feb/2012:00:20:11 +0900] "GET / HTTP/1.1" 200 9887.12.1.87 - - [04/Feb/2012:00:20:11 +0900] "GET / HTTP/1.1" 200 98... { “host”: “87.12.1.87”, “method”: “GET”, “code”: 200, “size”: 98, “path”: “/” } ...
  • Plugins• Bundled plugins > file writes event logs to files hourly or daily > forward forwards event logs (+fail-over and load balancing) > exec passes event logs to/from external commands > tail reads event logs from a file (like `tail -f`)
  • Plugins• 3rd party plugins > scribe integrates Fluentd with Scribe > s3 uploads log files to Amazon S3 hourly or daily > mongo writes logs to MongoDB > hoop puts log files on Hadoop HDFS via Hoop ...
  • Plugin developer API• Unit test framework (like “MRUnit”) > Fluent::Test::InputTestDriver > Fluent::Test::OutputTestDriver > Fluent::Test::BufferedOutputTestDriver• Fluent::TailInput (base class of “tail” plugin) > text parser is customizable def parse_line(line)
  • Fluentd• Documents > http://fluentd.org• Source code > http://github.com/fluent• Twitter > #fluentd• Mailing list > http://groups.google.com/group/fluentd