Fluentd meetup in japan
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Fluentd meetup in japan

on

  • 73,838 views

 

Statistics

Views

Total Views
73,838
Views on SlideShare
9,764
Embed Views
64,074

Actions

Likes
64
Downloads
180
Comments
1

39 Embeds 64,074

http://docs.fluentd.org 62087
http://www.fluentd.org 583
http://www.scoop.it 377
http://localhost 281
http://www.smallake.kr 225
http://wiki.dev.qops.net 121
http://play.daumcorp.com 95
http://fluentd-website-unify.herokuapp.com 88
http://dev.gluesys.com 87
http://rg443blog.wordpress.com 37
http://blogbigdatav5.blogspot.com.es 20
http://127.0.0.1 17
http://translate.googleusercontent.com 9
https://twitter.com 9
http://www-staging.fluentd.org 5
http://mighty-inlet-4687.herokuapp.com 4
http://us-w1.rockmelt.com 4
http://webcache.googleusercontent.com 2
http://polar-inlet-1645.herokuapp.com 2
http://live.swiftype.com 2
http://blogbigdatav5.blogspot.sg 1
http://fluentd-docs.herokuapp.com 1
http://blogbigdatav5.blogspot.com 1
http://news.google.com 1
http://morning-badlands-4779.herokuapp.com 1
http://secure-refuge-5547.herokuapp.com 1
http://protected-ridge-4086.herokuapp.com 1
http://10.160.1.106 1
http://tweetedtimes.com 1
http://www.twylah.com 1
http://www.googleadservices.com 1
http://www.adafruit.com 1
http://www.raspberrypithai.com 1
http://plus.url.google.com 1
http://vk-test.net.eximb.com 1
https://www.chatwork.com 1
http://m.facebook.com 1
http://www.google.co.in 1
http://blogbigdatav5.blogspot.co.il 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Fluentd meetup in japan Presentation Transcript

  • 1. Fluentd Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki
  • 2. • Sadayuki Furuhashi > twitter: @frsyuki• Treasure Data, Inc. > Software Engineer; founder• Author of MessagePack• Author of Fluentd
  • 3. What’s Fluentd?Its like syslogd, but uses JSON for log messages
  • 4. What’s Fluentd? Application 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage }
  • 5. What’s Fluentd? time Application tag 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage } record
  • 6. What’s Fluentd? Application Fluentd lter / bu er / routing Storage
  • 7. What’s Fluentd? Application Fluentd lter / bu er / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
  • 8. What’s Fluentd? syslogd Scribe Application File Plug-in tailPlug-in Plug-in Fluentd lter / bu er / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
  • 9. What’s Fluentd?• Client libraries > Ruby > Perl Application > PHP > Python > Java Fluentd > ...Fluent.open(“myapp”)Fluent.event(“login”, {“user”=>38})#=> 2012-02-04 04:56:01 myapp.login {“user”:38}
  • 10. Fluentd & Event logsBefore: App server App server App server Application Application Application File File File ... File File File ... File File File ... File High latency must wait for a day Log server Hard to analyze complex text parsers
  • 11. Fluentd & Event logsAfter: App server App server App server Application Application Application Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd
  • 12. Fluentd & Event logs Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd Hadoop Amazon Ready to MongoDB / Hive S3 / EMR Analyze!
  • 13. # receive events via HTTP # save alerts to a file<source> <match alert.**> type http type file port 8888 path /var/log/fluent/alerts</source> </match># read logs from a file # forward other logs to servers<source> # (load-balancing + fail-over) type tail <match **> path /var/log/httpd.log type forward format apache <server> tag apache.access host 192.168.0.11</source> weight 20 </server># save access logs to MongoDB <server><match apache.access> host 192.168.0.12 type mongo weight 60 host 127.0.0.1 </server></match> </match>
  • 14. Fluentd vs Scribe• Deals with structured logs• Easy to install > “gem install fluentd” > apt-get and yum http://packages.treasure-data.com/• Easy to customize• add/modify plugins without re-compiling > “gem search -rd fluent-plugin”
  • 15. Fluentd vs Flume• Easy to setup > “sudo fluentd --setup && fluentd”• Very small footprint > small engine (3,000 lines) + plugins• JVM-free• Easy to configure
  • 16. Architecture of Fluentd
  • 17. Architecture :: Input
  • 18. Architecture :: Bu er
  • 19. Architecture :: Output
  • 20. Plugins :: out_forwardforward event logs Fluentd Heartbeat out_forward in_forward Fluentd Fluentd ✓ load balancing
  • 21. Plugins :: out_forwardforward event logs Fluentd Heartbeat out_forwardφ accrual failure detector in_forward Fluentd Fluentd ✓ load balancing
  • 22. Plugins :: out_copyduplicate event logs Fluentd out_copy out_mongo out_forward out_ le MongoDB File Fluentd
  • 23. Plugins :: buf_ lereliable bu ering buf_ le Fluentd le le✓ Automatic retry le✓ 2^N retry interval le ✓ Persistent bu er
  • 24. Plugins :: out_execexecute external programs Fluentd out_exec TSV → stdin ✓ Python external ✓ Perl program ✓ C++
  • 25. Plugins :: out_exec_ lterexecute external programs out_exec_ lter stdin external Fluentd stdout program out_exec TSV → stdin ✓ Python external ✓ Perl program ✓ C++
  • 26. Plugins :: in_execexecute external programs in_exec out_exec_ lter external stdout stdin external Fluentd program stdout program out_exec TSV → stdin ✓ Python external ✓ Perl program ✓ C++
  • 27. Plugins :: in_tailRead event logs from a le Application File /var/log/access.log ✓ Apache log parser in_tail ✓ Syslog parser ✓ Custom parser Fluentd
  • 28. Plugins :: in_tailApache log parser87.12.1.87 - - [04/Feb/2012:00:20:11 +0900] "GET / HTTP/1.1" 200 9887.12.1.87 - - [04/Feb/2012:00:20:11 +0900] "GET / HTTP/1.1" 200 98... { “host”: “87.12.1.87”, “method”: “GET”, “code”: 200, “size”: 98, “path”: “/” } ...
  • 29. Plugins• Bundled plugins > file writes event logs to files hourly or daily > forward forwards event logs (+fail-over and load balancing) > exec passes event logs to/from external commands > tail reads event logs from a file (like `tail -f`)
  • 30. Plugins• 3rd party plugins > scribe integrates Fluentd with Scribe > s3 uploads log files to Amazon S3 hourly or daily > mongo writes logs to MongoDB > hoop puts log files on Hadoop HDFS via Hoop ...
  • 31. Plugin developer API• Unit test framework (like “MRUnit”) > Fluent::Test::InputTestDriver > Fluent::Test::OutputTestDriver > Fluent::Test::BufferedOutputTestDriver• Fluent::TailInput (base class of “tail” plugin) > text parser is customizable def parse_line(line)
  • 32. Fluentd• Documents > http://fluentd.org• Source code > http://github.com/fluent• Twitter > #fluentd• Mailing list > http://groups.google.com/group/fluentd