Fluentd                              Structured logging                              Pluggable architecture               ...
Fluentd in briefIts like syslogd, but uses JSON for log messages
Fluentd :: format of logs            Application                          2012-02-04 01:33:51                          mya...
Fluentd :: format of logs                                                time            Application                      ...
Fluentd :: plugins             Application              Fluentd      filter / buffer / routing    SaaS      Storage       ...
Fluentd :: plugins syslogd     Scribe   Application          File   Plug-in                                    tailPlug-in...
Fluentd :: client libraries•   Client libraries    > Ruby    > Perl                     Application    > PHP    > Python  ...
Typical architecture before Fluentd   App server                  App server              App server   Application        ...
Architecture after Fluentd App server        App server         App server Application       Application        Applicatio...
Architecture after Fluentd  Fluentd          Fluentd                 Fluentd                                        Realti...
Case study  Ruby on Rails          Ruby on Rails          Ruby on Rails     Fluentd                  Fluentd             F...
# read logs from a file         # forward other logs to servers<source>                        # (load-balancing + fail-ov...
Scribe         Scribe: log collector by Facebook   Frontend servers                      Aggregator nodes       scribe    ...
Scribe’s Pros & Cons•   Pros.    >   Fast (C++)•   Cons.    >   VERY hard to install    >   Deals with unstructured logs  ...
Fluentd vs Scribe•   Easy to install    >   “gem install fluentd”    >   stable RPM and DEB packages          http://packag...
FlumeFlume: distributed log collector by Cloudera Phisical            Flume MasterTopology             Flume      Flume   ...
Flume’s Pros & Cons•   Pros.    >   Central master server manages all nodes•   Cons.    >   Difficult to understand        ...
Fluentd vs Flume•   Easy to understand    >   “syslogd that understands JSON”•   Easy to setup    >   “sudo fluentd --setup...
Fluentd vs Scribe/Flume                       Fluentd           Scribe           FlumeInstallation          gem/rpm/deb   ...
Fluentd•   Documents    >   http://fluentd.org•   Source code    >   http://github.com/fluent    >   14 committers across   ...
•   Sadayuki Furuhashi    >   twitter: @frsyuki•   Treasure Data, Inc.    >   Software Engineer; founder•   Author of Mess...
Upcoming SlideShare
Loading in …5
×

Fluentd meetup

2,766 views

Published on

Published in: Technology

Fluentd meetup

  1. 1. Fluentd Structured logging Pluggable architecture Reliable forwardingThe Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki
  2. 2. Fluentd in briefIts like syslogd, but uses JSON for log messages
  3. 3. Fluentd :: format of logs Application 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage }
  4. 4. Fluentd :: format of logs time Application tag 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage } record
  5. 5. Fluentd :: plugins Application Fluentd filter / buffer / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
  6. 6. Fluentd :: plugins syslogd Scribe Application File Plug-in tailPlug-in Plug-in Fluentd filter / buffer / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
  7. 7. Fluentd :: client libraries• Client libraries > Ruby > Perl Application > PHP > Python > Java Fluentd > ...Fluent.open(“myapp”)Fluent.event(“login”, {“user”=>38})#=> 2012-02-04 04:56:01 myapp.login {“user”:38}
  8. 8. Typical architecture before Fluentd App server App server App server Application Application Application File File File ... File File File ... File File File ... File High latency must wait for a day Log server Hard to analyze complex text parsers
  9. 9. Architecture after Fluentd App server App server App server Application Application Application Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd
  10. 10. Architecture after Fluentd Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd Hadoop Amazon Ready to MongoDB / Hive S3 / EMR Analyze!
  11. 11. Case study Ruby on Rails Ruby on Rails Ruby on Rails Fluentd Fluentd Fluentd✓ 127 RoR servers✓ 70,000 msgs/sec Fluentd Fluentd routing✓ 120Mbps at peak✓ 650GB/day Hadoop User behavior PV logs MongoDB logs / Hive
  12. 12. # read logs from a file # forward other logs to servers<source> # (load-balancing + fail-over) type tail <match **> path /var/log/httpd.log type forward format apache <server> tag apache.access host 192.168.0.11</source> weight 20 </server># save access logs to MongoDB <server><match apache.access> host 192.168.0.12 type mongo weight 60 host 127.0.0.1 </server></match> </match>
  13. 13. Scribe Scribe: log collector by Facebook Frontend servers Aggregator nodes scribe scribe scribe Hadoop HDFS scribe scribe scribe
  14. 14. Scribe’s Pros & Cons• Pros. > Fast (C++)• Cons. > VERY hard to install > Deals with unstructured logs you must parse logs before analyzing them > Hard to extend you must re-compile C++ programs > No longer maintained?
  15. 15. Fluentd vs Scribe• Easy to install > “gem install fluentd” > stable RPM and DEB packages http://packages.treasure-data.com/• Easy to write plugins > you can use Ruby• Easy to distribute plugins > “gem search -rd fluent-plugin”
  16. 16. FlumeFlume: distributed log collector by Cloudera Phisical Flume MasterTopology Flume Flume Flume LogicalTopology Hadoop HDFS
  17. 17. Flume’s Pros & Cons• Pros. > Central master server manages all nodes• Cons. > Difficult to understand logical topologies, phisical servers and a configuration of the logical/phisical mapping > Dificult to configure replicated master servers, log servers and agents > Big footprint 50,000 lines of Java codes
  18. 18. Fluentd vs Flume• Easy to understand > “syslogd that understands JSON”• Easy to setup > “sudo fluentd --setup && fluentd”• Very small footprint > small engine (3,000 lines) + plugins• Easy to configure
  19. 19. Fluentd vs Scribe/Flume Fluentd Scribe FlumeInstallation gem/rpm/deb make rpm/deb 3000 lines of 8000 lines of 50,000 lines ofFootprint Ruby C++ JavaPlugin Ruby N/A JavaPlugin distribution RubyGems.org N/A N/AMaster Server No No YesLicense Apache License Apache License Apache License
  20. 20. Fluentd• Documents > http://fluentd.org• Source code > http://github.com/fluent > 14 committers across many organizations• Mailing list > Google groups
  21. 21. • Sadayuki Furuhashi > twitter: @frsyuki• Treasure Data, Inc. > Software Engineer; founder• Author of MessagePack• Author of Fluentd

×