Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Fluentd                               Structured logging                               Pluggable architecture             ...
•   Sadayuki Furuhashi    >   twitter: @frsyuki•   Treasure Data, Inc.    >   Software Engineer; founder•   Author of Mess...
What’s Fluentd?Its like syslogd, but uses JSON for log messages
What’s Fluentd?          Application                        2012-02-04 01:33:51                        myapp.buylog {     ...
What’s Fluentd?                                              time          Application                           tag      ...
What’s Fluentd?          Application           Fluentd      lter / bu er / routing           Storage
What’s Fluentd?             Application              Fluentd      lter / bu er / routing    SaaS      Storage            F...
What’s Fluentd? syslogd     Scribe   Application          File   Plug-in                                    tailPlug-in Pl...
What’s Fluentd?•   Client libraries    > Ruby    > Perl                     Application    > PHP    > Python    > Java    ...
Fluentd & Event logsBefore:   App server                  App server              App server   Application                ...
Fluentd & Event logsAfter:   App server        App server         App server   Application       Application        Applic...
Fluentd & Event logs  Fluentd          Fluentd                 Fluentd                                        Realtime!   ...
# receive events via HTTP       # save alerts to a file<source>                        <match alert.**>  type http        ...
Fluentd vs Scribe•   Deals with structured logs•   Easy to install    >   “gem install fluentd”    >   apt-get and yum     ...
Fluentd vs Flume•   Easy to setup    >   “sudo fluentd --setup && fluentd”•   Very small footprint    >   small engine (3,00...
Architecture of Fluentd
Architecture :: Input
Architecture :: Bu er
Architecture :: Output
Plugins :: out_forwardforward event logs                           Fluentd         Heartbeat                              ...
Plugins :: out_forwardforward event logs                                 Fluentd            Heartbeat                     ...
Plugins :: out_copyduplicate event logs                       Fluentd                                 out_copy        out_...
Plugins :: buf_ lereliable bu ering                                 buf_ le                       Fluentd             le  ...
Plugins :: out_execexecute external programs                        Fluentd                             out_exec          ...
Plugins :: out_exec_ lterexecute external programs                              out_exec_ lter                            ...
Plugins :: in_execexecute external programs              in_exec          out_exec_ lter   external   stdout              ...
Plugins :: in_tailRead event logs from a le                       Application                            File /var/log/acc...
Plugins :: in_tailApache log parser87.12.1.87 - - [04/Feb/2012:00:20:11 +0900] "GET / HTTP/1.1" 200 9887.12.1.87 - - [04/F...
Plugins•   Bundled plugins    >   file          writes event logs to files hourly or daily    >   forward          forwards ...
Plugins•   3rd party plugins    >     scribe            integrates Fluentd with Scribe    >     s3            uploads log ...
Plugin developer API•   Unit test framework (like “MRUnit”)    >   Fluent::Test::InputTestDriver    >   Fluent::Test::Outp...
Fluentd•   Documents    >   http://fluentd.org•   Source code    >   http://github.com/fluent•   Twitter    >   #fluentd•   M...
Upcoming SlideShare
Loading in …5
×

Fluentd meetup in japan

87,610 views

Published on

Published in: Technology

Fluentd meetup in japan

  1. 1. Fluentd Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki
  2. 2. • Sadayuki Furuhashi > twitter: @frsyuki• Treasure Data, Inc. > Software Engineer; founder• Author of MessagePack• Author of Fluentd
  3. 3. What’s Fluentd?Its like syslogd, but uses JSON for log messages
  4. 4. What’s Fluentd? Application 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage }
  5. 5. What’s Fluentd? time Application tag 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage } record
  6. 6. What’s Fluentd? Application Fluentd lter / bu er / routing Storage
  7. 7. What’s Fluentd? Application Fluentd lter / bu er / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
  8. 8. What’s Fluentd? syslogd Scribe Application File Plug-in tailPlug-in Plug-in Fluentd lter / bu er / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
  9. 9. What’s Fluentd?• Client libraries > Ruby > Perl Application > PHP > Python > Java Fluentd > ...Fluent.open(“myapp”)Fluent.event(“login”, {“user”=>38})#=> 2012-02-04 04:56:01 myapp.login {“user”:38}
  10. 10. Fluentd & Event logsBefore: App server App server App server Application Application Application File File File ... File File File ... File File File ... File High latency must wait for a day Log server Hard to analyze complex text parsers
  11. 11. Fluentd & Event logsAfter: App server App server App server Application Application Application Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd
  12. 12. Fluentd & Event logs Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd Hadoop Amazon Ready to MongoDB / Hive S3 / EMR Analyze!
  13. 13. # receive events via HTTP # save alerts to a file<source> <match alert.**> type http type file port 8888 path /var/log/fluent/alerts</source> </match># read logs from a file # forward other logs to servers<source> # (load-balancing + fail-over) type tail <match **> path /var/log/httpd.log type forward format apache <server> tag apache.access host 192.168.0.11</source> weight 20 </server># save access logs to MongoDB <server><match apache.access> host 192.168.0.12 type mongo weight 60 host 127.0.0.1 </server></match> </match>
  14. 14. Fluentd vs Scribe• Deals with structured logs• Easy to install > “gem install fluentd” > apt-get and yum http://packages.treasure-data.com/• Easy to customize• add/modify plugins without re-compiling > “gem search -rd fluent-plugin”
  15. 15. Fluentd vs Flume• Easy to setup > “sudo fluentd --setup && fluentd”• Very small footprint > small engine (3,000 lines) + plugins• JVM-free• Easy to configure
  16. 16. Architecture of Fluentd
  17. 17. Architecture :: Input
  18. 18. Architecture :: Bu er
  19. 19. Architecture :: Output
  20. 20. Plugins :: out_forwardforward event logs Fluentd Heartbeat out_forward in_forward Fluentd Fluentd ✓ load balancing
  21. 21. Plugins :: out_forwardforward event logs Fluentd Heartbeat out_forwardφ accrual failure detector in_forward Fluentd Fluentd ✓ load balancing
  22. 22. Plugins :: out_copyduplicate event logs Fluentd out_copy out_mongo out_forward out_ le MongoDB File Fluentd
  23. 23. Plugins :: buf_ lereliable bu ering buf_ le Fluentd le le✓ Automatic retry le✓ 2^N retry interval le ✓ Persistent bu er
  24. 24. Plugins :: out_execexecute external programs Fluentd out_exec TSV → stdin ✓ Python external ✓ Perl program ✓ C++
  25. 25. Plugins :: out_exec_ lterexecute external programs out_exec_ lter stdin external Fluentd stdout program out_exec TSV → stdin ✓ Python external ✓ Perl program ✓ C++
  26. 26. Plugins :: in_execexecute external programs in_exec out_exec_ lter external stdout stdin external Fluentd program stdout program out_exec TSV → stdin ✓ Python external ✓ Perl program ✓ C++
  27. 27. Plugins :: in_tailRead event logs from a le Application File /var/log/access.log ✓ Apache log parser in_tail ✓ Syslog parser ✓ Custom parser Fluentd
  28. 28. Plugins :: in_tailApache log parser87.12.1.87 - - [04/Feb/2012:00:20:11 +0900] "GET / HTTP/1.1" 200 9887.12.1.87 - - [04/Feb/2012:00:20:11 +0900] "GET / HTTP/1.1" 200 98... { “host”: “87.12.1.87”, “method”: “GET”, “code”: 200, “size”: 98, “path”: “/” } ...
  29. 29. Plugins• Bundled plugins > file writes event logs to files hourly or daily > forward forwards event logs (+fail-over and load balancing) > exec passes event logs to/from external commands > tail reads event logs from a file (like `tail -f`)
  30. 30. Plugins• 3rd party plugins > scribe integrates Fluentd with Scribe > s3 uploads log files to Amazon S3 hourly or daily > mongo writes logs to MongoDB > hoop puts log files on Hadoop HDFS via Hoop ...
  31. 31. Plugin developer API• Unit test framework (like “MRUnit”) > Fluent::Test::InputTestDriver > Fluent::Test::OutputTestDriver > Fluent::Test::BufferedOutputTestDriver• Fluent::TailInput (base class of “tail” plugin) > text parser is customizable def parse_line(line)
  32. 32. Fluentd• Documents > http://fluentd.org• Source code > http://github.com/fluent• Twitter > #fluentd• Mailing list > http://groups.google.com/group/fluentd

×