Fluentd                               Structured logging                               Pluggable architecture             ...
•   Sadayuki Furuhashi    >   twitter: @frsyuki•   Treasure Data, Inc.    >   Software Engineer; founder•   Author of Mess...
What’s Fluentd?Its like syslogd, but uses JSON for log messages
What’s Fluentd?          Application                        2012-02-04 01:33:51                        myapp.buylog {     ...
What’s Fluentd?                                              time          Application                           tag      ...
What’s Fluentd?          Application           Fluentd      lter / bu er / routing           Storage
What’s Fluentd?             Application              Fluentd      lter / bu er / routing    SaaS      Storage            F...
What’s Fluentd? syslogd     Scribe   Application          File   Plug-in                                    tailPlug-in Pl...
What’s Fluentd?•   Client libraries    > Ruby    > Perl                     Application    > PHP    > Python    > Java    ...
Fluentd & Event logsBefore:   App server                  App server              App server   Application                ...
Fluentd & Event logsAfter:   App server        App server         App server   Application       Application        Applic...
Fluentd & Event logs  Fluentd          Fluentd                 Fluentd                                        Realtime!   ...
# receive events via HTTP       # save alerts to a file<source>                        <match alert.**>  type http        ...
Fluentd vs Scribe•   Deals with structured logs•   Easy to install    >   “gem install fluentd”    >   apt-get and yum     ...
Fluentd vs Flume•   Easy to setup    >   “sudo fluentd --setup && fluentd”•   Very small footprint    >   small engine (3,00...
Architecture of Fluentd
Architecture :: Input
Architecture :: Bu er
Architecture :: Output
Plugins :: out_forwardforward event logs                           Fluentd         Heartbeat                              ...
Plugins :: out_forwardforward event logs                                 Fluentd            Heartbeat                     ...
Plugins :: out_copyduplicate event logs                       Fluentd                                 out_copy        out_...
Plugins :: buf_ lereliable bu ering                                 buf_ le                       Fluentd             le  ...
Plugins :: out_execexecute external programs                        Fluentd                             out_exec          ...
Plugins :: out_exec_ lterexecute external programs                              out_exec_ lter                            ...
Plugins :: in_execexecute external programs              in_exec          out_exec_ lter   external   stdout              ...
Plugins :: in_tailRead event logs from a le                       Application                            File /var/log/acc...
Plugins :: in_tailApache log parser87.12.1.87 - - [04/Feb/2012:00:20:11 +0900] "GET / HTTP/1.1" 200 9887.12.1.87 - - [04/F...
Plugins•   Bundled plugins    >   file          writes event logs to files hourly or daily    >   forward          forwards ...
Plugins•   3rd party plugins    >     scribe            integrates Fluentd with Scribe    >     s3            uploads log ...
Plugin developer API•   Unit test framework (like “MRUnit”)    >   Fluent::Test::InputTestDriver    >   Fluent::Test::Outp...
Fluentd•   Documents    >   http://fluentd.org•   Source code    >   http://github.com/fluent•   Twitter    >   #fluentd•   M...
Upcoming SlideShare
Loading in...5
×

Fluentd meetup in japan

84,640

Published on

Published in: Technology
1 Comment
66 Likes
Statistics
Notes
No Downloads
Views
Total Views
84,640
On Slideshare
0
From Embeds
0
Number of Embeds
40
Actions
Shares
0
Downloads
188
Comments
1
Likes
66
Embeds 0
No embeds

No notes for slide

Fluentd meetup in japan

  1. 1. Fluentd Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki
  2. 2. • Sadayuki Furuhashi > twitter: @frsyuki• Treasure Data, Inc. > Software Engineer; founder• Author of MessagePack• Author of Fluentd
  3. 3. What’s Fluentd?Its like syslogd, but uses JSON for log messages
  4. 4. What’s Fluentd? Application 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage }
  5. 5. What’s Fluentd? time Application tag 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage } record
  6. 6. What’s Fluentd? Application Fluentd lter / bu er / routing Storage
  7. 7. What’s Fluentd? Application Fluentd lter / bu er / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
  8. 8. What’s Fluentd? syslogd Scribe Application File Plug-in tailPlug-in Plug-in Fluentd lter / bu er / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
  9. 9. What’s Fluentd?• Client libraries > Ruby > Perl Application > PHP > Python > Java Fluentd > ...Fluent.open(“myapp”)Fluent.event(“login”, {“user”=>38})#=> 2012-02-04 04:56:01 myapp.login {“user”:38}
  10. 10. Fluentd & Event logsBefore: App server App server App server Application Application Application File File File ... File File File ... File File File ... File High latency must wait for a day Log server Hard to analyze complex text parsers
  11. 11. Fluentd & Event logsAfter: App server App server App server Application Application Application Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd
  12. 12. Fluentd & Event logs Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd Hadoop Amazon Ready to MongoDB / Hive S3 / EMR Analyze!
  13. 13. # receive events via HTTP # save alerts to a file<source> <match alert.**> type http type file port 8888 path /var/log/fluent/alerts</source> </match># read logs from a file # forward other logs to servers<source> # (load-balancing + fail-over) type tail <match **> path /var/log/httpd.log type forward format apache <server> tag apache.access host 192.168.0.11</source> weight 20 </server># save access logs to MongoDB <server><match apache.access> host 192.168.0.12 type mongo weight 60 host 127.0.0.1 </server></match> </match>
  14. 14. Fluentd vs Scribe• Deals with structured logs• Easy to install > “gem install fluentd” > apt-get and yum http://packages.treasure-data.com/• Easy to customize• add/modify plugins without re-compiling > “gem search -rd fluent-plugin”
  15. 15. Fluentd vs Flume• Easy to setup > “sudo fluentd --setup && fluentd”• Very small footprint > small engine (3,000 lines) + plugins• JVM-free• Easy to configure
  16. 16. Architecture of Fluentd
  17. 17. Architecture :: Input
  18. 18. Architecture :: Bu er
  19. 19. Architecture :: Output
  20. 20. Plugins :: out_forwardforward event logs Fluentd Heartbeat out_forward in_forward Fluentd Fluentd ✓ load balancing
  21. 21. Plugins :: out_forwardforward event logs Fluentd Heartbeat out_forwardφ accrual failure detector in_forward Fluentd Fluentd ✓ load balancing
  22. 22. Plugins :: out_copyduplicate event logs Fluentd out_copy out_mongo out_forward out_ le MongoDB File Fluentd
  23. 23. Plugins :: buf_ lereliable bu ering buf_ le Fluentd le le✓ Automatic retry le✓ 2^N retry interval le ✓ Persistent bu er
  24. 24. Plugins :: out_execexecute external programs Fluentd out_exec TSV → stdin ✓ Python external ✓ Perl program ✓ C++
  25. 25. Plugins :: out_exec_ lterexecute external programs out_exec_ lter stdin external Fluentd stdout program out_exec TSV → stdin ✓ Python external ✓ Perl program ✓ C++
  26. 26. Plugins :: in_execexecute external programs in_exec out_exec_ lter external stdout stdin external Fluentd program stdout program out_exec TSV → stdin ✓ Python external ✓ Perl program ✓ C++
  27. 27. Plugins :: in_tailRead event logs from a le Application File /var/log/access.log ✓ Apache log parser in_tail ✓ Syslog parser ✓ Custom parser Fluentd
  28. 28. Plugins :: in_tailApache log parser87.12.1.87 - - [04/Feb/2012:00:20:11 +0900] "GET / HTTP/1.1" 200 9887.12.1.87 - - [04/Feb/2012:00:20:11 +0900] "GET / HTTP/1.1" 200 98... { “host”: “87.12.1.87”, “method”: “GET”, “code”: 200, “size”: 98, “path”: “/” } ...
  29. 29. Plugins• Bundled plugins > file writes event logs to files hourly or daily > forward forwards event logs (+fail-over and load balancing) > exec passes event logs to/from external commands > tail reads event logs from a file (like `tail -f`)
  30. 30. Plugins• 3rd party plugins > scribe integrates Fluentd with Scribe > s3 uploads log files to Amazon S3 hourly or daily > mongo writes logs to MongoDB > hoop puts log files on Hadoop HDFS via Hoop ...
  31. 31. Plugin developer API• Unit test framework (like “MRUnit”) > Fluent::Test::InputTestDriver > Fluent::Test::OutputTestDriver > Fluent::Test::BufferedOutputTestDriver• Fluent::TailInput (base class of “tail” plugin) > text parser is customizable def parse_line(line)
  32. 32. Fluentd• Documents > http://fluentd.org• Source code > http://github.com/fluent• Twitter > #fluentd• Mailing list > http://groups.google.com/group/fluentd
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×