Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Fluentd: the missing log collectorKiyoto Tamura
Sorry about the corporate chrome!  I am a Keynote n00b. Recycling my CTO’s slides.  Thanks Melissa O’Neill@Harvey Mudd for...
Its like syslogd, but uses JSON for log                 messages                                                          ...
Logging in JSON? Why?• Machine-readable: More and more  logs are getting consumed by  machines, not homo sapiens. You  wan...
Logs As TEXT   Logs As JSON                         + Field Name                         + No Custom Parser               ...
Logs As TEXT         “2011-04-01 host1 myapp: cmessage size=12MB user=me”   Logs As JSON                         2011-04-0...
Fluentd ArchitectureTuesday, July 17, 2012
Fluentd: Log Format                         Application                          Fluentd                          Storage ...
Fluentd: Log Format                         Application                                       2012-02-04 01:33:51         ...
Fluentd: Log Format                                                       time                         Application        ...
Fluentd: Plugins                             Application                                           filter / buffer /       ...
Fluentd: Plugins                                       Application                                                     filt...
Fluentd: Plugins            syslogd         Scribe     Application          File Plug-in                                  ...
•       Client libraries                   > Ruby                   > Perl             Application         Buffering      ...
•       Client libraries                   > Ruby                   > Perl               Application         Buffering    ...
Typical Log Collection by `rsync`               Burst of traffic               rsync consumes               all bandwidth ...
Typical Log Collection by `rsync`                     App server              App server              App server          ...
Log Collection using Fluentd                         Fluentd        Fluentd          Fluentd                              ...
Log Collection using Fluentd                         Fluentd        Fluentd          Fluentd                              ...
Fluentd Case Study               Ruby on Rails              Ruby on Rails          Ruby on Rails                         F...
http://fluentd.org/                                              11Tuesday, July 17, 2012
Thanks!
Upcoming SlideShare
Loading in …5
×

Fluentd: the missing log collector

2,218 views

Published on

Published in: Technology, Business
  • Be the first to comment

Fluentd: the missing log collector

  1. 1. Fluentd: the missing log collectorKiyoto Tamura
  2. 2. Sorry about the corporate chrome! I am a Keynote n00b. Recycling my CTO’s slides. Thanks Melissa O’Neill@Harvey Mudd for “PDF to Keynote”. Saved my life. 2
  3. 3. Its like syslogd, but uses JSON for log messages 8Tuesday, July 17, 2012
  4. 4. Logging in JSON? Why?• Machine-readable: More and more logs are getting consumed by machines, not homo sapiens. You want some structure...• Schema-free: but not too much structure. Data evolve fast and you need to adapt quickly. 4
  5. 5. Logs As TEXT Logs As JSON + Field Name + No Custom Parser + Type Information + Schema Free 10Tuesday, July 17, 2012
  6. 6. Logs As TEXT “2011-04-01 host1 myapp: cmessage size=12MB user=me” Logs As JSON 2011-04-01 myapp.message { “on_host”: ”host1”, ”combined”: true, “size”: 12000000, + Field Name “user”: “me” + No Custom Parser + Type Information } + Schema Free 10Tuesday, July 17, 2012
  7. 7. Fluentd ArchitectureTuesday, July 17, 2012
  8. 8. Fluentd: Log Format Application Fluentd Storage 14Tuesday, July 17, 2012
  9. 9. Fluentd: Log Format Application 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage } 14Tuesday, July 17, 2012
  10. 10. Fluentd: Log Format time Application tag 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage } record 14Tuesday, July 17, 2012
  11. 11. Fluentd: Plugins Application filter / buffer / Fluentd routing Storage 15Tuesday, July 17, 2012
  12. 12. Fluentd: Plugins Application filter / buffer / Fluentd routing SaaS Storage Fluentd Plug-in Plug-in Plug-in 16Tuesday, July 17, 2012
  13. 13. Fluentd: Plugins syslogd Scribe Application File Plug-in tail Plug-in Plug-in filter / buffer / Fluentd routing SaaS Storage Fluentd Plug-in Plug-in Plug-in 16Tuesday, July 17, 2012
  14. 14. • Client libraries > Ruby > Perl Application Buffering > PHP HTTP / TCP / UDS > Python > Java Fluentd > ... 17Tuesday, July 17, 2012
  15. 15. • Client libraries > Ruby > Perl Application Buffering > PHP HTTP / TCP / UDS > Python > Java Fluentd > ... Fluent.open(“myapp”) Fluent.event(“login”, {“user”=>38}) #=> 2012-02-04 04:56:01 myapp.login {“user”:38} 17Tuesday, July 17, 2012
  16. 16. Typical Log Collection by `rsync` Burst of traffic rsync consumes all bandwidth 18Tuesday, July 17, 2012
  17. 17. Typical Log Collection by `rsync` App server App server App server Application Application Application File File File ... File File File ... File File File ... File Burst of traffic High latency rsync consumes must wait for a day all bandwidth Log server Hard to analyze complex text parsers 18Tuesday, July 17, 2012
  18. 18. Log Collection using Fluentd Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd 19Tuesday, July 17, 2012
  19. 19. Log Collection using Fluentd Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd Amazon Ready to Hadoop Mongo S3 / / Hive DB EMR Analyze! 19Tuesday, July 17, 2012
  20. 20. Fluentd Case Study Ruby on Rails Ruby on Rails Ruby on Rails Fluentd Fluentd Fluentd ✓ 127 RoR servers ✓ 100,000 msgs/sec Fluentd Fluentd routing ✓ 120Mbps at peak ✓ 1TB/day Hadoop Mongo User behavior PV logs / Hive DB logs 20Tuesday, July 17, 2012
  21. 21. http://fluentd.org/ 11Tuesday, July 17, 2012
  22. 22. Thanks!

×