Fluentd: the missing log collector

2,097 views
1,882 views

Published on

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,097
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Fluentd: the missing log collector

    1. 1. Fluentd: the missing log collectorKiyoto Tamura
    2. 2. Sorry about the corporate chrome! I am a Keynote n00b. Recycling my CTO’s slides. Thanks Melissa O’Neill@Harvey Mudd for “PDF to Keynote”. Saved my life. 2
    3. 3. Its like syslogd, but uses JSON for log messages 8Tuesday, July 17, 2012
    4. 4. Logging in JSON? Why?• Machine-readable: More and more logs are getting consumed by machines, not homo sapiens. You want some structure...• Schema-free: but not too much structure. Data evolve fast and you need to adapt quickly. 4
    5. 5. Logs As TEXT Logs As JSON + Field Name + No Custom Parser + Type Information + Schema Free 10Tuesday, July 17, 2012
    6. 6. Logs As TEXT “2011-04-01 host1 myapp: cmessage size=12MB user=me” Logs As JSON 2011-04-01 myapp.message { “on_host”: ”host1”, ”combined”: true, “size”: 12000000, + Field Name “user”: “me” + No Custom Parser + Type Information } + Schema Free 10Tuesday, July 17, 2012
    7. 7. Fluentd ArchitectureTuesday, July 17, 2012
    8. 8. Fluentd: Log Format Application Fluentd Storage 14Tuesday, July 17, 2012
    9. 9. Fluentd: Log Format Application 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage } 14Tuesday, July 17, 2012
    10. 10. Fluentd: Log Format time Application tag 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage } record 14Tuesday, July 17, 2012
    11. 11. Fluentd: Plugins Application filter / buffer / Fluentd routing Storage 15Tuesday, July 17, 2012
    12. 12. Fluentd: Plugins Application filter / buffer / Fluentd routing SaaS Storage Fluentd Plug-in Plug-in Plug-in 16Tuesday, July 17, 2012
    13. 13. Fluentd: Plugins syslogd Scribe Application File Plug-in tail Plug-in Plug-in filter / buffer / Fluentd routing SaaS Storage Fluentd Plug-in Plug-in Plug-in 16Tuesday, July 17, 2012
    14. 14. • Client libraries > Ruby > Perl Application Buffering > PHP HTTP / TCP / UDS > Python > Java Fluentd > ... 17Tuesday, July 17, 2012
    15. 15. • Client libraries > Ruby > Perl Application Buffering > PHP HTTP / TCP / UDS > Python > Java Fluentd > ... Fluent.open(“myapp”) Fluent.event(“login”, {“user”=>38}) #=> 2012-02-04 04:56:01 myapp.login {“user”:38} 17Tuesday, July 17, 2012
    16. 16. Typical Log Collection by `rsync` Burst of traffic rsync consumes all bandwidth 18Tuesday, July 17, 2012
    17. 17. Typical Log Collection by `rsync` App server App server App server Application Application Application File File File ... File File File ... File File File ... File Burst of traffic High latency rsync consumes must wait for a day all bandwidth Log server Hard to analyze complex text parsers 18Tuesday, July 17, 2012
    18. 18. Log Collection using Fluentd Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd 19Tuesday, July 17, 2012
    19. 19. Log Collection using Fluentd Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd Amazon Ready to Hadoop Mongo S3 / / Hive DB EMR Analyze! 19Tuesday, July 17, 2012
    20. 20. Fluentd Case Study Ruby on Rails Ruby on Rails Ruby on Rails Fluentd Fluentd Fluentd ✓ 127 RoR servers ✓ 100,000 msgs/sec Fluentd Fluentd routing ✓ 120Mbps at peak ✓ 1TB/day Hadoop Mongo User behavior PV logs / Hive DB logs 20Tuesday, July 17, 2012
    21. 21. http://fluentd.org/ 11Tuesday, July 17, 2012
    22. 22. Thanks!

    ×