Fluentd meetup in japan

85,923 views
85,940 views

Published on

Published in: Technology
1 Comment
67 Likes
Statistics
Notes
No Downloads
Views
Total views
85,923
On SlideShare
0
From Embeds
0
Number of Embeds
75,737
Actions
Shares
0
Downloads
195
Comments
1
Likes
67
Embeds 0
No embeds

No notes for slide

Fluentd meetup in japan

  1. 1. Fluentd Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki
  2. 2. • Sadayuki Furuhashi > twitter: @frsyuki• Treasure Data, Inc. > Software Engineer; founder• Author of MessagePack• Author of Fluentd
  3. 3. What’s Fluentd?Its like syslogd, but uses JSON for log messages
  4. 4. What’s Fluentd? Application 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage }
  5. 5. What’s Fluentd? time Application tag 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage } record
  6. 6. What’s Fluentd? Application Fluentd lter / bu er / routing Storage
  7. 7. What’s Fluentd? Application Fluentd lter / bu er / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
  8. 8. What’s Fluentd? syslogd Scribe Application File Plug-in tailPlug-in Plug-in Fluentd lter / bu er / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in
  9. 9. What’s Fluentd?• Client libraries > Ruby > Perl Application > PHP > Python > Java Fluentd > ...Fluent.open(“myapp”)Fluent.event(“login”, {“user”=>38})#=> 2012-02-04 04:56:01 myapp.login {“user”:38}
  10. 10. Fluentd & Event logsBefore: App server App server App server Application Application Application File File File ... File File File ... File File File ... File High latency must wait for a day Log server Hard to analyze complex text parsers
  11. 11. Fluentd & Event logsAfter: App server App server App server Application Application Application Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd
  12. 12. Fluentd & Event logs Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd Hadoop Amazon Ready to MongoDB / Hive S3 / EMR Analyze!
  13. 13. # receive events via HTTP # save alerts to a file<source> <match alert.**> type http type file port 8888 path /var/log/fluent/alerts</source> </match># read logs from a file # forward other logs to servers<source> # (load-balancing + fail-over) type tail <match **> path /var/log/httpd.log type forward format apache <server> tag apache.access host 192.168.0.11</source> weight 20 </server># save access logs to MongoDB <server><match apache.access> host 192.168.0.12 type mongo weight 60 host 127.0.0.1 </server></match> </match>
  14. 14. Fluentd vs Scribe• Deals with structured logs• Easy to install > “gem install fluentd” > apt-get and yum http://packages.treasure-data.com/• Easy to customize• add/modify plugins without re-compiling > “gem search -rd fluent-plugin”
  15. 15. Fluentd vs Flume• Easy to setup > “sudo fluentd --setup && fluentd”• Very small footprint > small engine (3,000 lines) + plugins• JVM-free• Easy to configure
  16. 16. Architecture of Fluentd
  17. 17. Architecture :: Input
  18. 18. Architecture :: Bu er
  19. 19. Architecture :: Output
  20. 20. Plugins :: out_forwardforward event logs Fluentd Heartbeat out_forward in_forward Fluentd Fluentd ✓ load balancing
  21. 21. Plugins :: out_forwardforward event logs Fluentd Heartbeat out_forwardφ accrual failure detector in_forward Fluentd Fluentd ✓ load balancing
  22. 22. Plugins :: out_copyduplicate event logs Fluentd out_copy out_mongo out_forward out_ le MongoDB File Fluentd
  23. 23. Plugins :: buf_ lereliable bu ering buf_ le Fluentd le le✓ Automatic retry le✓ 2^N retry interval le ✓ Persistent bu er
  24. 24. Plugins :: out_execexecute external programs Fluentd out_exec TSV → stdin ✓ Python external ✓ Perl program ✓ C++
  25. 25. Plugins :: out_exec_ lterexecute external programs out_exec_ lter stdin external Fluentd stdout program out_exec TSV → stdin ✓ Python external ✓ Perl program ✓ C++
  26. 26. Plugins :: in_execexecute external programs in_exec out_exec_ lter external stdout stdin external Fluentd program stdout program out_exec TSV → stdin ✓ Python external ✓ Perl program ✓ C++
  27. 27. Plugins :: in_tailRead event logs from a le Application File /var/log/access.log ✓ Apache log parser in_tail ✓ Syslog parser ✓ Custom parser Fluentd
  28. 28. Plugins :: in_tailApache log parser87.12.1.87 - - [04/Feb/2012:00:20:11 +0900] "GET / HTTP/1.1" 200 9887.12.1.87 - - [04/Feb/2012:00:20:11 +0900] "GET / HTTP/1.1" 200 98... { “host”: “87.12.1.87”, “method”: “GET”, “code”: 200, “size”: 98, “path”: “/” } ...
  29. 29. Plugins• Bundled plugins > file writes event logs to files hourly or daily > forward forwards event logs (+fail-over and load balancing) > exec passes event logs to/from external commands > tail reads event logs from a file (like `tail -f`)
  30. 30. Plugins• 3rd party plugins > scribe integrates Fluentd with Scribe > s3 uploads log files to Amazon S3 hourly or daily > mongo writes logs to MongoDB > hoop puts log files on Hadoop HDFS via Hoop ...
  31. 31. Plugin developer API• Unit test framework (like “MRUnit”) > Fluent::Test::InputTestDriver > Fluent::Test::OutputTestDriver > Fluent::Test::BufferedOutputTestDriver• Fluent::TailInput (base class of “tail” plugin) > text parser is customizable def parse_line(line)
  32. 32. Fluentd• Documents > http://fluentd.org• Source code > http://github.com/fluent• Twitter > #fluentd• Mailing list > http://groups.google.com/group/fluentd

×