Fluentd meetup #2

11,221 views
11,050 views

Published on

Published in: Technology, Education
1 Comment
61 Likes
Statistics
Notes
No Downloads
Views
Total views
11,221
On SlideShare
0
From Embeds
0
Number of Embeds
1,776
Actions
Shares
0
Downloads
120
Comments
1
Likes
61
Embeds 0
No embeds

No notes for slide

Fluentd meetup #2

  1. 1. Log everyting in JSON. Treasuare Data, Inc. Sadayuki Furuhashi
  2. 2. Self-introduction> Sadayuki Furuhashi twitter: @frsyuki> Original author of Fluentd> Treasure Data, Inc. Software Architect; Founder> open-source MessagePack - efficient serialization format
  3. 3. 0. Why logging?1. Why Fluentd? - Design of Fluentd > Extensibility > Unified log format > Simplicity2. Who uses Fluentd?3. Future of Fluentd
  4. 4. 0. Why logging?1. Why Fluentd? - Design of Fluentd > Extensibility > Unified log format > Simplicity2. Who uses Fluentd?3. Future of Fluentd
  5. 5. 0. Why logging? > Error notifications > Performance monitoring > User segment analysis > Funnel analysis > Heatmap analysis > Market prediction etc...
  6. 6. 0. Why logging? - Error notifications Error!
  7. 7. 0. Why logging? - Performance monitor
  8. 8. 0. Why logging? - User segment analysis
  9. 9. 0. Why logging? - Funnel analysis -28%! -27%!
  10. 10. 0. Why logging? - Heatmap analysis
  11. 11. 0. Why logging? - Market prediction
  12. 12. 0. Why logging?1. Why Fluentd? - Design of Fluentd > Extensibility > Unified log format > Simplicity2. Who uses Fluentd?3. Future of Fluentd
  13. 13. 0. Why logging?1. Why Fluentd? - Design of Fluentd > Extensibility > Unified log format > Simplicity2. Who uses Fluentd?3. Future of Fluentd
  14. 14. Alerting Nagios Analysis MongoDB MySQL Hadooplog utilization Archiving Amazon S3
  15. 15. Access logs Alerting Apache NagiosApp logs Analysis Frontend log sources MongoDB Backend MySQLSystem logs Hadoop syslogd log utilization ArchivingDatabases Amazon S3
  16. 16. Access logs Alerting Apache NagiosApp logs Analysis Frontend MongoDB Backend MySQLSystem logs Hadoop syslogd ArchivingDatabases Amazon S3
  17. 17. Access logs rsync servers Alerting Apache NagiosApp logs Analysis Frontend MongoDB Backend MySQL bash scriptsSystem logs Hadoop syslogd ArchivingDatabases Amazon S3 perl scripts
  18. 18. Problems...No unified method to collect logs> Too many bash/perl scripts Fragile for changes Less reliable> Mixed log formats Old-fashioned “Human-readable” text logs Not ready to analyze> High latency must wait a day for log rotation
  19. 19. Access logs Alerting Apache NagiosApp logs Analysis Frontend MongoDB Backend MySQLSystem logs Hadoop syslogd ArchivingDatabases Amazon S3
  20. 20. Access logs Alerting Apache NagiosApp logs Analysis Frontend MongoDB Backend MySQLSystem logs Hadoop syslogd Archiving filter / buffer / routingDatabases Amazon S3
  21. 21. Input Plugins Output Plugins Buffer Plugins Filter Plugins
  22. 22. Input Plugins Output Plugins 2012-02-04 01:33:51 JSON format myapp.buylog { “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” }
  23. 23. Input Plugins Output Plugins time tag 2012-02-04 01:33:51 JSON format myapp.buylog { “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” } record
  24. 24. Why Fluentd?> Extensibility - Plugin architecture collect logs from various systems forward logs to various systems> Unified log format - JSON format modern “Machine-readable” log format immediately ready to analyze> Reliable - HA configuration> Easy to install - RPM/deb packages deploy instantly to everywhere
  25. 25. Comparision with other log collectors:> Scribe Less extensible No unified log format No longer developped?> Flume Less simple No unified log format Little information about Flume-NG
  26. 26. 0. Why logging?1. Why Fluentd? - Design of Fluentd > Extensibility > Unified log format > Simplicity2. Who uses Fluentd?3. Future of Fluentd
  27. 27. 0. Why logging?1. Why Fluentd? - Design of Fluentd > Extensibility > Unified log format > Simplicity2. Who uses Fluentd?3. Future of Fluentd
  28. 28. NHN Japan COOKPAD NAVERCrocoshttp://www.quora.com/Who-uses-Fluentd-in-production
  29. 29. 0. Why logging?1. Why Fluentd? - Design of Fluentd > Extensibility > Unified log format > Simplicity2. Who uses Fluentd?3. Future of Fluentd
  30. 30. 0. Why logging?1. Why Fluentd? - Design of Fluentd > Extensibility > Unified log format > Simplicity2. Who uses Fluentd?3. Future of Fluentd
  31. 31. Future of Fluentd> <filter>> <match> in <source>> <label>> MessagePack for Ruby v5> td-agent-lite> Pub/Sub & Monitoring API> New process model & Live restart> Backward compatibility
  32. 32. <source> <match **> type tail type forward path /var/log/httpd.log host log.server format apache </match> tag not_filtered.apache</source> Mysterious tag<match not_filetered.**> type rewrite remove_prefix not_filtered tag operations <rule> key status pattern ^500$ ignore true </rule></match> Before
  33. 33. <source> <match **> type tail type forward path /var/log/httpd.log host log.server format apache </match> tag apache</source><filter **> Filter plugins! type rewrite <rule> key status pattern ^500$ ignore true </rule></match> After (v11)
  34. 34. <source> <match **> type tail type forward path /var/log/httpd.log host log.server format apache </match> tag apache <filter **> type rewrite <rule> key status pattern ^500$ <filter>/<match> in ignore true <source> </rule> </match></source> After (v11)
  35. 35. <source> <match **> type tail type forward path /var/log/httpd.log host log.server tag apache </match></source> I want to add flowcounter here... Before
  36. 36. <source> <store> type tail type forward path /var/log/httpd.log host log.server tag apache </store></source> </match><match flow.traffic> type forward host traffic.server</match> Nested!<match **> type copy <store> type flowcounter tag flow.traffic </store> Before
  37. 37. <source> <match **> type tail type forward path /var/log/httpd.log host log.server tag apache </match></source><filter **> type copy <match> type flowcounter tag flow.traffic <match> type forward host traffic.server </match> </match> Filtering pipeline</match> After (v11)
  38. 38. <source> # copy & label & forward type forward <filter **></source> type copy <match><filter **> type forward type copy label alert <match> host alerting.server type file </match> path /mnt/local_archive </filter> </match></filter> # copy & label & forward <filter **><label alert> type copy <match **> <match> ... type forward </match> label analysis</label> host analysis.server </match><label analysis> </filter> ...</label> After (v11)
  39. 39. MessagePack for Ruby v5(tweets/sec)40000300002000010000 0 Serialize Deserialize msgpack v5 msgpack v4 yajl json
  40. 40. td-agent-lite> in_tail + out_forward in “single” binary statically linked ruby binary + scripts tied with the binary
  41. 41. New process model & Live restartOld multiprocess model detached process fork() Supervisor Engine detached all data pass through process the central process
  42. 42. New process model & Live restartNew multiprocess model detached process Process Supervisor Engine Manager detached process direct communication
  43. 43. New process model & Live restartNew multiprocess model detached process Process Supervisor Engine Manager detached Live restart process Process Engine Manager
  44. 44. Backward compatibilityFluentd v11 includes 2 namespaces: > Fluentd:: new code base > Fluent:: old code base + wrapper classesCheckout the repository for details: > http://github.com/frsyuki/fluentd-v11
  45. 45. ConculutionFluentd makes logging better > Plugin architecture > JSON format > HA configuration > RPM/deb packageFluentd is under active developmentFluentd is suppored by many committers
  46. 46. contact: sales@treasure-data.com
  47. 47. ログ収集/解析に使っているツール
  48. 48. ログの保存先
  49. 49. Fluentdを導入するにあたっての障壁

×