Plugins by tagomoris #fluentdcasual

8,936 views
8,850 views

Published on

Published in: Technology

Plugins by tagomoris #fluentdcasual

  1. 1. Pluginsby tagomoris Fluentd Casual Talks LT 2012/05/18@tagomoris (NHN Japan Corp.)
  2. 2. @tagomoris NHN Japan Corp.Web Service Business DivisionDevelopment Department 2 Development Platforms
  3. 3. Appendix diff of 2012/02 and now• Our Fluentd cluster status
  4. 4. Fluentd Cluster status updates (1) 89 Fluentd processes 12 nodes 89 Fluentd processes 12 nodes
  5. 5. Fluentd Cluster status updates (2) ruby 1.9.2 + glibc malloc scribeline ruby 1.9.3 + jemalloc fluent-agent-lite
  6. 6. Fluentd Cluster status updates (3) from 127 servers 146 log streams from 205 servers 246 log streams
  7. 7. Fluentd Cluster status updates (4) 70,000 msgs/sec 120Mbps (at peak time) 92,000 msgs/sec 184Mbps
  8. 8. Fluentd Cluster status updates (5) 650GB/day (non-compressed bytes on HDFS) 995GB/day
  9. 9. Fluentd Cluster status updates (6) Before: 10 pluginsin_forward, out_forward, in_scribe, out_scribe out_copy, out_roundrobin, out_exec_filter out_hoop out_flowcounter, out_growthforecast After: +9 plugins out_route, in_gc_stat, in_object_space out_mongo, out_sampling_filter out_amplifier_filter, out_datacounter out_file_alternative, out_forest
  10. 10. Plugins!All your plugin are belong to us.
  11. 11. gem search -r fluent-plugin 41 pluginsfluent-plugin-amplifier-filter (0.1.1) fluent-plugin-msgpack-rpc (0.2.1) fluent-plugin-amqp (0.0.0) fluent-plugin-mysql (0.0.2) fluent-plugin-cassandra (0.0.2) fluent-plugin-notifier (0.0.1) fluent-plugin-couch (0.5.0) fluent-plugin-numeric-monitor (0.0.1)fluent-plugin-datacalculator (0.0.1) fluent-plugin-parser (0.1.0) fluent-plugin-datacounter (0.3.0) fluent-plugin-pghstore (0.1.2) fluent-plugin-delayed (0.0.1) fluent-plugin-redis (0.2.0) fluent-plugin-dstat (0.1.0) fluent-plugin-resque (0.2.1)fluent-plugin-file-alternative (0.1.2) fluent-plugin-s3 (0.2.2) fluent-plugin-flowcounter (0.1.4) fluent-plugin-sampling-filter (0.1.1) fluent-plugin-flume (0.1.0) fluent-plugin-say (0.1.1) fluent-plugin-forest (0.1.0) fluent-plugin-scribe (0.10.8) fluent-plugin-growl (0.0.5) fluent-plugin-sns (0.2.1)fluent-plugin-growthforecast (0.1.2) fluent-plugin-solr (0.1.1) fluent-plugin-hipchat (0.1.0) fluent-plugin-splunk (0.0.1.1) fluent-plugin-hoop (0.1.2) fluent-plugin-sqs (0.2.1)fluent-plugin-http-enhanced (0.0.3) fluent-plugin-td (0.10.6) fluent-plugin-ikachan (0.1.0) fluent-plugin-udp (0.0.1) fluent-plugin-kestrel (0.1.0) fluent-plugin-zabbix (0.0.1)fluent-plugin-librato-metrics (0.2.3) fluent-plugin-zmq (0.0.3) fluent-plugin-mongo (0.6.7)
  12. 12. gem search -r fluent-plugin 13/41 pluginsfluent-plugin-amplifier-filter (0.1.1) fluent-plugin-msgpack-rpc (0.2.1) fluent-plugin-amqp (0.0.0) fluent-plugin-mysql (0.0.2) fluent-plugin-cassandra (0.0.2) fluent-plugin-notifier (0.0.1) fluent-plugin-couch (0.5.0) fluent-plugin-numeric-monitor (0.0.1) fluent-plugin-datacalculator (0.0.1) fluent-plugin-parser (0.1.0) fluent-plugin-datacounter (0.3.0) fluent-plugin-pghstore (0.1.2) fluent-plugin-delayed (0.0.1) fluent-plugin-redis (0.2.0) fluent-plugin-dstat (0.1.0) fluent-plugin-resque (0.2.1)fluent-plugin-file-alternative (0.1.2) fluent-plugin-s3 (0.2.2) fluent-plugin-flowcounter (0.1.4) fluent-plugin-sampling-filter (0.1.1) fluent-plugin-flume (0.1.0) fluent-plugin-say (0.1.1) fluent-plugin-forest (0.1.0) fluent-plugin-scribe (0.10.8) fluent-plugin-growl (0.0.5) fluent-plugin-sns (0.2.1)fluent-plugin-growthforecast (0.1.2) fluent-plugin-solr (0.1.1) fluent-plugin-hipchat (0.1.0) fluent-plugin-splunk (0.0.1.1) fluent-plugin-hoop (0.1.2) fluent-plugin-sqs (0.2.1)fluent-plugin-http-enhanced (0.0.3) fluent-plugin-td (0.10.6) fluent-plugin-ikachan (0.1.0) fluent-plugin-udp (0.0.1) fluent-plugin-kestrel (0.1.0) fluent-plugin-zabbix (0.0.1)fluent-plugin-librato-metrics (0.2.3) fluent-plugin-zmq (0.0.3) fluent-plugin-mongo (0.6.7)
  13. 13. file-alternative (1/13)out_file time [TAB] tag [TAB] {"key1":"value1", "key2":"value2",... }out_file_alternative time [TAB] tag [TAB] {"key1":"value1", "key2":"value2",... } time [TAB] {"key1":"value1", "key2":"value2",... } tag [TAB] {"key1":"value1", "key2":"value2",... } time [TAB] tag [TAB] value1 [TAB] value2 ... time,tag,value1,value2 ...
  14. 14. hoop (2/13) HoopFluentd Server HDFSoutput format compatible with out_file_alternative
  15. 15. mysql (3/13) alpha!Fluentd MySQL JSON into single column or values into columns
  16. 16. ikachan (4/13) alpha!Fluentd ikachan IRC Server Server
  17. 17. growthforecast (5/13) growthforecast by @kazeburo
  18. 18. flowcounter (6/13) messages count_keys *flowcounter unit minute{"count":300, "bytes":3660,"count_rate":5, "bytes_rate":61}
  19. 19. datacounter (7/13) messages unit minutedatacounter aggregate all count_key referer pattern1 google google.com{"google_count":120, "google_rate":2, "google_percentage":20, "unmatched_count":480, ...}
  20. 20. alpha!numeric-monitor (8/13) messages unit minute aggregate allnumeric_monitor monitor_key duration percentiles 90,95{"min":100131,"max":483120, "avg":181343, "percentile_90":283201, "percentile_95":341210}
  21. 21. notifier (9/13) alpha!10 Notify msg12 suppressed10 notifier in specified interval... warn 10 crit 301513 Notify msg
  22. 22. sampling-filter (10/13)msg msgmsgmsg samplingmsg msg filtermsg interval 3msgmsg msg
  23. 23. amplifier-filter (11/13)10 3012 3610 amplifier 30 filter15 ratio 3 4513 39
  24. 24. beautiful real-time monitoring worldStreaming data Sampling Counting Amplifying Generatig Notification Output (Graph, IRC, ...)
  25. 25. parser (12/13) alpha!{"log":"2012/05/18:16:21 DRINK1 1"} format /^(?<time>S+) (?<item>S+) (?<num>d+)$/ numeric_monitor time_format %Y/%m/%d:%H:%M:%S key_name log reserve_data yeslog_based_time message{"log":"2012/05/18:16:21 DRINK1 1", "item":"DRINK1", "num":"1"}
  26. 26. <match input.A> forest (13/13) type foo .... <match input.*> conf_a xxxx.A.yyyy type forest option_x 30 subtype foo .... remove_prefix input</match> <template><match input.B> .... type foo conf_a xxxx.__TAG__.yyyy .... .... conf_a xxxx.B.yyyy </template> option_x 30 <case {C,D}> .... option_x 100</match> </case><match input.C> <case *> type foo option_x 30 .... </case> conf_a xxxx.C.yyyy </match> option_x 100 ....</match>
  27. 27. NO PLUGINS,NO FLUENTD Enjoy Fluentd!
  28. 28. One more thing
  29. 29. fluent-plugin-webhdfs Coming Soon!
  30. 30. Thanks!

×