Pluginsby tagomoris  Fluentd Casual Talks LT       2012/05/18@tagomoris (NHN Japan Corp.)
@tagomoris     NHN Japan Corp.Web Service Business DivisionDevelopment Department 2  Development Platforms
Appendix diff of 2012/02 and now• Our Fluentd cluster status
Fluentd Cluster status updates (1)      89 Fluentd processes            12 nodes      89 Fluentd processes            12 n...
Fluentd Cluster status updates (2)   ruby 1.9.2 + glibc malloc           scribeline     ruby 1.9.3 + jemalloc       fluent-...
Fluentd Cluster status updates (3)        from 127 servers         146 log streams        from 205 servers         246 log...
Fluentd Cluster status updates (4)        70,000 msgs/sec           120Mbps             (at peak time)        92,000 msgs/...
Fluentd Cluster status updates (5)            650GB/day        (non-compressed bytes on HDFS)            995GB/day
Fluentd Cluster status updates (6)              Before: 10 pluginsin_forward, out_forward, in_scribe, out_scribe  out_copy...
Plugins!All your plugin are belong to us.
gem search -r fluent-plugin                              41 pluginsfluent-plugin-amplifier-filter (0.1.1)      fluent-plugin-ms...
gem search -r fluent-plugin                         13/41 pluginsfluent-plugin-amplifier-filter (0.1.1)      fluent-plugin-msgp...
file-alternative (1/13)out_file time [TAB] tag [TAB] {"key1":"value1", "key2":"value2",... }out_file_alternative time [TAB] t...
hoop (2/13)                               HoopFluentd                               Server      HDFSoutput format compatib...
mysql (3/13)                alpha!Fluentd                                    MySQL          JSON into single column       ...
ikachan (4/13)       alpha!Fluentd                   ikachan    IRC                    Server                             ...
growthforecast (5/13)   growthforecast by @kazeburo
flowcounter (6/13)  messages               count_keys *flowcounter     unit minute{"count":300, "bytes":3660,"count_rate":5,...
datacounter (7/13)  messages               unit minutedatacounter    aggregate all               count_key referer        ...
alpha!numeric-monitor (8/13)  messages                  unit minute                  aggregate allnumeric_monitor   monito...
notifier (9/13)          alpha!10                        Notify msg12                        suppressed10     notifier     i...
sampling-filter (10/13)msg                            msgmsgmsg            samplingmsg                            msg      ...
amplifier-filter (11/13)10                            3012                            3610         amplifier           30    ...
beautiful real-time       monitoring worldStreaming data                    Sampling                                     C...
parser (12/13)                   alpha!{"log":"2012/05/18:16:21 DRINK1 1"}                   format /^(?<time>S+) (?<item>...
<match input.A>                   forest (13/13)  type foo  ....                          <match input.*>  conf_a xxxx.A.y...
NO PLUGINS,NO FLUENTD  Enjoy Fluentd!
One more thing
fluent-plugin-webhdfs   Coming Soon!
Thanks!
Upcoming SlideShare
Loading in...5
×

Plugins by tagomoris #fluentdcasual

8,468

Published on

Published in: Technology

Plugins by tagomoris #fluentdcasual

  1. 1. Pluginsby tagomoris Fluentd Casual Talks LT 2012/05/18@tagomoris (NHN Japan Corp.)
  2. 2. @tagomoris NHN Japan Corp.Web Service Business DivisionDevelopment Department 2 Development Platforms
  3. 3. Appendix diff of 2012/02 and now• Our Fluentd cluster status
  4. 4. Fluentd Cluster status updates (1) 89 Fluentd processes 12 nodes 89 Fluentd processes 12 nodes
  5. 5. Fluentd Cluster status updates (2) ruby 1.9.2 + glibc malloc scribeline ruby 1.9.3 + jemalloc fluent-agent-lite
  6. 6. Fluentd Cluster status updates (3) from 127 servers 146 log streams from 205 servers 246 log streams
  7. 7. Fluentd Cluster status updates (4) 70,000 msgs/sec 120Mbps (at peak time) 92,000 msgs/sec 184Mbps
  8. 8. Fluentd Cluster status updates (5) 650GB/day (non-compressed bytes on HDFS) 995GB/day
  9. 9. Fluentd Cluster status updates (6) Before: 10 pluginsin_forward, out_forward, in_scribe, out_scribe out_copy, out_roundrobin, out_exec_filter out_hoop out_flowcounter, out_growthforecast After: +9 plugins out_route, in_gc_stat, in_object_space out_mongo, out_sampling_filter out_amplifier_filter, out_datacounter out_file_alternative, out_forest
  10. 10. Plugins!All your plugin are belong to us.
  11. 11. gem search -r fluent-plugin 41 pluginsfluent-plugin-amplifier-filter (0.1.1) fluent-plugin-msgpack-rpc (0.2.1) fluent-plugin-amqp (0.0.0) fluent-plugin-mysql (0.0.2) fluent-plugin-cassandra (0.0.2) fluent-plugin-notifier (0.0.1) fluent-plugin-couch (0.5.0) fluent-plugin-numeric-monitor (0.0.1)fluent-plugin-datacalculator (0.0.1) fluent-plugin-parser (0.1.0) fluent-plugin-datacounter (0.3.0) fluent-plugin-pghstore (0.1.2) fluent-plugin-delayed (0.0.1) fluent-plugin-redis (0.2.0) fluent-plugin-dstat (0.1.0) fluent-plugin-resque (0.2.1)fluent-plugin-file-alternative (0.1.2) fluent-plugin-s3 (0.2.2) fluent-plugin-flowcounter (0.1.4) fluent-plugin-sampling-filter (0.1.1) fluent-plugin-flume (0.1.0) fluent-plugin-say (0.1.1) fluent-plugin-forest (0.1.0) fluent-plugin-scribe (0.10.8) fluent-plugin-growl (0.0.5) fluent-plugin-sns (0.2.1)fluent-plugin-growthforecast (0.1.2) fluent-plugin-solr (0.1.1) fluent-plugin-hipchat (0.1.0) fluent-plugin-splunk (0.0.1.1) fluent-plugin-hoop (0.1.2) fluent-plugin-sqs (0.2.1)fluent-plugin-http-enhanced (0.0.3) fluent-plugin-td (0.10.6) fluent-plugin-ikachan (0.1.0) fluent-plugin-udp (0.0.1) fluent-plugin-kestrel (0.1.0) fluent-plugin-zabbix (0.0.1)fluent-plugin-librato-metrics (0.2.3) fluent-plugin-zmq (0.0.3) fluent-plugin-mongo (0.6.7)
  12. 12. gem search -r fluent-plugin 13/41 pluginsfluent-plugin-amplifier-filter (0.1.1) fluent-plugin-msgpack-rpc (0.2.1) fluent-plugin-amqp (0.0.0) fluent-plugin-mysql (0.0.2) fluent-plugin-cassandra (0.0.2) fluent-plugin-notifier (0.0.1) fluent-plugin-couch (0.5.0) fluent-plugin-numeric-monitor (0.0.1) fluent-plugin-datacalculator (0.0.1) fluent-plugin-parser (0.1.0) fluent-plugin-datacounter (0.3.0) fluent-plugin-pghstore (0.1.2) fluent-plugin-delayed (0.0.1) fluent-plugin-redis (0.2.0) fluent-plugin-dstat (0.1.0) fluent-plugin-resque (0.2.1)fluent-plugin-file-alternative (0.1.2) fluent-plugin-s3 (0.2.2) fluent-plugin-flowcounter (0.1.4) fluent-plugin-sampling-filter (0.1.1) fluent-plugin-flume (0.1.0) fluent-plugin-say (0.1.1) fluent-plugin-forest (0.1.0) fluent-plugin-scribe (0.10.8) fluent-plugin-growl (0.0.5) fluent-plugin-sns (0.2.1)fluent-plugin-growthforecast (0.1.2) fluent-plugin-solr (0.1.1) fluent-plugin-hipchat (0.1.0) fluent-plugin-splunk (0.0.1.1) fluent-plugin-hoop (0.1.2) fluent-plugin-sqs (0.2.1)fluent-plugin-http-enhanced (0.0.3) fluent-plugin-td (0.10.6) fluent-plugin-ikachan (0.1.0) fluent-plugin-udp (0.0.1) fluent-plugin-kestrel (0.1.0) fluent-plugin-zabbix (0.0.1)fluent-plugin-librato-metrics (0.2.3) fluent-plugin-zmq (0.0.3) fluent-plugin-mongo (0.6.7)
  13. 13. file-alternative (1/13)out_file time [TAB] tag [TAB] {"key1":"value1", "key2":"value2",... }out_file_alternative time [TAB] tag [TAB] {"key1":"value1", "key2":"value2",... } time [TAB] {"key1":"value1", "key2":"value2",... } tag [TAB] {"key1":"value1", "key2":"value2",... } time [TAB] tag [TAB] value1 [TAB] value2 ... time,tag,value1,value2 ...
  14. 14. hoop (2/13) HoopFluentd Server HDFSoutput format compatible with out_file_alternative
  15. 15. mysql (3/13) alpha!Fluentd MySQL JSON into single column or values into columns
  16. 16. ikachan (4/13) alpha!Fluentd ikachan IRC Server Server
  17. 17. growthforecast (5/13) growthforecast by @kazeburo
  18. 18. flowcounter (6/13) messages count_keys *flowcounter unit minute{"count":300, "bytes":3660,"count_rate":5, "bytes_rate":61}
  19. 19. datacounter (7/13) messages unit minutedatacounter aggregate all count_key referer pattern1 google google.com{"google_count":120, "google_rate":2, "google_percentage":20, "unmatched_count":480, ...}
  20. 20. alpha!numeric-monitor (8/13) messages unit minute aggregate allnumeric_monitor monitor_key duration percentiles 90,95{"min":100131,"max":483120, "avg":181343, "percentile_90":283201, "percentile_95":341210}
  21. 21. notifier (9/13) alpha!10 Notify msg12 suppressed10 notifier in specified interval... warn 10 crit 301513 Notify msg
  22. 22. sampling-filter (10/13)msg msgmsgmsg samplingmsg msg filtermsg interval 3msgmsg msg
  23. 23. amplifier-filter (11/13)10 3012 3610 amplifier 30 filter15 ratio 3 4513 39
  24. 24. beautiful real-time monitoring worldStreaming data Sampling Counting Amplifying Generatig Notification Output (Graph, IRC, ...)
  25. 25. parser (12/13) alpha!{"log":"2012/05/18:16:21 DRINK1 1"} format /^(?<time>S+) (?<item>S+) (?<num>d+)$/ numeric_monitor time_format %Y/%m/%d:%H:%M:%S key_name log reserve_data yeslog_based_time message{"log":"2012/05/18:16:21 DRINK1 1", "item":"DRINK1", "num":"1"}
  26. 26. <match input.A> forest (13/13) type foo .... <match input.*> conf_a xxxx.A.yyyy type forest option_x 30 subtype foo .... remove_prefix input</match> <template><match input.B> .... type foo conf_a xxxx.__TAG__.yyyy .... .... conf_a xxxx.B.yyyy </template> option_x 30 <case {C,D}> .... option_x 100</match> </case><match input.C> <case *> type foo option_x 30 .... </case> conf_a xxxx.C.yyyy </match> option_x 100 ....</match>
  27. 27. NO PLUGINS,NO FLUENTD Enjoy Fluentd!
  28. 28. One more thing
  29. 29. fluent-plugin-webhdfs Coming Soon!
  30. 30. Thanks!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×