Plugins by tagomoris #fluentdcasual

Pluginsby tagomoris Fluentd Casual Talks LT 2012/05/18@tagomoris (NHN Japan Corp.)

@tagomoris NHN Japan Corp.Web Service Business DivisionDevelopment Department 2 Development Platforms

Appendix diff of 2012/02 and now• Our Fluentd cluster status

Fluentd Cluster status updates (1) 89 Fluentd processes 12 nodes 89 Fluentd processes 12 nodes

Fluentd Cluster status updates (2) ruby 1.9.2 + glibc malloc scribeline ruby 1.9.3 + jemalloc ﬂuent-agent-lite

Fluentd Cluster status updates (3) from 127 servers 146 log streams from 205 servers 246 log streams

Fluentd Cluster status updates (4) 70,000 msgs/sec 120Mbps (at peak time) 92,000 msgs/sec 184Mbps

Fluentd Cluster status updates (5) 650GB/day (non-compressed bytes on HDFS) 995GB/day

Fluentd Cluster status updates (6) Before: 10 pluginsin_forward, out_forward, in_scribe, out_scribe out_copy, out_roundrobin, out_exec_filter out_hoop out_flowcounter, out_growthforecast After: +9 plugins out_route, in_gc_stat, in_object_space out_mongo, out_sampling_filter out_amplifier_filter, out_datacounter out_file_alternative, out_forest

Plugins!All your plugin are belong to us.

gem search -r fluent-plugin 41 pluginsfluent-plugin-amplifier-filter (0.1.1) fluent-plugin-msgpack-rpc (0.2.1) fluent-plugin-amqp (0.0.0) fluent-plugin-mysql (0.0.2) fluent-plugin-cassandra (0.0.2) fluent-plugin-notifier (0.0.1) fluent-plugin-couch (0.5.0) fluent-plugin-numeric-monitor (0.0.1)fluent-plugin-datacalculator (0.0.1) fluent-plugin-parser (0.1.0) fluent-plugin-datacounter (0.3.0) fluent-plugin-pghstore (0.1.2) fluent-plugin-delayed (0.0.1) fluent-plugin-redis (0.2.0) fluent-plugin-dstat (0.1.0) fluent-plugin-resque (0.2.1)fluent-plugin-file-alternative (0.1.2) fluent-plugin-s3 (0.2.2) fluent-plugin-flowcounter (0.1.4) fluent-plugin-sampling-filter (0.1.1) fluent-plugin-flume (0.1.0) fluent-plugin-say (0.1.1) fluent-plugin-forest (0.1.0) fluent-plugin-scribe (0.10.8) fluent-plugin-growl (0.0.5) fluent-plugin-sns (0.2.1)fluent-plugin-growthforecast (0.1.2) fluent-plugin-solr (0.1.1) fluent-plugin-hipchat (0.1.0) fluent-plugin-splunk (0.0.1.1) fluent-plugin-hoop (0.1.2) fluent-plugin-sqs (0.2.1)fluent-plugin-http-enhanced (0.0.3) fluent-plugin-td (0.10.6) fluent-plugin-ikachan (0.1.0) fluent-plugin-udp (0.0.1) fluent-plugin-kestrel (0.1.0) fluent-plugin-zabbix (0.0.1)fluent-plugin-librato-metrics (0.2.3) fluent-plugin-zmq (0.0.3) fluent-plugin-mongo (0.6.7)

gem search -r fluent-plugin 13/41 pluginsfluent-plugin-amplifier-filter (0.1.1) fluent-plugin-msgpack-rpc (0.2.1) fluent-plugin-amqp (0.0.0) fluent-plugin-mysql (0.0.2) fluent-plugin-cassandra (0.0.2) fluent-plugin-notifier (0.0.1) fluent-plugin-couch (0.5.0) fluent-plugin-numeric-monitor (0.0.1) fluent-plugin-datacalculator (0.0.1) fluent-plugin-parser (0.1.0) fluent-plugin-datacounter (0.3.0) fluent-plugin-pghstore (0.1.2) fluent-plugin-delayed (0.0.1) fluent-plugin-redis (0.2.0) fluent-plugin-dstat (0.1.0) fluent-plugin-resque (0.2.1)fluent-plugin-file-alternative (0.1.2) fluent-plugin-s3 (0.2.2) fluent-plugin-flowcounter (0.1.4) fluent-plugin-sampling-filter (0.1.1) fluent-plugin-flume (0.1.0) fluent-plugin-say (0.1.1) fluent-plugin-forest (0.1.0) fluent-plugin-scribe (0.10.8) fluent-plugin-growl (0.0.5) fluent-plugin-sns (0.2.1)fluent-plugin-growthforecast (0.1.2) fluent-plugin-solr (0.1.1) fluent-plugin-hipchat (0.1.0) fluent-plugin-splunk (0.0.1.1) fluent-plugin-hoop (0.1.2) fluent-plugin-sqs (0.2.1)fluent-plugin-http-enhanced (0.0.3) fluent-plugin-td (0.10.6) fluent-plugin-ikachan (0.1.0) fluent-plugin-udp (0.0.1) fluent-plugin-kestrel (0.1.0) fluent-plugin-zabbix (0.0.1)fluent-plugin-librato-metrics (0.2.3) fluent-plugin-zmq (0.0.3) fluent-plugin-mongo (0.6.7)

file-alternative (1/13)out_file time [TAB] tag [TAB] {"key1":"value1", "key2":"value2",... }out_file_alternative time [TAB] tag [TAB] {"key1":"value1", "key2":"value2",... } time [TAB] {"key1":"value1", "key2":"value2",... } tag [TAB] {"key1":"value1", "key2":"value2",... } time [TAB] tag [TAB] value1 [TAB] value2 ... time,tag,value1,value2 ...

hoop (2/13) HoopFluentd Server HDFSoutput format compatible with out_ﬁle_alternative

mysql (3/13) alpha!Fluentd MySQL JSON into single column or values into columns

ikachan (4/13) alpha!Fluentd ikachan IRC Server Server

growthforecast (5/13) growthforecast by @kazeburo

ﬂowcounter (6/13) messages count_keys *ﬂowcounter unit minute{"count":300, "bytes":3660,"count_rate":5, "bytes_rate":61}

datacounter (7/13) messages unit minutedatacounter aggregate all count_key referer pattern1 google google.com{"google_count":120, "google_rate":2, "google_percentage":20, "unmatched_count":480, ...}

alpha!numeric-monitor (8/13) messages unit minute aggregate allnumeric_monitor monitor_key duration percentiles 90,95{"min":100131,"max":483120, "avg":181343, "percentile_90":283201, "percentile_95":341210}

notifier (9/13) alpha!10 Notify msg12 suppressed10 notifier in specified interval... warn 10 crit 301513 Notify msg

sampling-ﬁlter (10/13)msg msgmsgmsg samplingmsg msg ﬁltermsg interval 3msgmsg msg

amplifier-filter (11/13)10 3012 3610 amplifier 30 filter15 ratio 3 4513 39

beautiful real-time monitoring worldStreaming data Sampling Counting Amplifying Generatig Notiﬁcation Output (Graph, IRC, ...)

parser (12/13) alpha!{"log":"2012/05/18:16:21 DRINK1 1"} format /^(?<time>S+) (?<item>S+) (?<num>d+)$/ numeric_monitor time_format %Y/%m/%d:%H:%M:%S key_name log reserve_data yeslog_based_time message{"log":"2012/05/18:16:21 DRINK1 1", "item":"DRINK1", "num":"1"}

<match input.A> forest (13/13) type foo .... <match input.*> conf_a xxxx.A.yyyy type forest option_x 30 subtype foo .... remove_prefix input</match> <template><match input.B> .... type foo conf_a xxxx.__TAG__.yyyy .... .... conf_a xxxx.B.yyyy </template> option_x 30 <case {C,D}> .... option_x 100</match> </case><match input.C> <case *> type foo option_x 30 .... </case> conf_a xxxx.C.yyyy </match> option_x 100 ....</match>

NO PLUGINS,NO FLUENTD Enjoy Fluentd!

One more thing

ﬂuent-plugin-webhdfs Coming Soon!

Thanks!

http://d.hatena.ne.jp	1781
http://okochang.hatenablog.jp	589
http://www.socialtoprunners.jp	387
http://springmt.github.com	75
http://hatenatunnel.appspot.com	30
http://www.socialtoprunners.com	24
http://webcache.googleusercontent.com	10
http://springmt.github.io	8
http://feedplanets.com	6
http://www35.jimdo.com	5
http://us-w1.rockmelt.com	4
http://socialtoprunners.jimdo.com	3
http://tweetedtimes.com	2
http://translate.googleusercontent.com	2
https://si0.twimg.com	1

Plugins by tagomoris #fluentdcasual

by SATOSHI TAGOMORI

Accessibility

Categories

Upload Details

Usage Rights

15 Embeds 2,927

Statistics