The document summarizes updates to Tagomoris' Fluentd cluster and new Fluentd plugins. It states that the cluster has grown from 89 processes on 12 nodes to 92,000 messages per second and 184Mbps, and the number of plugins has increased from 10 to 19, including new plugins for routing, monitoring, filtering, and outputting data. It promotes developing plugins to expand Fluentd's capabilities for real-time data processing.

1. Plugins
by tagomoris
Fluentd Casual Talks LT
2012/05/18
@tagomoris (NHN Japan Corp.)

2. @tagomoris
NHN Japan Corp.
Web Service Business Division
Development Department 2
Development Platforms

3. Appendix
diff of 2012/02 and now
• Our Fluentd cluster status

4. Fluentd Cluster status updates (1)
89 Fluentd processes
12 nodes
89 Fluentd processes
12 nodes

5. Fluentd Cluster status updates (2)
ruby 1.9.2 + glibc malloc
scribeline
ruby 1.9.3 + jemalloc
fluent-agent-lite

6. Fluentd Cluster status updates (3)
from 127 servers
146 log streams
from 205 servers
246 log streams

7. Fluentd Cluster status updates (4)
70,000 msgs/sec
120Mbps
(at peak time)
92,000 msgs/sec
184Mbps

8. Fluentd Cluster status updates (5)
650GB/day
(non-compressed bytes on HDFS)
995GB/day

9. Fluentd Cluster status updates (6)
Before: 10 plugins
in_forward, out_forward, in_scribe, out_scribe
out_copy, out_roundrobin, out_exec_filter
out_hoop
out_flowcounter, out_growthforecast
After: +9 plugins
out_route, in_gc_stat, in_object_space
out_mongo, out_sampling_filter
out_amplifier_filter, out_datacounter
out_file_alternative, out_forest

10. Plugins!
All your plugin are belong to us.

11. gem search -r fluent-plugin
41 plugins
fluent-plugin-amplifier-filter (0.1.1) fluent-plugin-msgpack-rpc (0.2.1)
fluent-plugin-amqp (0.0.0) fluent-plugin-mysql (0.0.2)
fluent-plugin-cassandra (0.0.2) fluent-plugin-notifier (0.0.1)
fluent-plugin-couch (0.5.0) fluent-plugin-numeric-monitor (0.0.1)
fluent-plugin-datacalculator (0.0.1) fluent-plugin-parser (0.1.0)
fluent-plugin-datacounter (0.3.0) fluent-plugin-pghstore (0.1.2)
fluent-plugin-delayed (0.0.1) fluent-plugin-redis (0.2.0)
fluent-plugin-dstat (0.1.0) fluent-plugin-resque (0.2.1)
fluent-plugin-file-alternative (0.1.2) fluent-plugin-s3 (0.2.2)
fluent-plugin-flowcounter (0.1.4) fluent-plugin-sampling-filter (0.1.1)
fluent-plugin-flume (0.1.0) fluent-plugin-say (0.1.1)
fluent-plugin-forest (0.1.0) fluent-plugin-scribe (0.10.8)
fluent-plugin-growl (0.0.5) fluent-plugin-sns (0.2.1)
fluent-plugin-growthforecast (0.1.2) fluent-plugin-solr (0.1.1)
fluent-plugin-hipchat (0.1.0) fluent-plugin-splunk (0.0.1.1)
fluent-plugin-hoop (0.1.2) fluent-plugin-sqs (0.2.1)
fluent-plugin-http-enhanced (0.0.3) fluent-plugin-td (0.10.6)
fluent-plugin-ikachan (0.1.0) fluent-plugin-udp (0.0.1)
fluent-plugin-kestrel (0.1.0) fluent-plugin-zabbix (0.0.1)
fluent-plugin-librato-metrics (0.2.3) fluent-plugin-zmq (0.0.3)
fluent-plugin-mongo (0.6.7)

12. gem search -r fluent-plugin
13/41 plugins
fluent-plugin-amplifier-filter (0.1.1) fluent-plugin-msgpack-rpc (0.2.1)
fluent-plugin-amqp (0.0.0) fluent-plugin-mysql (0.0.2)
fluent-plugin-cassandra (0.0.2) fluent-plugin-notifier (0.0.1)
fluent-plugin-couch (0.5.0) fluent-plugin-numeric-monitor (0.0.1)
fluent-plugin-datacalculator (0.0.1) fluent-plugin-parser (0.1.0)
fluent-plugin-datacounter (0.3.0) fluent-plugin-pghstore (0.1.2)
fluent-plugin-delayed (0.0.1) fluent-plugin-redis (0.2.0)
fluent-plugin-dstat (0.1.0) fluent-plugin-resque (0.2.1)
fluent-plugin-file-alternative (0.1.2) fluent-plugin-s3 (0.2.2)
fluent-plugin-flowcounter (0.1.4) fluent-plugin-sampling-filter (0.1.1)
fluent-plugin-flume (0.1.0) fluent-plugin-say (0.1.1)
fluent-plugin-forest (0.1.0) fluent-plugin-scribe (0.10.8)
fluent-plugin-growl (0.0.5) fluent-plugin-sns (0.2.1)
fluent-plugin-growthforecast (0.1.2) fluent-plugin-solr (0.1.1)
fluent-plugin-hipchat (0.1.0) fluent-plugin-splunk (0.0.1.1)
fluent-plugin-hoop (0.1.2) fluent-plugin-sqs (0.2.1)
fluent-plugin-http-enhanced (0.0.3) fluent-plugin-td (0.10.6)
fluent-plugin-ikachan (0.1.0) fluent-plugin-udp (0.0.1)
fluent-plugin-kestrel (0.1.0) fluent-plugin-zabbix (0.0.1)
fluent-plugin-librato-metrics (0.2.3) fluent-plugin-zmq (0.0.3)
fluent-plugin-mongo (0.6.7)

13. file-alternative (1/13)
out_file
time [TAB] tag [TAB] {"key1":"value1", "key2":"value2",... }
out_file_alternative
time [TAB] tag [TAB] {"key1":"value1", "key2":"value2",... }
time [TAB] {"key1":"value1", "key2":"value2",... }
tag [TAB] {"key1":"value1", "key2":"value2",... }
time [TAB] tag [TAB] value1 [TAB] value2 ...
time,tag,value1,value2 ...

14. hoop (2/13)
Hoop
Fluentd
Server HDFS
output format compatible with out_file_alternative

15. mysql (3/13) alpha!
Fluentd
MySQL
JSON into single column
or
values into columns

16. ikachan (4/13) alpha!
Fluentd
ikachan IRC
Server
Server

17. growthforecast (5/13)
growthforecast by @kazeburo

18. flowcounter (6/13)
messages
count_keys *
flowcounter unit minute
{"count":300, "bytes":3660,
"count_rate":5, "bytes_rate":61}

19. datacounter (7/13)
messages
unit minute
datacounter aggregate all
count_key referer
pattern1 google google.com
{"google_count":120,
"google_rate":2,
"google_percentage":20,
"unmatched_count":480, ...}

20. alpha!
numeric-monitor (8/13)
messages
unit minute
aggregate all
numeric_monitor monitor_key duration
percentiles 90,95
{"min":100131,"max":483120,
"avg":181343,
"percentile_90":283201,
"percentile_95":341210}

21. notifier (9/13) alpha!
10 Notify msg
12
suppressed
10 notifier in specified interval...
warn 10
crit 30
15
13 Notify msg

22. sampling-filter (10/13)
msg msg
msg
msg
sampling
msg msg
filter
msg interval 3
msg
msg msg

23. amplifier-filter (11/13)
10 30
12 36
10 amplifier 30
filter
15 ratio 3 45
13 39

24. beautiful real-time
monitoring world
Streaming data
Sampling
Counting
Amplifying
Generatig
Notification
Output
(Graph, IRC, ...)

25. parser (12/13) alpha!
{"log":"2012/05/18:16:21 DRINK1 1"}
format /^(?<time>S+) (?<item>S+)
(?<num>d+)$/
numeric_monitor time_format %Y/%m/%d:%H:%M:%S
key_name log
reserve_data yes
log_based_time message
{"log":"2012/05/18:16:21 DRINK1 1",
"item":"DRINK1", "num":"1"}

26. <match input.A>
forest (13/13)
type foo
....
<match input.*>
conf_a xxxx.A.yyyy
type forest
option_x 30
subtype foo
....
remove_prefix input
</match>
<template>
<match input.B>
....
type foo
conf_a xxxx.__TAG__.yyyy
....
....
conf_a xxxx.B.yyyy
</template>
option_x 30
<case {C,D}>
....
option_x 100
</match>
</case>
<match input.C>
<case *>
type foo
option_x 30
....
</case>
conf_a xxxx.C.yyyy
</match>
option_x 100
....
</match>

27. NO PLUGINS,
NO FLUENTD
Enjoy Fluentd!

28. One more thing

29. fluent-plugin-webhdfs
Coming Soon!

30. Thanks!