• Like
Plugins by tagomoris #fluentdcasual
Upcoming SlideShare
Loading in...5
×
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
8,122
On Slideshare
0
From Embeds
0
Number of Embeds
8

Actions

Shares
Downloads
20
Comments
0
Likes
11

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Pluginsby tagomoris Fluentd Casual Talks LT 2012/05/18@tagomoris (NHN Japan Corp.)
  • 2. @tagomoris NHN Japan Corp.Web Service Business DivisionDevelopment Department 2 Development Platforms
  • 3. Appendix diff of 2012/02 and now• Our Fluentd cluster status
  • 4. Fluentd Cluster status updates (1) 89 Fluentd processes 12 nodes 89 Fluentd processes 12 nodes
  • 5. Fluentd Cluster status updates (2) ruby 1.9.2 + glibc malloc scribeline ruby 1.9.3 + jemalloc fluent-agent-lite
  • 6. Fluentd Cluster status updates (3) from 127 servers 146 log streams from 205 servers 246 log streams
  • 7. Fluentd Cluster status updates (4) 70,000 msgs/sec 120Mbps (at peak time) 92,000 msgs/sec 184Mbps
  • 8. Fluentd Cluster status updates (5) 650GB/day (non-compressed bytes on HDFS) 995GB/day
  • 9. Fluentd Cluster status updates (6) Before: 10 pluginsin_forward, out_forward, in_scribe, out_scribe out_copy, out_roundrobin, out_exec_filter out_hoop out_flowcounter, out_growthforecast After: +9 plugins out_route, in_gc_stat, in_object_space out_mongo, out_sampling_filter out_amplifier_filter, out_datacounter out_file_alternative, out_forest
  • 10. Plugins!All your plugin are belong to us.
  • 11. gem search -r fluent-plugin 41 pluginsfluent-plugin-amplifier-filter (0.1.1) fluent-plugin-msgpack-rpc (0.2.1) fluent-plugin-amqp (0.0.0) fluent-plugin-mysql (0.0.2) fluent-plugin-cassandra (0.0.2) fluent-plugin-notifier (0.0.1) fluent-plugin-couch (0.5.0) fluent-plugin-numeric-monitor (0.0.1)fluent-plugin-datacalculator (0.0.1) fluent-plugin-parser (0.1.0) fluent-plugin-datacounter (0.3.0) fluent-plugin-pghstore (0.1.2) fluent-plugin-delayed (0.0.1) fluent-plugin-redis (0.2.0) fluent-plugin-dstat (0.1.0) fluent-plugin-resque (0.2.1)fluent-plugin-file-alternative (0.1.2) fluent-plugin-s3 (0.2.2) fluent-plugin-flowcounter (0.1.4) fluent-plugin-sampling-filter (0.1.1) fluent-plugin-flume (0.1.0) fluent-plugin-say (0.1.1) fluent-plugin-forest (0.1.0) fluent-plugin-scribe (0.10.8) fluent-plugin-growl (0.0.5) fluent-plugin-sns (0.2.1)fluent-plugin-growthforecast (0.1.2) fluent-plugin-solr (0.1.1) fluent-plugin-hipchat (0.1.0) fluent-plugin-splunk (0.0.1.1) fluent-plugin-hoop (0.1.2) fluent-plugin-sqs (0.2.1)fluent-plugin-http-enhanced (0.0.3) fluent-plugin-td (0.10.6) fluent-plugin-ikachan (0.1.0) fluent-plugin-udp (0.0.1) fluent-plugin-kestrel (0.1.0) fluent-plugin-zabbix (0.0.1)fluent-plugin-librato-metrics (0.2.3) fluent-plugin-zmq (0.0.3) fluent-plugin-mongo (0.6.7)
  • 12. gem search -r fluent-plugin 13/41 pluginsfluent-plugin-amplifier-filter (0.1.1) fluent-plugin-msgpack-rpc (0.2.1) fluent-plugin-amqp (0.0.0) fluent-plugin-mysql (0.0.2) fluent-plugin-cassandra (0.0.2) fluent-plugin-notifier (0.0.1) fluent-plugin-couch (0.5.0) fluent-plugin-numeric-monitor (0.0.1) fluent-plugin-datacalculator (0.0.1) fluent-plugin-parser (0.1.0) fluent-plugin-datacounter (0.3.0) fluent-plugin-pghstore (0.1.2) fluent-plugin-delayed (0.0.1) fluent-plugin-redis (0.2.0) fluent-plugin-dstat (0.1.0) fluent-plugin-resque (0.2.1)fluent-plugin-file-alternative (0.1.2) fluent-plugin-s3 (0.2.2) fluent-plugin-flowcounter (0.1.4) fluent-plugin-sampling-filter (0.1.1) fluent-plugin-flume (0.1.0) fluent-plugin-say (0.1.1) fluent-plugin-forest (0.1.0) fluent-plugin-scribe (0.10.8) fluent-plugin-growl (0.0.5) fluent-plugin-sns (0.2.1)fluent-plugin-growthforecast (0.1.2) fluent-plugin-solr (0.1.1) fluent-plugin-hipchat (0.1.0) fluent-plugin-splunk (0.0.1.1) fluent-plugin-hoop (0.1.2) fluent-plugin-sqs (0.2.1)fluent-plugin-http-enhanced (0.0.3) fluent-plugin-td (0.10.6) fluent-plugin-ikachan (0.1.0) fluent-plugin-udp (0.0.1) fluent-plugin-kestrel (0.1.0) fluent-plugin-zabbix (0.0.1)fluent-plugin-librato-metrics (0.2.3) fluent-plugin-zmq (0.0.3) fluent-plugin-mongo (0.6.7)
  • 13. file-alternative (1/13)out_file time [TAB] tag [TAB] {"key1":"value1", "key2":"value2",... }out_file_alternative time [TAB] tag [TAB] {"key1":"value1", "key2":"value2",... } time [TAB] {"key1":"value1", "key2":"value2",... } tag [TAB] {"key1":"value1", "key2":"value2",... } time [TAB] tag [TAB] value1 [TAB] value2 ... time,tag,value1,value2 ...
  • 14. hoop (2/13) HoopFluentd Server HDFSoutput format compatible with out_file_alternative
  • 15. mysql (3/13) alpha!Fluentd MySQL JSON into single column or values into columns
  • 16. ikachan (4/13) alpha!Fluentd ikachan IRC Server Server
  • 17. growthforecast (5/13) growthforecast by @kazeburo
  • 18. flowcounter (6/13) messages count_keys *flowcounter unit minute{"count":300, "bytes":3660,"count_rate":5, "bytes_rate":61}
  • 19. datacounter (7/13) messages unit minutedatacounter aggregate all count_key referer pattern1 google google.com{"google_count":120, "google_rate":2, "google_percentage":20, "unmatched_count":480, ...}
  • 20. alpha!numeric-monitor (8/13) messages unit minute aggregate allnumeric_monitor monitor_key duration percentiles 90,95{"min":100131,"max":483120, "avg":181343, "percentile_90":283201, "percentile_95":341210}
  • 21. notifier (9/13) alpha!10 Notify msg12 suppressed10 notifier in specified interval... warn 10 crit 301513 Notify msg
  • 22. sampling-filter (10/13)msg msgmsgmsg samplingmsg msg filtermsg interval 3msgmsg msg
  • 23. amplifier-filter (11/13)10 3012 3610 amplifier 30 filter15 ratio 3 4513 39
  • 24. beautiful real-time monitoring worldStreaming data Sampling Counting Amplifying Generatig Notification Output (Graph, IRC, ...)
  • 25. parser (12/13) alpha!{"log":"2012/05/18:16:21 DRINK1 1"} format /^(?<time>S+) (?<item>S+) (?<num>d+)$/ numeric_monitor time_format %Y/%m/%d:%H:%M:%S key_name log reserve_data yeslog_based_time message{"log":"2012/05/18:16:21 DRINK1 1", "item":"DRINK1", "num":"1"}
  • 26. <match input.A> forest (13/13) type foo .... <match input.*> conf_a xxxx.A.yyyy type forest option_x 30 subtype foo .... remove_prefix input</match> <template><match input.B> .... type foo conf_a xxxx.__TAG__.yyyy .... .... conf_a xxxx.B.yyyy </template> option_x 30 <case {C,D}> .... option_x 100</match> </case><match input.C> <case *> type foo option_x 30 .... </case> conf_a xxxx.C.yyyy </match> option_x 100 ....</match>
  • 27. NO PLUGINS,NO FLUENTD Enjoy Fluentd!
  • 28. One more thing
  • 29. fluent-plugin-webhdfs Coming Soon!
  • 30. Thanks!