What's new in v11 - Fluentd Casual Talks #3 #fluentdcasual

8,221 views

Published on

Published in: Technology, Business

What's new in v11 - Fluentd Casual Talks #3 #fluentdcasual

  1. 1. What s new in v11 Sadayuki Furuhashi fluentd.org
  2. 2. Non-stop restart v10 (old): supervisor process listen TCP socket worker process heartbeat
  3. 3. Non-stop restart v10 (old): supervisor process Downtime! heartbeat worker process restart initializing...
  4. 4. Non-stop restart v11: supervisor process listen TCP socket worker process heartbeat
  5. 5. Non-stop restart Zero downtime! v11: supervisor process listen TCP socket heartbeat worker process restart initializing...
  6. 6. Multiprocess supervisor process heartbeat worker process <worker> ... </worker> <worker> ... </worker> <worker> ... </worker>
  7. 7. Multiprocess worker process <worker> <worker> ... </worker> <worker> ・ in_tail ・ in_forward ・ out_groupcounter ・ out_VERY_UNSTABLE ・ out_webhdfs </worker> </worker> Separate important logging from experimental plugins
  8. 8. Error stream v10 (old): <source> <match **> {“event”: 1, ...} chunk event OK {“event”: 2, ...} ERROR! {“event”: 3, ...} ? {“event”: 4, ...} chunk ERROR! {“event”: 5, ...} ? {“event”: 6, ...} ? Error handling depends on plugin (throw away! etc.)
  9. 9. Error stream v10 (old): <source> 特殊ラベル <match **> {“event”: 1, ...} event chunk OK {“event”: 2, ...} ERROR! {“event”: 3, ...} OK {“event”: 4, ...} chunk ERROR! {“event”: 5, ...} OK {“event”: 6, ...} OK <label @ERROR> <match **> ... OK OK Fluentd core handles error events
  10. 10. Plugin version management v11: $ cat /etc/td-agent/Gemfile gem “fluent-plugin-s3”, “~> 0.11.3” gem “fluent-plugin-webhdfs”, “~> 0.2.2” gem “fluentd”, “~> 0.11.2” $
  11. 11. Log level per plugin v11: <source> <match **> type forward type mongo log_level error log_level debug </source> </match>
  12. 12. Variables in a config file v11: <source> type forward port “#{ENV[‘FLUENTD_LISTEN_PORT’]}” </source>
  13. 13. Variables in a config file v11?: <server> @set port 9800 @include “/etc/fluentd/server.conf” </server> <server> @set port 9801 @include “/etc/fluentd/server.conf” </server>
  14. 14. Streaming processing without tag hacking v10 (old): <match raw.**> type flowcounter add_tag_prefix counted </match> <match counted.**> type growthforecast </match> v11: <match **> type flowcounter <match **> type growthforecast </match> </match>
  15. 15. v11 Reliability & Non-stop restart Performance Multiprocess Error stream Management Plugin version management Log level per plugins Configuration Variables in a config file Streaming processing without tag hacking

×