What s new in v11

Sadayuki Furuhashi
fluentd.org
Non-stop restart
v10 (old):

supervisor process
listen TCP socket

worker process

heartbeat
Non-stop restart
v10 (old):

supervisor process
Downtime!

heartbeat

worker process
restart
initializing...
Non-stop restart
v11:

supervisor process
listen TCP socket

worker process

heartbeat
Non-stop restart

Zero downtime!

v11:

supervisor process
listen TCP socket

heartbeat

worker process
restart
initializi...
Multiprocess

supervisor process
heartbeat

worker process

<worker>
...
</worker>

<worker>
...
</worker>

<worker>
...
<...
Multiprocess
worker process

<worker>

<worker>
...
</worker>

<worker>

・ in_tail

・ in_forward

・ out_groupcounter

・ ou...
Error stream
v10 (old):
<source>

<match **>

{“event”: 1, ...}
chunk
event

OK

{“event”: 2, ...}

ERROR!

{“event”: 3, ....
Error stream
v10 (old):
<source>

特殊ラベル
<match **>

{“event”: 1, ...}

event

chunk

OK

{“event”: 2, ...}

ERROR!

{“even...
Plugin version management
v11:

$ cat /etc/td-agent/Gemfile
gem “fluent-plugin-s3”, “~> 0.11.3”
gem “fluent-plugin-webhdfs...
Log level per plugin
v11:

<source>

<match **>

type forward

type mongo

log_level error

log_level debug

</source>

</...
Variables in a config file
v11:

<source>
type forward
port “#{ENV[‘FLUENTD_LISTEN_PORT’]}”
</source>
Variables in a config file
v11?:
<server>
@set port 9800
@include “/etc/fluentd/server.conf”
</server>
<server>
@set port 98...
Streaming processing without tag hacking
v10 (old):

<match raw.**>
type flowcounter
add_tag_prefix counted
</match>
<matc...
v11
Reliability &

Non-stop restart

Performance

Multiprocess
Error stream

Management

Plugin version management
Log lev...
Upcoming SlideShare
Loading in...5
×

What's new in v11 - Fluentd Casual Talks #3 #fluentdcasual

6,829

Published on

Published in: Technology, Business

What's new in v11 - Fluentd Casual Talks #3 #fluentdcasual

  1. 1. What s new in v11 Sadayuki Furuhashi fluentd.org
  2. 2. Non-stop restart v10 (old): supervisor process listen TCP socket worker process heartbeat
  3. 3. Non-stop restart v10 (old): supervisor process Downtime! heartbeat worker process restart initializing...
  4. 4. Non-stop restart v11: supervisor process listen TCP socket worker process heartbeat
  5. 5. Non-stop restart Zero downtime! v11: supervisor process listen TCP socket heartbeat worker process restart initializing...
  6. 6. Multiprocess supervisor process heartbeat worker process <worker> ... </worker> <worker> ... </worker> <worker> ... </worker>
  7. 7. Multiprocess worker process <worker> <worker> ... </worker> <worker> ・ in_tail ・ in_forward ・ out_groupcounter ・ out_VERY_UNSTABLE ・ out_webhdfs </worker> </worker> Separate important logging from experimental plugins
  8. 8. Error stream v10 (old): <source> <match **> {“event”: 1, ...} chunk event OK {“event”: 2, ...} ERROR! {“event”: 3, ...} ? {“event”: 4, ...} chunk ERROR! {“event”: 5, ...} ? {“event”: 6, ...} ? Error handling depends on plugin (throw away! etc.)
  9. 9. Error stream v10 (old): <source> 特殊ラベル <match **> {“event”: 1, ...} event chunk OK {“event”: 2, ...} ERROR! {“event”: 3, ...} OK {“event”: 4, ...} chunk ERROR! {“event”: 5, ...} OK {“event”: 6, ...} OK <label @ERROR> <match **> ... OK OK Fluentd core handles error events
  10. 10. Plugin version management v11: $ cat /etc/td-agent/Gemfile gem “fluent-plugin-s3”, “~> 0.11.3” gem “fluent-plugin-webhdfs”, “~> 0.2.2” gem “fluentd”, “~> 0.11.2” $
  11. 11. Log level per plugin v11: <source> <match **> type forward type mongo log_level error log_level debug </source> </match>
  12. 12. Variables in a config file v11: <source> type forward port “#{ENV[‘FLUENTD_LISTEN_PORT’]}” </source>
  13. 13. Variables in a config file v11?: <server> @set port 9800 @include “/etc/fluentd/server.conf” </server> <server> @set port 9801 @include “/etc/fluentd/server.conf” </server>
  14. 14. Streaming processing without tag hacking v10 (old): <match raw.**> type flowcounter add_tag_prefix counted </match> <match counted.**> type growthforecast </match> v11: <match **> type flowcounter <match **> type growthforecast </match> </match>
  15. 15. v11 Reliability & Non-stop restart Performance Multiprocess Error stream Management Plugin version management Log level per plugins Configuration Variables in a config file Streaming processing without tag hacking
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×