Fluentd version 11 includes several new features and improvements including non-stop restart capability, support for multiprocess architecture with separate worker processes, dedicated error stream handling, improved plugin version management, ability to set log levels for individual plugins, use of variables in configuration files, and streaming processing without needing to modify record tags.

1. What s new in v11
Sadayuki Furuhashi
fluentd.org

2. Non-stop restart
v10 (old):
supervisor process
listen TCP socket
worker process
heartbeat

3. Non-stop restart
v10 (old):
supervisor process
Downtime!
heartbeat
worker process
restart
initializing...

4. Non-stop restart
v11:
supervisor process
listen TCP socket
worker process
heartbeat

5. Non-stop restart
Zero downtime!
v11:
supervisor process
listen TCP socket
heartbeat
worker process
restart
initializing...

6. Multiprocess
supervisor process
heartbeat
worker process
<worker>
...
</worker>
<worker>
...
</worker>
<worker>
...
</worker>

7. Multiprocess
worker process
<worker>
<worker>
...
</worker>
<worker>
･ in_tail
･ in_forward
･ out_groupcounter
･ out_VERY_UNSTABLE
･ out_webhdfs
</worker>
</worker>
Separate important
logging from
experimental plugins

8. Error stream
v10 (old):
<source>
<match **>
{“event”: 1, ...}
chunk
event
OK
{“event”: 2, ...}
ERROR!
{“event”: 3, ...}
?
{“event”: 4, ...}
chunk
ERROR!
{“event”: 5, ...}
?
{“event”: 6, ...}
?
Error handling
depends on plugin
(throw away! etc.)

9. Error stream
v10 (old):
<source>
特殊ラベル
<match **>
{“event”: 1, ...}
event
chunk
OK
{“event”: 2, ...}
ERROR!
{“event”: 3, ...}
OK
{“event”: 4, ...}
chunk
ERROR!
{“event”: 5, ...}
OK
{“event”: 6, ...}
OK
<label @ERROR>
<match **>
...
OK
OK
Fluentd core
handles error
events

10. Plugin version management
v11:
$ cat /etc/td-agent/Gemfile
gem “fluent-plugin-s3”, “~> 0.11.3”
gem “fluent-plugin-webhdfs”, “~> 0.2.2”
gem “fluentd”, “~> 0.11.2”
$

11. Log level per plugin
v11:
<source>
<match **>
type forward
type mongo
log_level error
log_level debug
</source>
</match>

12. Variables in a config file
v11:
<source>
type forward
port “#{ENV[‘FLUENTD_LISTEN_PORT’]}”
</source>

13. Variables in a config file
v11?：
<server>
@set port 9800
@include “/etc/fluentd/server.conf”
</server>
<server>
@set port 9801
@include “/etc/fluentd/server.conf”
</server>

14. Streaming processing without tag hacking
v10 (old):
<match raw.**>
type flowcounter
add_tag_prefix counted
</match>
<match counted.**>
type growthforecast
</match>
v11:
<match **>
type flowcounter
<match **>
type growthforecast
</match>
</match>

15. v11
Reliability &
Non-stop restart
Performance
Multiprocess
Error stream
Management
Plugin version management
Log level per plugins
Configuration
Variables in a config file
Streaming processing
without tag hacking