The document summarizes the key features and changes in Fluentd v0.14, including new plugin APIs, plugin storage and helpers, time with nanosecond resolution, a ServerEngine-based supervisor for Windows support, and plans for symmetric multi-core processing, a counter API, and TLS/authentication in future versions. It also benchmarks some performance improvements and outlines the roadmap for Treasure Agent 3.0 based on Fluentd v0.14.

1. v0.14 Overview
May 30, 2016
Masahiro Nakagawa

2. • Released at May 31, 2016
• New Features
• New Plugin APIs, Plugin Helpers & Plugin Storage
• Time with Nanosecond resolution
• ServerEngine based Supervisor
• Windows support
v0.14.0

3. New Plugin APIs
• Input/Output plugin APIs w/ well-controlled lifecycle
• stop, shutdown, close, terminate
• New Buffer API for delayed commit of chunks
• parallel/async "commit" operation for chunks
• 100% Compatible w/ v0.12 plugins
• compatibility layer for traditional APIs
• it will be supported between v1.x versions

4. Router
buffer_chunk_limit
enqueue: exceed flush_interval
or buffer_chunk_limit
Key pattern:
- BufferedOutput
empty string or specified key
-ObjectBufferedOutput tag
-TimeSlicedOutput time slice
emit emit
Buffer
Queue
buffer_queue_limit
Output
OutputInput / Filter
Tag Time
Record Chunk
Chunk
Chunk Chunk
Chunk
key:foo
key:bar
key:baz
v0.12 buffer design

5. v0.14 buffer design

6. Plugin Storage & Helpers
• Plugin Storage: new plugin type for plugins
• provides key-value storage for plugins
• to persistent intermediate status of plugins
• built-in plugins (in plan): in-memory, local file
• pluggable: 3rd party plugin to store data to Redis?
• Plugin Helpers:
• collections of utility methods for plugins
• making threads, sockets, network servers, ...
• fully integrated with test drivers to run test codes after setup
phase of helpers (e.g., after created threads started)

7. v0.12 plugins
ParserInput Buffer Output FormatterFilter
“output-ish”“input-ish”

8. v0.14 plugins
ParserInput Buffer Output FormatterFilter
“output-ish”“input-ish”
Storage
Helper

9. Time with nanosecond
• For sub-second systems: Elasticsearch, InfluxData and etc
• Fluent::EventTime
• behaves as Integer (used as time in v0.12)
• has methods to get sub-second resolution
• be serialized into msgpack using Ext type
• Fluentd core can handle both of Integer and EventTime as
time
• compatible with older versions and software in eco-
system (e.g., fluent-logger, Docker logging driver)

10. ServerEngine based Supervisor
• Replacing supervisor process with ServerEngine
• it has SocketManager to share listening sockets
between 2 or more worker processes
• Replacing Fluentd's processing model from fork to
spawn
• to support Windows environment

11. Windows support
• Fluentd and core plugin work on Windows
• several companies have already used
v0.14.0.pre version on production
• We will send a patch to popular plugins if
it doesn’t work on Windows
• Use HTTP RPC instead of signals

12. v0.14.x - v1
• v0.14.x (some versions in 2016)
• Symmetric multi-core processing
• Counter API
• TLS/authentication/authorization support
(merging secure forward)
• https://github.com/fluent/fluentd/issues/1000
• v1 (4Q in 2016 or 1Q in 2017)
• Stable version for new APIs / features
• Fully compatible with v0.12
• exclude v0 config syntax and detach_process

13. Symmetric multi core processing
• 2 or more workers share a configuration file
• and share listening sockets via PluginHelper
• under a supervisor process (ServerEngine)
• Multi core scalability for huge traffic
• one input plugin for a tcp port, some filters and
one (or some) output plugin
• buffer paths are managed automatically by
Fluentd core

14. Worker
Supervisor
Worker Worker
Worker
Supervisor
Worker Worker
Supervisor Supervisor
Using fluent-plugin-multiprocess
v0.14

15. Counter API
• APIs to increment/decrement values
• shared by some processes
• persisted on disk backed by Storage API
• Useful for collecting metrics or stats filters

16. TLS/Authn/Authz support for forward plugin
• secure-forward will be merged into built-in forward
• TLS w/ at-least-one semantics
• Simple authentication/authorization w/ non-SSL
forwarding
• Authentication and Authorization providers
• Who can connect to input plugins?
What tags are permitted for clients?
• New plugin types (3rd party authors can write it)
• Mainly for in/out forward, but available from others

17. Benchmark (1 CPU usage)
100,000msgs/sec v0.14 v0.12
in_tail (none) +
out_forward
70% 66%
in_forward +
flowcounter_simple
11% 11%
in_forward + tdlog 43% 38%
※ Use EC2 c3.8xlarge ※ Not fully optimized yet

18. Treasure Agent 3.0 (td-agent 3)
• fluentd v0.14
• Ruby 2.3 and latest core components
• Environments
• Add msi Windows package
• Remove CentOS 5, Ubuntu 10.04 support
• Release date is not fixed…