Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Fluentd v1.0 in a nutshell
March 30, 2017
Masahiro Nakagawa
Fluentd v0.12
• Current stable and widely used on production
• Input, Parser, Filter, Formatter, Buffer, Output plugins
• ...
• Development version of v1
• Latest version is v0.14.14: March 24, 2017
• Implemented New features
• New Plugin APIs
• Ti...
Fluentd v1
• Stable announcement for APIs / features
• No breaking API changes in v1.x
• Compatible with v0.12 and v0.14
•...
New Plugin APIs
• Input/Output plugin APIs w/ well-controlled lifecycle
• stop, shutdown, close, terminate
• Integrate all...
Router
buffer_chunk_limit
enqueue: exceed flush_interval
or buffer_chunk_limit
Key pattern:
- BufferedOutput
empty string or s...
v0.14 buffer design
Buffer keys and placeholders
• Dynamic parameters for table name, object path and more
• We can embed time, tag and any fie...
Time with nanosecond
• For sub-second systems: Elasticsearch, InfluxData, etc…
• Fluent::EventTime
• behaves as Integer for...
ServerEngine based Supervisor
• ServerEngine is a framework for building robust server
• https://github.com/treasure-data/...
Windows support
• Fluentd and core plugins work on Windows
• Windows service registration is also supported
• http://docs....
Symmetric multi core processing
• 2 or more workers share a configuration file
• and share listening sockets via PluginHelpe...
Worker0
Supervisor
v0.14’s multi process feature
grep
forward
tdlog
Worker1 Worker2
grep
forward
tdlog
grep
forward
tdlog
...
Configuration example
<system>
workers 2
root_dir /var/log/fluentd
</system>
<source>
@type forward
</source>
<filter pattern...
TLS/Authn/Authz support for forward plugin
• secure-forward is merged into built-in forward
• TLS w/ at-least-one semantic...
Plugin Storage & Helpers
• Plugin Storage: new plugin type for plugins
• provides key-value storage to persistent intermed...
server helper: before
def start
@loop = Coolio::Loop.new
@handler = Coolio::TCPServer.new(@bind, @port, SocketUtil::TcpHan...
server helper: after
def start
server_create(:foo_server, @port, bind: @bind) do |data, conn|

# body

end
end
v0.12 plugins
ParserInput Buffer Output FormatterFilter
“output-ish”“input-ish”
v0.14 plugins
ParserInput Buffer Output FormatterFilter
“output-ish”“input-ish”
Storage
Helper
TODO list for v1
• <worker N> directive to separate plugins
• https://github.com/fluent/fluentd/pull/1507
• Counter API to s...
<worker N> directive
• To execute plugins under one process
• Good for non-multiprocess supported plugins like in_tail



...
Treasure Agent 3.0 (td-agent 3)
• fluentd v0.14, Ruby 2.4, and latest core components
• Environments
• Add msi Windows pack...
Upcoming SlideShare
Loading in …5
×

Fluentd v1.0 in a nutshell

4,848 views

Published on

Fluentd salon at CNCon Europe 2017

Published in: Technology
  • Be the first to comment

Fluentd v1.0 in a nutshell

  1. 1. Fluentd v1.0 in a nutshell March 30, 2017 Masahiro Nakagawa
  2. 2. Fluentd v0.12 • Current stable and widely used on production • Input, Parser, Filter, Formatter, Buffer, Output plugins • Known issues • Event time is second unit • No multi core support • No Windows support • Need to improve plugin API to support more various use cases
  3. 3. • Development version of v1 • Latest version is v0.14.14: March 24, 2017 • Implemented New features • New Plugin APIs • Time with Nanosecond resolution • ServerEngine based Supervisor • Windows support • Plugin Helpers & Plugin Storage Fluentd v0.14
  4. 4. Fluentd v1 • Stable announcement for APIs / features • No breaking API changes in v1.x • Compatible with v0.12 and v0.14 • exclude v0 config syntax and detach_process • Release plan • Q2, 2017 • Need v0.14 feedback from developers and users
 https://hub.docker.com/r/fluent/fluentd/
  5. 5. New Plugin APIs • Input/Output plugin APIs w/ well-controlled lifecycle • stop, shutdown, close, terminate • Integrate all output plugin into Fluent::Plugin::Output • New Buffer API for delayed commit and flexible chunking with metadata • parallel/async "commit" operation for chunks • For high latency case: forward’s at-least-once, issuing job, etc… • Users can choose chunk keys by configuration for dynamic parameters • Compatible w/ v0.12 plugins • compatibility layer for traditional APIs • it will be supported between v1.x versions
  6. 6. Router buffer_chunk_limit enqueue: exceed flush_interval or buffer_chunk_limit Key pattern: - BufferedOutput empty string or specified key -ObjectBufferedOutput tag -TimeSlicedOutput time slice emit emit Buffer Queue buffer_queue_limit Output OutputInput / Filter Tag Time Record Chunk Chunk Chunk Chunk Chunk key:foo key:bar key:baz v0.12 buffer design
  7. 7. v0.14 buffer design
  8. 8. Buffer keys and placeholders • Dynamic parameters for table name, object path and more • We can embed time, tag and any field with placeholder
 
 
 
 
 
 
 <match s3.**> @type s3 aws_key_id "#{ENV['AWS_ACCESS_KEY']}" aws_sec_key "#{ENV['AWS_SECRETA_KEY']}" s3_bucket fluent-plugin-s3 path test/%Y/%m/${tag}/${key}/ <buffer time,tag,key> timekey 3600 </buffer> </match> http://docs.fluentd.org/v0.14/articles/buffer-section for more details time: 2017-03-30 12:00:00 +0200 tag: “test” record: {“key”:”hello”} - Event sample test/2017/3/test/hello/ - Generated “path”
  9. 9. Time with nanosecond • For sub-second systems: Elasticsearch, InfluxData, etc… • Fluent::EventTime • behaves as Integer for v0.12’s second unit compatibility • has methods to get sub-second resolution • be serialized into msgpack using Ext type • Fluent::Engine.now now returns EventTime, not Integer • Fluentd core can handle both of Integer and EventTime as time • compatible with older versions and software in eco-system (e.g., fluent-logger, Docker logging driver)
  10. 10. ServerEngine based Supervisor • ServerEngine is a framework for building robust server • https://github.com/treasure-data/serverengine • Replacing supervisor process with ServerEngine • it has SocketManager to share listening sockets between 2 or more worker processes • Replacing Fluentd's processing model from fork to spawn • to support Windows environment
  11. 11. Windows support • Fluentd and core plugins work on Windows • Windows service registration is also supported • http://docs.fluentd.org/v0.14/articles/install-by-msi • Use HTTP RPC instead of signals • https://github.com/fluent/fluent-plugin-windows-eventlog • We can collect windows eventlog :)
  12. 12. Symmetric multi core processing • 2 or more workers share a configuration file • and share listening sockets via PluginHelper • under a supervisor process (ServerEngine) • Multi core scalability for huge traffic • one input plugin for a tcp port, some filters and one (or some) output plugin • buffer paths are managed automatically by Fluentd. Need root_dir and @id parameters
  13. 13. Worker0 Supervisor v0.14’s multi process feature grep forward tdlog Worker1 Worker2 grep forward tdlog grep forward tdlog socket
  14. 14. Configuration example <system> workers 2 root_dir /var/log/fluentd </system> <source> @type forward </source> <filter pattern> @type grep </filter> <match pattern> @type tdlog @id out_td </match> /var/log/fluentd/worker0/out_td/buffer/buffer.xxx.log /var/log/fluentd/worker0/out_td/buffer/buffer.xxx.log.meta - buf_file’s path is automatically generated worker id root_dir plugin’s @id
  15. 15. TLS/Authn/Authz support for forward plugin • secure-forward is merged into built-in forward • TLS w/ at-least-one semantics • Simple authentication/authorization w/o SSL • Different points • secure-forward uses keep-alive, but forward doesn’t • secure-forward uses thread per connection, but forward uses libev based IO. http://www.fluentd.org/blog/fluentd-v0.14.12-has-been-released
  16. 16. Plugin Storage & Helpers • Plugin Storage: new plugin type for plugins • provides key-value storage to persistent intermediate status • built-in plugins: in-memory, local file • pluggable: 3rd party plugin to store data into storage • storage-redis • Plugin Helpers: • collections of utility methods for plugins • making threads, sockets, network servers, ... • fully integrated with test drivers to run test code after setup phase of helpers (e.g. test started after created threads)
  17. 17. server helper: before def start @loop = Coolio::Loop.new @handler = Coolio::TCPServer.new(@bind, @port, SocketUtil::TcpHandler, log, @delimiter, method(:on_message)) @loop.attach(@handler) @thread = Thread.new(&method(:run)) end def shutdown @loop.watchers.each { |w| w.detach } @loop.stop @handler.close @thread.join end def run @loop.run(@blocking_timeout) rescue => e log.error "unexpected error", error: e log.error_backtrace end def on_message(msg, addr) # body end
  18. 18. server helper: after def start server_create(:foo_server, @port, bind: @bind) do |data, conn|
 # body
 end end
  19. 19. v0.12 plugins ParserInput Buffer Output FormatterFilter “output-ish”“input-ish”
  20. 20. v0.14 plugins ParserInput Buffer Output FormatterFilter “output-ish”“input-ish” Storage Helper
  21. 21. TODO list for v1 • <worker N> directive to separate plugins • https://github.com/fluent/fluentd/pull/1507 • Counter API to store metrics between processes • https://github.com/fluent/fluentd/tree/counter-api • Migrate more plugin into v0.14 API • Fix regressions and bugs • Write docuemnt: design, plugin, operation, etc…
  22. 22. <worker N> directive • To execute plugins under one process • Good for non-multiprocess supported plugins like in_tail
 
 
 
 
 
 
 
 <system> workers 4 </system> <source> @type forward </source> <match pattern> @type mongo </match> <worker 0> <source> @type in_tail </source> <match pattern> @type s3 </match> </worker> in_tail/out_s3 works under worker 0 in_forward/out_mongo works
 under multiprocess environment with
 worker 1, worker 2, and worker 3
  23. 23. Treasure Agent 3.0 (td-agent 3) • fluentd v0.14, Ruby 2.4, and latest core components • Environments • Add msi Windows package • Remove CentOS 5, Ubuntu 10.04 support • Beta packages have beed released • http://docs.fluentd.org/v0.14/categories/installation • beta will be removed after v0.14 becomes stable :)

×