Download as PDF, PPTX
![Buffer keys and placeholders
• Dynamic parameters for table name, object path and more
• We can embed time, tag and any field with placeholder
<match s3.**>
@type s3
aws_key_id "#{ENV['AWS_ACCESS_KEY']}"
aws_sec_key "#{ENV['AWS_SECRETA_KEY']}"
s3_bucket fluent-plugin-s3
path test/%Y/%m/${tag}/${key}/
<buffer time,tag,key>
timekey 3600
</buffer>
</match>
http://docs.fluentd.org/v1.0/articles/buffer-section
time: 2018-02-15 12:00:00 +0700
tag: “test”
record: {“key”:”hello”}
- Event sample
test/2018/2/test/hello/
- Generated “path”](https://image.slidesharecdn.com/fluentd-v1-and-future-at-techtalk-180418050650/75/Fluentd-v1-and-future-at-techtalk-9-2048.jpg)
![record_accessor helper
• access / delete support for nested field
• e.g. parser’s key_name parameter uses this helper
• Provide two syntax for configuration
• $.field1.field2 == record[“field1”][“field2”]
• $[“field1”][“field2”] == record[“field1”][“field2”]
ra = record_accessor_create(”$.user.name”)
ra.call(record) # access record[”$.user”][”name”]
ra.delete(record) # delete record[”$.user”][”name”]](https://image.slidesharecdn.com/fluentd-v1-and-future-at-techtalk-180418050650/75/Fluentd-v1-and-future-at-techtalk-21-2048.jpg)
Uploaded byN Masahiro
Fluentd v1 and future at techtalk
Fluentd v1 provides major improvements over v0.12 including nanosecond event time resolution, multi-core support, Windows support, and new plugin APIs. The new plugin APIs provide well-controlled lifecycles and integrate all output plugins. v1 also introduces a server engine based supervisor, dynamic buffering capabilities, and various plugin helpers. While maintaining compatibility with v0.12 plugins, v1 focuses on ease of use, stability, performance and flexibility.
More Related Content
Similar to Fluentd v1 and future at techtalk
Fluentd v1 and future at techtalk
- 1. Fluentd v1 andthe future Feb 15, 2018 Masahiro Nakagawa
- 2.
- 3. Fluentd v0.12 • Oldstable and widely used on production • Input, Parser, Filter, Formatter, Buffer, Output plugins • Known issues • Event time is second unit • No Windows support • No multi core support • Need to improve plugin API to support more various use cases`
- 4. Fluentd v0.14 • Developmentversion of v1 • Implemented New features • New Plugin APIs • Event Time with Nanosecond resolution • ServerEngine based Supervisor • Windows support • Multicore support • New Plugin Helpers & Plugin Storage
- 5. New Plugin APIs •v1.0 Annoucement at CNCon + KubeCon NA. • Stable announcement for APIs / features • No breaking API changes in v1.x • Compatible with v0.12 and v0.14 • exclude v0 config syntax and detach_process • Latest version is v1.1.0: Jan 18, 2018
- 6. New Plugin APIs •Input/Output plugin APIs w/ well-controlled lifecycle • stop, shutdown, close, terminate • Integrate all output plugin into Fluent::Plugin::Output • New Buffer API for delayed commit and flexible chunking with metadata • parallel/async "commit" operation for chunks • For high latency case: forward’s at-least-once, issuing job, etc… • Users can choose chunk keys by configuration for dynamic parameters • Compatible w/ v0.12 plugins • compatibility layer for traditional APIs • it will be supported between v1.x versions
- 7. Router buffer_chunk_limit enqueue: exceed flush_interval orbuffer_chunk_limit Key pattern: - BufferedOutput empty string or specified key -ObjectBufferedOutput tag -TimeSlicedOutput time slice emit emit Buffer Queue buffer_queue_limit Output OutputInput / Filter Tag Time Record Chunk Chunk Chunk Chunk Chunk key:foo key:bar key:baz v0.12 buffer design
- 8.
- 9. Buffer keys andplaceholders • Dynamic parameters for table name, object path and more • We can embed time, tag and any field with placeholder <match s3.**> @type s3 aws_key_id "#{ENV['AWS_ACCESS_KEY']}" aws_sec_key "#{ENV['AWS_SECRETA_KEY']}" s3_bucket fluent-plugin-s3 path test/%Y/%m/${tag}/${key}/ <buffer time,tag,key> timekey 3600 </buffer> </match> http://docs.fluentd.org/v1.0/articles/buffer-section time: 2018-02-15 12:00:00 +0700 tag: “test” record: {“key”:”hello”} - Event sample test/2018/2/test/hello/ - Generated “path”
- 10. Time with nanosecond •For sub-second systems: Elasticsearch, InfluxData, etc… • Fluent::EventTime • behaves as Integer for v0.12’s second unit compatibility • has methods to get sub-second resolution • be serialized into msgpack using Ext type • Fluent::Engine.now now returns EventTime, not Integer • Fluentd core can handle both of Integer and EventTime as time • compatible with older versions and software in eco-system (e.g., fluent-logger, Docker logging driver)
- 11. ServerEngine based Supervisor •ServerEngine is a framework for building robust server • https://github.com/treasure-data/serverengine • Replacing supervisor process with ServerEngine • it has SocketManager to share listening sockets between 2 or more worker processes • Replacing Fluentd's processing model from fork to spawn • to support Windows environment • Log rotation support
- 12. Windows support • Fluentdand core plugins work on Windows • Windows service registration is also supported • http://docs.fluentd.org/v1.0/articles/install-by-msi • Use HTTP RPC instead of signals • https://github.com/fluent/fluent-plugin-windows-eventlog • We can collect windows eventlog :)
- 13. Symmetric multi coreprocessing • 2 or more workers share a configuration file • and share listening sockets via PluginHelper • under a supervisor process (ServerEngine) • Multi core scalability for huge traffic • one input plugin for a tcp port, some filters and one (or some) output plugin • buffer paths are managed by Fluentd core. Need root_dir and @id parameters
- 14. Worker0 Supervisor v1’s multi processfeature grep forward tdlog Worker1 Worker2 grep forward tdlog grep forward tdlog socket
- 15. Configuration example <system> workers 2 root_dir/var/log/fluentd </system> <source> @type forward </source> <filter pattern> @type grep </filter> <match pattern> @type tdlog @id out_td </match> /var/log/fluentd/worker0/out_td/buffer/buffer.xxx.log /var/log/fluentd/worker0/out_td/buffer/buffer.xxx.log.meta - buf_file’s path is automatically generated worker id root_dir plugin’s @id
- 16. <worker N> directive •To execute plugins under one process • Good for non-multiprocess supported plugins like in_tail in_tail/out_s3 works under worker 0 in_forward/out_kafka works under multiprocess environment with worker 1, worker 2, and worker 3 <worker 0> <source> @type in_tail </source> <match pattern> @type s3 </match> </worker> <system> workers 4 </system> <source> @type forward </source> <match pattern> @type mongo </match>
- 17. TLS/Authn/Authz support forforward plugin • Support v1 forward protocol spec • secure-forward is merged into built-in forward • TLS w/ at-least-one semantics • Simple authentication/authorization w/o SSL • Different points • secure-forward uses keep-alive, but forward doesn’t • secure-forward uses thread per connection, but forward uses cool.io, libev based IO. http://www.fluentd.org/blog/fluentd-v0.14.12-has-been-released
- 18. Plugin Storage &Helpers • Plugin Storage: new plugin type for plugins • provides key-value storage to persistent intermediate status • built-in plugins: in-memory, local file • pluggable: 3rd party plugin to store data into storage • storage-redis, storage-memcached • Plugin Helpers: • collections of utility methods for plugins • fully integrated with test drivers to run test code after setup phase of helpers (e.g. test started after created threads)
- 19. server helper: before defstart @loop = Coolio::Loop.new @handler = Coolio::TCPServer.new(@bind, @port, SocketUtil::TcpHandler, log, @delimiter, method(:on_message)) @loop.attach(@handler) @thread = Thread.new(&method(:run)) end def shutdown @loop.watchers.each { |w| w.detach } @loop.stop @handler.close @thread.join end def run @loop.run rescue => e log.error "unexpected error", error: e log.error_backtrace end def on_message(msg, addr) # body end
- 20. server helper: after defstart server_create(:foo_server, @port, bind: @bind) { |data, conn| # body } end https://docs.fluentd.org/v1.0/articles/api-plugin-helper-server
- 21. record_accessor helper • access/ delete support for nested field • e.g. parser’s key_name parameter uses this helper • Provide two syntax for configuration • $.field1.field2 == record[“field1”][“field2”] • $[“field1”][“field2”] == record[“field1”][“field2”] ra = record_accessor_create(”$.user.name”) ra.call(record) # access record[”$.user”][”name”] ra.delete(record) # delete record[”$.user”][”name”]
- 22. v0.12 plugins ParserInput BufferOutput FormatterFilter “output-ish”“input-ish”
- 23. v1 plugins ParserInput BufferOutput FormatterFilter “output-ish”“input-ish” Storage Helper
- 24. Other helpers • Timer:one-shot / periodic timer • Event Loop: Low-layer event loop • Socket: TCP/UDP/TLS support • Formatter/Parser: Manage parser/formatter plugins • Chile Process: Manage process for exec like plugin • etc… https://docs.fluentd.org/v1.0/categories/plugin-helpers
- 25. v1.2.0 • Counter API:store metrics between processes • Need for limit calculation in multi processes • https://github.com/fluent/fluentd/pull/1857 • Backup feature for problematic chunks • Improve retry mechanizm for bad records • https://github.com/fluent/fluentd/issues/1856
- 26. Focus • Easy touse • Stability • Performance • Flexibility • Avoid fat core
- 27. v2! • No plan… •Remove v0.12 or earlier features
- 28.
- 29. Treasure Agent 3(td-agent 3) • fluentd v1, Ruby 2.4, systemd support and latest components • Latest version is 3.1.1: Dec 20, 2017 • 3.2.0 will be released in March • Environments • Add msi Windows package, Amazon Linux 2 • Remove CentOS 5, Ubuntu 10.04 support
- 30. Containers • Docker • Alpineand Debian for v0.12 and v1.x • https://github.com/fluent/fluentd-docker-image • Kubernetes DaemonSet • Alpine and Debian for v0.12 • Debian for v1.x (WIP) • https://github.com/fluent/fluentd-kubernetes-daemonset • Need other container support?
- 31. Integrations • Kafka • kafka-connect-fluentdfor high performance ingestion • Promethuse • fluent-plugin-prometheus to push / pull for prometheus • Integrate internal metrics with monitor_agent • gRPC? • Distributed tracing?
- 32. Benchmark set (WIP) •Check configuration and performance • Current fluentd-benchmak is not enough • Automated test • Various combo: ruby, fluentd, plugins • Collect metrics: CPU, Memory, etc… • Running on: Docker, AWS, etc…
- 33. fluent-bit • Lightweight agentwritten in C • Running on lots of environment including embedded systems with small resource • Pluggable architecute: Input / Parser / Filter / Buffer / Output • fluent-bit is useful for forwarders with fluentd in distributed logging http://fluentbit.io/
- 34. Community • Plugins /Libraries • Thanks for maintaining the project • Users • Experts help new users • Documentation • Need feedback!