Fluentd is a data collector for unified logging that provides a robust core and plugins. It allows for reliable data transfer through error handling and retries. The core handles common concerns like parsing, buffering, and writing data, while plugins handle input, output, and other use cases. Fluentd has a pluggable architecture and processes data through a pipeline of input, parser, filter, buffer, formatter, and output plugins.

1. Fluentd:
Unified logging layer
June 25, 2016 / HKOScon
Masahiro Nakagawa

2. Who am I
• Masahiro Nakagawa
• github: @repeatedly
• Treasure Data, Inc.
• Senior Software Engineer
• Fluentd / td-agent developer
• Living at OSS :)
• D language - Phobos, a.k.a standard library, committer
• Fluentd - Main maintainer
• MessagePack / RPC
• etc…

3. Background

4. Before
✓ duplicated code for error handling...
✓ messy code for retrying mechanism...

5. After

6. Structured logging
Reliable forwarding
Pluggable architecture
http://fluentd.org/

7. Concept
/
Design

8. What’s Fluentd?
• Data collector for unified logging layer
• Streaming data transfer based on JSON /
MessagePack
• Robust core and plugins written in Ruby
• Rubygems based various plugins by the community
• www.fluentd.org/plugins
• Apache License, Version 2.0
• https://github.com/fluent/fluentd

9. Reliable data transfer
error retry
error retry retry
retry
Batch
Stream
Other stream
(micro-batch)

10. Core
Common concerns Use case specific
PLUGINSCORE
• Read Data
• Parse Data
• Buffer Data
• Write Data
• Format Data
• Divide & Conquer
• Buffering & Retries
• Error Handling
• Message Routing
• Parallelism

11. > logged time
Event structure(log message)
✓ Time
> for message routing
✓ Tag
> Actual content
> JSON / MessagePack
✓ Record
127.0.0.1 - - [11/Dec/2012:07:26:27] "GET / ...
127.0.0.1 - - [11/Dec/2012:07:26:30] "GET / ...
127.0.0.1 - - [11/Dec/2012:07:26:32] "GET / ...
127.0.0.1 - - [11/Dec/2012:07:26:40] "GET / ...
...
2012-12-11 07:26:27
apache.log
{
"host": "127.0.0.1",
"method": "GET",
...
}
Apache log Fluentd event
convert
time
tag
record

12. Architecture

13. Data processing pipeline
ParserInput Buffer Output FormatterFilter
“output-ish”“input-ish”

14. Input plugins
File tail (in_tail)
Syslog (in_syslog)
HTTP (in_http)
RDBMS (in_sql)
...
✓ Receive logs
✓ Or pull logs from data sources
✓ non-blocking
InputInput

15. Parser plugins
JSON
Regexp
Apache/Nginx
Syslog
CSV/TSV
etc.
✓ Parse into JSON
✓ Common formats out of the box
✓ Some inputs plugin depends on
Parser plugin
ParseParser

16. Filter plugins
grep
record_transformer
suppress
…
✓ Filter / Mutate record
✓ Record level and Stream level
✓ v0.12 and above
Filter

17. Buffer plugins
✓ Improve performance
✓ Provide reliability
✓ Provide thread-safetyMemory (buf_memory)
File (buf_file)
Buffer

18. Buffer internal
✓ Chunk = adjustable unit of data
✓ Buffer = Queue of chunks
chunk
chunk
chunk output
Input

19. Formatter plugins
✓ Format output
✓ Convert json into other format
✓ Some plugins depends on
Formatter plugins
JSON
CSV/TSV
“single value”
msgpack
Formatter

20. Output plugins
✓ Write to external systems
✓ Buffered & Non-buffered
✓ 300+ pluginsFile (out_file)
Amazon S3 (out_s3)
Kafka (out_kafka)
...
Output

21. Use
cases

22. Simple Forwarding

23. # logs from a file
<source>
@type tail
path /var/log/httpd.log
pos_file /tmp/pos_file
format apache2
tag backend.apache
</source>
# logs from client libraries
<source>
@type forward
port 24224
</source>
# store logs to MongoDB
<match backend.*>
@type mongo
database fluent
collection test
</match>

24. Less Simple Forwarding
- At-most-once / At-least-once
- HA (failover)
- Load-balancing

25. Container Logging

26. Treasure Data
Frontend
Job Queue
Worker
Hadoop
Presto
Fluentd
Applications push
metrics to Fluentd
(via local Fluentd)
Datadog
for realtime monitoring / alerting
Treasure Data
for historical analysis
Fluentd sums up data minutes
(partial aggregation)

27. Slideshare
http://engineering.slideshare.net/2014/04/skynet-project-monitor-scale-and-auto-heal-a-system-in-the-cloud/

28. Other companies…
https://www.fluentd.org/testimonials

29. Roadmap

30. History / Roadmap
• v0.10.0 (In Oct 2011)
• v0.12.0 (In Dec 2014) -> Current stable
• v0.14.0 (In May 2016)
• v0.14.x (some versions in 2016)
• v1 (4Q in 2016 / 1Q in 2017)

31. v0.10 (old stable)
> First stable release
> Mainly for log forwarding
> Only Input and Output
> No complex event handling support
> Only At-most-once semantics in forwarding
> Treasure Data provides td-agent 1 for v0.10

32. v0.12 (current stable)
> Event handling improvement
> Filter, Label, Error Stream
> New configuration format
> Add at-least-once semantics in forwarding
> Use require_ack_response parameter
> HTTP RPC based process management
> Treasure Data provides td-agent 2 for v0.12

33. • New Plugin APIs, Plugin Helpers & Plugin Storage
• Supervisor using ServerEngine
• Time with Nanosecond precision
• Windows support
v0.14 (Next stable)

34. Time with nanosecond
• Fluent::EventTime
• behaves as Integer (used as time in v0.12)
• has methods to get sub-second precision
• be serialized into msgpack using Ext type
• Fluentd core can handle both of Integer and
EventTime as time
• compatible with older versions and software in eco-
system (e.g., fluent-logger, Docker logging driver)

35. Windows support
• Fluentd and core plugin work on Windows
• several companies have already used
v0.14.0.pre version on production
• We will send a patch to popular plugins if
it doesn’t work on Windows
• Use HTTP RPC instead of signals
• Treasure Data will provide td-agent msi package

36. v0.14.x - v1
• v0.14.x (some versions in 2016)
• Symmetric multi processing model
• Counter API
• TLS/authentication/authorization support
(merging secure-forward into core)
• v1 (4Q in 2016 / 1Q in 2017)
• Stable version for new APIs/features
• fully compatible with v0.12.x and v0.14.x

37. Symmetric multi processing model
• 2 or more workers share a configuration file
• and share listening sockets via PluginHelper
• under a supervisor process (ServerEngine)
• Multi core scalability for huge traffic
• one input plugin for a tcp port, some filters and
one (or some) output plugin
• No more fluent-plugin-multiprocess

38. Worker
Supervisor
Worker Worker
Worker
Supervisor
Worker Worker
Supervisor Supervisor
Using fluent-plugin-multiprocess
v0.14

39. TLS/Authn/Authz support for forward plugin
• secure-forward will be merged into built-in forward
• TLS w/ at-least-one semantics
• Simple authentication/authorization w/ non-SSL
forwarding
• Authentication and Authorization providers
• Who can connect to input plugins?
What tags are permitted for clients?
• New plugin types (3rd party authors can write it)
• Mainly for in/out forward, but available from others

40. Ecosystem

41. Treasure Agent (td-agent)
> Treasure Data distribution of Fluentd
> include Ruby runtime, fluentd and popular plugins
> deb, rpm, dmg are supported
> Treasure Agent 2 is current stable
> fluentd v0.12 and Ruby 2.1
> Treasure Agent 3 will be released at fall
> fluentd v0.14 and Ruby 2.3
> msi will be supported for Windows

42. fluentd-ui
> Manage Fluentd instance via Web UI
> https://github.com/fluent/fluentd-ui

43. fluent-bit
> http://fluentbit.io/
> Log collector mainly for embedded systems
> Written in C
> Can communicate with Fluentd
> There are several plugins
> Treasure Data, Elasticsearch, MQTT, etc…

44. > Bulk Loader version of Fluentd
> Pluggable architecture
> JRuby, JVM languages
> High throughput by parallel processing
> Invented by Treasure Data
> Share your script as a plugin
> https://github.com/embulk
Slide: Fighting Against Chaotically SeparatedValues with Embulk

45. treasuredata.com
Enjoy logging!