The document summarizes Masahiro Nakagawa's presentation on Fluentd at the Cloudera World Tokyo conference. Fluentd is an open source log collector written in Ruby that uses a pluggable architecture and JSON format for log messages. It provides unified logging and data processing capabilities. The presentation covered Fluentd's core functionality, related products from Treasure Data, use cases, and the company's roadmap.

1. Masahiro Nakagawa
Nov 6, 2014
Cloudera World Tokyo
Fluentd
Unified Logging Layer

2. Who are you?
> Masahiro Nakagawa
> github/twitter: @repeatedly
> Treasure Data, Inc.
> Senior Software Engineer
> Fluentd / td-agent developer
> I love OSS :)
> D language - Phobos committer
> Fluentd - Main maintainer
> MessagePack / RPC- D and Python (only RPC)
> The organizer of Presto Source Code Reading
> etc…

3. Structured logging
!
Reliable forwarding
!
Pluggable architecture
http://fluentd.org/

4. Data Processing
Data source
Collect Store Process Visualize
Reporting
Monitoring

5. Related Products
Collect Store Process
Visualize
Cloudera
Horton Works
Treasure Data
Excel
Tableau
R
easier & shorter time
???

6. What’s Fluentd?
• Open sourced log collector written in Ruby
• Customization is essential
small core + many plugins
6
It’s like syslogd, but
uses JSON for log messages

7. Divide & Conquer & Retry
7
error retry
retry
error retry retry

8. Core Plugins
!
• Divide & Conquer
• Buffring & Retrying
• Error handling
• Message routing
• Parallelism
!
!
• read / receive data
• write / send data
8

9. Pluggable Architecture
9
Pluggable Pluggable
Buffer Output
Input
> Forward
> HTTP
> File tail
> dstat
> ...
> Forward
> File
> MongoDB
> ...
> File
> Memory
Engine
Output
> rewrite
> ...

10. Before
10

11. After
11

12. M x N → M + N
12
Alerting
Nagios
Analysis
MongoDB
MySQL
Hadoop
Archiving
Amazon S3
Access logs
Apache
App logs
Frontend
Backend
System logs
syslogd
Databases
buffer / buffer / routing

13. Ecosystem

14. Treasure Agent
• Treasure Data distribution of Fluentd
• including Ruby, core libraries and
third party plugins
• Current stable is 2.1.1
• rpm / deb / dmg
• 2.1.2 will be released soon with fluentd-ui
14

15. fluentd-forwarder
• Forwarding agent written in Go
• mainly for Windows support
• immature than Fluentd
• Bundle TCP input/output and TD output
• No plugin mechanizm
• Similar product: ik, fluent-agent-hydra, etc
15

16. fluentd-ui
• Manage Fluentd instance using Web GUI
16

17. Use-case

18. Users
• Companies
• Web service, Ad Tech, Game, Media,
EC, IoT, PoS, etc..
• Products
• Kubernetes, Cloudn, Harvester,
Bloomberg Clustered Private Cloud, etc..
18

19. Simple forwarding
19
HDFS
HBase
Mongo
S3

20. Separate log stream
20

21. CEP for stream processing
21

22. Roadmap
• v0.10 (current)
• v0.12 (Nov, 2014)
• Filter / Label / at-least-once
• v0.14 (TBD)
• New Plugin APIs, ServerEngine
• v1 (TBD. In the first half of 2015)
22

23. Cloud service for the entire data pipeline
Check: treasuredata.com