Fluentd is an open source log collector that allows flexible collection and routing of log data. It uses JSON format for log messages and supports many input and output plugins. Fluentd can collect logs from files, network services, and applications before routing them to storage and analysis services like MongoDB, HDFS, and Treasure Data. The open source project has grown a large community contributing over 100 plugins to make log collection and processing easier.

1. Muga Nishizawa
Treasure Data, Inc.
the missing log collector

2. Muga Nishizawa (@muga_nishizawa)
Chief Software Architect, Treasure Data

3. 3
Treasure Data Overview
 Founded to deliver big data analytics in days not months without
specialist IT resources for one-tenth the cost of other alternatives
 Service based subscription business model
 World class open source team
• Founded world’s largest Hadoop User Group
• Developed Fluentd and MessagePack
• Contributed to Memcached, Hibernate, etc.
 Treasure Data is in production
• 60+ customers incl. Fortune 500 companies
• 400+ billion records stored
 Processing 40,000 messages per second

4. =
Fluentd
syslogd
+
many

5. =
Fluentd
syslogd
+
many
✓ Plugins
✓ JSON

6. > Open sourced log collector written in Ruby
> Using rubygems ecosystem for plugins
In short
It’s like syslogd, but
uses JSON for log messages

7. Make log collection easy
using Fluentd

8. Reporting & Monitoring

9. Reporting & Monitoring
Collect Store Process Visualize

10. Collect Store Process Visualize
easier & shorter time
Hadoop / Hive
MongoDB
Treasure Data
Tableau
Excel
R
Reporting & Monitoring

11. Collect Store Process Visualize
easier & shorter timeHow to shorten here?
Hadoop / Hive
MongoDB
Treasure Data
Tableau
Excel
R

12. Collect Store Process Visualize
easier & shorter timeHow to shorten here?
Hadoop / Hive
MongoDB
Treasure Data
Tableau
Excel
R

13. Before Fluentd
Application
･･･
Server2
Application
･･･
Server3
Application
･･･
Server1
FluentLog Server
High Latency!
must wait for a day...

14. After Fluentd
Application
･･･
Server2
Application
･･･
Server3
Application
･･･
Server1
In streaming!
Fluentd Fluentd Fluentd
Fluentd Fluentd

15. Many Users

16. Many Meetups

17. Growth by Community

18. Why did we develop Fluentd?

19. Apache
App
App
Other data sources
td-agent
RDBMS
Treasure Data
columnar data
warehouse
Query
Processing
Cluster
Query
API
HIVE, PIG (to be supported)
JDBC, REST
MAPREDUCE JOBS
User
td-command
BI apps
Treasure Data Service Architecture

20. Apache
App
App
Other data sources
td-agent
RDBMS
Treasure Data
columnar data
warehouse
Query
Processing
Cluster
Query
API
HIVE, PIG (to be supported)
JDBC, REST
MAPREDUCE JOBS
User
td-command
BI apps
Treasure Data Service Architecture
Open Sourced

21. writes logs to text files
Rails app
Google
Spreadsheet
MySQL
MySQL
MySQL
MySQL
writes logs to text files
Nightly
INSERT
hundreds of app servers
Daily/Hourly
Batch
KPI
visualizationFeedback rankings
Rails app
writes logs to text files
Rails app
- Limited scalability
- Fixed schema
- Not realtime
- Unexpected INSERT latency
Example Use Case – MySQL to TD

22. hundreds of app servers
sends event logs
sends event logs
sends event logs
Rails app td-agent
td-agent
td-agent
Google
Spreadsheet
Treasure Data
MySQL
Logs are available
after several mins.
Daily/Hourly
Batch
KPI
visualizationFeedback rankings
Rails app
Rails app
Unlimited scalability
Flexible schema
Realtime
Less performance impact
Example Use Case – MySQL to TD

23. td-agent
> Open sourced distribution package of fluentd
> ETL part of Treasure Data
> Including useful components
> ruby, jemalloc, fluentd
> 3rd party gems: td, mongo, webhdfs, etc...
td plugin is for TD
> http://packages.treasure-data.com/

24. How Fluentd works?

25. =
Fluentd
syslogd
+
many
✓ Plugins
✓ JSON

26. Nagios
MongoDB
Hadoop
Alerting
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Databases
filter / buffer / routing

27. Nagios
MongoDB
Hadoop
Alerting
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Databases
filter / buffer / routing

28. Nagios
MongoDB
Hadoop
Alerting
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Databases
filter / buffer / routing

29. Input Plugins Output Plugins
Buffer Plugins
(Filter Plugins)
Nagios
MongoDB
Hadoop
Alerting
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Databases
filter / buffer / routing

30. Architecture
Buffer OutputInput
> Forward
> HTTP
> File tail
> dstat
> ...
> Forward
> File
> Amazon S3
> MongoDB
> ...
> Memory
> File
Pluggable Pluggable Pluggable

31. Architecture
Buffer OutputInput
> Forward
> HTTP
> File tail
> dstat
> ...
> Forward
> File
> Amazon S3
> MongoDB
> ...
> Memory
> File
Pluggable Pluggable Pluggable
117 plugins!
Contributions by Community

32. Input Plugins Output Plugins
2012-02-04 01:33:51
myapp.buylog {
“user”: ”me”,
“path”: “/buyItem”,
“price”: 150,
“referer”: “/landing”
}
time
tag
record
JSON
log

33. > second unit
> from data source or
adding parsed time
Event structure(log message)
✓ Time
> for message routing
✓ Tag
> JSON format
> MessagePack
internally
> non-unstructured
✓ Record

34. in_tail: reads file and parses lines
fluentd
apache
access.log
✓ read a log file
✓ custom regexp
✓ custom parser in Ruby
in_tail

35. out_mongo: writes buffered
chunks
fluentd
apache
access.log buffer
in_tail

36. failure handling & retrying
fluentd
apache
access.log buffer
✓ retry automatically
✓ exponential retry wait
✓ persistent on a file
in_tail

37. out_s3
fluentd
apache
access.log buffer
✓ retry automatically
✓ exponential retry wait
✓ persistent on a file
Amazon S3
✓ slice files based on time
in_tail
2013-01-01/01/access.log.gz
2013-01-01/02/access.log.gz
2013-01-01/03/access.log.gz
...

38. out_hdfs
fluentd
apache
access.log buffer
✓ retry automatically
✓ exponential retry wait
✓ persistent on a file
✓ slice files based on time
in_tail
2013-01-01/01/access.log.gz
2013-01-01/02/access.log.gz
2013-01-01/03/access.log.gz
...
HDFS
✓ custom text formater

39. routing / copying
fluentd
apache
access.log buffer
✓ routing based on tags
✓ copy to multiple storages
in_tail
Amazon S3
Hadoop

40. Fluentd
# Ruby
Fluent.open(“myapp”)
Fluent.event(“login”, {“user” => 38})
#=> 2012-12-11 07:56:01 myapp.login {“user”:38}
> Ruby
> Java
> Perl
> PHP
> Python
> D
> Scala
> ...
Application
Time:Tag:Record
Client libraries

41. # logs from a file
<source>
type tail
path /var/log/httpd.log
format apache2
tag web.access
</source>
# logs from client libraries
<source>
type forward
port 24224
</source>
# store logs to MongoDB and S3
<match **>
type copy
<match>
type mongo
host mongo.example.com
capped
capped_size 200m
</match>
<match>
type s3
path archive/
</match>
</match>
Fluentd

42. out_forward
fluentd
apache
access.log buffer
✓ retry automatically
✓ exponential retry wait
✓ persistent on a file
✓ slice files based on time
in_tail
2013-01-01/01/access.log.gz
2013-01-01/02/access.log.gz
2013-01-01/03/access.log.gz
...
fluentd
fluentd
fluentd
✓ automatic fail-over
✓ load balancing

43. forwarding
fluentd
fluentd
fluentd
fluentd
fluentd
fluentd
fluentd
send / ack
Fluentd

44. Fluentd - plugin distribution
platform
$ fluent-gem search -rd fluent-plugin
$ fluent-gem install fluent-plugin-mongo

45. Use cases

46. hundreds of app servers
sends event logs
sends event logs
sends event logs
Rails app td-agent
td-agent
td-agent
Google
Spreadsheet
Treasure Data
MySQL
Logs are available
after several mins.
Daily/Hourly
Batch
KPI
visualizationFeedback rankings
Rails app
Rails app
Unlimited scalability
Flexible schema
Realtime
Less performance impact
Cookpad
✓ Over 100 RoR servers (2012/2/4)

47. http://www.slideshare.net/tagomoris/log-analysis-with-hadoop-in-livedoor-2013
NHN Japan
by @tagomoris
✓ 16 nodes
✓ 120,000+ lines/sec
✓ 400Mbps at peak
✓ 1.5+ TB/day (raw)
Web
Servers Fluentd
Cluster
Archive
Storage
(scribed)
Fluentd
Watchers
Graph
Tools
Notifications
(IRC)
Hadoop Cluster
CDH4
(HDFS, YARN)
webhdfs
Huahin
Manager
hive
server
STREAM
Shib ShibUI
BATCH
SCHEDULED
BATCH

48. Treasure Data
Frontend
Job Queue
Worker
Hadoop
Hadoop
Fluentd
Applications push
metrics to Fluentd
(via local Fluentd)
Librato
Metrics
for realtime analysis
Treasure
Data
for historical analysis
Fluentd sums up data minutes
(partial aggregation)

49. Key to Fluentd’s growth is...

50. =
Fluentd
syslogd
+
many
+
Community
✓ Plugins
✓ JSON

51. Muga Nishizawa
Treasure Data, Inc.
the missing log collector