6. Fluentd
Fluentd is an open source data collector
to simplify log management.
Fluentd is designed to process high-volume data streams reliably. Use cases
include real-time search and monitoring, Big Data analytics, reliable archiving and
more.
http://www.fluentd.org/
14年6月26日木曜日
10. Before Fluentd:
Access logs
App logs
System logs
Metrics
Analytics
Archives
apache
nginx
frontend
backend
syslogd
snmp data
graphs
Hadoop
MySQL
MongoDB
Redshift
Amazon S3
Filesystem
tail -f
scp
python
ruby
ruby
cmd
file ruby
logger
Error handling? Buffering?
Routing? API Keys? Formats?
14年6月26日木曜日
11. Before Fluentd: CHAOS
Access logs
App logs
System logs
Various logs
Metrics
Analytics
Archives
apache
nginx
frontend
backend
syslogd
snmp data
graphs
Hadoop
MySQL
MongoDB
Redshift
Amazon S3
Filesystem
tail -f
scp
python
ruby
ruby
cmd
file ruby
logger
file
logger
ruby
cmd
ruby
Error handling? Buffering?
Routing? API Keys? Formats?
14年6月26日木曜日
12. After Fluentd: Controllable
Access logs
App logs
System logs
Various logs
Metrics
Analytics
Archives
apache
nginx
frontend
backend
syslogd
snmp data
graphs
Hadoop
MySQL
MongoDB
Redshift
Amazon S3
Filesystem
14年6月26日木曜日
13. Access logs
App logs
System logs
Various logs
Metrics
Analytics
Archives
apache
nginx
frontend
backend
syslogd
snmp data
graphs
Hadoop
MySQL
MongoDB
Redshift
Amazon S3
Filesystem
Fluentd does:
Format, Buffer, Retry, Route
After Fluentd: Controllable
14年6月26日木曜日
14. Fluentd
Open source data collector
Written in Ruby, runs on CRuby on UNIX-like OS
With error handling and routing in core
Plugin systems
Input, Output and Buffer (w/ many built-in plugins)
Distributed on rubygems.org
Fluentd and its plugins: gem install fluentd
rpm/deb are also available (td-agent)
14年6月26日木曜日
18. Why Fluentd?
Simple data structure
tag, time and record(hash)
Apache-like configuration syntax
Simple / powerful routing
Many public plugins
Just few steps for custom plugins
Scalability
14年6月26日木曜日
21. Fluentd Event
app.device.ios
1403512916 (2014-06-23 16:41:56 +0800)
{
“username”: “tagomoris”,
“fullname”: “TAGOMORI Satoshi”,
“age”: 34,
“device”: “iPhone 5”,
...
}
tag for routing
record
structured data
time by unix time
14年6月26日木曜日
22. # read from a file and parse
<source>
type tail
path /var/log/httpd.log
format apache2
tag web.access
</source>
# logs from client libraries
<source>
type forward
port 24224
</source>
# store logs to MongoDB and S3
<match app.**>
type copy
<store>
type mongo
host mongo.example.com
capped
capped_size 200m
</store>
<store>
type s3
path archive/
</store>
</match>
Fluentd Configuration
14年6月26日木曜日
23. # read from a file and parse
<source>
type tail
path /var/log/httpd.log
format apache2
tag web.access
</source>
# logs from client libraries
<source>
type forward
port 24224
</source>
# store logs to MongoDB and S3
<match app.**>
type copy
<store>
type mongo
host mongo.example.com
capped
capped_size 200m
</store>
<store>
type s3
path archive/
</store>
</match>
Fluentd Configuration
for input for output
14年6月26日木曜日
24. # read from a file and parse
source {
type ”tail”
path “/var/log/httpd.log”
format “apache2”
tag ”web.access”
}
# logs from client libraries
source {
type ”forward”
port 24224
}
# store logs to MongoDB and S3
match(“app.**”) {
type ”copy”
store {
type ”mongo”
host “mongo.example.com”
capped
capped_size “200m”
}
store {
type ”s3”
path “archive/”
}
}
Fluentd Configuration DSL
14年6月26日木曜日
29. 1.
read logs from file
and write these on storages
file in_tail
read, parse
out_file
format, write
file
14年6月26日木曜日
30. 1.
read logs from file
and write these on storages
file
read, parse insert
MongoDBout_mongo
https://github.com/fluent/fluent-plugin-mongo
in_tail
14年6月26日木曜日
31. 1.
read logs from file
and write these on storages
file
read, parse
out_mysql
insert
MySQL
https://github.com/tagomoris/fluent-plugin-mysql
in_tail
14年6月26日木曜日
32. 1.
read logs from file
and write these on storages
file
read, parse
out_elasticsearch
send
Elasticsearch
https://github.com/uken/fluent-plugin-elasticsearch
in_tail
14年6月26日木曜日
33. 1.
read logs from file
and write these on storages
file
read, parse
out_webhdfs
format, write
Hadoop HDFS
https://github.com/fluent/fluent-plugin-webhdfs
in_tail
14年6月26日木曜日
34. 1.
read logs from file
and write these on storages
file
read, parse
out_s3
format, write
Amazon S3
https://github.com/fluent/fluent-plugin-s3
in_tail
14年6月26日木曜日
35. 1.
read logs from file
and write these on storages
file
read, parse
out_redshift
insert
Amazon
Redshift
https://github.com/hapyrus/fluent-plugin-redshift
in_tail
14年6月26日木曜日
36. 1.
read logs from file
and write these on storages
file
read, parse
out_bigquery
insert
Google
BigQuery
https://github.com/tagomoris/fluent-plugin-bigquery
in_tail
14年6月26日木曜日
37. 2.
receive and forward data
from/to other node
forward
forward
forward
input
events
input
events
output
events
fluent-logger-ruby
fluent-logger-java
...
send events over TCP
14年6月26日木曜日
38. 2.
receive and forward data
from/to other node
forward
forward
forward
load balance, active-standby forward
forward
forward
14年6月26日木曜日
39. datacenter
2’.
receive and forward data
from/to other node, over internet & SSL
secure-forward
secure-forward
datacenter
secure-forward
send events over SSL
with authentication
https://github.com/tagomoris/fluent-plugin-secure-forward
14年6月26日木曜日
45. 6.
various inputs: Linux performance (dstat)
in_dstatdstat
collect server performance data
https://github.com/shun0102/fluent-plugin-dstat
14年6月26日木曜日
46. 6.
various inputs: SQL execution
in_sql
input from SELECT
RDBMS
https://github.com/fluent/fluent-plugin-sql
14年6月26日木曜日
50. 7.
various outpus: notification on HipChat
out_hipchat
notice on HipChat
HipChat
https://github.com/hotchpotch/fluent-plugin-hipchat
14年6月26日木曜日
51. 7.
various outpus: graph tools
out_growthforecast
POST data into graph tools
GrowthForecast
or
Focuslight
https://github.com/tagomoris/fluent-plugin-growthforecast
14年6月26日木曜日
54. 8. filters:
stream processing: external command
any inputs any outputs
format & write
into STDIN
exec_filter
any commands
read & parse
from STDOUT
read from STDIN
do WHATEVER you want
write into STDOUT
ex: tail -f | grep ... | sed ... | cat
events
14年6月26日木曜日
55. 8. filters:
stream processing w/ external server RPC
any inputs any outputs
send
out_norikra
fetch
stream processing w/ SQL
in_norikra
http://norikra.github.io/
SELECT stage, score, COUNT(*) AS c
FROM results.win:time_batch(1 min)
WHERE stage > 1 AND user.valid
GROUP BY stage, score
events
14年6月26日木曜日
61. Fluentd v1
Planned as the first major release
someday in 2014 (?)
100% Compatible with v0.10.x
New (and additional) features on v1.x loadmap
https://github.com/fluent/fluentd/issues/251
new configuration syntax, plugin backends
daemon process management
multi core CPU supports
14年6月26日木曜日
70. Fluentd is the best partner
for stream-processing newbies
and rubyists!
Check out sites and code!
http://fluentd.org/
https://github.com/fluent/fluentd
14年6月26日木曜日