18. Problems...
No unified method to collect logs
> Too many bash/perl scripts
Fragile for changes
Less reliable
> Mixed log formats
Old-fashioned “Human-readable” text logs
Not ready to analyze
> High latency
must wait a day for log rotation
23. Input Plugins Output Plugins
time
tag
2012-02-04 01:33:51
JSON format
myapp.buylog {
“user”: ”me”,
“path”: “/buyItem”,
“price”: 150,
“referer”: “/landing”
}
record
24. Why Fluentd?
> Extensibility - Plugin architecture
collect logs from various systems
forward logs to various systems
> Unified log format - JSON format
modern “Machine-readable” log format
immediately ready to analyze
> Reliable - HA configuration
> Easy to install - RPM/deb packages
deploy instantly to everywhere
25.
26. Comparision with other log collectors:
> Scribe
Less extensible
No unified log format
No longer developped?
> Flume
Less simple
No unified log format
Little information about Flume-NG
27. 0. Why logging?
1. Why Fluentd? - Design of Fluentd
> Extensibility
> Unified log format
> Simplicity
2. Who uses Fluentd?
3. Future of Fluentd
28. 0. Why logging?
1. Why Fluentd? - Design of Fluentd
> Extensibility
> Unified log format
> Simplicity
2. Who uses Fluentd?
3. Future of Fluentd
29. NHN Japan COOKPAD NAVER
Crocos
http://www.quora.com/Who-uses-Fluentd-in-production
30. 0. Why logging?
1. Why Fluentd? - Design of Fluentd
> Extensibility
> Unified log format
> Simplicity
2. Who uses Fluentd?
3. Future of Fluentd
31. 0. Why logging?
1. Why Fluentd? - Design of Fluentd
> Extensibility
> Unified log format
> Simplicity
2. Who uses Fluentd?
3. Future of Fluentd
32. Future of Fluentd
> <filter>
> <match> in <source>
> <label>
> MessagePack for Ruby v5
> td-agent-lite
> Pub/Sub & Monitoring API
> New process model & Live restart
> Backward compatibility
33. <source> <match **>
type tail type forward
path /var/log/httpd.log host log.server
format apache </match>
tag not_filtered.apache
</source>
Mysterious tag
<match not_filetered.**>
type rewrite
remove_prefix not_filtered tag operations
<rule>
key status
pattern ^500$
ignore true
</rule>
</match>
Before
34. <source> <match **>
type tail type forward
path /var/log/httpd.log host log.server
format apache </match>
tag apache
</source>
<filter **> Filter plugins!
type rewrite
<rule>
key status
pattern ^500$
ignore true
</rule>
</match>
After (v11)
35. <source> <match **>
type tail type forward
path /var/log/httpd.log host log.server
format apache </match>
tag apache
<filter **>
type rewrite
<rule>
key status
pattern ^500$
<filter>/<match> in
ignore true <source>
</rule>
</match>
</source>
After (v11)
36. <source> <match **>
type tail type forward
path /var/log/httpd.log host log.server
tag apache </match>
</source>
I want to add flowcounter here...
Before
37. <source> <store>
type tail type forward
path /var/log/httpd.log host log.server
tag apache </store>
</source> </match>
<match flow.traffic>
type forward
host traffic.server
</match> Nested!
<match **>
type copy
<store>
type flowcounter
tag flow.traffic
</store>
Before
38. <source> <match **>
type tail type forward
path /var/log/httpd.log host log.server
tag apache </match>
</source>
<filter **>
type copy
<match>
type flowcounter
tag flow.traffic
<match>
type forward
host traffic.server
</match>
</match> Filtering pipeline
</match>
After (v11)
39. <source> # copy & label & forward
type forward <filter **>
</source> type copy
<match>
<filter **> type forward
type copy label alert
<match> host alerting.server
type file </match>
path /mnt/local_archive </filter>
</match>
</filter> # copy & label & forward
<filter **>
<label alert> type copy
<match **> <match>
... type forward
</match> label analysis
</label> host analysis.server
</match>
<label analysis> </filter>
...
</label>
After (v11)
41. td-agent-lite
> in_tail + out_forward in “single” binary
statically linked ruby binary + scripts tied with the binary
42. New process model & Live restart
Old multiprocess model
detached
process
fork()
Supervisor Engine
detached
all data pass through process
the central process
43. New process model & Live restart
New multiprocess model
detached
process
Process
Supervisor Engine
Manager
detached
process
direct communication
44. New process model & Live restart
New multiprocess model
detached
process
Process
Supervisor Engine
Manager
detached
Live restart process
Process
Engine
Manager
45. Backward compatibility
Fluentd v11 includes 2 namespaces:
> Fluentd:: new code base
> Fluent:: old code base + wrapper classes
Checkout the repository for details:
> http://github.com/frsyuki/fluentd-v11
46. Conculution
Fluentd makes logging better
> Plugin architecture
> JSON format
> HA configuration
> RPM/deb package
Fluentd is under active development
Fluentd is suppored by many committers
