Fluentd & Event logsBefore: App server App server App server Application Application Application File File File ... File File File ... File File File ... File High latency must wait for a day Log server Hard to analyze complex text parsers
Fluentd & Event logsAfter: App server App server App server Application Application Application Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd
# receive events via HTTP # save alerts to a file<source> <match alert.**> type http type file port 8888 path /var/log/fluent/alerts</source> </match># read logs from a file # forward other logs to servers<source> # (load-balancing + fail-over) type tail <match **> path /var/log/httpd.log type forward format apache <server> tag apache.access host 192.168.0.11</source> weight 20 </server># save access logs to MongoDB <server><match apache.access> host 192.168.0.12 type mongo weight 60 host 127.0.0.1 </server></match> </match>
Fluentd vs Scribe• Deals with structured logs• Easy to install > “gem install ﬂuentd” > apt-get and yum http://packages.treasure-data.com/• Easy to customize• add/modify plugins without re-compiling > “gem search -rd ﬂuent-plugin”
Fluentd vs Flume• Easy to setup > “sudo ﬂuentd --setup && ﬂuentd”• Very small footprint > small engine (3,000 lines) + plugins• JVM-free• Easy to conﬁgure
Plugins• Bundled plugins > ﬁle writes event logs to ﬁles hourly or daily > forward forwards event logs (+fail-over and load balancing) > exec passes event logs to/from external commands > tail reads event logs from a ﬁle (like `tail -f`)
Plugins• 3rd party plugins > scribe integrates Fluentd with Scribe > s3 uploads log ﬁles to Amazon S3 hourly or daily > mongo writes logs to MongoDB > hoop puts log ﬁles on Hadoop HDFS via Hoop ...
Plugin developer API• Unit test framework (like “MRUnit”) > Fluent::Test::InputTestDriver > Fluent::Test::OutputTestDriver > Fluent::Test::BufferedOutputTestDriver• Fluent::TailInput (base class of “tail” plugin) > text parser is customizable def parse_line(line)