This document discusses centralized and unified logging. It describes how Fluentd provides a pluggable architecture for collecting, transporting, storing, analyzing, and alerting on logs from various sources in a centralized and scalable way. Examples are given of using Fluentd plugins to collect Apache logs, parse and enrich the data, forward to multiple outputs like Elasticsearch and Graphite, and more.

1. Centralized + Unified Logging
Gabor Kozma / gabo@ustream.tv / @kozmag82

2. Everybody wants to write logs!
✓ Application Logs (frontend / backend)
➢ php, java, ruby, python, bash
✓ Access Logs
➢ apache, nginx, tomcat, jetty
✓ System Logs
➢ syslog, hardware error log
✓ Database Logs
➢ history, transaction

3. Centralized Logging

4. Central Logging Architecture
✓ Collection
➢ file, syslog, database
✓ Transport
➢ chukwa, heka, syslog, logstash, flume, fluentd,
kafka, nsq, nxlog, other custom solution.
Typical: syslog-ng, rsyslog
✓ Storage / Store
➢ Amazon S3, Glacier, NAS ...

5. Central Logging Architecture
✓ Analysis (You need a way to analyze them!)
➢ Apache Hadoop + HDFS + Map-Reduce jobs
■ Hive, Pig, HBase, Impala....
➢ Elasticsearch + Graylog2 / Kibana
➢ MongoDB + Map-Reduce/Aggregation Framework
➢ Graphite, Statsd + Dashboards
✓ Alerting (Errors almost always indicate a problem!)
➢ Airbreak/Errbit, Sentry, Honeybadger, Nagios,
Zabbix, Open/PagerDuty

6. Unified Logging Layer

7. Unified Logging Layer
✓ Ubiquity
➢ Various format problem
➢ Various source and destination
➢ You must be optimize most of use case!
✓ Rigidity vs. Flexibility
➢ Apache Thrift , Apache Avro, Protocol Buffer , JSON /
BSON, MessagePack

8. Unified Logging Layer
✓ Reliability and Scalability
➢ Scalable
➢ Support retryable data transfer
➢ Sync / Async data transfer
➢ Push / Pull base system
✓ Extensibility
➢ Support new input / output
■ You don’t have to modify anything else.

10. Fluentd - Pluggable architecture
✓ Input, Output, Buffer, Parser, Formatter
300+ plugins

11. Fluentd - Minimum res. require
✓ Combination of C language and Ruby
✓ 1 node
✓ 30/40 Mbyte RAM
✓ 1 CPU core
13.000 event / sec

12. Fluentd - Built-in Reliability
✓ Buffer
➢ file or memory
✓ Retrying
✓ Error handling
➢ transaction, failover, secondary node support
(heartbeat)

13. Fluentd - Event structure (log)
✓ Time
➢ Second unit
➢ From data source or adding parsed time
✓ Tag
➢ for message routing
✓ Record
➢ JSON format
■ MessagePack internally :)
■ none structured

14. Fluentd - Useful plugins
✓ Output
➢ stdout, file, forest, graphite, mongo, mysql,
elasticsearch, splunk, null, s3, geoip, webhdfs
✓ Input
➢ syslog, tail, http, udp, tcp, scribe
✓ Buffer
➢ memory, file
✓ Formatter and/or Parser
➢ lstv, json, multiline

15. Examples

16. Fluentd - Examples
<source>
type tail
format /^(?<host>[^ ]*):(?<port>[^ ]*) (?<ip>[^ ]*) (?<user>[^ ]*) (?<remotelog>[^ ]*)
[(?<time>[^]]*)] "(?<method>S+)(?: +(?<path>[^ ]*) +S*)?" (?<code>[^ ]*) (?<size>[^ ]*)
(?: "(?<referer>[^"]*)" ""(?<agent>[^"]*)"")?(?: "(?<referer>[^"]*)" "(?<agent>
[^"]*)")?$/
path /var/log/apache2/other_vhosts_access.log.*
pos_file /var/log/fluent/apache2.other_vhosts_access.log.pos
time_format %d/%b/%Y:%H:%M:%S %z
tag apache2.access.raw
read_from_head true
</source>

17. Fluentd - Examples
<match apache2.*.raw>
type record_reformer
enable_ruby false
renew_record false
remove_keys remotelog
tag ${tag_prefix[-2]}.reformed
<record>
hostname ${hostname}
</record>
</match>
<match apache2.*.reformed>
type geoip
geoip_lookup_key ip
geoip_database /usr/share/GeoIP/GeoIPCity.dat
<record>
geo_city ${city['ip']}
...
geo_region ${region['ip']}
</record>
add_tag_suffix .geoip
flush_interval 5s
</match>

18. Fluentd - Examples
<match apache2.access.reformed.geoip>
type forward
flush_interval 5s
buffer_type file
buffer_queue_limit 512
buffer_chunk_limit 100M
buffer_path /opt/fluent/buffer/apache2/
<server>
name hostname
host xxx.xxx.xxx.xxx
weight 10
</server>
...
<server>
name hostname
host xxx.xxx.xxx.xxx
standby
</server>
<secondary>
type file
path /var/log/fluent/forward-failed/
apache2/
</secondary>
</match>

19. Fluentd - Examples
<match apache2.access.**>
type copy
<store>
type file
path /opt/fluent/apache2/access
time_format %Y%m%dT%H%M%S%z
flush_interval 60s
append true
compress gzip
utc
num_threads 4
...
...
</store>
<store>
type datacounter
...
</store>
<store>
type graphite
...
</store>
</match>

20. Fluentd - Testimonials

21. Questions?
http://www.fluentd.org
http://docs.fluentd.org/
http://fluentular.herokuapp.com/
https://rubygems.org/search?query=fluent-plugin-http://
msgpack.org/