Fluentd meetup in japan

Fluentd Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki

• Sadayuki Furuhashi > twitter: @frsyuki• Treasure Data, Inc. > Software Engineer; founder• Author of MessagePack• Author of Fluentd

What’s Fluentd?Its like syslogd, but uses JSON for log messages

What’s Fluentd? Application 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage }

What’s Fluentd? time Application tag 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage } record

What’s Fluentd? Application Fluentd lter / bu er / routing Storage

What’s Fluentd? Application Fluentd lter / bu er / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in

What’s Fluentd? syslogd Scribe Application File Plug-in tailPlug-in Plug-in Fluentd lter / bu er / routing SaaS Storage Fluentd Plug-in Plug-in Plug-in

What’s Fluentd?• Client libraries > Ruby > Perl Application > PHP > Python > Java Fluentd > ...Fluent.open(“myapp”)Fluent.event(“login”, {“user”=>38})#=> 2012-02-04 04:56:01 myapp.login {“user”:38}

Fluentd & Event logsBefore: App server App server App server Application Application Application File File File ... File File File ... File File File ... File High latency must wait for a day Log server Hard to analyze complex text parsers

Fluentd & Event logsAfter: App server App server App server Application Application Application Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd

Fluentd & Event logs Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd Hadoop Amazon Ready to MongoDB / Hive S3 / EMR Analyze!

# receive events via HTTP # save alerts to a file<source> <match alert.**> type http type file port 8888 path /var/log/fluent/alerts</source> </match># read logs from a file # forward other logs to servers<source> # (load-balancing + fail-over) type tail <match **> path /var/log/httpd.log type forward format apache <server> tag apache.access host 192.168.0.11</source> weight 20 </server># save access logs to MongoDB <server><match apache.access> host 192.168.0.12 type mongo weight 60 host 127.0.0.1 </server></match> </match>

Fluentd vs Scribe• Deals with structured logs• Easy to install > “gem install ﬂuentd” > apt-get and yum http://packages.treasure-data.com/• Easy to customize• add/modify plugins without re-compiling > “gem search -rd ﬂuent-plugin”

Fluentd vs Flume• Easy to setup > “sudo fluentd --setup && fluentd”• Very small footprint > small engine (3,000 lines) + plugins• JVM-free• Easy to configure

Architecture of Fluentd

Architecture :: Input

Architecture :: Bu er

Architecture :: Output

Plugins :: out_forwardforward event logs Fluentd Heartbeat out_forward in_forward Fluentd Fluentd ✓ load balancing

Plugins :: out_forwardforward event logs Fluentd Heartbeat out_forwardφ accrual failure detector in_forward Fluentd Fluentd ✓ load balancing

Plugins :: out_copyduplicate event logs Fluentd out_copy out_mongo out_forward out_ le MongoDB File Fluentd

Plugins :: buf_ lereliable bu ering buf_ le Fluentd le le✓ Automatic retry le✓ 2^N retry interval le ✓ Persistent bu er

Plugins :: out_execexecute external programs Fluentd out_exec TSV → stdin ✓ Python external ✓ Perl program ✓ C++

Plugins :: out_exec_ lterexecute external programs out_exec_ lter stdin external Fluentd stdout program out_exec TSV → stdin ✓ Python external ✓ Perl program ✓ C++

Plugins :: in_execexecute external programs in_exec out_exec_ lter external stdout stdin external Fluentd program stdout program out_exec TSV → stdin ✓ Python external ✓ Perl program ✓ C++

Plugins :: in_tailRead event logs from a le Application File /var/log/access.log ✓ Apache log parser in_tail ✓ Syslog parser ✓ Custom parser Fluentd

Plugins :: in_tailApache log parser87.12.1.87 - - [04/Feb/2012:00:20:11 +0900] "GET / HTTP/1.1" 200 9887.12.1.87 - - [04/Feb/2012:00:20:11 +0900] "GET / HTTP/1.1" 200 98... { “host”: “87.12.1.87”, “method”: “GET”, “code”: 200, “size”: 98, “path”: “/” } ...

Plugins• Bundled plugins > file writes event logs to files hourly or daily > forward forwards event logs (+fail-over and load balancing) > exec passes event logs to/from external commands > tail reads event logs from a file (like `tail -f`)

Plugins• 3rd party plugins > scribe integrates Fluentd with Scribe > s3 uploads log ﬁles to Amazon S3 hourly or daily > mongo writes logs to MongoDB > hoop puts log ﬁles on Hadoop HDFS via Hoop ...

Plugin developer API• Unit test framework (like “MRUnit”) > Fluent::Test::InputTestDriver > Fluent::Test::OutputTestDriver > Fluent::Test::BufferedOutputTestDriver• Fluent::TailInput (base class of “tail” plugin) > text parser is customizable def parse_line(line)

Fluentd• Documents > http://fluentd.org• Source code > http://github.com/fluent• Twitter > #fluentd• Mailing list > http://groups.google.com/group/fluentd

http://docs.fluentd.org	2250
http://www.scoop.it	376
http://rg443blog.wordpress.com	37
http://localhost	17
http://us-w1.rockmelt.com	4
https://twitter.com	4
http://tweetedtimes.com	1
http://www.twylah.com	1

Fluentd meetup in japan

by Treasure Data, Inc.

Accessibility

Categories

Upload Details

Usage Rights

8 Embeds 2,690

Statistics