Fluentd loves MongoDB, at MongoDB SV User Group, July 17, 2012

Fluentd ♥ MongoDB Log Everything As JSON Kazuki Ohta, CTO at Treasure Data, Inc.Tuesday, July 17, 2012

Self-Introduction • Kazuki Ohta > twitter: @kzk_mover > github: kzk • Treasure Data, Inc. > Chief Technology Ofﬁcer; Founder > Original Fluentd Author @frsyuki is another co-founder. • Open-Source Enthusiast > KDE, uim, Hadoop, memcached, Mozilla, Mongo, etc. > Fluentd rpm/deb package manager 2Tuesday, July 17, 2012

Logging? Why?Tuesday, July 17, 2012

Figure 1: Common Logging Purposes Analytics Error Notiﬁcation Recommendation 4Tuesday, July 17, 2012

Figure 2: Types of Logs App Log Access Log (Apache, Rails, etc.) System Log (syslog etc.) Others 5Tuesday, July 17, 2012

From “Scaling Lessons learned at Dropbox” 6Tuesday, July 17, 2012

Fragile for format change, No type information, No ﬁeld name, etc. From “Scaling Lessons learned at Dropbox” 6Tuesday, July 17, 2012

About FluentdTuesday, July 17, 2012

Its like syslogd, but uses JSON for log messages 8Tuesday, July 17, 2012

Logs in JSON? Why? 1. Machine-Readable > machine is goint to be a main consumer of logs 2. Schema-Free > you want to add/remove fields from logs at anytime Write Logs for Machines, use JSON http://journal.paul.querna.org/articles/2011/12/26/log-for-machines-in-json/ 9Tuesday, July 17, 2012

Logs As TEXT Logs As JSON + Field Name + No Custom Parser + Type Information + Schema Free 10Tuesday, July 17, 2012

Logs As TEXT “2011-04-01 host1 myapp: cmessage size=12MB user=me” Logs As JSON 2011-04-01 myapp.message { “on_host”: ”host1”, ”combined”: true, “size”: 12000000, + Field Name “user”: “me” + No Custom Parser + Type Information } + Schema Free 10Tuesday, July 17, 2012

http://ﬂuentd.org/ 11Tuesday, July 17, 2012

• Website > http://ﬂuentd.org/ • Community > http://github.com/ﬂuent > 16 committers across many organizations > web, game, enterprise • Mailing list > Google groups 12Tuesday, July 17, 2012

Fluentd ArchitectureTuesday, July 17, 2012

Fluentd: Log Format Application Fluentd Storage 14Tuesday, July 17, 2012

Fluentd: Log Format Application 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage } 14Tuesday, July 17, 2012

Fluentd: Log Format time Application tag 2012-02-04 01:33:51 myapp.buylog { Fluentd “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” Storage } record 14Tuesday, July 17, 2012

Fluentd: Plugins Application ﬁlter / buffer / Fluentd routing Storage 15Tuesday, July 17, 2012

Fluentd: Plugins Application ﬁlter / buffer / Fluentd routing SaaS Storage Fluentd Plug-in Plug-in Plug-in 15Tuesday, July 17, 2012

Fluentd: Plugins Application ﬁlter / buffer / Fluentd routing SaaS Storage Fluentd Plug-in Plug-in Plug-in 16Tuesday, July 17, 2012

Fluentd: Plugins syslogd Scribe Application File Plug-in tail Plug-in Plug-in ﬁlter / buffer / Fluentd routing SaaS Storage Fluentd Plug-in Plug-in Plug-in 16Tuesday, July 17, 2012

• Client libraries > Ruby > Perl Application Buffering > PHP HTTP / TCP / UDS > Python > Java Fluentd > ... 17Tuesday, July 17, 2012

• Client libraries > Ruby > Perl Application Buffering > PHP HTTP / TCP / UDS > Python > Java Fluentd > ... Fluent.open(“myapp”) Fluent.event(“login”, {“user”=>38}) #=> 2012-02-04 04:56:01 myapp.login {“user”:38} 17Tuesday, July 17, 2012

Typical Log Collection by `rsync` Burst of traffic rsync consumes all bandwidth 18Tuesday, July 17, 2012

Typical Log Collection by `rsync` App server App server App server Application Application Application File File File ... File File File ... File File File ... File Burst of traffic High latency rsync consumes must wait for a day all bandwidth Log server Hard to analyze complex text parsers 18Tuesday, July 17, 2012

Log Collection using Fluentd Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd 19Tuesday, July 17, 2012

Log Collection using Fluentd Fluentd Fluentd Fluentd Realtime! Fluentd Fluentd Amazon Ready to Hadoop Mongo S3 / / Hive DB EMR Analyze! 19Tuesday, July 17, 2012

Fluentd Case Study Ruby on Rails Ruby on Rails Ruby on Rails Fluentd Fluentd Fluentd ✓ 127 RoR servers ✓ 100,000 msgs/sec Fluentd Fluentd routing ✓ 120Mbps at peak ✓ 1TB/day Hadoop Mongo User behavior PV logs / Hive DB logs 20Tuesday, July 17, 2012

# read logs from a file # forward other logs to servers <source> # (load-balancing + fail-over) type tail <match **> path /var/log/httpd.log type forward format apache <server> tag apache.access host 192.168.0.11 </source> weight 20 </server> # save access logs to MongoDB <server> <match apache.access> host 192.168.0.12 type mongo weight 60 host 127.0.0.1 </server> </match> </match>Tuesday, July 17, 2012

ComparisonTuesday, July 17, 2012

Scribe: log collector by Facebook Frontend servers Aggregator nodes scribe scribe scribe Hadoop HDFS scribe scribe scribe 23Tuesday, July 17, 2012

Scribe’s Pros & Cons • Pros. • Fast (written in C++) • Cons. • VERY HARD to install • nightmare of boost, thrift, libhdfs, etc. • Unstructured Logs • parsing must be required before the analysis • Hard to extend • recompiling C++ programs are required • No longer maintained 24Tuesday, July 17, 2012

Fluentd vs Scribe • Easy to install • “gem install fluentd” • Stable RPM and Deb packages • http://packages.treasure-data.com/ • Easy to write plugins • you can use Ruby • Easy plugin distribution • “gem search -rd fluent-plugin” 25Tuesday, July 17, 2012

Flume: distributed log collector by Cloudera Phisical Flume Master Topology Flume Flume Flume Logical Hadoop Topology HDFS 26Tuesday, July 17, 2012

Flume’s Pros & Cons • Pros. • Central master server manages all nodes • Cons. • Difficult to understand • logical topologies, phisical servers and a configuration of the logical/phisical mapping • Difficult to configure • replicated master servers, log servers and agents • Big footprint • 50,000 lines of Java 27Tuesday, July 17, 2012

Fluentd vs Flume • Easy to understand • “syslogd that understands JSON” • Easy to setup • “sudo fluentd --setup && fluentd” • Very small footprint • small engine (3,000) lines + plugins • small, but battle-tested! • Easy to configure 28Tuesday, July 17, 2012

Fluentd Scribe Flume Installation gem/rpm/deb make jar/rpm/deb 3000 lines of 8000 lines of 50,000 lines of Footprint Ruby C++ Java Plugin Ruby N/A Java Plugin distribution RubyGems.org N/A N/A Master Server No No Yes License Apache License Apache License Apache License 29Tuesday, July 17, 2012

Fluentd Plugin forTuesday, July 17, 2012

fluent-plugin-mongo • Included within rpm/deb by default! • http://github.com/fluent/fluent-plugin-mongo • #1 plugin among 50+ Fluentd plugins • Logs As JSON. WHY NOT Put Them Into Mongo?? • http://fluentd.org/plugin/ • Supports most of the MongoDB features • Authentication • ReplicaSet • Capped Collection 31Tuesday, July 17, 2012

• MongoDB Output Plugin Application • Maintain JSON Structure • Reliable Buffering • Batch Insertion Fluentd Buffering • Handle Broken Records • Ruby Driver #82 Authentication MongoDB MongoDB MongoDB MongoDB MongoDB MongoDB Single Instance MongoDB MongoDB (Capped or Not) MongoDB MongoDB Sharding ReplicaSet 32Tuesday, July 17, 2012

ReplicaSet (Capped Collection) Single Instance (Capped Collection) MongoDB MongoDB MongoDB MongoDB Authentication Fluentd Buffering • MongoDB Input Plugin • Tailing Capped Collections 33Tuesday, July 17, 2012

Realtime Analytics with Fluentd + MongoDB App App App Fluentd Fluentd Fluentd routing Fluentd Fluentd Nagios, Zabbix, etc. Mongo query Charting Alert DB 34Tuesday, July 17, 2012

Realtime or Batch? No, BOTH! App App App Fluentd Fluentd Fluentd routing Fluentd Fluentd Hadoop Amazon Mongo query Charting / Hive S3 DB batch archive realtime 35Tuesday, July 17, 2012

Intro of our company’s service: Treasure Data App App App Fluentd Fluentd Fluentd routing Fluentd Fluentd Treasure Mongo Hadoop-based Data DB Cloud Data Warehouse batch realtime 36Tuesday, July 17, 2012

Exercise: Apache Logs into MongoDBTuesday, July 17, 2012

Log File 38Tuesday, July 17, 2012

39Tuesday, July 17, 2012

40Tuesday, July 17, 2012

Conclusion • Log Everything as JSON • Machine Readability • Schema Freeness • MongoDB fits into Fluentd’s backend perfectly • Both using JSON representation 41Tuesday, July 17, 2012