An Introduction to Fluent & MongoDB Plugins

Fluent Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki

log log log log

Fluent Structured logging Pluggable architecture Reliable forwardinge Event Collector Service

log log log log

log log

<match ...> type mongo_backup capped_size 100m log <store> tcp type tcp host 192.168.0.13 ... </store></match>

log log log log

<match mongo.**> type mongo database fluent collection test log # Following attibutes are optional host fluenter port 10000 # Other buffer configurations here</match>

Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 }Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { replSetGetStatus: 1 } ntoreturn:1 reslen:571 0msMon Nov 14 23:36:22 [conn13] run command admin.$cmd { ismaster: 1 }Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 reslen:234 0msMon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 } Input { _id : ..., time: Mon Nov 14 23:36:22, log key1 : “[conn13]”, key2 : “command”, key3 : ”admin.$cmd”, key4 : { “ismaster”: 1 }, value : “0ms”, } Output

aggregate aggregate aggregate aggregate log log log log key1 key2 key3 shufﬂe aggregate aggregate aggregate aggregate

<source> type tail format /^(?<time>[^ ]* [^ ]* [^ ]* [^ ]*) (?<key1>[^ ]*) (?<key2>[^ ]*) (?<key3>[^ ]*)(?<value1>[^ ]*)$/ time_format %a %b %e %H:%M:%S path /var/log/something.log tag aggr_hostneme</source> <metrics> <server> name one_key name host1 partition_by m host host1 each_key key1 port 24224 </metrics> </server> <metrics> <server> name two_keys name host2 partition_by m host host2 each_key key2,key3 port 24224 value_key value1 </server> type float ... </metrics>

Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 }Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { replSetGetStatus: 1 } ntoreturn:1 reslen:571 0msMon Nov 14 23:36:22 [conn13] run command admin.$cmd { ismaster: 1 }Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 reslen:234 0msMon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 } Input{ _id : "399e94941cacf13eeb3f808e8ac00981", delta1 delta2 delta3 name : one_key, partition : "2011-11-14 19:17" key : { key1 : "PeriodicTask::Runner" }, delta4 delta5 count : 30, value : { response : 1024 } delta6} Output

key1 key3 shufﬂe key2aggregate aggregate aggregate ... shard key1 shard key3 shard key2 aggregate aggregate ...

http://foreign-business.blogspot.com	29
http://us-w1.rockmelt.com	26
http://a0.twimg.com	23
http://factory.joshfire.com	13
http://www.twylah.com	5
http://www.onlydoo.com	3
http://foreign-business.blogspot.mx	2
http://teknikjawabbmstpm.blogspot.com	1
http://paper.li	1
http://tweetedtimes.com	1
http://www.blogger.com	1
http://factory-joshfire.herokuapp.com	1

An Introduction to Fluent & MongoDB Plugins

by Takahiro Inoue on Nov 14, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

12 Embeds 106

Statistics

An Introduction to Fluent & MongoDB Plugins — Presentation Transcript