Fluent                               Structured logging                               Pluggable architecture              ...
Fluent                               Structured logging                               Pluggable architecture              ...
log
log   log   log   log
log   log   log   log
log   log   log   log
Fluent                            Structured logging                            Pluggable architecture                    ...
Fluent                               Structured logging                               Pluggable architecture              ...
Fluent                               Structured logging                               Pluggable architecture              ...
log   log   log   log
log   log   log   log
log   log
log   log
<match ...>  type mongo_backup  capped_size 100m                          log  <store>                        tcp    type ...
log   log   log   log
<match mongo.**>  type mongo  database fluent  collection test                                       log  # Following atti...
Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 }Mon Nov 14 23:36:22 [conn13] command admin.$cmd...
aggregate       aggregate           aggregate                aggregate     log             log                 log        ...
<source>    type tail    format /^(?<time>[^ ]* [^ ]* [^ ]* [^ ]*) (?<key1>[^ ]*) (?<key2>[^ ]*) (?<key3>[^ ]*)(?<value1>[...
Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 }Mon Nov 14 23:36:22 [conn13] command admin.$cmd...
key1                                            key3           shuffle                                    key2aggregate    ...
Fluent                               Structured logging                               Pluggable architecture              ...
An Introduction to Fluent & MongoDB Plugins
An Introduction to Fluent & MongoDB Plugins
An Introduction to Fluent & MongoDB Plugins
An Introduction to Fluent & MongoDB Plugins
An Introduction to Fluent & MongoDB Plugins
An Introduction to Fluent & MongoDB Plugins
Upcoming SlideShare
Loading in...5
×

An Introduction to Fluent & MongoDB Plugins

6,104

Published on

Published in: Technology, Business
1 Comment
16 Likes
Statistics
Notes
No Downloads
Views
Total Views
6,104
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
101
Comments
1
Likes
16
Embeds 0
No embeds

No notes for slide

An Introduction to Fluent & MongoDB Plugins

  1. 1. Fluent Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki
  2. 2. Fluent Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki
  3. 3. log
  4. 4. log log log log
  5. 5. log log log log
  6. 6. log log log log
  7. 7. Fluent Structured logging Pluggable architecture Reliable forwardinge Event Collector Service
  8. 8. Fluent Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki
  9. 9. Fluent Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki
  10. 10. log log log log
  11. 11. log log log log
  12. 12. log log
  13. 13. log log
  14. 14. <match ...> type mongo_backup capped_size 100m log <store> tcp type tcp host 192.168.0.13 ... </store></match>
  15. 15. log log log log
  16. 16. <match mongo.**> type mongo database fluent collection test log # Following attibutes are optional host fluenter port 10000 # Other buffer configurations here</match>
  17. 17. Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 }Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { replSetGetStatus: 1 } ntoreturn:1 reslen:571 0msMon Nov 14 23:36:22 [conn13] run command admin.$cmd { ismaster: 1 }Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 reslen:234 0msMon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 } Input { _id : ..., time: Mon Nov 14 23:36:22, log key1 : “[conn13]”, key2 : “command”, key3 : ”admin.$cmd”, key4 : { “ismaster”: 1 }, value : “0ms”, } Output
  18. 18. aggregate aggregate aggregate aggregate log log log log key1 key2 key3 shuffle aggregate aggregate aggregate aggregate
  19. 19. <source> type tail format /^(?<time>[^ ]* [^ ]* [^ ]* [^ ]*) (?<key1>[^ ]*) (?<key2>[^ ]*) (?<key3>[^ ]*)(?<value1>[^ ]*)$/ time_format %a %b %e %H:%M:%S path /var/log/something.log tag aggr_hostneme</source> <metrics> <server> name one_key name host1 partition_by m host host1 each_key key1 port 24224 </metrics> </server> <metrics> <server> name two_keys name host2 partition_by m host host2 each_key key2,key3 port 24224 value_key value1 </server> type float ... </metrics>
  20. 20. Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 }Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { replSetGetStatus: 1 } ntoreturn:1 reslen:571 0msMon Nov 14 23:36:22 [conn13] run command admin.$cmd { ismaster: 1 }Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 reslen:234 0msMon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 } Input{ _id : "399e94941cacf13eeb3f808e8ac00981", delta1 delta2 delta3 name : one_key, partition : "2011-11-14 19:17" key : { key1 : "PeriodicTask::Runner" }, delta4 delta5 count : 30, value : { response : 1024 } delta6} Output
  21. 21. key1 key3 shuffle key2aggregate aggregate aggregate ... shard key1 shard key3 shard key2 aggregate aggregate ...
  22. 22. Fluent Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×