An Introduction to Fluent & MongoDB Plugins

7,777 views
7,090 views

Published on

Published in: Technology, Business
1 Comment
16 Likes
Statistics
Notes
No Downloads
Views
Total views
7,777
On SlideShare
0
From Embeds
0
Number of Embeds
230
Actions
Shares
0
Downloads
103
Comments
1
Likes
16
Embeds 0
No embeds

No notes for slide

An Introduction to Fluent & MongoDB Plugins

  1. Fluent Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki
  2. Fluent Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki
  3. log
  4. log log log log
  5. log log log log
  6. log log log log
  7. Fluent Structured logging Pluggable architecture Reliable forwardinge Event Collector Service
  8. Fluent Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki
  9. Fluent Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki
  10. log log log log
  11. log log log log
  12. log log
  13. log log
  14. <match ...> type mongo_backup capped_size 100m log <store> tcp type tcp host 192.168.0.13 ... </store></match>
  15. log log log log
  16. <match mongo.**> type mongo database fluent collection test log # Following attibutes are optional host fluenter port 10000 # Other buffer configurations here</match>
  17. Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 }Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { replSetGetStatus: 1 } ntoreturn:1 reslen:571 0msMon Nov 14 23:36:22 [conn13] run command admin.$cmd { ismaster: 1 }Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 reslen:234 0msMon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 } Input { _id : ..., time: Mon Nov 14 23:36:22, log key1 : “[conn13]”, key2 : “command”, key3 : ”admin.$cmd”, key4 : { “ismaster”: 1 }, value : “0ms”, } Output
  18. aggregate aggregate aggregate aggregate log log log log key1 key2 key3 shuffle aggregate aggregate aggregate aggregate
  19. <source> type tail format /^(?<time>[^ ]* [^ ]* [^ ]* [^ ]*) (?<key1>[^ ]*) (?<key2>[^ ]*) (?<key3>[^ ]*)(?<value1>[^ ]*)$/ time_format %a %b %e %H:%M:%S path /var/log/something.log tag aggr_hostneme</source> <metrics> <server> name one_key name host1 partition_by m host host1 each_key key1 port 24224 </metrics> </server> <metrics> <server> name two_keys name host2 partition_by m host host2 each_key key2,key3 port 24224 value_key value1 </server> type float ... </metrics>
  20. Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 }Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { replSetGetStatus: 1 } ntoreturn:1 reslen:571 0msMon Nov 14 23:36:22 [conn13] run command admin.$cmd { ismaster: 1 }Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 reslen:234 0msMon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 } Input{ _id : "399e94941cacf13eeb3f808e8ac00981", delta1 delta2 delta3 name : one_key, partition : "2011-11-14 19:17" key : { key1 : "PeriodicTask::Runner" }, delta4 delta5 count : 30, value : { response : 1024 } delta6} Output
  21. key1 key3 shuffle key2aggregate aggregate aggregate ... shard key1 shard key3 shard key2 aggregate aggregate ...
  22. Fluent Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki

×