• Like
An Introduction to Fluent & MongoDB Plugins
Upcoming SlideShare
Loading in...5
×

An Introduction to Fluent & MongoDB Plugins

  • 5,839 views
Uploaded on

 

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
5,839
On Slideshare
0
From Embeds
0
Number of Embeds
4

Actions

Shares
Downloads
99
Comments
1
Likes
16

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Fluent Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki
  • 2. Fluent Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki
  • 3. log
  • 4. log log log log
  • 5. log log log log
  • 6. log log log log
  • 7. Fluent Structured logging Pluggable architecture Reliable forwardinge Event Collector Service
  • 8. Fluent Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki
  • 9. Fluent Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki
  • 10. log log log log
  • 11. log log log log
  • 12. log log
  • 13. log log
  • 14. <match ...> type mongo_backup capped_size 100m log <store> tcp type tcp host 192.168.0.13 ... </store></match>
  • 15. log log log log
  • 16. <match mongo.**> type mongo database fluent collection test log # Following attibutes are optional host fluenter port 10000 # Other buffer configurations here</match>
  • 17. Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 }Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { replSetGetStatus: 1 } ntoreturn:1 reslen:571 0msMon Nov 14 23:36:22 [conn13] run command admin.$cmd { ismaster: 1 }Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 reslen:234 0msMon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 } Input { _id : ..., time: Mon Nov 14 23:36:22, log key1 : “[conn13]”, key2 : “command”, key3 : ”admin.$cmd”, key4 : { “ismaster”: 1 }, value : “0ms”, } Output
  • 18. aggregate aggregate aggregate aggregate log log log log key1 key2 key3 shuffle aggregate aggregate aggregate aggregate
  • 19. <source> type tail format /^(?<time>[^ ]* [^ ]* [^ ]* [^ ]*) (?<key1>[^ ]*) (?<key2>[^ ]*) (?<key3>[^ ]*)(?<value1>[^ ]*)$/ time_format %a %b %e %H:%M:%S path /var/log/something.log tag aggr_hostneme</source> <metrics> <server> name one_key name host1 partition_by m host host1 each_key key1 port 24224 </metrics> </server> <metrics> <server> name two_keys name host2 partition_by m host host2 each_key key2,key3 port 24224 value_key value1 </server> type float ... </metrics>
  • 20. Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 }Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { replSetGetStatus: 1 } ntoreturn:1 reslen:571 0msMon Nov 14 23:36:22 [conn13] run command admin.$cmd { ismaster: 1 }Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 reslen:234 0msMon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 } Input{ _id : "399e94941cacf13eeb3f808e8ac00981", delta1 delta2 delta3 name : one_key, partition : "2011-11-14 19:17" key : { key1 : "PeriodicTask::Runner" }, delta4 delta5 count : 30, value : { response : 1024 } delta6} Output
  • 21. key1 key3 shuffle key2aggregate aggregate aggregate ... shard key1 shard key3 shard key2 aggregate aggregate ...
  • 22. Fluent Structured logging Pluggable architecture Reliable forwarding e Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki