This document discusses Fluent, an open source data collector. It has structured logging, a pluggable architecture, and reliable forwarding. It also mentions that Sadayuki Furuhashi of Treasure Data presented information on Fluent and its event collector service.

3. Fluent
Structured logging
Pluggable architecture
Reliable forwarding
e Event Collector Service
Sadayuki Furuhashi
Treasure Data, Inc.
@frsyuki

4. Fluent
Structured logging
Pluggable architecture
Reliable forwarding
e Event Collector Service
Sadayuki Furuhashi
Treasure Data, Inc.
@frsyuki

6. log

7. log log log log

8. log log log log

9. log log log log

11. Fluent
Structured logging
Pluggable architecture
Reliable forwarding
e Event Collector Service

12. Fluent
Structured logging
Pluggable architecture
Reliable forwarding
e Event Collector Service
Sadayuki Furuhashi
Treasure Data, Inc.
@frsyuki

13. Fluent
Structured logging
Pluggable architecture
Reliable forwarding
e Event Collector Service
Sadayuki Furuhashi
Treasure Data, Inc.
@frsyuki

15. log log log log

16. log log log log

17. log log

18. log log

19. <match ...>
type mongo_backup
capped_size 100m
log
<store>
tcp
type tcp
host 192.168.0.13
...
</store>
</match>

20. log log log log

21. <match mongo.**>
type mongo
database fluent
collection test
log
# Following attibutes are optional
host fluenter
port 10000
# Other buffer configurations here
</match>

22. Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 }
Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { replSetGetStatus: 1 } ntoreturn:1 reslen:571 0ms
Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { ismaster: 1 }
Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 reslen:234 0ms
Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 } Input
{
_id : ...,
time: Mon Nov 14 23:36:22,
log
key1 : “[conn13]”,
key2 : “command”,
key3 : ”admin.$cmd”,
key4 : {
“ismaster”: 1
},
value : “0ms”,
} Output

24. aggregate aggregate aggregate aggregate
log log log log
key1 key2 key3 shuffle
aggregate aggregate aggregate
aggregate

25. <source>
type tail
format /^(?<time>[^ ]* [^ ]* [^ ]* [^ ]*) (?<key1>[^ ]*) (?<key2>[^ ]*) (?<key3>[^ ]*)
(?<value1>[^ ]*)$/
time_format %a %b %e %H:%M:%S
path /var/log/something.log
tag aggr_hostneme
</source>
<metrics>
<server>
name one_key
name host1
partition_by m
host host1
each_key key1
port 24224
</metrics>
</server>
<metrics>
<server>
name two_keys
name host2
partition_by m
host host2
each_key key2,key3
port 24224
value_key value1
</server>
type float
...
</metrics>

26. Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 }
Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { replSetGetStatus: 1 } ntoreturn:1 reslen:571 0ms
Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { ismaster: 1 }
Mon Nov 14 23:36:22 [conn13] command admin.$cmd command: { ismaster: 1 } ntoreturn:1 reslen:234 0ms
Mon Nov 14 23:36:22 [conn13] run command admin.$cmd { replSetGetStatus: 1 } Input
{
_id : "399e94941cacf13eeb3f808e8ac00981",
delta1 delta2 delta3
name : one_key,
partition : "2011-11-14 19:17"
key : {
key1 : "PeriodicTask::Runner"
}, delta4 delta5
count : 30,
value : {
response : 1024
} delta6
} Output

27. key1 key3 shuffle
key2
aggregate aggregate aggregate
...
shard key1 shard key3
shard key2
aggregate aggregate
...

28. Fluent
Structured logging
Pluggable architecture
Reliable forwarding
e Event Collector Service
Sadayuki Furuhashi
Treasure Data, Inc.
@frsyuki