Who are you?
• Masahiro Nakagawa
• github: @repeatedly
• Treasure Data Inc.
• Fluentd / td-agent developer
• Fluentd Enterprise support
• I love OSS :)
• D Language, MessagePack, The organizer of several meetups, etc…
• Agent for each purpose by Elastic
• ofﬁcial: topbeat, ﬁlebeat, packetbeat
• 3rd party: dockerbeat, nginxbeat, etc…
• Beats support several outputs: elasticsearch,
logstash, stdout and etc.
• logstash output uses lumberjack protocol so
we can use it for communicating with Beats.
• Pluggable streaming event collector
• Lightweight, robust and ﬂexible
• Lots of plugins on rubygems
• Used by AWS, GCP, MS and more companies
• Webinar: https://www.youtube.com/watch?v=6uPB_M7cbYk
• Input plugin for Elastic Beats
• Use lumberjack protocol to handle events
• Tested with topbeat, ﬁlebeat, packetbeat
• Beats use same event format so it should work
with 3rd party Beats.
• Tested on Mac Book Pro, not 2 machines.
2.6 GHz Intel Core i7, 16 GB 1600 MHz DDR3
ﬂuentd with in_tail ﬂuent-agent-hydra ﬁlebeat
80,000 events/sec 100,000+ events/sec 18,000 events/sec
Read nginx 100000 logs and count by ﬂowcounter_simple
1. Lumberjack protocol doesn’t focus on throughput
• lumberjack sends/receives ack on each record
2. Beats framework is slow? [Issue #587]
• ﬁlebeat is slower than logstash-forwarder
Why ﬁlebeat is slow?
• Beats are useful for collecting various metrics
• ﬂuent-plugin-beats can handle Beats event
and route events to elasitcsearch properly
• Thanks ﬂuent-plugin-elasticsearch plugin ;)
• Note that ﬁlebeat is slow so it is not good
on high volume environment
• Use ﬂuentd or ﬂuent-agent-hydra instead