Update version of Fluentd plugin guide : http://www.slideshare.net/repeatedly/fluentd-meetup-dive-into-fluent-plugin

1. Dive into Fluentd Plugin

2. Site:
repeatedly.github.com
Company:
Treasure Data Inc.
Love plugins:
input: tail
buffer: memory
output: mongo

3. developed by
The missing log collector
What's Fluentd?

4. Fluentd is a
buffer
router
collector
converter
aggregator
etc...

5. ... but,
Fluentd doesn’t have such features as a built-in.

6. Instead,
Fluentd has flexible plugin architecture which
consists of Input, Filter, Output and Buffer.

7. We can customize Fluentd using plugins :)

8. Agenda: This slide
talk about
- an example of Fluentd plugins
- Fluentd and libraries
- how to develop a Fluentd plugin
don’t talk about
- the details of each plugin
- the experience of production

9. See also:
http://docs.fluentd.org/articles/plugin-development

10. Example
based on bit.ly/fluentd-with-mongo

11. Install
Plugin name is ,
and fluent-gem is included in Fluentd gem.
fluent-plugin-xxx

12. Let’s type!
$ fluent-gem install fluent-plugin-mongo

13. <source>
type tail
format apache
path /path/to/log
tag mongo.apache
</source>
<match mongo.**>
type mongo
database apache
collection access
host otherhost
</match>
Input Output
fluentd.conf

14. Start!
$ fluentd -c fluentd.conf
2015-10-04 00:00:14 +0900: starting fluentd-0.12.16
2015-10-04 00:00:14 +0900: reading config file path="fluentd.conf"
2015-10-04 00:00:14 +0900: adding source type="tail"
2015-10-04 00:00:14 +0900: adding match pattern="mongo.**" type="mongo"
█

15. Attack!
$ ab -n 100 -c 10 http://localhost/

16. $ mongo --host otherhost
> use apache
> db.access.find()
{
"type": "127.0.0.1",
"method": "GET",
"path": "/",
"code": "200",
"size": "44",
"time": ISODate("2015-10-04T00:01:00Z")
...
}
has more...

17. Mongo
Apache
Fluentd
write
tail
insert
I’m a log!
event
buffering

18. Warming up

19. Ruby
Fluentd Stack
OS
Cool.io
MessagePack
Buffer
Input
Outpu

20. Ruby
ruby-lang.org

21. Fluentd and plugins are written in Ruby.

22. Fluentd works on Ruby 1.9.3 or later

23. Note that
Fluentd v0.14 works on Ruby 2.1 or later.
We will remove 1.9 or 2.0 support.

24. MessagePack
msgpack.org

25. Serialization:
JSON like fast and compact format.
RPC:
Async and parallelism for high performance.
IDL:
Easy to integrate and maintain the service.

26. Binary format,
Header + Body,
and
Variable length.

27. Note that Ruby version can’t handle
a Time object natively.

28. So, we use an Integer object , second unit,
instead of a Time for now.

29. Since v0.14, Fluentd supports nano-second
unit with new EventTime object.
https://github.com/fluent/fluentd/pull/653

30. Source:
github.com/msgpack
Wiki:
wiki.msgpack.org/display/MSGPACK
Mailing List:
groups.google.com/group/msgpack

31. Cool.io
coolio.github.com

32. Event driven framework built on top of libev.

33. Cool.io has Loop and Watchers with
Transport wrappers.
Lots of input plugins use cool.io.

34. Configuration

35. Fluentd loads plugins from $LOAD_PATH.

36. Input:
$LOAD_PATH/fluent/plugin/in_<type>.rb
Buffer:
$LOAD_PATH/fluent/plugin/buf_<type>.rb
Output:
$LOAD_PATH/fluent/plugin/out_<type>.rb
Filter:
$LOAD_PATH/fluent/plugin/filter_<type>.rb

37. We use ‘register_xxx’ to register a plugin.
’xxx’ is ’input’, ’filter’, ’buffer’ and ’output’

38. We can load the plugin configuration using
config_param and configure method.
config_param set config value to
@<config name> automatically.
Supported types are below:
http://docs.fluentd.org/articles/config-
file#supported-data-types-for-values

39. class TailInput < Input
Plugin.register_input(’tail’, self)
config_param :path, :string
...
end
<source>
type tail
path /path/to/log
...
</source> fluentd.conf
in_tail.rb

40. One trick is here:
Fluentd’s configuration module does not
verify a default value. So,
we can use the nil like Tribool :)
config_param :tag, :string, :default => nil
Fluentd does not check the type

41. Fluentd provides some useful mixins for
input and output plugins.

42. SetTagKeyMixin:
Provide ‘tag_key’ and ‘include_tag_key’.
SetTimeKeyMixin:
Provide ‘time_key’ and ‘include_time_key’.
HandleTagNameMixin:
Provide
‘remove_tag_prefix’ and ‘add_tag_prefix’
‘remove_tag_suffix’ and ‘add_tag_suffix’

43. Code Flow
Mixin usage
class MongoOutput <
BufferedOutput
...
include SetTagKeyMixin
config_set_default
:include_tag_key, false
...
end MongoOutput
SetTagKeyMixin
BufferedOutput
super
super
super

44. Input

45. Default 3rd party
Available plugins
exec
forward
http
syslog
tail
monitor_agent
etc...
mongo_tail
scribe
msgpack
dstat
kafka
amqp2
etc...

46. class NewInput < Input
...
def configure(conf)
# parse a configuration manually
end
def start
# invoke action
end
def shutdown
# cleanup resources
end
end

47. In action method,
we use router.emit to input data.
tag = "app.tag"
time = Engine.now
record = {"key" => "value", ...}
router.emit(tag, time, record)
Sample:

48. How to read an input in an efficient way?
We use a thread and an event loop.

49. class ForwardInput < Fluent::Input
...
def start
...
@thread = Thread.new(&method(:run))
end
def run
...
end
end
Thread

50. class ForwardInput < Fluent::Input
...
def start
@loop = Coolio::Loop.new
@lsock = listen
@loop.attach(@lsock)
...
end
...
end
Event loop

51. Note that
We must use Engine.now instead of Time.now

52. Filter

53. Default 3rd party
Available plugins
grep
stdout
record_transformer
parser
geoip
record_map
typecast
record_modifier
etc...

54. class NewFilter < Filter
# configure, start and shutdown
# are same as input plugin
def filter(tag, time, record)
# Modify record and return it.
# If returns nil, that records are ignored
record
end
end

55. Buffer

56. Default 3rd party
Available plugins
memory
file

57. In most cases,
Memory and File are enough.

58. Memory type is default.
It’s fast but can’t resume data.

59. File type is persistent type.
It can resume data from file.
TimeSlicedOutput’s default is file.

60. Output

61. Default 3rd party
Available plugins
copy
exec
file
forward
null
stdout
etc...
mongo
s3
scribe
elasticsearch
webhdfs
kafka
Norikra
etc...

62. class NewOutput < BufferedOutput
# configure, start and shutdown
# are same as input plugin
def format(tag, time, record)
# convert event to raw string
end
def write(chunk)
# write chunk to target
# chunk has multiple formatted data
end
end

63. Output has 3 buffering modes.
None
Buffered
ObjectBuffered
Time sliced

64. Buffered Time sliced
Buffering type
chunk
go out
chunk
chunk
from in
chunk
limit
queue
limit
Buffer has an internal
map to manage a chunk.
A key is “” in Buffered,
but a key is time slice in
TimeSliced buffer.
def write(chunk)
# chunk.key is time slice
end

65. Test

66. Input:
Fluent::Test::InputTestDriver
Buffer:
Fluent::Test::BufferedOutputTestDriver
Output:
Fluent::Test::OutputTestDriver
Filter:
Fluent::Test::FilterTestDriver

67. class MongoOutputTest < Test::Unit::TestCase
def setup
Fluent::Test.setup
require 'fluent/plugin/out_mongo'
end
def create_driver(conf = CONFIG)
Fluent::Test::BufferedOutputTestDriver.new
(Fluent::MongoOutput) {
def start # prevent external access
super
end
...
}.configure(conf)
end

68. ...
def test_format
# test format using emit and expect_format
end
def test_write
d = create_driver
t = emit_documents(d)
# return a result of write method
collection_name, documents = d.run
assert_equal([{...}, {...}, ...], documents)
assert_equal('test', collection_name)
end
...
end

69. Release

70. Gem Structure
Plugin root
|-- lib/
| |-- fluent/
| |-- plugin/
| |- out_<name>.rb
|- Gemfile
|- fluent-plugin-<name>.gemspec
|- Rakefile
|- README.md(rdoc)
|- VERSION

71. Bundle with git
$ edit lib/fluent/plugin/out_<name>.rb
$ git add / commit
$ cat VERSION
0.1.0
$ bunlde exec rake release
See: rubygems.org/gems/fluent-plugin-<name>

72. See released plugins
for more details about each file.