Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Dive into Fluentd plugin v0.12

12,044 views

Published on

Update version of Fluentd plugin guide : http://www.slideshare.net/repeatedly/fluentd-meetup-dive-into-fluent-plugin

Published in: Technology

Dive into Fluentd plugin v0.12

  1. 1. Dive into Fluentd Plugin
  2. 2. Site: repeatedly.github.com Company: Treasure Data Inc. Love plugins: input: tail
 buffer: memory
 output: mongo
  3. 3. developed by The missing log collector What's Fluentd?
  4. 4. Fluentd is a buffer router collector converter aggregator etc...
  5. 5. ... but, Fluentd doesn’t have such features as a built-in.
  6. 6. Instead, Fluentd has flexible plugin architecture which consists of Input, Filter, Output and Buffer.
  7. 7. We can customize Fluentd using plugins :)
  8. 8. Agenda: This slide talk about - an example of Fluentd plugins - Fluentd and libraries - how to develop a Fluentd plugin don’t talk about - the details of each plugin - the experience of production
  9. 9. See also:
 http://docs.fluentd.org/articles/plugin-development
  10. 10. Example based on bit.ly/fluentd-with-mongo
  11. 11. Install Plugin name is , and fluent-gem is included in Fluentd gem. fluent-plugin-xxx
  12. 12. Let’s type! $ fluent-gem install fluent-plugin-mongo
  13. 13. <source> type tail format apache path /path/to/log tag mongo.apache </source> <match mongo.**> type mongo database apache collection access host otherhost </match> Input Output fluentd.conf
  14. 14. Start! $ fluentd -c fluentd.conf 2015-10-04 00:00:14 +0900: starting fluentd-0.12.16 2015-10-04 00:00:14 +0900: reading config file path="fluentd.conf" 2015-10-04 00:00:14 +0900: adding source type="tail" 2015-10-04 00:00:14 +0900: adding match pattern="mongo.**" type="mongo" █
  15. 15. Attack! $ ab -n 100 -c 10 http://localhost/
  16. 16. $ mongo --host otherhost > use apache > db.access.find() { "type": "127.0.0.1", "method": "GET", "path": "/", "code": "200", "size": "44", "time": ISODate("2015-10-04T00:01:00Z") ... } has more...
  17. 17. Mongo Apache Fluentd write tail insert I’m a log! event buffering
  18. 18. Warming up
  19. 19. Ruby Fluentd Stack OS Cool.io MessagePack Buffer Input Outpu
  20. 20. Ruby ruby-lang.org
  21. 21. Fluentd and plugins are written in Ruby.
  22. 22. Fluentd works on Ruby 1.9.3 or later
  23. 23. Note that Fluentd v0.14 works on Ruby 2.1 or later. We will remove 1.9 or 2.0 support.
  24. 24. MessagePack msgpack.org
  25. 25. Serialization: JSON like fast and compact format. RPC: Async and parallelism for high performance. IDL: Easy to integrate and maintain the service.
  26. 26. Binary format, Header + Body, and Variable length.
  27. 27. Note that Ruby version can’t handle a Time object natively.
  28. 28. So, we use an Integer object , second unit, instead of a Time for now.
  29. 29. Since v0.14, Fluentd supports nano-second unit with new EventTime object.
 https://github.com/fluent/fluentd/pull/653
  30. 30. Source: github.com/msgpack Wiki: wiki.msgpack.org/display/MSGPACK Mailing List: groups.google.com/group/msgpack
  31. 31. Cool.io coolio.github.com
  32. 32. Event driven framework built on top of libev.
  33. 33. Cool.io has Loop and Watchers with Transport wrappers. Lots of input plugins use cool.io.
  34. 34. Configuration
  35. 35. Fluentd loads plugins from $LOAD_PATH.
  36. 36. Input: $LOAD_PATH/fluent/plugin/in_<type>.rb Buffer: $LOAD_PATH/fluent/plugin/buf_<type>.rb Output: $LOAD_PATH/fluent/plugin/out_<type>.rb Filter: $LOAD_PATH/fluent/plugin/filter_<type>.rb
  37. 37. We use ‘register_xxx’ to register a plugin. ’xxx’ is ’input’, ’filter’, ’buffer’ and ’output’
  38. 38. We can load the plugin configuration using
 config_param and configure method. config_param set config value to @<config name> automatically. Supported types are below: http://docs.fluentd.org/articles/config- file#supported-data-types-for-values
  39. 39. class TailInput < Input Plugin.register_input(’tail’, self) config_param :path, :string ... end <source> type tail path /path/to/log ... </source> fluentd.conf in_tail.rb
  40. 40. One trick is here: Fluentd’s configuration module does not verify a default value. So, we can use the nil like Tribool :) config_param :tag, :string, :default => nil Fluentd does not check the type
  41. 41. Fluentd provides some useful mixins for input and output plugins.
  42. 42. SetTagKeyMixin: Provide ‘tag_key’ and ‘include_tag_key’. SetTimeKeyMixin: Provide ‘time_key’ and ‘include_time_key’. HandleTagNameMixin: Provide ‘remove_tag_prefix’ and ‘add_tag_prefix’ ‘remove_tag_suffix’ and ‘add_tag_suffix’
  43. 43. Code Flow Mixin usage class MongoOutput < BufferedOutput ... include SetTagKeyMixin config_set_default :include_tag_key, false ... end MongoOutput SetTagKeyMixin BufferedOutput super super super
  44. 44. Input
  45. 45. Default 3rd party Available plugins exec forward http syslog tail monitor_agent etc... mongo_tail scribe msgpack dstat kafka amqp2 etc...
  46. 46. class NewInput < Input ... def configure(conf) # parse a configuration manually end def start # invoke action end def shutdown # cleanup resources end end
  47. 47. In action method, we use router.emit to input data. tag = "app.tag" time = Engine.now record = {"key" => "value", ...} router.emit(tag, time, record) Sample:
  48. 48. How to read an input in an efficient way? We use a thread and an event loop.
  49. 49. class ForwardInput < Fluent::Input ... def start ... @thread = Thread.new(&method(:run)) end def run ... end end Thread
  50. 50. class ForwardInput < Fluent::Input ... def start @loop = Coolio::Loop.new @lsock = listen @loop.attach(@lsock) ... end ... end Event loop
  51. 51. Note that We must use Engine.now instead of Time.now
  52. 52. Filter
  53. 53. Default 3rd party Available plugins grep stdout record_transformer parser geoip record_map typecast record_modifier etc...
  54. 54. class NewFilter < Filter # configure, start and shutdown # are same as input plugin def filter(tag, time, record) # Modify record and return it. # If returns nil, that records are ignored record end end
  55. 55. Buffer
  56. 56. Default 3rd party Available plugins memory file
  57. 57. In most cases, Memory and File are enough.
  58. 58. Memory type is default. It’s fast but can’t resume data.
  59. 59. File type is persistent type. It can resume data from file.
 TimeSlicedOutput’s default is file.
  60. 60. Output
  61. 61. Default 3rd party Available plugins copy exec file forward null stdout etc... mongo s3 scribe elasticsearch webhdfs kafka Norikra etc...
  62. 62. class NewOutput < BufferedOutput # configure, start and shutdown # are same as input plugin def format(tag, time, record) # convert event to raw string end def write(chunk) # write chunk to target # chunk has multiple formatted data end end
  63. 63. Output has 3 buffering modes. None Buffered ObjectBuffered Time sliced
  64. 64. Buffered Time sliced Buffering type chunk go out chunk chunk from in chunk limit queue limit Buffer has an internal map to manage a chunk. A key is “” in Buffered, but a key is time slice in TimeSliced buffer. def write(chunk) # chunk.key is time slice end
  65. 65. Test
  66. 66. Input: Fluent::Test::InputTestDriver Buffer: Fluent::Test::BufferedOutputTestDriver Output: Fluent::Test::OutputTestDriver Filter: Fluent::Test::FilterTestDriver
  67. 67. class MongoOutputTest < Test::Unit::TestCase def setup Fluent::Test.setup require 'fluent/plugin/out_mongo' end def create_driver(conf = CONFIG) Fluent::Test::BufferedOutputTestDriver.new (Fluent::MongoOutput) { def start # prevent external access super end ... }.configure(conf) end
  68. 68. ... def test_format # test format using emit and expect_format end def test_write d = create_driver t = emit_documents(d) # return a result of write method collection_name, documents = d.run assert_equal([{...}, {...}, ...], documents) assert_equal('test', collection_name) end ... end
  69. 69. Release
  70. 70. Gem Structure Plugin root |-- lib/ | |-- fluent/ | |-- plugin/ | |- out_<name>.rb |- Gemfile |- fluent-plugin-<name>.gemspec |- Rakefile |- README.md(rdoc) |- VERSION
  71. 71. Bundle with git $ edit lib/fluent/plugin/out_<name>.rb $ git add / commit $ cat VERSION 0.1.0 $ bunlde exec rake release See: rubygems.org/gems/fluent-plugin-<name>
  72. 72. See released plugins for more details about each file.

×