Fluentd meetup dive into fluent plugin (outdated)

28,625 views

Published on

Fluentd meetup in Japan. I talked about "Dive into Fluent plugin".

Some contents are outdated. See this slide: http://www.slideshare.net/repeatedly/dive-into-fluentd-plugin-v012

Published in: Technology
1 Comment
43 Likes
Statistics
Notes
No Downloads
Views
Total views
28,625
On SlideShare
0
From Embeds
0
Number of Embeds
16,671
Actions
Shares
0
Downloads
181
Comments
1
Likes
43
Embeds 0
No embeds

No notes for slide

Fluentd meetup dive into fluent plugin (outdated)

  1. 1. Dive into Fluent P lugin2012 2 4
  2. 2. Site: repeatedly.github.com Company: Preferred Infrastructure, Inc. Love plugins: input: tail buffer: memory output: mongo2012 2 4
  3. 3. Whats Fluentd? The missing log collector See keynote developed by2012 2 4
  4. 4. Fluentd is a buffer router collector converter aggregator etc...2012 2 4
  5. 5. ... but, Fluentd doesn’t have such features as a built-in.2012 2 4
  6. 6. Instead, Fluentd has flexible plugin architecture which consists of Input, Output and Buffer.2012 2 4
  7. 7. We can customize Fluentd using plugins :)2012 2 4
  8. 8. Agenda Yes, I talk about - an example of Fluentd plugins - Fluentd and libraries - how to develop a Fluentd plugins No, I don’t talk about - the details of each plugin - the experience of production2012 2 4
  9. 9. Example based on bit.ly/fluentd-with-mongo2012 2 4
  10. 10. Install Plugin name is fluent-plugin-xxx , and fluent-gem is included in Fluentd gem.2012 2 4
  11. 11. Let’s type! $ fluent-gem install fluent-plugin-mongo2012 2 4
  12. 12. Me! Many Many Many Plugins!2012 2 4
  13. 13. fluentd.conf Input Output <source> <match mongo.**> type tail type mongo format apache database apache path /path/to/log collection access tag mongo.apache host otherhost </source> </match>2012 2 4
  14. 14. Start! $ fluentd -c fluentd.conf 2012-02-04 00:00:14 +0900: starting fluentd-0.10.8 2012-02-04 00:00:14 +0900: reading config file path="fluentd.conf" 2012-02-04 00:00:14 +0900: adding source type="tail" 2012-02-04 00:00:14 +0900: adding match pattern="mongo.**" type="mongo"2012 2 4
  15. 15. Attack! $ ab -n 100 -c 10 http://localhost/2012 2 4
  16. 16. $ mongo --host otherhost > use apache > db.access.find() { "type": "127.0.0.1", "method": "GET", "path": "/", "code": "200", "size": "44", "time": ISODate("2011-11-27T07:56:27Z") ... } has more...2012 2 4
  17. 17. Apache I’m a log! tail write insert Fluentd event buffering Mongo2012 2 4
  18. 18. Warming up2012 2 4
  19. 19. Fluentd Stack Output Input Buffer Ruby MessagePack Cool.io OS2012 2 4
  20. 20. Ruby ruby-lang.org2012 2 4
  21. 21. Fluentd and plugins are written in Ruby.2012 2 4
  22. 22. ... but note that Fluentd works on Ruby 1.9, goodbye 1.8!2012 2 4
  23. 23. MessagePack msgpack.org2012 2 4
  24. 24. Serialization: JSON like fast and compact format. RPC: Async and parallelism for high performance. IDL: Easy to integrate and maintain the service.2012 2 4
  25. 25. Binary format, Header + Body, and Variable length.2012 2 4
  26. 26. Note that Ruby version can’t handle a Time object.2012 2 4
  27. 27. So, we use an Integer object instead of a Time.2012 2 4
  28. 28. Source: github.com/msgpack Wiki: wiki.msgpack.org/display/MSGPACK Mailing List: groups.google.com/group/msgpack2012 2 4
  29. 29. Cool.io coolio.github.com2012 2 4
  30. 30. Event driven framework built on top of libev.2012 2 4
  31. 31. Cool.io has Loop and Watchers with Transport wrappers.2012 2 4
  32. 32. Fluentd has a default event loop. We can use @default_loop in the plugin.2012 2 4
  33. 33. Configuration2012 2 4
  34. 34. Fluentd loads plugins from $LOAD_PATH.2012 2 4
  35. 35. Input: $LOAD_PATH/fluent/plugin/in_<type>.rb Buffer: $LOAD_PATH/fluent/plugin/buf_<type>.rb Output: $LOAD_PATH/fluent/plugin/out_<type>.rb2012 2 4
  36. 36. We use ‘register_input’, ‘register_buffer’ and ‘register_output’ to register a plugin.2012 2 4
  37. 37. We can load the plugin configuration using config_param and configure method. config_param set config value to @<config name> automatically.2012 2 4
  38. 38. <source> type tail path /path/to/log ... </source> fluentd.conf class TailInput < Input Plugin.register_input(’tail’, self) config_param :path, :string ... end in_tail.rb2012 2 4
  39. 39. One trick is here: Fluentd’s configuration module does not verify a default value. So, we can use the nil like Tribool :) config_param :tag, :string, :default => nil Fluentd does not check the type2012 2 4
  40. 40. Fluentd provides some useful mixins for input and output plugins.2012 2 4
  41. 41. SetTagKeyMixin: Provide ‘tag_key’ and ‘include_tag_key’. SetTimeKeyMixin: Provide ‘time_key’ and ‘include_time_key’. DetachMultiProcessMixin: Provide ‘detach_process’ and execute an action in the multi-process.2012 2 4
  42. 42. Mixin usage Code Flow super class MongoOutput < BufferedOutput BufferedOutput ... super include SetTagKeyMixin config_set_default SetTagKeyMixin :include_tag_key, false super ... end MongoOutput2012 2 4
  43. 43. Input2012 2 4
  44. 44. Available plugins Default 3rd party exec mongo_tail forward scribe http msgpack stream dstat syslog zmq tail amqp etc... etc...2012 2 4
  45. 45. class NewInput < Input ... def configure(conf) # parse a configuration manually end def start # invoke action end def shutdown # cleanup resources end end2012 2 4
  46. 46. In action method, we use Engine.emit to input data. tag = "app.tag" time = Engine.now Sample: record = {"key" => "value", ...} Engine.emit(tag, time, record)2012 2 4
  47. 47. How to read an input in an efficient way? We use a thread and an event loop.2012 2 4
  48. 48. Thread class ForwardInput < Fluent::Input ... def start ... @thread = Thread.new(&method(:run)) end def run ... end end2012 2 4
  49. 49. Event loop class ForwardInput < Fluent::Input ... def start @loop = Coolio::Loop.new @lsock = listen @loop.attach(@lsock) ... end ... end2012 2 4
  50. 50. Note that We must use Engine.now instead of Time.now2012 2 4
  51. 51. Buffer2012 2 4
  52. 52. Available plugins Default 3rd party memory file zfile (?)2012 2 4
  53. 53. In most cases, Memory and File are enough.2012 2 4
  54. 54. Memory type is default. It’s fast but can’t resume data.2012 2 4
  55. 55. File type is persistent type. It can resume data from file.2012 2 4
  56. 56. Output2012 2 4
  57. 57. Available plugins Default 3rd party copy mongo exec s3 file scribe forward couch null hoop stdout splunk etc... etc...2012 2 4
  58. 58. class NewOutput < BufferedOutput # configure, start and shutdown # are same as input plugin def format(tag, time, record) # convert event to raw string end def write(chunk) # write chunk to target # chunk has multiple formatted data end end2012 2 4
  59. 59. Output has 3 buffering modes. None Buffered Time sliced2012 2 4
  60. 60. Buffering type Buffered Time sliced from in Buffer has an internal chunk map to manage a chunk. A key is tag in Buffered, chunk queue but a key is time slice in limit chunk limit TimeSliced buffer. go out def write(chunk) chunk # chunk.key is time slice end2012 2 4
  61. 61. How to write an output in an efficient way? We can use multi-process (input too). See: DetachMultiProcessMixin with detach_multi_process2012 2 4
  62. 62. Test2012 2 4
  63. 63. Input: Fluent::Test::InputTestDriver Buffer: Fluent::Test::BufferedOutputTestDriver Output: Fluent::Test::OutputTestDriver2012 2 4
  64. 64. class MongoOutputTest < Test::Unit::TestCase def setup Fluent::Test.setup require fluent/plugin/out_mongo end def create_driver(conf = CONFIG) Fluent::Test::BufferedOutputTestDriver.new (Fluent::MongoOutput) { def start # prevent external access super end ... }.configure(conf) end2012 2 4
  65. 65. ... def test_format # test format using emit and expect_format end def test_write d = create_driver t = emit_documents(d) # return a result of write method collection_name, documents = d.run assert_equal([{...}, {...}, ...], documents) assert_equal(test, collection_name) end ... end2012 2 4
  66. 66. It’s a weak point in Fluentd... right?2012 2 4
  67. 67. Release2012 2 4
  68. 68. Gem Structure Plugin root |-- lib/ | |-- fluent/ | |-- plugin/ | |- out_<name>.rb |- Gemfile |- fluent-plugin-<name>.gemspec |- Rakefile |- README.md(rdoc) |- VERSION2012 2 4
  69. 69. Bundle with git $ edit lib/fluent/plugin/out_<name>.rb $ git add / commit $ cat VERSION 0.1.0 $ bunlde exec rake release See: rubygems.org/gems/fluent-plugin-<name>2012 2 4
  70. 70. See released plugins for more details about each file.2012 2 4
  71. 71. Lastly...2012 2 4
  72. 72. Help!2012 2 4
  73. 73. Question?2012 2 4

×