Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
Fluentd Hacking Guide at RubyKaigi 2014
Next
Download to read offline and view in fullscreen.

Share

Fluentd meetup dive into fluent plugin (outdated)

Download to read offline

Fluentd meetup in Japan. I talked about "Dive into Fluent plugin".

Some contents are outdated. See this slide: http://www.slideshare.net/repeatedly/dive-into-fluentd-plugin-v012

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Fluentd meetup dive into fluent plugin (outdated)

  1. 1. Dive into Fluent P lugin2012 2 4
  2. 2. Site: repeatedly.github.com Company: Preferred Infrastructure, Inc. Love plugins: input: tail buffer: memory output: mongo2012 2 4
  3. 3. Whats Fluentd? The missing log collector See keynote developed by2012 2 4
  4. 4. Fluentd is a buffer router collector converter aggregator etc...2012 2 4
  5. 5. ... but, Fluentd doesn’t have such features as a built-in.2012 2 4
  6. 6. Instead, Fluentd has flexible plugin architecture which consists of Input, Output and Buffer.2012 2 4
  7. 7. We can customize Fluentd using plugins :)2012 2 4
  8. 8. Agenda Yes, I talk about - an example of Fluentd plugins - Fluentd and libraries - how to develop a Fluentd plugins No, I don’t talk about - the details of each plugin - the experience of production2012 2 4
  9. 9. Example based on bit.ly/fluentd-with-mongo2012 2 4
  10. 10. Install Plugin name is fluent-plugin-xxx , and fluent-gem is included in Fluentd gem.2012 2 4
  11. 11. Let’s type! $ fluent-gem install fluent-plugin-mongo2012 2 4
  12. 12. Me! Many Many Many Plugins!2012 2 4
  13. 13. fluentd.conf Input Output <source> <match mongo.**> type tail type mongo format apache database apache path /path/to/log collection access tag mongo.apache host otherhost </source> </match>2012 2 4
  14. 14. Start! $ fluentd -c fluentd.conf 2012-02-04 00:00:14 +0900: starting fluentd-0.10.8 2012-02-04 00:00:14 +0900: reading config file path="fluentd.conf" 2012-02-04 00:00:14 +0900: adding source type="tail" 2012-02-04 00:00:14 +0900: adding match pattern="mongo.**" type="mongo"2012 2 4
  15. 15. Attack! $ ab -n 100 -c 10 http://localhost/2012 2 4
  16. 16. $ mongo --host otherhost > use apache > db.access.find() { "type": "127.0.0.1", "method": "GET", "path": "/", "code": "200", "size": "44", "time": ISODate("2011-11-27T07:56:27Z") ... } has more...2012 2 4
  17. 17. Apache I’m a log! tail write insert Fluentd event buffering Mongo2012 2 4
  18. 18. Warming up2012 2 4
  19. 19. Fluentd Stack Output Input Buffer Ruby MessagePack Cool.io OS2012 2 4
  20. 20. Ruby ruby-lang.org2012 2 4
  21. 21. Fluentd and plugins are written in Ruby.2012 2 4
  22. 22. ... but note that Fluentd works on Ruby 1.9, goodbye 1.8!2012 2 4
  23. 23. MessagePack msgpack.org2012 2 4
  24. 24. Serialization: JSON like fast and compact format. RPC: Async and parallelism for high performance. IDL: Easy to integrate and maintain the service.2012 2 4
  25. 25. Binary format, Header + Body, and Variable length.2012 2 4
  26. 26. Note that Ruby version can’t handle a Time object.2012 2 4
  27. 27. So, we use an Integer object instead of a Time.2012 2 4
  28. 28. Source: github.com/msgpack Wiki: wiki.msgpack.org/display/MSGPACK Mailing List: groups.google.com/group/msgpack2012 2 4
  29. 29. Cool.io coolio.github.com2012 2 4
  30. 30. Event driven framework built on top of libev.2012 2 4
  31. 31. Cool.io has Loop and Watchers with Transport wrappers.2012 2 4
  32. 32. Fluentd has a default event loop. We can use @default_loop in the plugin.2012 2 4
  33. 33. Configuration2012 2 4
  34. 34. Fluentd loads plugins from $LOAD_PATH.2012 2 4
  35. 35. Input: $LOAD_PATH/fluent/plugin/in_<type>.rb Buffer: $LOAD_PATH/fluent/plugin/buf_<type>.rb Output: $LOAD_PATH/fluent/plugin/out_<type>.rb2012 2 4
  36. 36. We use ‘register_input’, ‘register_buffer’ and ‘register_output’ to register a plugin.2012 2 4
  37. 37. We can load the plugin configuration using config_param and configure method. config_param set config value to @<config name> automatically.2012 2 4
  38. 38. <source> type tail path /path/to/log ... </source> fluentd.conf class TailInput < Input Plugin.register_input(’tail’, self) config_param :path, :string ... end in_tail.rb2012 2 4
  39. 39. One trick is here: Fluentd’s configuration module does not verify a default value. So, we can use the nil like Tribool :) config_param :tag, :string, :default => nil Fluentd does not check the type2012 2 4
  40. 40. Fluentd provides some useful mixins for input and output plugins.2012 2 4
  41. 41. SetTagKeyMixin: Provide ‘tag_key’ and ‘include_tag_key’. SetTimeKeyMixin: Provide ‘time_key’ and ‘include_time_key’. DetachMultiProcessMixin: Provide ‘detach_process’ and execute an action in the multi-process.2012 2 4
  42. 42. Mixin usage Code Flow super class MongoOutput < BufferedOutput BufferedOutput ... super include SetTagKeyMixin config_set_default SetTagKeyMixin :include_tag_key, false super ... end MongoOutput2012 2 4
  43. 43. Input2012 2 4
  44. 44. Available plugins Default 3rd party exec mongo_tail forward scribe http msgpack stream dstat syslog zmq tail amqp etc... etc...2012 2 4
  45. 45. class NewInput < Input ... def configure(conf) # parse a configuration manually end def start # invoke action end def shutdown # cleanup resources end end2012 2 4
  46. 46. In action method, we use Engine.emit to input data. tag = "app.tag" time = Engine.now Sample: record = {"key" => "value", ...} Engine.emit(tag, time, record)2012 2 4
  47. 47. How to read an input in an efficient way? We use a thread and an event loop.2012 2 4
  48. 48. Thread class ForwardInput < Fluent::Input ... def start ... @thread = Thread.new(&method(:run)) end def run ... end end2012 2 4
  49. 49. Event loop class ForwardInput < Fluent::Input ... def start @loop = Coolio::Loop.new @lsock = listen @loop.attach(@lsock) ... end ... end2012 2 4
  50. 50. Note that We must use Engine.now instead of Time.now2012 2 4
  51. 51. Buffer2012 2 4
  52. 52. Available plugins Default 3rd party memory file zfile (?)2012 2 4
  53. 53. In most cases, Memory and File are enough.2012 2 4
  54. 54. Memory type is default. It’s fast but can’t resume data.2012 2 4
  55. 55. File type is persistent type. It can resume data from file.2012 2 4
  56. 56. Output2012 2 4
  57. 57. Available plugins Default 3rd party copy mongo exec s3 file scribe forward couch null hoop stdout splunk etc... etc...2012 2 4
  58. 58. class NewOutput < BufferedOutput # configure, start and shutdown # are same as input plugin def format(tag, time, record) # convert event to raw string end def write(chunk) # write chunk to target # chunk has multiple formatted data end end2012 2 4
  59. 59. Output has 3 buffering modes. None Buffered Time sliced2012 2 4
  60. 60. Buffering type Buffered Time sliced from in Buffer has an internal chunk map to manage a chunk. A key is tag in Buffered, chunk queue but a key is time slice in limit chunk limit TimeSliced buffer. go out def write(chunk) chunk # chunk.key is time slice end2012 2 4
  61. 61. How to write an output in an efficient way? We can use multi-process (input too). See: DetachMultiProcessMixin with detach_multi_process2012 2 4
  62. 62. Test2012 2 4
  63. 63. Input: Fluent::Test::InputTestDriver Buffer: Fluent::Test::BufferedOutputTestDriver Output: Fluent::Test::OutputTestDriver2012 2 4
  64. 64. class MongoOutputTest < Test::Unit::TestCase def setup Fluent::Test.setup require fluent/plugin/out_mongo end def create_driver(conf = CONFIG) Fluent::Test::BufferedOutputTestDriver.new (Fluent::MongoOutput) { def start # prevent external access super end ... }.configure(conf) end2012 2 4
  65. 65. ... def test_format # test format using emit and expect_format end def test_write d = create_driver t = emit_documents(d) # return a result of write method collection_name, documents = d.run assert_equal([{...}, {...}, ...], documents) assert_equal(test, collection_name) end ... end2012 2 4
  66. 66. It’s a weak point in Fluentd... right?2012 2 4
  67. 67. Release2012 2 4
  68. 68. Gem Structure Plugin root |-- lib/ | |-- fluent/ | |-- plugin/ | |- out_<name>.rb |- Gemfile |- fluent-plugin-<name>.gemspec |- Rakefile |- README.md(rdoc) |- VERSION2012 2 4
  69. 69. Bundle with git $ edit lib/fluent/plugin/out_<name>.rb $ git add / commit $ cat VERSION 0.1.0 $ bunlde exec rake release See: rubygems.org/gems/fluent-plugin-<name>2012 2 4
  70. 70. See released plugins for more details about each file.2012 2 4
  71. 71. Lastly...2012 2 4
  72. 72. Help!2012 2 4
  73. 73. Question?2012 2 4
  • AndrewXia

    Apr. 7, 2016
  • yikster

    Aug. 31, 2015
  • blahstyle

    Aug. 7, 2015
  • FrankChen6

    Jul. 1, 2015
  • timesking

    Apr. 25, 2015
  • campeterson

    Mar. 2, 2015
  • cynial

    Feb. 24, 2015
  • tagnee

    Jan. 16, 2015
  • seohoseok14

    Jan. 15, 2015
  • allenkk

    Dec. 21, 2014
  • benjamincheng

    Jul. 8, 2014
  • YoshihiroNomura

    Apr. 30, 2014
  • shunwen

    Nov. 4, 2013
  • ellis.weng

    Oct. 31, 2013
  • udomsakc

    Aug. 16, 2013
  • iyagi15

    Aug. 12, 2013
  • tetsuyaonoda

    Jul. 4, 2013
  • wolfg

    Feb. 28, 2013
  • alkchen

    Feb. 20, 2013
  • arpit_bansal

    Jan. 23, 2013

Fluentd meetup in Japan. I talked about "Dive into Fluent plugin". Some contents are outdated. See this slide: http://www.slideshare.net/repeatedly/dive-into-fluentd-plugin-v012

Views

Total views

29,662

On Slideshare

0

From embeds

0

Number of embeds

16,685

Actions

Downloads

188

Shares

0

Comments

0

Likes

41

×