Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Modern Black Mages Fighting in the Real World

7,567 views

Published on

RubyKaigi 2016 Day2 talk

Published in: Software
  • Be the first to comment

Modern Black Mages Fighting in the Real World

  1. 1. Modern Black Mages Fighting in the Real World Sep 9, 2016 in RubyKaigi 2016 @tagomoris Satoshi "Moris" Tagomori
  2. 2. Satoshi "Moris" Tagomori (@tagomoris) Fluentd, MessagePack-Ruby, Norikra, ... Treasure Data, Inc.
  3. 3. https://github.com/tagomoris/msgpack-inspect
  4. 4. http://docs.fluentd.org/articles/logo http://www.fluentd.org/
  5. 5. Fluentd • What is Fluentd? • Open Source Log Collector • Pluggable, Reliable, Less resource usage, Ease to use • Versions of Fluentd • v0.12: stable versions (2014/12 - Now) • v0.14: versions for next stable (2016/05 - Now)
  6. 6. http://docs.fluentd.org/articles/logo
  7. 7. What's about this session? • Introduce some patterns of "Black Magic"s (a.k.a. meta programming) in Ruby • Show you some PRAGMATIC use of Black Magics
  8. 8. Fluentd v0.14 Release
  9. 9. Fluentd v0.14 API Update • Everything changed :) • Plugin namespace • before: Fluent::* (Top level classes even for plugins!) • after: Fluent::Plugin::* • Plugin base class for common methods • Inconsistent Output plugin hierarchy • Plugin must call `super` in common methods http://www.slideshare.net/tagomoris/fluentd-v014-plugin-api-details
  10. 10. Classes hierarchy (v0.12) Fluent::Input F::Filter F::Output BufferedOutput Object Buffered Time Sliced Multi Output F::Buffer F::Parser F::Formatter 3rd party plugins
  11. 11. Classes hierarchy (v0.14) F::P::Input F::P::Filter F::P::Output Fluent::Plugin::Base F::P::Buffer F::P::Parser F::P::Formatter F::P::Storage both of buffered/non-buffered F::P:: BareOutput (not for 3rd party plugins) F::P:: MultiOutput copy roundrobin
  12. 12. diff v0.12 v0.14 F::P::Output Fluent::Plugin::Base both of buffered/non-buffered F::P:: BareOutput (not for 3rd party plugins) F::P:: MultiOutput copy roundrobin F::Output BufferedOutput Object Buffered Time Sliced Multi Output Super classes by how to buffer data All output plugins are just "Output"
  13. 13. Basic Black Magics: Class and Mixin in Ruby
  14. 14. Class and Subclass in Ruby class A #bar class B #bar super B.new.bar
  15. 15. class A #bar class B #bar super B.new.bar module M #bar Introducing Methods by Mixin
  16. 16. class A #bar class B #bar super B.new.bar module M #bar Singleton Class of Ruby #bar B.new.singleton_class
  17. 17. class A #bar class B #bar super b=B.new b.singleton_class.include M2 b.bar module M #bar Adding Methods on An Instance (1) B.new.singleton_class #bar M2 #bar
  18. 18. class A #bar class B #bar super b=B.new b.extend M2 b.bar module M #bar Adding Methods on An Instance (2) B.new.singleton_class #bar M2 #bar
  19. 19. Back to Fluentd code :)
  20. 20. diff v0.12 v0.14 F::P::Output Fluent::Plugin::Base both of buffered/non-buffered F::P:: BareOutput (not for 3rd party plugins) F::P:: MultiOutput copy roundrobin F::Output BufferedOutput Object Buffered Time Sliced Multi Output Super classes by how to buffer data All output plugins are just "Output"
  21. 21. Fluentd v0.12 Fluent::Output class Fluent::Output #emit(tag, es, chain) MyOutput Engine calls plugin.emit(tag, es, chain) @buffer
  22. 22. Fluentd v0.12 Fluent::BufferedOutput (1) class Fluent::Outputclass BufferedOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) super(tag, es, chain, any_key) Engine calls plugin.emit(tag, es, chain) @buffer #emit(key, data, chain) #format(tag,time,record) #format_stream(tag,es)
  23. 23. Fluentd v0.12 Fluent::BufferedOutput (2) class Fluent::Outputclass BufferedOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) super(tag, es, chain, any_key) Engine calls plugin.emit(tag, es, chain) @buffer #emit(key, data, chain) #format_stream(tag,es) #format_stream(tag,es) #format(tag,time,record)
  24. 24. Fluentd v0.12 Fluent::TimeSlicedOutput class Fluent::Outputclass BufferedOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) Engine calls plugin.emit(tag, es, chain) @buffer #emit(key, data, chain) #emit(tag, es, chain) class TimeSlicedOutput #format(tag,time,record)
  25. 25. Fluentd v0.12 Fluent::ObjectBufferedOutput class Fluent::Outputclass BufferedOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) Engine calls plugin.emit(tag, es, chain) @buffer #emit(key, data, chain) #emit(tag, es, chain) class ObjectBufferedOutput
  26. 26. Fluentd v0.12 Fluent::BufferedOutput class Fluent::Outputclass BufferedOutputMyOutput @buffer calls #write in OutputThread @buffer chunk #write(chunk) OutputThread #pop
  27. 27. Fluentd v0.12 Fluent::TimeSlicedOutput class Fluent::Outputclass BufferedOutput @buffer MyOutput class TimeSlicedOutput OutputThread #write(chunk) @buffer calls #write in OutputThread #write calls chunk.key chunk #pop
  28. 28. Fluentd v0.12 Fluent::ObjectBufferedOutput class Fluent::Outputclass BufferedOutput @buffer MyOutput class ObjectBufferedOutput OutputThread #write(chunk) #write(chunk) #write_object(chunk_key, chunk) @buffer calls #write in OutputThread chunk #pop
  29. 29. Fluentd v0.12 API Problems • Entry point method is implemented by Plugin subclasses • Fluentd core cannot add any processes • counting input events • hook arguments/return values to update API • Fluentd core didn't show fixed API • Plugins have different call stacks • It's not clear what should be implemented for authors • It's not clear what interfaces are supported for arguments/return values
  30. 30. How can we solve this problem?
  31. 31. Fluent::Plugin::Output (v0.14)
  32. 32. Fluentd v0.14 Fluent::Plugin::Output class Outputclass MyOutput #process(tag, es) Engine calls plugin.emit_events(tag, es) @buffer #write #emit_events(tag, es) #format(tag, time, record) #write(chunk) #try_write(chunk) #emit_sync(tag, es) #emit_buffered(tag, es)
  33. 33. Fluentd v0.14 Fluent::Plugin::Output class Outputclass MyOutput Output calls plugin.write (or try_write) @buffer chunk #write(chunk) #try_write(chunk) flush thread #process(tag, es) #format(tag, time, record)
  34. 34. Fluentd v0.14 Design Policy • Separate entry points from implementations • Methods in superclass control everything • Do NOT override these methods! • Methods in subclass do things only for themselves • not for data flow, control flow nor others • Plugins have simple/straightforward call stack • Easy to understand/maintain
  35. 35.
  36. 36. How about existing v0.12 plugins?
  37. 37. Requirement: (Almost) All Existing Plugins SHOULD
 Work Well WITHOUT ANY MODIFICATION
  38. 38. • Fluent::Compat namespace for compatibility layer v0.14 Plugins & Compat Layer F::P::Output F::P::Base v0.14 Plugins Fluent:: Compat:: Output F::C:: Buffered Output F::C:: TimeSliced Output F::C:: ObjectBuffered Output Fluent::Output F:: Buffered Output F:: TimeSliced Output F:: ObjectBuffered Output v0.12 Plugins
  39. 39. Double Decker Compat Layer? • Existing plugins inherits Fluent::Output or others • No more codes in Fluent top level :-( • Separate code into Fluent::Compat • and import it into Fluent top level
  40. 40. Fluentd v0.14 Fluent::Plugin::Output class Outputclass MyOutput #process(tag, es) Engine calls plugin.emit_events(tag, es) @buffer #write #emit_events(tag, es) #format(tag, time, record) #write(chunk) #try_write(chunk) #emit_sync(tag, es) #emit_buffered(tag, es)
  41. 41. Fluentd v0.12 Fluent::BufferedOutput (2) class Fluent::Outputclass BufferedOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) super(tag, es, chain, any_key) Engine calls plugin.emit(tag, es, chain) @buffer #emit(key, data, chain) #format_stream(tag,es) #format_stream(tag,es) #format(tag,time,record)
  42. 42. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) v0.12 Plugins via Compat Layer: Best case (virtual) Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain, key) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread
  43. 43. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) v0.12 Plugins via Compat Layer: Best case (real) Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain, key) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread
  44. 44. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #format_stream Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain, key) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread
  45. 45. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #format_stream Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain, key) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread default implementation for calling "super"
  46. 46. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #emit Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread
  47. 47. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread When plugin overrides #emit
  48. 48. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread This call doesn't happen, in fact #emit doesn't return values! When plugin overrides #emit
  49. 49. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #emit Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread #emit calls @buffer.emit → NoMethodError !
  50. 50. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #emit Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread
  51. 51. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #emit Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread #emit 1. #emit calls @buffer.emit with data to be written in buffer 0. plugin calls @buffer.extend to add #emit 2. @buffer.emit stores arguments into plugin's attribute 3. get stored data 4. call @buffer.write with data
  52. 52. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #emit Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread
  53. 53. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) Thinking about "chunk" instance ... Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread #write may call "chunk.key", but v0.14 chunk doesn't have #key !
  54. 54. Fluent::Plugin::Outputclass MyOutput @buffer #write Compat::BufferedOutput #write(chunk) flush thread "chunk" has #metadata, and values of #key can be created via #metadata Let's "chunk.extend" ! Where to do so? ? Thinking about "chunk" instance ...
  55. 55. Fluent::Plugin::OutputMyOutput @buffer #write C::BufferedOutput #write(chunk) flush thread Thinking about "chunk" instance ... #write(chunk) BufferedChunkMixin plugin.extend BufferedChunkMixin in #configure
  56. 56. Similar hacks for TimeSlicedOutput and ObjectBufferedOutput ...
  57. 57. Controlling Plugin Lifecycle
  58. 58. Plugin Lifecycle Updated Methods(v0.12) • #configure • #start • #before_shutdown • #shutdown v0.12 Plugins often doesn't call "super"! Methods(v0.14) • #configure • #start • #stop • #before_shutdown • #shutdown • #after_shutdown • #close • #terminate In v0.14, these methods MUST call "super" • #configured? • #started? • #stopped? • #before_shutdown? • #shutdown? • #after_shutdown? • #closed? • #terminated?
  59. 59. For Example: shutdown compat plugins Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown It doesn't call "super"! We want to call this...
  60. 60. What We Want To Do: Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown 1. call #shutdown anyway 0. Fluentd core calls #shutdown 2. call #shutdown? to check "super" is called or not 3. call #shutdown of superclass forcedly!
  61. 61. What We Want To Do: Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown How to make this point?
  62. 62. One More Magic! Module#prepend
  63. 63. class A #bar class B #bar super B.new.bar Wrapping Methods on a Class (1) B.new.singleton_class #bar
  64. 64. class A #bar class B #bar super B.new.bar module M Wrapping Methods on a Class (2) B.new.singleton_class #bar #bar Using extend is powerful, but it should be done for all instances How about wrapping methods for all instances of the class?
  65. 65. class A #bar class B #bar super module M;def bar;super;end;end B.prepend M B.new.bar module M Wrapping Methods on a Class (3): Module#prepend B.new.singleton_class #bar #bar module M wraps B, and M#bar is called at first
  66. 66. What We Want To Do: Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown THIS ONE !!!
  67. 67. What We Got :-) Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown 1. call #shutdown anyway 0. prepend CallSuperMixin at first 2. call #shutdown? to check "super" is called or not 3. if not, get method of superclass, bind self with it, then call it Thank you @unak -san!
  68. 68. Beating Test Code
  69. 69. Testing: Capturing return values by Test Driver OutputMyOutput @buffer #write #emit_events #format #emit_buffered Output Plugin Test Driver Create plugin instances Feed test data into plugin
  70. 70. Testing: Capturing return values by Test Driver OutputMyOutput @buffer #write #emit_events #format #emit_buffered We want to assert this return value! Output Plugin Test Driver Feed test data into plugin
  71. 71. Using #prepend to capture return values OutputMyOutput @buffer #write #emit_events #format #emit_buffered Output Plugin Test Driver Feed test data into plugin module M #format Store return value of "super"
  72. 72. Using #prepend ... doesn't work :-( OutputMyOutput @buffer #write #emit_events #format #emit_buffered Output Plugin Test Driver Feed test data into plugin #format Test code sometimes overwrites methods for many reasons :P singleton class #format 😱 module M
  73. 73. One Another Magic: Stronger Than Anything
  74. 74. class A #bar class B #bar super b=B.new b.singleton_class.module_eval{define_method(:bar){"1"}} b.bar Another Study: How To Wrap Singleton Method? B.new.singleton_class #bar module M #bar
  75. 75. class A #bar class B #bar super module P Another Study: How To Wrap Singleton Method? B.new.singleton_class #bar b=B.new b.singleton_class.module_eval{define_method(:bar){"1"}} b.singleton_class.prepend P b.bar #bar module M #bar Singleton class is a class, so it can be prepended :) It's actually done in Test Driver implementation...
  76. 76. Using #prepend on singleton_class: Yay! OutputMyOutput @buffer #write #emit_events #format #emit_buffered Output Plugin Test Driver Feed test data into plugin #format singleton class #format😃 module M Prepending modules on singleton_class overrides everything!
  77. 77. IS BUILT ON A TOP OF BUNCH OF BLACK MAGICS :P
  78. 78. Do Whatever You Can For Users! It Makes Everyone Happier! ... Except for Maintainers :(

×