Fighting API Compatibility
On Fluentd Using "Black Magic"
Jul 2 2016 in YAPC::Asia Hachioji
#yapc8oji
Satoshi "Moris" Tagomori (@tagomoris)

Satoshi "Moris" Tagomori
(@tagomoris)
Fluentd, MessagePack-Ruby, Norikra, ...
Treasure Data, Inc.

http://docs.fluentd.org/articles/logo

Fluentd v0.14 Release

Fluentd v0.14 API Update
• Everything changed :)
• Plugin namespace
• before: Fluent::* (Top level classes even for plugins!)
• after: Fluent::Plugin::*
• Plugin base class for common methods
• Inconsistent Output plugin hierarchy
• Plugin must call `super` in common methods
http://www.slideshare.net/tagomoris/fluentd-v014-plugin-api-details

Classes hierarchy (v0.12)
Fluent::Input F::Filter
F::Output
BufferedOutput
Object
Buffered
Time
Sliced
Multi
Output F::Buffer
F::Parser
F::Formatter
3rd party plugins

Classes hierarchy (v0.14)
F::P::Input F::P::Filter F::P::Output
Fluent::Plugin::Base
F::P::Buffer
F::P::Parser
F::P::Formatter
F::P::Storage
both of
buffered/non-buffered
F::P::
BareOutput
(not for 3rd party
plugins)
F::P::
MultiOutput
copy
roundrobin

diff v0.12 v0.14
F::P::Output
Fluent::Plugin::Base
both of
buffered/non-buffered
F::P::
BareOutput
(not for 3rd party
plugins)
F::P::
MultiOutput
copy
roundrobin
F::Output
BufferedOutput
Object
Buffered
Time
Sliced
Multi
Output
Super classes by
how to buffer data
All output plugins
are just "Output"

Basic Weapons:
Class and Mixin in Ruby

Class and Subclass in Ruby
class A
#bar
class B
#bar
super
B.new.bar

class A
#bar
class B
#bar
super
B.new.bar
module M
#bar
Introducing Methods by Mixin

class A
#bar
class B
#bar
super
B.new.bar
module M
#bar
Singleton Class of Ruby
#bar
B.new.singleton_class

class A
#bar
class B
#bar
super
b=B.new
b.singleton_class.include M2
b.bar
module M
#bar
Adding Methods on An Instance (1)
B.new.singleton_class
#bar
M2
#bar

class A
#bar
class B
#bar
super
b=B.new
b.extend M2
b.bar
module M
#bar
Adding Methods on An Instance (2)
B.new.singleton_class
#bar
M2
#bar

Back to Fluentd code :)

diff v0.12 v0.14
F::P::Output
Fluent::Plugin::Base
both of
buffered/non-buffered
F::P::
BareOutput
(not for 3rd party
plugins)
F::P::
MultiOutput
copy
roundrobin
F::Output
BufferedOutput
Object
Buffered
Time
Sliced
Multi
Output
Super classes by
how to buffer data
All output plugins
are just "Output"

Fluentd v0.12 Fluent::Output
class Fluent::Output
#emit(tag, es, chain)
MyOutput
Engine calls plugin.emit(tag, es, chain)
@buffer

Fluentd v0.12 Fluent::BufferedOutput (1)
class Fluent::Outputclass BufferedOutput
#emit(tag, es, chain, key)
MyOutput
#emit(tag, es, chain)
super(tag, es, chain, any_key)
Engine calls plugin.emit(tag, es, chain)
@buffer
#emit(key, data, chain)
#format(tag,time,record)
#format_stream(tag,es)

Fluentd v0.12 Fluent::BufferedOutput (2)
class Fluent::Outputclass BufferedOutput
#emit(tag, es, chain, key)
MyOutput
#emit(tag, es, chain)
super(tag, es, chain, any_key)
Engine calls plugin.emit(tag, es, chain)
@buffer
#emit(key, data, chain)
#format_stream(tag,es)
#format_stream(tag,es)
#format(tag,time,record)

Fluentd v0.12 Fluent::TimeSlicedOutput
class Fluent::Outputclass BufferedOutput
#emit(tag, es, chain, key)
MyOutput
#emit(tag, es, chain)
Engine calls plugin.emit(tag, es, chain)
@buffer
#emit(key, data, chain)
#emit(tag, es, chain) class TimeSlicedOutput
#format(tag,time,record)

Fluentd v0.12 Fluent::ObjectBufferedOutput
class Fluent::Outputclass BufferedOutput
#emit(tag, es, chain, key)
MyOutput
#emit(tag, es, chain)
Engine calls plugin.emit(tag, es, chain)
@buffer
#emit(key, data, chain)
#emit(tag, es, chain) class ObjectBufferedOutput

Fluentd v0.12 Fluent::BufferedOutput
class Fluent::Outputclass BufferedOutputMyOutput
@buffer calls #write in OutputThread
@buffer
chunk
#write(chunk)
OutputThread
#pop

Fluentd v0.12 Fluent::TimeSlicedOutput
class Fluent::Outputclass BufferedOutput
@buffer
MyOutput
class TimeSlicedOutput
OutputThread
#write(chunk)
@buffer calls #write in OutputThread
#write calls chunk.key
chunk
#pop

Fluentd v0.12 Fluent::ObjectBufferedOutput
class Fluent::Outputclass BufferedOutput
@buffer
MyOutput
class ObjectBufferedOutput
OutputThread
#write(chunk)
#write(chunk)
#write_object(chunk_key, chunk)
@buffer calls #write in OutputThread
chunk
#pop

Fluentd v0.12 API Problems
• Entry point method is implemented by Plugin subclasses
• Fluentd core cannot add any processes
• counting input events
• hook arguments/return values to update API
• Fluentd core didn't show fixed API
• Plugins have different call stacks
• It's not clear what should be implemented for authors
• It's not clear what interfaces are supported for
arguments/return values

How can we solve this problem?

Fluent::Plugin::Output (v0.14)

Fluentd v0.14 Fluent::Plugin::Output
class Outputclass MyOutput
#process(tag, es)
Engine calls plugin.emit_events(tag, es)
@buffer
#write
#emit_events(tag, es)
#format(tag, time, record)
#write(chunk)
#try_write(chunk)
#emit_sync(tag, es)
#emit_buffered(tag, es)

Fluentd v0.14 Fluent::Plugin::Output
class Outputclass MyOutput
Output calls plugin.write (or try_write)
@buffer
chunk
#write(chunk)
#try_write(chunk)
flush thread
#process(tag, es)
#format(tag, time, record)

Fluentd v0.14 Design Policy
• Separate entry points from implementations
• Methods in superclass control everything
• Do NOT override these methods!
• Methods in subclass do things only for themselves
• not for data flow, control flow nor others
• Plugins have simple/straightforward call stack
• Easy to understand/maintain

❤

How about existing
v0.12 plugins?

Requirement:
(Almost)
All Existing Plugins SHOULD
Work Well
WITHOUT ANY MODIFICATION

• Fluent::Compat namespace for compatibility layer
v0.14 Plugins & Compat Layer
F::P::Output
F::P::Base
v0.14 Plugins
Fluent::
Compat::
Output
F::C::
Buffered
Output
F::C::
TimeSliced
Output
F::C::
ObjectBuffered
Output
Fluent::Output
F::
Buffered
Output
F::
TimeSliced
Output
F::
ObjectBuffered
Output
v0.12 Plugins

Double Decker Compat Layer?
• Existing plugins inherits Fluent::Output or others
• No more codes in Fluent top level :-(
• Separate code into Fluent::Compat
• and import it into Fluent top level

Fluentd v0.14 Fluent::Plugin::Output
class Outputclass MyOutput
#process(tag, es)
Engine calls plugin.emit_events(tag, es)
@buffer
#write
#emit_events(tag, es)
#format(tag, time, record)
#write(chunk)
#try_write(chunk)
#emit_sync(tag, es)
#emit_buffered(tag, es)

Fluentd v0.12 Fluent::BufferedOutput (2)
class Fluent::Outputclass BufferedOutput
#emit(tag, es, chain, key)
MyOutput
#emit(tag, es, chain)
super(tag, es, chain, any_key)
Engine calls plugin.emit(tag, es, chain)
@buffer
#emit(key, data, chain)
#format_stream(tag,es)
#format_stream(tag,es)
#format(tag,time,record)

Fluent::Plugin::Outputclass MyOutput
@buffer
#write
#emit_events(tag, es)
v0.12 Plugins via Compat Layer: Best case (virtual)
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain, key) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk)
flush thread

Fluent::Plugin::Outputclass MyOutput
@buffer
#write
#emit_events(tag, es)
v0.12 Plugins via Compat Layer: Best case (real)
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain, key) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk)
flush thread

Fluent::Plugin::Outputclass MyOutput
@buffer
#write
#emit_events(tag, es)
When plugin overrides #format_stream
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain, key) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk)
flush thread

Fluent::Plugin::Outputclass MyOutput
@buffer
#write
#emit_events(tag, es)
When plugin overrides #format_stream
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain, key) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk)
flush thread
default implementation
for calling "super"

Fluent::Plugin::Outputclass MyOutput
@buffer
#write
#emit_events(tag, es)
When plugin overrides #emit
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk)
flush thread

Fluent::Plugin::Outputclass MyOutput
@buffer
#write
#emit_events(tag, es)
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk)
flush thread
When plugin overrides #emit

Fluent::Plugin::Outputclass MyOutput
@buffer
#write
#emit_events(tag, es)
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk)
flush thread
This call doesn't happen, in fact
#emit doesn't return values!
When plugin overrides #emit

Fluent::Plugin::Outputclass MyOutput
@buffer
#write
#emit_events(tag, es)
When plugin overrides #emit
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk)
flush thread
#emit calls @buffer.emit
→ NoMethodError !

Fluent::Plugin::Outputclass MyOutput
@buffer
#write
#emit_events(tag, es)
When plugin overrides #emit
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk)
flush thread

Fluent::Plugin::Outputclass MyOutput
@buffer
#write
#emit_events(tag, es)
When plugin overrides #emit
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk)
flush thread
#emit
1. #emit calls @buffer.emit with data to be written in buffer
0. plugin calls @buffer.extend to add #emit
2. @buffer.emit stores arguments into plugin's attribute
3. get stored data
4. call @buffer.write with data

Fluent::Plugin::Outputclass MyOutput
@buffer
#write
#emit_events(tag, es)
When plugin overrides #emit
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk)
flush thread

Fluent::Plugin::Outputclass MyOutput
@buffer
#write
#emit_events(tag, es)
Thinking about "chunk" instance ...
Compat::BufferedOutput
#emit_buffered(tag, es)
#format(tag, time, record)
#format_stream(tag,es) #handle_stream_*
#handle_stream_simple
#emit(tag, es, chain) #emit(tag, es, chain, key)
#format_stream(tag,es)
#write(chunk)
flush thread
#write may call "chunk.key",
but v0.14 chunk doesn't have #key !

Fluent::Plugin::Outputclass MyOutput
@buffer
#write
Compat::BufferedOutput
#write(chunk)
flush thread
"chunk" has #metadata, and values
of #key can be created via #metadata
Let's "chunk.extend" !
Where to do so?
?
Thinking about "chunk" instance ...

Fluent::Plugin::OutputMyOutput
@buffer
#write
C::BufferedOutput
#write(chunk) flush thread
Thinking about "chunk" instance ...
#write(chunk)
BufferedChunkMixin
plugin.extend BufferedChunkMixin
in #configure

Similar hacks for
TimeSlicedOutput and
ObjectBufferedOutput ...

Controlling Plugin Lifecycle

Plugin Lifecycle Updated
Methods(v0.12)
• #configure
• #start
• #before_shutdown
• #shutdown
v0.12 Plugins often
doesn't call "super"!
Methods(v0.14)
• #configure
• #start
• #stop
• #before_shutdown
• #shutdown
• #after_shutdown
• #close
• #terminate
In v0.14, these methods MUST call "super"
• #configured?
• #started?
• #stopped?
• #before_shutdown?
• #shutdown?
• #after_shutdown?
• #closed?
• #terminated?

For Example: shutdown compat plugins
Fluent::Plugin::Base
#shutdown
F::P::Output
super
#shutdown?
#shutdown
F::C::Output
#shutdown
MyOutput
#shutdown
It doesn't call "super"! We want to call this...

What We Want To Do:
Fluent::Plugin::Base
#shutdown
F::P::Output
super
#shutdown?
#shutdown
F::C::Output
#shutdown
MyOutput
#shutdown
1. call #shutdown anyway
0. Fluentd core calls #shutdown
2. call #shutdown? to check "super" is called or not
3. call #shutdown of superclass forcedly!

What We Want To Do:
Fluent::Plugin::Base
#shutdown
F::P::Output
super
#shutdown?
#shutdown
F::C::Output
#shutdown
MyOutput
#shutdown
How to make this point?

More Weapon!
Module#prepend

class A
#bar
class B
#bar
super
B.new.bar
Wrapping Methods on a Class (1)
B.new.singleton_class
#bar

class A
#bar
class B
#bar
super
B.new.bar
module M
Wrapping Methods on a Class (2)
B.new.singleton_class
#bar
#bar
Using extend is powerful,
but it should be done for all instances
How about wrapping methods for
all instances of the class?

class A
#bar
class B
#bar
super
module M;def bar;super;end;end
B.prepend M
B.new.bar
module M
Wrapping Methods on a Class (3): Module#prepend
B.new.singleton_class
#bar
#bar
module M wraps B, and
M#bar is called at first

class A
#bar
class B
#bar
super
b=B.new
b.singleton_class.module_eval{define_method(:bar){"1"}}
b.bar
Another Study: How To Wrap Singleton Method?
B.new.singleton_class
#bar

class A
#bar
class B
#bar
super
module M
Another Study: How To Wrap Singleton Method?
B.new.singleton_class
#bar
Singleton class is a class,
so it can be prepended :)
b=B.new
b.singleton_class.module_eval{define_method(:bar){"1"}}
b.singleton_class.prepend M
b.bar
#bar
It's actually done in Test Driver implementation...

What We Want To Do:
Fluent::Plugin::Base
#shutdown
F::P::Output
super
#shutdown?
#shutdown
F::C::Output
#shutdown
MyOutput
#shutdown
THIS ONE !!!

What We Got :-)
Fluent::Plugin::Base
#shutdown
F::P::Output
super
#shutdown?
#shutdown
F::C::Output
#shutdown
MyOutput
#shutdown
1. call #shutdown anyway
0. prepend CallSuperMixin at first
2. call #shutdown? to check "super" is called or not
3. if not, get method of superclass, bind self with it, then call it
Thank you @unak -san!

IS BUILT ON A TOP OF
BUNCH OF BLACK MAGICS :P

Do Whatever You Can
For Users!
It Makes Everyone Happier
... Except for Maintainers :(