SlideShare a Scribd company logo
1 of 56
Download to read offline
Fluentd v0.14
Plugin API Details
Fluentd meetup 2016 Summer
Jun 1, 2016
Satoshi "Moris" Tagomori (@tagomoris)
Satoshi "Moris" Tagomori
(@tagomoris)
Fluentd, MessagePack-Ruby, Norikra, ...
Treasure Data, Inc.
Topics
• Why Fluentd v0.14 has a new API set for plugins
• Compatibility of v0.12 plugins/configurations
• Plugin APIs: Input, Filter, Output & Buffer
• Storage Plugin, Plugin Helpers
• New Test Drivers for plugins
• Plans for v0.14.x & v1
Why Fluentd v0.14
has a New API set for plugins?
Fluentd v0.12 Plugins
• No supports to write plugins by Fluentd core
• plugins creates threads, sockets, timers and event loops
• writing tests is very hard and messy with sleeps
• Fragmented implementations
• Output, BufferedOutput, ObjectBufferedOutput and TimeSlicedOutput
• Mixture of configuration parameters from output&buffer
• Uncontrolled plugin instance lifecycle (no "super" in start/shutdown)
• Imperfect buffering control and useless configurations
• the reason why fluent-plugin-forest exists and be used widely
Fluentd v0.12 Plugins
• Insufficient buffer chunking control
• only by size, without number of events in chunks
• Forcedly synchronized buffer flushing
• no way to flush-and-commit chunks asynchronously
• Ultimate freedom for using mix-ins
• everything overrides Plugin#emit ... (the only one entry point for
events to plugins)
• no valid hook points to get metrics or something else
• Bad Ruby coding rules and practices
• too many classes at "Fluent::*" in fluent/plugin, no "require", ...
And many others!
Compatibility
of v0.12 plugins/configurations
Compatibility of plugins
• v0.12 plugins are subclass of Fluent::*
• Fluent::Input, Fluent::Filter, Fluent::Output, ...
• Compatibility layers for v0.12 plugins in v0.14
• Fluent::Compat::Klass -> Fluent::Klass (e.g., Input, Output, ...)
• it provides transformation of:
• namespaces, configuration parameters
• internal APIs, argument objects
• IT SHOULD WORK, except for :P
• 3rd party buffer plugin, part of test code
• "Engine.emit"
Compatibility of configurations
• v0.14 plugins have another set of parameters
• many old-fashioned parameters are removed
• "buffer_type", "num_threads", "timezone", "time_slice_format",
"buffer_chunk_limit", "buffer_queue_limit", ...
• Plugin helper "compat_parameters"
• transform parameters between v0.12 style
configuration and v0.14 plugin
v0.12 v0.14
convert
internally
FAQ:
Can we create plugins like this?
* it uses v0.14 API
* it runs on Fluentd v0.12
Impossible :P
Overview of
v0.14 Plugin classes
v0.14 plugin classes
• All files MUST be in `fluent/plugin/*.rb` (in gems)
• or just a "*.rb" file in directory specified by "-r"
• All classes MUST be under Fluent::Plugin
• All plugins MUST be subclasses of Fluent::Plugin::Base
• All plugins MUST call `super` in methods overriding
default implementation (e.g., #configure, #start, #shutdown, ...)
Classes hierarchy (v0.12)
Fluent::Input F::Filter
F::Output
BufferedOutput
Object
Buffered
Time
Sliced
Multi
Output F::Buffer
F::Parser
F::Formatter
3rd party plugins
Classes hierarchy (v0.14)
F::P::Input F::P::Filter F::P::Output
Fluent::Plugin::Base
F::P::Buffer
F::P::Parser
F::P::Formatter
F::P::Storage
both of
buffered/non-buffered
F::P::
BareOutput
(not for 3rd party
plugins)
F::P::
MultiOutput
copy
roundrobin
Tour of New Plugin APIs:
Fluent::Plugin::Input
Fluent::Plugin::Input
• Nothing changed :)
• except for overall rules
• But it's much easier

to write plugins

than v0.12 :)
• fetch HTTP resource per
specified interval
• parse response body
with format specified in
config
• emit parse result
Fluent::Plugin::Input
Tour of New Plugin APIs:
Fluent::Plugin::Filter
Fluent::Plugin::Filter
• Almost nothing changed :)
• Required:

#filter(tag, time, record)

#=> record | nil
• Optional:

#filter_stream(tag, es)

#=> event_stream
Tour of New Plugin APIs:
Fluent::Plugin::Output
Fluent::Plugin::Output
• Many things changed!
• Merged Output, BufferedOutput, ObjectBufferedOutput, TimeSlicedOutput
• Output plugins can be
• with buffering
• without buffering
• both (do/doesn't buffering by configuration)
• Buffers chunks events by:
• byte size, interval, tag
• number of records (new!)
• time (by any unit(new!): 30s, 5m, 15m, 3h, ...)
• any specified field in records (new!)
• any combination of above (new!)
Variations of buffering
NO MORE forest plugin!
Output Plugin:
Methods to be implemented
• Non-buffered: #process(tag, es)
• Buffered synchronous: #write(chunk)
• Buffered Asynchronous: #try_write(chunk)
• New feature for destinations with huge latency to write
chunks
• Plugins must call #commit_write(chunk_id) (otherwise,
#try_write will be retried)
• Buffered w/ custom format: #format(tag, time, record)
• Without this method, output uses standard format
implement?
#process
implement?
#process or #write or #try_write
NO error
YES
#prefer_buffered_processing
called (default true)
NO
non-buffered
YES
exists?
<buffer> section
YES implement?
#write or #try_write
error
NO
YES
implement?
#write or
#try_write
NO
NO
YES
false
implement?
#write and
#try_write
YES
#prefer_delayed_commit
called (default true)
implement?
#try_write
sync
buffered
async
buffered
In other words :P
• If users configure "<buffer>" section
• plugin try to do buffering
• Else if plugin implements both (buffering/non-buf)
• plugin call #prefer_buffer_processing to decide
• Else plugin does as implemented
• When plugin does buffering

If plugin implements both (sync/async write)
• plugin call #prefer_delayed_commit to decide
• Else plugin does as implemented
Delayed commit (1)
• high latency #write operations locks a flush thread for long time

(e.g., ACK in forward)
destination w/ high latency
#write
Output Plugin
send data send ACK
return #write
a flush thread locked
Delayed commit (2)
• #try_write & delayed #commit_write
destination w/ high latency
#try_write
Output Plugin
send data
send ACK
return
#try_write
async check thread
#commit_write
Use cases: delayed commit
• Forward protocol w/ ACK
• Distributed file systems or databases
• put data -> confirm to read data -> commit
• Submit tasks to job queues
• submit a job -> detect executed -> commit
Standard chunk format
• Buffering w/o #format method
• Almost same with ObjectBufferedOutput
• No need to implement #format always
• Implement it for performance/low-latency
• Tool to dump & read buffer chunks on disk w/
standard format
• To be implemented in v0.14.x :)
<buffer CHUNK_KEYS>
• comma-separated tag, time or ANY_KEYS
• Nothing specified: all events are in same chunk
• flushed when chunk is full
• (optional) "flush_interval" after first event in chunk
• tag: events w/ same tag are in same chunks
• time: buffer chunks will be split by timekey
• timekey: unit of time to be chunked (1m, 15m, 3h, ...)
• flushed after expiration of timekey unit + timekey_wait
• ANY_KEYS: any key names in records
• comma-separated tag, time or ANY_KEYS
• Nothing specified: all events are in same chunk
• flushed when chunk is full
• (optional) "flush_interval" after first event in chunk
• tag: events w/ same tag are in same chunks
• time: buffer chunks will be split by timekey
• timekey: unit of time to be chunked (1m, 15m, 3h, ...)
• flushed after expiration of timekey unit + timekey_wait
• ANY_KEYS: any key names in records
<buffer CHUNK_KEYS>
BufferedOutput
TimeSlicedOutput
ObjectBufferedOutput
in v0.12
in v0.12
in v0.12
configurations:

flushing buffers
• flush_mode: lazy, interval, immediate
• default: lazy if "time" specified, otherwise interval
• flush_interval, flush_thread_count
• flush_thread_count: number of threads for flushing
• delayed_commit_timeout
• output plugin will retry #try_write when expires
Retries, Secondary
• Explicit timeout for retries:
• retry_timeout: timeout not to retry anymore
• retry_max_times: how many times to retry
• retry_type: "periodic" w/ fixed retry_wait
• retry_secondary_threshold (percentage)
• output will use secondary if specified percentage
of retry_timeout elapsed after first error
Buffer parameters
• chunk_limit_size
• maximum bytesize per chunks
• chunk_records_limit (default: not specified)
• maximum number of records per chunks
• total_limit_size
• maximum bytesize which a buffer plugin can use
• (optional) queue_length_limit: no need to specify
Chunk metadata
• Stores various information of buffer chunks
• key-values of chunking unit
• number of records
• created_at, modified_at
• `chunk.metadata`
• extract_placeholders(@path, chunk.metadata)
Tour of New Plugin APIs:
Other plugin types
Classes hierarchy (v0.14)
F::P::Input F::P::Filter F::P::Output
Fluent::Plugin::Base
F::P::Buffer
F::P::Parser
F::P::Formatter
F::P::Storage
both of
buffered/non-buffered
F::P::
BareOutput
(not for 3rd party
plugins)
F::P::
MultiOutput
copy
roundrobin
Classes hierarchy (v0.14)
F::P::Input F::P::Filter F::P::Output
Fluent::Plugin::Base
F::P::Buffer
F::P::Parser
F::P::Formatter
F::P::Storage
both of
buffered/non-buffered
F::P::
BareOutput
(not for 3rd party
plugins)
F::P::
MultiOutput
copy
roundrobin"Owned" plugins
"Owned" plugins
• Primary plugins: Input, Output, Filter
• Instantiated by Fluentd core
• "Owned" plugins are owned by primary plugins
• Buffer, Parser, Formatter, Storage, ...
• It can refer owner's plugin id, logger, ...
• Fluent::Plugin.new_xxx("kind", parent:@input)
• "Owned" plugins can be configured by owner plugins
Owner plugins can control defaults of owned plugins
Fluentd provides standard way to configure owned
plugins
Tour of New Plugin APIs:
Fluent::Plugin::Storage
Storage plugins
• Pluggable Key-Value store for plugins
• configurable: autosave, persistent, save_at_shutdown
• get, fetch, put, delete, update (transactional)
• Various possible implementations
• built-in: local (json) on-disk / on-memory
• possible: Redis, Consul,

or whatever supports serialize/deserialize of json-like object
• To store states of plugins:
• counter values of data-counter plugin
• pos data of file plugin
• To load configuration dynamically for plugins:
• load configurations from any file systems
Tour of New Plugin APIs:
Plugin Helpers
Plugin Helpers
• No more mixin!
• declare to use helpers by "helpers :name"
• Utility functions to support difficult things
• creating threads, timers, child processes...
• created timers will be stopped automatically in
plugin's shutdown sequence
• Integrated w/ New Test Drivers
• tests runs after helpers started everything requested
Plugin Helpers Example
• Thread: thread_create, thread_current_running?
• Timer: timer_execute
• ChildProcess: child_process_execute
• command, arguments, subprocess_name, interval, immediate,
parallel, mode, stderr, env, unsetenv, chdir, ...
• EventEmitter: router (Output doesn't have router in v0.14 default)
• Storage: storage_create
• (TBD) Socket/Server for TCP/UDP/TLS, Parser, Formatter
Tour of New Plugin APIs:
New Test Drivers
New Test Drivers
• Instead of old drivers Fluent::Test::*TestDriver
• Fluent::Test::Driver::Input, Output or Filter
• fully emulates actual plugin behavior
• w/ override SystemConfig
• capturing emitted events & error event streams
• inserting TestLogger to capture/test logs of plugins
• capturing "format" result of output plugins
• controlling "flush" timing of output plugins
• Running tests under control
• Plugin Helper integration
• conditions to keep/break running tests
• timeouts, number of emits/events to stop tests
• automatic start/shutdown call for plugins
Plans for v0.14.x
New Features
• Symmetric multi processing
• to use 2 or more CPU cores!
• by sharing a configuration between all processes
• "detach_process" will be deprecated
• forward: TLS + authentication/authorization support
• secure-forward integration
• Buffer supports compression & forward it
• Plugin generator & template
New APIs
• Controlling global configuration from SystemConfig
• configured via <system> tag
• root buffer path + plugin id: remove paths from
each buffers
• process total buffer size control
• Counter APIs
• counting everything over processes via RPC
• creating metrics for a whole fluentd cluster
For v1
v1: stable version of v0.14
• v0.12 plugins will be still supported at v1.0.0
• deprecated, and will be obsoleted at v1.x
• Will be obsoleted:
• v0 (traditional) configuration syntax
• "detach_process" feature
• Q4 2016?
To Be Written by me :-)
• As soooooooooon as possible...
• Plugin developers' guide for
• Updating v0.12 plugins with v0.14 APIs
• Writing plugins with v0.14 APIs
• Writing tests of plugins with v0.14 APIs
• Users' guide for
• How to use buffering in general (w/ <buffer>)
• Updated plugin documents
Enjoy logging!

More Related Content

What's hot

Tame the small files problem and optimize data layout for streaming ingestion...
Tame the small files problem and optimize data layout for streaming ingestion...Tame the small files problem and optimize data layout for streaming ingestion...
Tame the small files problem and optimize data layout for streaming ingestion...Flink Forward
 
Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...Flink Forward
 
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...HostedbyConfluent
 
Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...
Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...
Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...Flink Forward
 
Distributed stream processing with Apache Kafka
Distributed stream processing with Apache KafkaDistributed stream processing with Apache Kafka
Distributed stream processing with Apache Kafkaconfluent
 
Click-Through Example for Flink’s KafkaConsumer Checkpointing
Click-Through Example for Flink’s KafkaConsumer CheckpointingClick-Through Example for Flink’s KafkaConsumer Checkpointing
Click-Through Example for Flink’s KafkaConsumer CheckpointingRobert Metzger
 
Tuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptxTuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptxFlink Forward
 
Using the New Apache Flink Kubernetes Operator in a Production Deployment
Using the New Apache Flink Kubernetes Operator in a Production DeploymentUsing the New Apache Flink Kubernetes Operator in a Production Deployment
Using the New Apache Flink Kubernetes Operator in a Production DeploymentFlink Forward
 
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah
 
ksqlDB - Stream Processing simplified!
ksqlDB - Stream Processing simplified!ksqlDB - Stream Processing simplified!
ksqlDB - Stream Processing simplified!Guido Schmutz
 
Fluentd v1.0 in a nutshell
Fluentd v1.0 in a nutshellFluentd v1.0 in a nutshell
Fluentd v1.0 in a nutshellN Masahiro
 
Using ClickHouse for Experimentation
Using ClickHouse for ExperimentationUsing ClickHouse for Experimentation
Using ClickHouse for ExperimentationGleb Kanterov
 
Apache Spark Listeners: A Crash Course in Fast, Easy Monitoring
Apache Spark Listeners: A Crash Course in Fast, Easy MonitoringApache Spark Listeners: A Crash Course in Fast, Easy Monitoring
Apache Spark Listeners: A Crash Course in Fast, Easy MonitoringDatabricks
 
Apache Flink and what it is used for
Apache Flink and what it is used forApache Flink and what it is used for
Apache Flink and what it is used forAljoscha Krettek
 
How Apache Kafka® Works
How Apache Kafka® WorksHow Apache Kafka® Works
How Apache Kafka® Worksconfluent
 
Deep dive into stateful stream processing in structured streaming by Tathaga...
Deep dive into stateful stream processing in structured streaming  by Tathaga...Deep dive into stateful stream processing in structured streaming  by Tathaga...
Deep dive into stateful stream processing in structured streaming by Tathaga...Databricks
 
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...confluent
 
Pinot: Near Realtime Analytics @ Uber
Pinot: Near Realtime Analytics @ UberPinot: Near Realtime Analytics @ Uber
Pinot: Near Realtime Analytics @ UberXiang Fu
 
ksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database SystemksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database Systemconfluent
 
Apache kafka performance(latency)_benchmark_v0.3
Apache kafka performance(latency)_benchmark_v0.3Apache kafka performance(latency)_benchmark_v0.3
Apache kafka performance(latency)_benchmark_v0.3SANG WON PARK
 

What's hot (20)

Tame the small files problem and optimize data layout for streaming ingestion...
Tame the small files problem and optimize data layout for streaming ingestion...Tame the small files problem and optimize data layout for streaming ingestion...
Tame the small files problem and optimize data layout for streaming ingestion...
 
Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...
 
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...
Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...
 
Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...
Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...
Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Juli...
 
Distributed stream processing with Apache Kafka
Distributed stream processing with Apache KafkaDistributed stream processing with Apache Kafka
Distributed stream processing with Apache Kafka
 
Click-Through Example for Flink’s KafkaConsumer Checkpointing
Click-Through Example for Flink’s KafkaConsumer CheckpointingClick-Through Example for Flink’s KafkaConsumer Checkpointing
Click-Through Example for Flink’s KafkaConsumer Checkpointing
 
Tuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptxTuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptx
 
Using the New Apache Flink Kubernetes Operator in a Production Deployment
Using the New Apache Flink Kubernetes Operator in a Production DeploymentUsing the New Apache Flink Kubernetes Operator in a Production Deployment
Using the New Apache Flink Kubernetes Operator in a Production Deployment
 
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
 
ksqlDB - Stream Processing simplified!
ksqlDB - Stream Processing simplified!ksqlDB - Stream Processing simplified!
ksqlDB - Stream Processing simplified!
 
Fluentd v1.0 in a nutshell
Fluentd v1.0 in a nutshellFluentd v1.0 in a nutshell
Fluentd v1.0 in a nutshell
 
Using ClickHouse for Experimentation
Using ClickHouse for ExperimentationUsing ClickHouse for Experimentation
Using ClickHouse for Experimentation
 
Apache Spark Listeners: A Crash Course in Fast, Easy Monitoring
Apache Spark Listeners: A Crash Course in Fast, Easy MonitoringApache Spark Listeners: A Crash Course in Fast, Easy Monitoring
Apache Spark Listeners: A Crash Course in Fast, Easy Monitoring
 
Apache Flink and what it is used for
Apache Flink and what it is used forApache Flink and what it is used for
Apache Flink and what it is used for
 
How Apache Kafka® Works
How Apache Kafka® WorksHow Apache Kafka® Works
How Apache Kafka® Works
 
Deep dive into stateful stream processing in structured streaming by Tathaga...
Deep dive into stateful stream processing in structured streaming  by Tathaga...Deep dive into stateful stream processing in structured streaming  by Tathaga...
Deep dive into stateful stream processing in structured streaming by Tathaga...
 
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...
 
Pinot: Near Realtime Analytics @ Uber
Pinot: Near Realtime Analytics @ UberPinot: Near Realtime Analytics @ Uber
Pinot: Near Realtime Analytics @ Uber
 
ksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database SystemksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database System
 
Apache kafka performance(latency)_benchmark_v0.3
Apache kafka performance(latency)_benchmark_v0.3Apache kafka performance(latency)_benchmark_v0.3
Apache kafka performance(latency)_benchmark_v0.3
 

Viewers also liked

Fluentd v0.14 Overview
Fluentd v0.14 OverviewFluentd v0.14 Overview
Fluentd v0.14 OverviewN Masahiro
 
Fluentd Meetup 2016 - ServerEngine Integration & Windows support
Fluentd Meetup 2016 - ServerEngine Integration & Windows supportFluentd Meetup 2016 - ServerEngine Integration & Windows support
Fluentd Meetup 2016 - ServerEngine Integration & Windows supportRitta Narita
 
Big Data Day LA 2016/ Big Data Track - Fluentd and Embulk: Collect More Data,...
Big Data Day LA 2016/ Big Data Track - Fluentd and Embulk: Collect More Data,...Big Data Day LA 2016/ Big Data Track - Fluentd and Embulk: Collect More Data,...
Big Data Day LA 2016/ Big Data Track - Fluentd and Embulk: Collect More Data,...Data Con LA
 
Fluentd with MySQL
Fluentd with MySQLFluentd with MySQL
Fluentd with MySQLI Goo Lee
 
Bai toan va thuat toan
Bai toan va thuat toanBai toan va thuat toan
Bai toan va thuat toanHữu Duy Duy
 
DSLを学ぶ - 設定式によるルールの表現を試す -
DSLを学ぶ - 設定式によるルールの表現を試す - DSLを学ぶ - 設定式によるルールの表現を試す -
DSLを学ぶ - 設定式によるルールの表現を試す - kumamidori
 
20160730 fluentd meetup in matsue slide
20160730 fluentd meetup in matsue slide20160730 fluentd meetup in matsue slide
20160730 fluentd meetup in matsue slidecosmo0920
 
Fluentd Overview, Now and Then
Fluentd Overview, Now and ThenFluentd Overview, Now and Then
Fluentd Overview, Now and ThenSATOSHI TAGOMORI
 
Modern Black Mages Fighting in the Real World
Modern Black Mages Fighting in the Real WorldModern Black Mages Fighting in the Real World
Modern Black Mages Fighting in the Real WorldSATOSHI TAGOMORI
 
Fighting API Compatibility On Fluentd Using "Black Magic"
Fighting API Compatibility On Fluentd Using "Black Magic"Fighting API Compatibility On Fluentd Using "Black Magic"
Fighting API Compatibility On Fluentd Using "Black Magic"SATOSHI TAGOMORI
 
Open Source Software, Distributed Systems, Database as a Cloud Service
Open Source Software, Distributed Systems, Database as a Cloud ServiceOpen Source Software, Distributed Systems, Database as a Cloud Service
Open Source Software, Distributed Systems, Database as a Cloud ServiceSATOSHI TAGOMORI
 
How to Make Norikra Perfect
How to Make Norikra PerfectHow to Make Norikra Perfect
How to Make Norikra PerfectSATOSHI TAGOMORI
 
How To Write Middleware In Ruby
How To Write Middleware In RubyHow To Write Middleware In Ruby
How To Write Middleware In RubySATOSHI TAGOMORI
 
Fluentd Hacking Guide at RubyKaigi 2014
Fluentd Hacking Guide at RubyKaigi 2014Fluentd Hacking Guide at RubyKaigi 2014
Fluentd Hacking Guide at RubyKaigi 2014Naotoshi Seo
 
AWSにおけるバッチ処理の ベストプラクティス - Developers.IO Meetup 05
AWSにおけるバッチ処理の ベストプラクティス - Developers.IO Meetup 05AWSにおけるバッチ処理の ベストプラクティス - Developers.IO Meetup 05
AWSにおけるバッチ処理の ベストプラクティス - Developers.IO Meetup 05都元ダイスケ Miyamoto
 
To Have Own Data Analytics Platform, Or NOT To
To Have Own Data Analytics Platform, Or NOT ToTo Have Own Data Analytics Platform, Or NOT To
To Have Own Data Analytics Platform, Or NOT ToSATOSHI TAGOMORI
 
Perfect Norikra 2nd Season
Perfect Norikra 2nd SeasonPerfect Norikra 2nd Season
Perfect Norikra 2nd SeasonSATOSHI TAGOMORI
 
Ruby and Distributed Storage Systems
Ruby and Distributed Storage SystemsRuby and Distributed Storage Systems
Ruby and Distributed Storage SystemsSATOSHI TAGOMORI
 
Java SE 9の紹介: モジュール・システムを中心に
Java SE 9の紹介: モジュール・システムを中心にJava SE 9の紹介: モジュール・システムを中心に
Java SE 9の紹介: モジュール・システムを中心にTaku Miyakawa
 

Viewers also liked (20)

Fluentd v0.14 Overview
Fluentd v0.14 OverviewFluentd v0.14 Overview
Fluentd v0.14 Overview
 
Fluentd Meetup 2016 - ServerEngine Integration & Windows support
Fluentd Meetup 2016 - ServerEngine Integration & Windows supportFluentd Meetup 2016 - ServerEngine Integration & Windows support
Fluentd Meetup 2016 - ServerEngine Integration & Windows support
 
Keynote - Fluentd meetup v14
Keynote - Fluentd meetup v14Keynote - Fluentd meetup v14
Keynote - Fluentd meetup v14
 
Big Data Day LA 2016/ Big Data Track - Fluentd and Embulk: Collect More Data,...
Big Data Day LA 2016/ Big Data Track - Fluentd and Embulk: Collect More Data,...Big Data Day LA 2016/ Big Data Track - Fluentd and Embulk: Collect More Data,...
Big Data Day LA 2016/ Big Data Track - Fluentd and Embulk: Collect More Data,...
 
Fluentd with MySQL
Fluentd with MySQLFluentd with MySQL
Fluentd with MySQL
 
Bai toan va thuat toan
Bai toan va thuat toanBai toan va thuat toan
Bai toan va thuat toan
 
DSLを学ぶ - 設定式によるルールの表現を試す -
DSLを学ぶ - 設定式によるルールの表現を試す - DSLを学ぶ - 設定式によるルールの表現を試す -
DSLを学ぶ - 設定式によるルールの表現を試す -
 
20160730 fluentd meetup in matsue slide
20160730 fluentd meetup in matsue slide20160730 fluentd meetup in matsue slide
20160730 fluentd meetup in matsue slide
 
Fluentd Overview, Now and Then
Fluentd Overview, Now and ThenFluentd Overview, Now and Then
Fluentd Overview, Now and Then
 
Modern Black Mages Fighting in the Real World
Modern Black Mages Fighting in the Real WorldModern Black Mages Fighting in the Real World
Modern Black Mages Fighting in the Real World
 
Fighting API Compatibility On Fluentd Using "Black Magic"
Fighting API Compatibility On Fluentd Using "Black Magic"Fighting API Compatibility On Fluentd Using "Black Magic"
Fighting API Compatibility On Fluentd Using "Black Magic"
 
Open Source Software, Distributed Systems, Database as a Cloud Service
Open Source Software, Distributed Systems, Database as a Cloud ServiceOpen Source Software, Distributed Systems, Database as a Cloud Service
Open Source Software, Distributed Systems, Database as a Cloud Service
 
How to Make Norikra Perfect
How to Make Norikra PerfectHow to Make Norikra Perfect
How to Make Norikra Perfect
 
How To Write Middleware In Ruby
How To Write Middleware In RubyHow To Write Middleware In Ruby
How To Write Middleware In Ruby
 
Fluentd Hacking Guide at RubyKaigi 2014
Fluentd Hacking Guide at RubyKaigi 2014Fluentd Hacking Guide at RubyKaigi 2014
Fluentd Hacking Guide at RubyKaigi 2014
 
AWSにおけるバッチ処理の ベストプラクティス - Developers.IO Meetup 05
AWSにおけるバッチ処理の ベストプラクティス - Developers.IO Meetup 05AWSにおけるバッチ処理の ベストプラクティス - Developers.IO Meetup 05
AWSにおけるバッチ処理の ベストプラクティス - Developers.IO Meetup 05
 
To Have Own Data Analytics Platform, Or NOT To
To Have Own Data Analytics Platform, Or NOT ToTo Have Own Data Analytics Platform, Or NOT To
To Have Own Data Analytics Platform, Or NOT To
 
Perfect Norikra 2nd Season
Perfect Norikra 2nd SeasonPerfect Norikra 2nd Season
Perfect Norikra 2nd Season
 
Ruby and Distributed Storage Systems
Ruby and Distributed Storage SystemsRuby and Distributed Storage Systems
Ruby and Distributed Storage Systems
 
Java SE 9の紹介: モジュール・システムを中心に
Java SE 9の紹介: モジュール・システムを中心にJava SE 9の紹介: モジュール・システムを中心に
Java SE 9の紹介: モジュール・システムを中心に
 

Similar to Fluentd v0.14 Plugin API Details: New Classes, Buffering, Storage Plugins

Treasure Data Summer Internship 2016
Treasure Data Summer Internship 2016Treasure Data Summer Internship 2016
Treasure Data Summer Internship 2016Yuta Iwama
 
Fluentd v1 and future at techtalk
Fluentd v1 and future at techtalkFluentd v1 and future at techtalk
Fluentd v1 and future at techtalkN Masahiro
 
Fluentd v1.0 in a nutshell
Fluentd v1.0 in a nutshellFluentd v1.0 in a nutshell
Fluentd v1.0 in a nutshellN Masahiro
 
HHVM and Hack: A quick introduction
HHVM and Hack: A quick introductionHHVM and Hack: A quick introduction
HHVM and Hack: A quick introductionKuan Yen Heng
 
Continuous Go Profiling & Observability
Continuous Go Profiling & ObservabilityContinuous Go Profiling & Observability
Continuous Go Profiling & ObservabilityScyllaDB
 
Systems Programming Assignment Help - Processes
Systems Programming Assignment Help - ProcessesSystems Programming Assignment Help - Processes
Systems Programming Assignment Help - ProcessesHelpWithAssignment.com
 
From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...Yury Bushmelev
 
¡El mejor lenguaje para automatizar pruebas!
¡El mejor lenguaje para automatizar pruebas!¡El mejor lenguaje para automatizar pruebas!
¡El mejor lenguaje para automatizar pruebas!Antonio Robres Turon
 
Wait, IPython can do that?! (30 minutes)
Wait, IPython can do that?! (30 minutes)Wait, IPython can do that?! (30 minutes)
Wait, IPython can do that?! (30 minutes)Sebastian Witowski
 
web programming UNIT VIII python by Bhavsingh Maloth
web programming UNIT VIII python by Bhavsingh Malothweb programming UNIT VIII python by Bhavsingh Maloth
web programming UNIT VIII python by Bhavsingh MalothBhavsingh Maloth
 
Fluentd at HKOScon
Fluentd at HKOSconFluentd at HKOScon
Fluentd at HKOSconN Masahiro
 
Tutorial on-python-programming
Tutorial on-python-programmingTutorial on-python-programming
Tutorial on-python-programmingChetan Giridhar
 
Tuenti Release Workflow
Tuenti Release WorkflowTuenti Release Workflow
Tuenti Release WorkflowTuenti
 
Python 3.6 Features 20161207
Python 3.6 Features 20161207Python 3.6 Features 20161207
Python 3.6 Features 20161207Jay Coskey
 
How to Write the Fastest JSON Parser/Writer in the World
How to Write the Fastest JSON Parser/Writer in the WorldHow to Write the Fastest JSON Parser/Writer in the World
How to Write the Fastest JSON Parser/Writer in the WorldMilo Yip
 
Source version control using subversion
Source version control using subversionSource version control using subversion
Source version control using subversionMangesh Bhujbal
 
Writing Asynchronous Programs with Scala & Akka
Writing Asynchronous Programs with Scala & AkkaWriting Asynchronous Programs with Scala & Akka
Writing Asynchronous Programs with Scala & AkkaYardena Meymann
 
Dynamic Instrumentation- OpenEBS Golang Meetup July 2017
Dynamic Instrumentation- OpenEBS Golang Meetup July 2017Dynamic Instrumentation- OpenEBS Golang Meetup July 2017
Dynamic Instrumentation- OpenEBS Golang Meetup July 2017OpenEBS
 
Effizientere WordPress-Plugin-Entwicklung mit Softwaretests
Effizientere WordPress-Plugin-Entwicklung mit SoftwaretestsEffizientere WordPress-Plugin-Entwicklung mit Softwaretests
Effizientere WordPress-Plugin-Entwicklung mit SoftwaretestsDECK36
 

Similar to Fluentd v0.14 Plugin API Details: New Classes, Buffering, Storage Plugins (20)

Treasure Data Summer Internship 2016
Treasure Data Summer Internship 2016Treasure Data Summer Internship 2016
Treasure Data Summer Internship 2016
 
Fluentd v1 and future at techtalk
Fluentd v1 and future at techtalkFluentd v1 and future at techtalk
Fluentd v1 and future at techtalk
 
Fluentd v1.0 in a nutshell
Fluentd v1.0 in a nutshellFluentd v1.0 in a nutshell
Fluentd v1.0 in a nutshell
 
HHVM and Hack: A quick introduction
HHVM and Hack: A quick introductionHHVM and Hack: A quick introduction
HHVM and Hack: A quick introduction
 
Continuous Go Profiling & Observability
Continuous Go Profiling & ObservabilityContinuous Go Profiling & Observability
Continuous Go Profiling & Observability
 
Systems Programming Assignment Help - Processes
Systems Programming Assignment Help - ProcessesSystems Programming Assignment Help - Processes
Systems Programming Assignment Help - Processes
 
From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...
 
¡El mejor lenguaje para automatizar pruebas!
¡El mejor lenguaje para automatizar pruebas!¡El mejor lenguaje para automatizar pruebas!
¡El mejor lenguaje para automatizar pruebas!
 
Wait, IPython can do that?! (30 minutes)
Wait, IPython can do that?! (30 minutes)Wait, IPython can do that?! (30 minutes)
Wait, IPython can do that?! (30 minutes)
 
web programming UNIT VIII python by Bhavsingh Maloth
web programming UNIT VIII python by Bhavsingh Malothweb programming UNIT VIII python by Bhavsingh Maloth
web programming UNIT VIII python by Bhavsingh Maloth
 
Fluentd at HKOScon
Fluentd at HKOSconFluentd at HKOScon
Fluentd at HKOScon
 
Functions
FunctionsFunctions
Functions
 
Tutorial on-python-programming
Tutorial on-python-programmingTutorial on-python-programming
Tutorial on-python-programming
 
Tuenti Release Workflow
Tuenti Release WorkflowTuenti Release Workflow
Tuenti Release Workflow
 
Python 3.6 Features 20161207
Python 3.6 Features 20161207Python 3.6 Features 20161207
Python 3.6 Features 20161207
 
How to Write the Fastest JSON Parser/Writer in the World
How to Write the Fastest JSON Parser/Writer in the WorldHow to Write the Fastest JSON Parser/Writer in the World
How to Write the Fastest JSON Parser/Writer in the World
 
Source version control using subversion
Source version control using subversionSource version control using subversion
Source version control using subversion
 
Writing Asynchronous Programs with Scala & Akka
Writing Asynchronous Programs with Scala & AkkaWriting Asynchronous Programs with Scala & Akka
Writing Asynchronous Programs with Scala & Akka
 
Dynamic Instrumentation- OpenEBS Golang Meetup July 2017
Dynamic Instrumentation- OpenEBS Golang Meetup July 2017Dynamic Instrumentation- OpenEBS Golang Meetup July 2017
Dynamic Instrumentation- OpenEBS Golang Meetup July 2017
 
Effizientere WordPress-Plugin-Entwicklung mit Softwaretests
Effizientere WordPress-Plugin-Entwicklung mit SoftwaretestsEffizientere WordPress-Plugin-Entwicklung mit Softwaretests
Effizientere WordPress-Plugin-Entwicklung mit Softwaretests
 

More from SATOSHI TAGOMORI

Ractor's speed is not light-speed
Ractor's speed is not light-speedRactor's speed is not light-speed
Ractor's speed is not light-speedSATOSHI TAGOMORI
 
Good Things and Hard Things of SaaS Development/Operations
Good Things and Hard Things of SaaS Development/OperationsGood Things and Hard Things of SaaS Development/Operations
Good Things and Hard Things of SaaS Development/OperationsSATOSHI TAGOMORI
 
Invitation to the dark side of Ruby
Invitation to the dark side of RubyInvitation to the dark side of Ruby
Invitation to the dark side of RubySATOSHI TAGOMORI
 
Hijacking Ruby Syntax in Ruby (RubyConf 2018)
Hijacking Ruby Syntax in Ruby (RubyConf 2018)Hijacking Ruby Syntax in Ruby (RubyConf 2018)
Hijacking Ruby Syntax in Ruby (RubyConf 2018)SATOSHI TAGOMORI
 
Make Your Ruby Script Confusing
Make Your Ruby Script ConfusingMake Your Ruby Script Confusing
Make Your Ruby Script ConfusingSATOSHI TAGOMORI
 
Hijacking Ruby Syntax in Ruby
Hijacking Ruby Syntax in RubyHijacking Ruby Syntax in Ruby
Hijacking Ruby Syntax in RubySATOSHI TAGOMORI
 
Lock, Concurrency and Throughput of Exclusive Operations
Lock, Concurrency and Throughput of Exclusive OperationsLock, Concurrency and Throughput of Exclusive Operations
Lock, Concurrency and Throughput of Exclusive OperationsSATOSHI TAGOMORI
 
Data Processing and Ruby in the World
Data Processing and Ruby in the WorldData Processing and Ruby in the World
Data Processing and Ruby in the WorldSATOSHI TAGOMORI
 
Planet-scale Data Ingestion Pipeline: Bigdam
Planet-scale Data Ingestion Pipeline: BigdamPlanet-scale Data Ingestion Pipeline: Bigdam
Planet-scale Data Ingestion Pipeline: BigdamSATOSHI TAGOMORI
 
Technologies, Data Analytics Service and Enterprise Business
Technologies, Data Analytics Service and Enterprise BusinessTechnologies, Data Analytics Service and Enterprise Business
Technologies, Data Analytics Service and Enterprise BusinessSATOSHI TAGOMORI
 
Distributed Logging Architecture in Container Era
Distributed Logging Architecture in Container EraDistributed Logging Architecture in Container Era
Distributed Logging Architecture in Container EraSATOSHI TAGOMORI
 
Overview of data analytics service: Treasure Data Service
Overview of data analytics service: Treasure Data ServiceOverview of data analytics service: Treasure Data Service
Overview of data analytics service: Treasure Data ServiceSATOSHI TAGOMORI
 
Hive dirty/beautiful hacks in TD
Hive dirty/beautiful hacks in TDHive dirty/beautiful hacks in TD
Hive dirty/beautiful hacks in TDSATOSHI TAGOMORI
 
Data Analytics Service Company and Its Ruby Usage
Data Analytics Service Company and Its Ruby UsageData Analytics Service Company and Its Ruby Usage
Data Analytics Service Company and Its Ruby UsageSATOSHI TAGOMORI
 
Tale of ISUCON and Its Bench Tools
Tale of ISUCON and Its Bench ToolsTale of ISUCON and Its Bench Tools
Tale of ISUCON and Its Bench ToolsSATOSHI TAGOMORI
 
Data Analytics Service Company and Its Ruby Usage
Data Analytics Service Company and Its Ruby UsageData Analytics Service Company and Its Ruby Usage
Data Analytics Service Company and Its Ruby UsageSATOSHI TAGOMORI
 
Data-Driven Development Era and Its Technologies
Data-Driven Development Era and Its TechnologiesData-Driven Development Era and Its Technologies
Data-Driven Development Era and Its TechnologiesSATOSHI TAGOMORI
 
Engineer as a Leading Role
Engineer as a Leading RoleEngineer as a Leading Role
Engineer as a Leading RoleSATOSHI TAGOMORI
 

More from SATOSHI TAGOMORI (20)

Ractor's speed is not light-speed
Ractor's speed is not light-speedRactor's speed is not light-speed
Ractor's speed is not light-speed
 
Good Things and Hard Things of SaaS Development/Operations
Good Things and Hard Things of SaaS Development/OperationsGood Things and Hard Things of SaaS Development/Operations
Good Things and Hard Things of SaaS Development/Operations
 
Maccro Strikes Back
Maccro Strikes BackMaccro Strikes Back
Maccro Strikes Back
 
Invitation to the dark side of Ruby
Invitation to the dark side of RubyInvitation to the dark side of Ruby
Invitation to the dark side of Ruby
 
Hijacking Ruby Syntax in Ruby (RubyConf 2018)
Hijacking Ruby Syntax in Ruby (RubyConf 2018)Hijacking Ruby Syntax in Ruby (RubyConf 2018)
Hijacking Ruby Syntax in Ruby (RubyConf 2018)
 
Make Your Ruby Script Confusing
Make Your Ruby Script ConfusingMake Your Ruby Script Confusing
Make Your Ruby Script Confusing
 
Hijacking Ruby Syntax in Ruby
Hijacking Ruby Syntax in RubyHijacking Ruby Syntax in Ruby
Hijacking Ruby Syntax in Ruby
 
Lock, Concurrency and Throughput of Exclusive Operations
Lock, Concurrency and Throughput of Exclusive OperationsLock, Concurrency and Throughput of Exclusive Operations
Lock, Concurrency and Throughput of Exclusive Operations
 
Data Processing and Ruby in the World
Data Processing and Ruby in the WorldData Processing and Ruby in the World
Data Processing and Ruby in the World
 
Planet-scale Data Ingestion Pipeline: Bigdam
Planet-scale Data Ingestion Pipeline: BigdamPlanet-scale Data Ingestion Pipeline: Bigdam
Planet-scale Data Ingestion Pipeline: Bigdam
 
Technologies, Data Analytics Service and Enterprise Business
Technologies, Data Analytics Service and Enterprise BusinessTechnologies, Data Analytics Service and Enterprise Business
Technologies, Data Analytics Service and Enterprise Business
 
Fluentd 101
Fluentd 101Fluentd 101
Fluentd 101
 
Distributed Logging Architecture in Container Era
Distributed Logging Architecture in Container EraDistributed Logging Architecture in Container Era
Distributed Logging Architecture in Container Era
 
Overview of data analytics service: Treasure Data Service
Overview of data analytics service: Treasure Data ServiceOverview of data analytics service: Treasure Data Service
Overview of data analytics service: Treasure Data Service
 
Hive dirty/beautiful hacks in TD
Hive dirty/beautiful hacks in TDHive dirty/beautiful hacks in TD
Hive dirty/beautiful hacks in TD
 
Data Analytics Service Company and Its Ruby Usage
Data Analytics Service Company and Its Ruby UsageData Analytics Service Company and Its Ruby Usage
Data Analytics Service Company and Its Ruby Usage
 
Tale of ISUCON and Its Bench Tools
Tale of ISUCON and Its Bench ToolsTale of ISUCON and Its Bench Tools
Tale of ISUCON and Its Bench Tools
 
Data Analytics Service Company and Its Ruby Usage
Data Analytics Service Company and Its Ruby UsageData Analytics Service Company and Its Ruby Usage
Data Analytics Service Company and Its Ruby Usage
 
Data-Driven Development Era and Its Technologies
Data-Driven Development Era and Its TechnologiesData-Driven Development Era and Its Technologies
Data-Driven Development Era and Its Technologies
 
Engineer as a Leading Role
Engineer as a Leading RoleEngineer as a Leading Role
Engineer as a Leading Role
 

Recently uploaded

Keeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldKeeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldRoberto Pérez Alcolea
 
What’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 UpdatesWhat’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 UpdatesVictoriaMetrics
 
Preparing BitVisor for Supporting Multiple Architectures
Preparing BitVisor for Supporting Multiple ArchitecturesPreparing BitVisor for Supporting Multiple Architectures
Preparing BitVisor for Supporting Multiple ArchitecturesAke Koomsin
 
OpenMetadata Community Meeting - 4th April, 2024
OpenMetadata Community Meeting - 4th April, 2024OpenMetadata Community Meeting - 4th April, 2024
OpenMetadata Community Meeting - 4th April, 2024OpenMetadata
 
Advantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptxAdvantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptxRTS corp
 
Santander Stream Processing with Apache Flink
Santander Stream Processing with Apache FlinkSantander Stream Processing with Apache Flink
Santander Stream Processing with Apache Flinkconfluent
 
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...Bert Jan Schrijver
 
Zer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdfZer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdfmaor17
 
Key Steps in Agile Software Delivery Roadmap
Key Steps in Agile Software Delivery RoadmapKey Steps in Agile Software Delivery Roadmap
Key Steps in Agile Software Delivery RoadmapIshara Amarasekera
 
Business Analyzopedia - Your Pocket Gita for Business Analysis
Business Analyzopedia - Your Pocket Gita for Business AnalysisBusiness Analyzopedia - Your Pocket Gita for Business Analysis
Business Analyzopedia - Your Pocket Gita for Business AnalysisDEEPRAJ PATHAK
 
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxUnderstanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxSasikiranMarri
 
Chapter -5 Agile Testing types and its examples.pptx
Chapter -5 Agile Testing types and its examples.pptxChapter -5 Agile Testing types and its examples.pptx
Chapter -5 Agile Testing types and its examples.pptxManishaPatil932723
 
SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?Alexandre Beguel
 
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdfSteve Caron
 
Tech Tuesday Slides - Getting Started with the Portfolio Module.
Tech Tuesday Slides - Getting Started with the Portfolio Module.Tech Tuesday Slides - Getting Started with the Portfolio Module.
Tech Tuesday Slides - Getting Started with the Portfolio Module.OnePlan Solutions
 
Revolutionize Your Video Editing with InVideo.io: A Comprehensive Review
Revolutionize Your Video Editing with InVideo.io: A Comprehensive ReviewRevolutionize Your Video Editing with InVideo.io: A Comprehensive Review
Revolutionize Your Video Editing with InVideo.io: A Comprehensive Reviewjw364beach
 
full course of software engineering mid term.pdf
full course of software engineering mid term.pdffull course of software engineering mid term.pdf
full course of software engineering mid term.pdfAbdul salam
 
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...kalichargn70th171
 
logical backup of Oracle Datapump-detailed.pptx
logical backup of Oracle Datapump-detailed.pptxlogical backup of Oracle Datapump-detailed.pptx
logical backup of Oracle Datapump-detailed.pptxRemote DBA Services
 
eSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolseSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolsosttopstonverter
 

Recently uploaded (20)

Keeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldKeeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository world
 
What’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 UpdatesWhat’s New in VictoriaMetrics: Q1 2024 Updates
What’s New in VictoriaMetrics: Q1 2024 Updates
 
Preparing BitVisor for Supporting Multiple Architectures
Preparing BitVisor for Supporting Multiple ArchitecturesPreparing BitVisor for Supporting Multiple Architectures
Preparing BitVisor for Supporting Multiple Architectures
 
OpenMetadata Community Meeting - 4th April, 2024
OpenMetadata Community Meeting - 4th April, 2024OpenMetadata Community Meeting - 4th April, 2024
OpenMetadata Community Meeting - 4th April, 2024
 
Advantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptxAdvantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptx
 
Santander Stream Processing with Apache Flink
Santander Stream Processing with Apache FlinkSantander Stream Processing with Apache Flink
Santander Stream Processing with Apache Flink
 
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
JavaLand 2024 - Going serverless with Quarkus GraalVM native images and AWS L...
 
Zer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdfZer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdf
 
Key Steps in Agile Software Delivery Roadmap
Key Steps in Agile Software Delivery RoadmapKey Steps in Agile Software Delivery Roadmap
Key Steps in Agile Software Delivery Roadmap
 
Business Analyzopedia - Your Pocket Gita for Business Analysis
Business Analyzopedia - Your Pocket Gita for Business AnalysisBusiness Analyzopedia - Your Pocket Gita for Business Analysis
Business Analyzopedia - Your Pocket Gita for Business Analysis
 
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxUnderstanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
 
Chapter -5 Agile Testing types and its examples.pptx
Chapter -5 Agile Testing types and its examples.pptxChapter -5 Agile Testing types and its examples.pptx
Chapter -5 Agile Testing types and its examples.pptx
 
SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?SAM Training Session - How to use EXCEL ?
SAM Training Session - How to use EXCEL ?
 
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
 
Tech Tuesday Slides - Getting Started with the Portfolio Module.
Tech Tuesday Slides - Getting Started with the Portfolio Module.Tech Tuesday Slides - Getting Started with the Portfolio Module.
Tech Tuesday Slides - Getting Started with the Portfolio Module.
 
Revolutionize Your Video Editing with InVideo.io: A Comprehensive Review
Revolutionize Your Video Editing with InVideo.io: A Comprehensive ReviewRevolutionize Your Video Editing with InVideo.io: A Comprehensive Review
Revolutionize Your Video Editing with InVideo.io: A Comprehensive Review
 
full course of software engineering mid term.pdf
full course of software engineering mid term.pdffull course of software engineering mid term.pdf
full course of software engineering mid term.pdf
 
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
The Ultimate Guide to Performance Testing in Low-Code, No-Code Environments (...
 
logical backup of Oracle Datapump-detailed.pptx
logical backup of Oracle Datapump-detailed.pptxlogical backup of Oracle Datapump-detailed.pptx
logical backup of Oracle Datapump-detailed.pptx
 
eSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolseSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration tools
 

Fluentd v0.14 Plugin API Details: New Classes, Buffering, Storage Plugins

  • 1. Fluentd v0.14 Plugin API Details Fluentd meetup 2016 Summer Jun 1, 2016 Satoshi "Moris" Tagomori (@tagomoris)
  • 2. Satoshi "Moris" Tagomori (@tagomoris) Fluentd, MessagePack-Ruby, Norikra, ... Treasure Data, Inc.
  • 3. Topics • Why Fluentd v0.14 has a new API set for plugins • Compatibility of v0.12 plugins/configurations • Plugin APIs: Input, Filter, Output & Buffer • Storage Plugin, Plugin Helpers • New Test Drivers for plugins • Plans for v0.14.x & v1
  • 4. Why Fluentd v0.14 has a New API set for plugins?
  • 5. Fluentd v0.12 Plugins • No supports to write plugins by Fluentd core • plugins creates threads, sockets, timers and event loops • writing tests is very hard and messy with sleeps • Fragmented implementations • Output, BufferedOutput, ObjectBufferedOutput and TimeSlicedOutput • Mixture of configuration parameters from output&buffer • Uncontrolled plugin instance lifecycle (no "super" in start/shutdown) • Imperfect buffering control and useless configurations • the reason why fluent-plugin-forest exists and be used widely
  • 6. Fluentd v0.12 Plugins • Insufficient buffer chunking control • only by size, without number of events in chunks • Forcedly synchronized buffer flushing • no way to flush-and-commit chunks asynchronously • Ultimate freedom for using mix-ins • everything overrides Plugin#emit ... (the only one entry point for events to plugins) • no valid hook points to get metrics or something else • Bad Ruby coding rules and practices • too many classes at "Fluent::*" in fluent/plugin, no "require", ...
  • 9. Compatibility of plugins • v0.12 plugins are subclass of Fluent::* • Fluent::Input, Fluent::Filter, Fluent::Output, ... • Compatibility layers for v0.12 plugins in v0.14 • Fluent::Compat::Klass -> Fluent::Klass (e.g., Input, Output, ...) • it provides transformation of: • namespaces, configuration parameters • internal APIs, argument objects • IT SHOULD WORK, except for :P • 3rd party buffer plugin, part of test code • "Engine.emit"
  • 10. Compatibility of configurations • v0.14 plugins have another set of parameters • many old-fashioned parameters are removed • "buffer_type", "num_threads", "timezone", "time_slice_format", "buffer_chunk_limit", "buffer_queue_limit", ... • Plugin helper "compat_parameters" • transform parameters between v0.12 style configuration and v0.14 plugin v0.12 v0.14 convert internally
  • 11. FAQ: Can we create plugins like this? * it uses v0.14 API * it runs on Fluentd v0.12 Impossible :P
  • 13. v0.14 plugin classes • All files MUST be in `fluent/plugin/*.rb` (in gems) • or just a "*.rb" file in directory specified by "-r" • All classes MUST be under Fluent::Plugin • All plugins MUST be subclasses of Fluent::Plugin::Base • All plugins MUST call `super` in methods overriding default implementation (e.g., #configure, #start, #shutdown, ...)
  • 14. Classes hierarchy (v0.12) Fluent::Input F::Filter F::Output BufferedOutput Object Buffered Time Sliced Multi Output F::Buffer F::Parser F::Formatter 3rd party plugins
  • 15. Classes hierarchy (v0.14) F::P::Input F::P::Filter F::P::Output Fluent::Plugin::Base F::P::Buffer F::P::Parser F::P::Formatter F::P::Storage both of buffered/non-buffered F::P:: BareOutput (not for 3rd party plugins) F::P:: MultiOutput copy roundrobin
  • 16. Tour of New Plugin APIs: Fluent::Plugin::Input
  • 17. Fluent::Plugin::Input • Nothing changed :) • except for overall rules • But it's much easier
 to write plugins
 than v0.12 :) • fetch HTTP resource per specified interval • parse response body with format specified in config • emit parse result
  • 19. Tour of New Plugin APIs: Fluent::Plugin::Filter
  • 20. Fluent::Plugin::Filter • Almost nothing changed :) • Required:
 #filter(tag, time, record)
 #=> record | nil • Optional:
 #filter_stream(tag, es)
 #=> event_stream
  • 21. Tour of New Plugin APIs: Fluent::Plugin::Output
  • 22. Fluent::Plugin::Output • Many things changed! • Merged Output, BufferedOutput, ObjectBufferedOutput, TimeSlicedOutput • Output plugins can be • with buffering • without buffering • both (do/doesn't buffering by configuration) • Buffers chunks events by: • byte size, interval, tag • number of records (new!) • time (by any unit(new!): 30s, 5m, 15m, 3h, ...) • any specified field in records (new!) • any combination of above (new!)
  • 23. Variations of buffering NO MORE forest plugin!
  • 24. Output Plugin: Methods to be implemented • Non-buffered: #process(tag, es) • Buffered synchronous: #write(chunk) • Buffered Asynchronous: #try_write(chunk) • New feature for destinations with huge latency to write chunks • Plugins must call #commit_write(chunk_id) (otherwise, #try_write will be retried) • Buffered w/ custom format: #format(tag, time, record) • Without this method, output uses standard format
  • 25. implement? #process implement? #process or #write or #try_write NO error YES #prefer_buffered_processing called (default true) NO non-buffered YES exists? <buffer> section YES implement? #write or #try_write error NO YES implement? #write or #try_write NO NO YES false implement? #write and #try_write YES #prefer_delayed_commit called (default true) implement? #try_write sync buffered async buffered
  • 26. In other words :P • If users configure "<buffer>" section • plugin try to do buffering • Else if plugin implements both (buffering/non-buf) • plugin call #prefer_buffer_processing to decide • Else plugin does as implemented • When plugin does buffering
 If plugin implements both (sync/async write) • plugin call #prefer_delayed_commit to decide • Else plugin does as implemented
  • 27. Delayed commit (1) • high latency #write operations locks a flush thread for long time
 (e.g., ACK in forward) destination w/ high latency #write Output Plugin send data send ACK return #write a flush thread locked
  • 28. Delayed commit (2) • #try_write & delayed #commit_write destination w/ high latency #try_write Output Plugin send data send ACK return #try_write async check thread #commit_write
  • 29. Use cases: delayed commit • Forward protocol w/ ACK • Distributed file systems or databases • put data -> confirm to read data -> commit • Submit tasks to job queues • submit a job -> detect executed -> commit
  • 30. Standard chunk format • Buffering w/o #format method • Almost same with ObjectBufferedOutput • No need to implement #format always • Implement it for performance/low-latency • Tool to dump & read buffer chunks on disk w/ standard format • To be implemented in v0.14.x :)
  • 31. <buffer CHUNK_KEYS> • comma-separated tag, time or ANY_KEYS • Nothing specified: all events are in same chunk • flushed when chunk is full • (optional) "flush_interval" after first event in chunk • tag: events w/ same tag are in same chunks • time: buffer chunks will be split by timekey • timekey: unit of time to be chunked (1m, 15m, 3h, ...) • flushed after expiration of timekey unit + timekey_wait • ANY_KEYS: any key names in records
  • 32. • comma-separated tag, time or ANY_KEYS • Nothing specified: all events are in same chunk • flushed when chunk is full • (optional) "flush_interval" after first event in chunk • tag: events w/ same tag are in same chunks • time: buffer chunks will be split by timekey • timekey: unit of time to be chunked (1m, 15m, 3h, ...) • flushed after expiration of timekey unit + timekey_wait • ANY_KEYS: any key names in records <buffer CHUNK_KEYS> BufferedOutput TimeSlicedOutput ObjectBufferedOutput in v0.12 in v0.12 in v0.12
  • 33. configurations:
 flushing buffers • flush_mode: lazy, interval, immediate • default: lazy if "time" specified, otherwise interval • flush_interval, flush_thread_count • flush_thread_count: number of threads for flushing • delayed_commit_timeout • output plugin will retry #try_write when expires
  • 34. Retries, Secondary • Explicit timeout for retries: • retry_timeout: timeout not to retry anymore • retry_max_times: how many times to retry • retry_type: "periodic" w/ fixed retry_wait • retry_secondary_threshold (percentage) • output will use secondary if specified percentage of retry_timeout elapsed after first error
  • 35. Buffer parameters • chunk_limit_size • maximum bytesize per chunks • chunk_records_limit (default: not specified) • maximum number of records per chunks • total_limit_size • maximum bytesize which a buffer plugin can use • (optional) queue_length_limit: no need to specify
  • 36. Chunk metadata • Stores various information of buffer chunks • key-values of chunking unit • number of records • created_at, modified_at • `chunk.metadata` • extract_placeholders(@path, chunk.metadata)
  • 37. Tour of New Plugin APIs: Other plugin types
  • 38. Classes hierarchy (v0.14) F::P::Input F::P::Filter F::P::Output Fluent::Plugin::Base F::P::Buffer F::P::Parser F::P::Formatter F::P::Storage both of buffered/non-buffered F::P:: BareOutput (not for 3rd party plugins) F::P:: MultiOutput copy roundrobin
  • 39. Classes hierarchy (v0.14) F::P::Input F::P::Filter F::P::Output Fluent::Plugin::Base F::P::Buffer F::P::Parser F::P::Formatter F::P::Storage both of buffered/non-buffered F::P:: BareOutput (not for 3rd party plugins) F::P:: MultiOutput copy roundrobin"Owned" plugins
  • 40. "Owned" plugins • Primary plugins: Input, Output, Filter • Instantiated by Fluentd core • "Owned" plugins are owned by primary plugins • Buffer, Parser, Formatter, Storage, ... • It can refer owner's plugin id, logger, ... • Fluent::Plugin.new_xxx("kind", parent:@input) • "Owned" plugins can be configured by owner plugins
  • 41. Owner plugins can control defaults of owned plugins Fluentd provides standard way to configure owned plugins
  • 42. Tour of New Plugin APIs: Fluent::Plugin::Storage
  • 43. Storage plugins • Pluggable Key-Value store for plugins • configurable: autosave, persistent, save_at_shutdown • get, fetch, put, delete, update (transactional) • Various possible implementations • built-in: local (json) on-disk / on-memory • possible: Redis, Consul,
 or whatever supports serialize/deserialize of json-like object • To store states of plugins: • counter values of data-counter plugin • pos data of file plugin • To load configuration dynamically for plugins: • load configurations from any file systems
  • 44. Tour of New Plugin APIs: Plugin Helpers
  • 45. Plugin Helpers • No more mixin! • declare to use helpers by "helpers :name" • Utility functions to support difficult things • creating threads, timers, child processes... • created timers will be stopped automatically in plugin's shutdown sequence • Integrated w/ New Test Drivers • tests runs after helpers started everything requested
  • 46. Plugin Helpers Example • Thread: thread_create, thread_current_running? • Timer: timer_execute • ChildProcess: child_process_execute • command, arguments, subprocess_name, interval, immediate, parallel, mode, stderr, env, unsetenv, chdir, ... • EventEmitter: router (Output doesn't have router in v0.14 default) • Storage: storage_create • (TBD) Socket/Server for TCP/UDP/TLS, Parser, Formatter
  • 47.
  • 48. Tour of New Plugin APIs: New Test Drivers
  • 49. New Test Drivers • Instead of old drivers Fluent::Test::*TestDriver • Fluent::Test::Driver::Input, Output or Filter • fully emulates actual plugin behavior • w/ override SystemConfig • capturing emitted events & error event streams • inserting TestLogger to capture/test logs of plugins • capturing "format" result of output plugins • controlling "flush" timing of output plugins • Running tests under control • Plugin Helper integration • conditions to keep/break running tests • timeouts, number of emits/events to stop tests • automatic start/shutdown call for plugins
  • 51. New Features • Symmetric multi processing • to use 2 or more CPU cores! • by sharing a configuration between all processes • "detach_process" will be deprecated • forward: TLS + authentication/authorization support • secure-forward integration • Buffer supports compression & forward it • Plugin generator & template
  • 52. New APIs • Controlling global configuration from SystemConfig • configured via <system> tag • root buffer path + plugin id: remove paths from each buffers • process total buffer size control • Counter APIs • counting everything over processes via RPC • creating metrics for a whole fluentd cluster
  • 54. v1: stable version of v0.14 • v0.12 plugins will be still supported at v1.0.0 • deprecated, and will be obsoleted at v1.x • Will be obsoleted: • v0 (traditional) configuration syntax • "detach_process" feature • Q4 2016?
  • 55. To Be Written by me :-) • As soooooooooon as possible... • Plugin developers' guide for • Updating v0.12 plugins with v0.14 APIs • Writing plugins with v0.14 APIs • Writing tests of plugins with v0.14 APIs • Users' guide for • How to use buffering in general (w/ <buffer>) • Updated plugin documents