SlideShare a Scribd company logo
1 of 24
Download to read offline
Fluentd v1.0 in a nutshell
June 1, 2017
Masahiro Nakagawa
Fluentd v0.12
• Current stable and widely used on production
• Input, Parser, Filter, Formatter, Buffer, Output plugins
• Known issues
• Event time is second unit
• No Windows support
• No multi core support
• Need to improve plugin API to support more various
use cases
• Development version of v1
• Latest version is v0.14.17: March 30, 2017
• Implemented New features
• New Plugin APIs
• Event Time with Nanosecond resolution
• ServerEngine based Supervisor
• Windows support
• Multicore support
• New Plugin Helpers & Plugin Storage
Fluentd v0.14
Fluentd v1
• Stable announcement for APIs / features
• No breaking API changes in v1.x
• Compatible with v0.12 and v0.14
• exclude v0 config syntax and detach_process
• Release plan
• Q3, 2017
• Need v0.14 feedback from developers and users

https://hub.docker.com/r/fluent/fluentd/
New Plugin APIs
• Input/Output plugin APIs w/ well-controlled lifecycle
• stop, shutdown, close, terminate
• Integrate all output plugin into Fluent::Plugin::Output
• New Buffer API for delayed commit and flexible chunking with metadata
• parallel/async "commit" operation for chunks
• For high latency case: forward’s at-least-once, issuing job, etc…
• Users can choose chunk keys by configuration for dynamic parameters
• Compatible w/ v0.12 plugins
• compatibility layer for traditional APIs
• it will be supported between v1.x versions
Router
buffer_chunk_limit
enqueue: exceed flush_interval
or buffer_chunk_limit
Key pattern:
- BufferedOutput
empty string or specified key
-ObjectBufferedOutput tag
-TimeSlicedOutput time slice
emit emit
Buffer
Queue
buffer_queue_limit
Output
OutputInput / Filter
Tag Time
Record Chunk
Chunk
Chunk Chunk
Chunk
key:foo
key:bar
key:baz
v0.12 buffer design
v0.14 buffer design
Buffer keys and placeholders
• Dynamic parameters for table name, object path and more
• We can embed time, tag and any field with placeholder













<match s3.**>
@type s3
aws_key_id "#{ENV['AWS_ACCESS_KEY']}"
aws_sec_key "#{ENV['AWS_SECRETA_KEY']}"
s3_bucket fluent-plugin-s3
path test/%Y/%m/${tag}/${key}/
<buffer time,tag,key>
timekey 3600
</buffer>
</match>
http://docs.fluentd.org/v0.14/articles/buffer-section for more details
time: 2017-06-01 12:00:00 +0700
tag: “test”
record: {“key”:”hello”}
- Event sample
test/2017/6/test/hello/
- Generated “path”
Time with nanosecond
• For sub-second systems: Elasticsearch, InfluxData, etc…
• Fluent::EventTime
• behaves as Integer for v0.12’s second unit compatibility
• has methods to get sub-second resolution
• be serialized into msgpack using Ext type
• Fluent::Engine.now now returns EventTime, not Integer
• Fluentd core can handle both of Integer and EventTime as time
• compatible with older versions and software in eco-system
(e.g., fluent-logger, Docker logging driver)
ServerEngine based Supervisor
• ServerEngine is a framework for building robust server
• https://github.com/treasure-data/serverengine
• Replacing supervisor process with ServerEngine
• it has SocketManager to share listening sockets
between 2 or more worker processes
• Replacing Fluentd's processing model from fork to
spawn
• to support Windows environment
Windows support
• Fluentd and core plugins work on Windows
• Windows service registration is also supported
• http://docs.fluentd.org/v0.14/articles/install-by-msi
• Use HTTP RPC instead of signals
• https://github.com/fluent/fluent-plugin-windows-eventlog
• We can collect windows eventlog :)
Symmetric multi core processing
• 2 or more workers share a configuration file
• and share listening sockets via PluginHelper
• under a supervisor process (ServerEngine)
• Multi core scalability for huge traffic
• one input plugin for a tcp port, some filters and one
(or some) output plugin
• buffer paths are managed automatically by
Fluentd. Need root_dir and @id parameters
Worker0
Supervisor
v0.14’s multi process feature
grep
forward
tdlog
Worker1 Worker2
grep
forward
tdlog
grep
forward
tdlog
socket
Configuration example
<system>
workers 2
root_dir /var/log/fluentd
</system>
<source>
@type forward
</source>
<filter pattern>
@type grep
</filter>
<match pattern>
@type tdlog
@id out_td
</match>
/var/log/fluentd/worker0/out_td/buffer/buffer.xxx.log
/var/log/fluentd/worker0/out_td/buffer/buffer.xxx.log.meta
- buf_file’s path is automatically generated
worker id
root_dir
plugin’s @id
TLS/Authn/Authz support for forward plugin
• Support v1 forward protocol spec
• secure-forward is merged into built-in forward
• TLS w/ at-least-one semantics
• Simple authentication/authorization w/o SSL
• Different points
• secure-forward uses keep-alive, but forward doesn’t
• secure-forward uses thread per connection, but
forward uses cool.io, libev based IO.
http://www.fluentd.org/blog/fluentd-v0.14.12-has-been-released
Plugin Storage & Helpers
• Plugin Storage: new plugin type for plugins
• provides key-value storage to persistent intermediate status
• built-in plugins: in-memory, local file
• pluggable: 3rd party plugin to store data into storage
• storage-redis
• Plugin Helpers:
• collections of utility methods for plugins
• fully integrated with test drivers to run test code after setup
phase of helpers (e.g. test started after created threads)
server helper: before
def start
@loop = Coolio::Loop.new
@handler = Coolio::TCPServer.new(@bind, @port, SocketUtil::TcpHandler, log,
@delimiter, method(:on_message))
@loop.attach(@handler)
@thread = Thread.new(&method(:run))
end
def shutdown
@loop.watchers.each { |w| w.detach }
@loop.stop
@handler.close
@thread.join
end
def run
@loop.run
rescue => e
log.error "unexpected error", error: e
log.error_backtrace
end
def on_message(msg, addr)
# body
end
server helper: after
def start
server_create(:foo_server, @port, bind: @bind) do |data, conn|

# body

end
end
v0.12 plugins
ParserInput Buffer Output FormatterFilter
“output-ish”“input-ish”
v0.14 plugins
ParserInput Buffer Output FormatterFilter
“output-ish”“input-ish”
Storage
Helper
Other helpers
• Timer: one-shot / periodic timer
• Event Loop: Low-layer event loop
• Socket: TCP/UDP/TLS support
• Formatter/Parser: Manage parser/formatter plugins
• Chile Process: Manage process for exec like plugin
• etc…
Is v0.14 stable?
• Not yet but it will be stable in this month (I hope!)
• Remaining tickets:
• Nested record support
• Fix core bugs
• v0.14 stable means ready for production, not
feature freeze. Feature freeze is done by v1 release
TODO list for v1
• Counter API to store metrics between processes
• https://github.com/fluent/fluentd/tree/counter-api
• Optimize performance and remove deprecated code
• Migrate more plugin into v0.14 API
• Core plugins are migrated
• Popular plugins release beta version for v0.14
• Write document: design, plugin, operation, etc…
Treasure Agent 3.0 serise (td-agent 3)
• fluentd v0.14/v1, Ruby 2.4, and latest core components
• Latest beta version is 3.0.1: May 19, 2017
• Environments
• Add msi Windows package
• Remove CentOS 5, Ubuntu 10.04 support
• Beta packages have beed released
• http://docs.fluentd.org/v0.14/categories/installation
• beta will be removed after v0.14 becomes stable :)

More Related Content

What's hot

fluent-plugin-beats at Elasticsearch meetup #14
fluent-plugin-beats at Elasticsearch meetup #14fluent-plugin-beats at Elasticsearch meetup #14
fluent-plugin-beats at Elasticsearch meetup #14N Masahiro
 
Fluentd Overview, Now and Then
Fluentd Overview, Now and ThenFluentd Overview, Now and Then
Fluentd Overview, Now and ThenSATOSHI TAGOMORI
 
FluentD for end to end monitoring
FluentD for end to end monitoringFluentD for end to end monitoring
FluentD for end to end monitoringPhil Wilkins
 
Fluentd v0.14 Plugin API Details
Fluentd v0.14 Plugin API DetailsFluentd v0.14 Plugin API Details
Fluentd v0.14 Plugin API DetailsSATOSHI TAGOMORI
 
Dive into Fluentd plugin v0.12
Dive into Fluentd plugin v0.12Dive into Fluentd plugin v0.12
Dive into Fluentd plugin v0.12N Masahiro
 
JRuby with Java Code in Data Processing World
JRuby with Java Code in Data Processing WorldJRuby with Java Code in Data Processing World
JRuby with Java Code in Data Processing WorldSATOSHI TAGOMORI
 
Fluentd - Set Up Once, Collect More
Fluentd - Set Up Once, Collect MoreFluentd - Set Up Once, Collect More
Fluentd - Set Up Once, Collect MoreSadayuki Furuhashi
 
Life of an Fluentd event
Life of an Fluentd eventLife of an Fluentd event
Life of an Fluentd eventKiyoto Tamura
 
Like loggly using open source
Like loggly using open sourceLike loggly using open source
Like loggly using open sourceThomas Alrin
 
Writing a fast HTTP parser
Writing a fast HTTP parserWriting a fast HTTP parser
Writing a fast HTTP parserfukamachi
 

What's hot (20)

fluent-plugin-beats at Elasticsearch meetup #14
fluent-plugin-beats at Elasticsearch meetup #14fluent-plugin-beats at Elasticsearch meetup #14
fluent-plugin-beats at Elasticsearch meetup #14
 
Fluentd Overview, Now and Then
Fluentd Overview, Now and ThenFluentd Overview, Now and Then
Fluentd Overview, Now and Then
 
Docker and Fluentd
Docker and FluentdDocker and Fluentd
Docker and Fluentd
 
Fluentd vs. Logstash for OpenStack Log Management
Fluentd vs. Logstash for OpenStack Log ManagementFluentd vs. Logstash for OpenStack Log Management
Fluentd vs. Logstash for OpenStack Log Management
 
The basics of fluentd
The basics of fluentdThe basics of fluentd
The basics of fluentd
 
FluentD for end to end monitoring
FluentD for end to end monitoringFluentD for end to end monitoring
FluentD for end to end monitoring
 
Fluentd v0.14 Plugin API Details
Fluentd v0.14 Plugin API DetailsFluentd v0.14 Plugin API Details
Fluentd v0.14 Plugin API Details
 
Dive into Fluentd plugin v0.12
Dive into Fluentd plugin v0.12Dive into Fluentd plugin v0.12
Dive into Fluentd plugin v0.12
 
Fluentd meetup in japan
Fluentd meetup in japanFluentd meetup in japan
Fluentd meetup in japan
 
JRuby with Java Code in Data Processing World
JRuby with Java Code in Data Processing WorldJRuby with Java Code in Data Processing World
JRuby with Java Code in Data Processing World
 
Fluentd - Set Up Once, Collect More
Fluentd - Set Up Once, Collect MoreFluentd - Set Up Once, Collect More
Fluentd - Set Up Once, Collect More
 
Fluentd introduction at ipros
Fluentd introduction at iprosFluentd introduction at ipros
Fluentd introduction at ipros
 
Life of an Fluentd event
Life of an Fluentd eventLife of an Fluentd event
Life of an Fluentd event
 
Fluentd meetup #2
Fluentd meetup #2Fluentd meetup #2
Fluentd meetup #2
 
Fluentd meetup
Fluentd meetupFluentd meetup
Fluentd meetup
 
Fluent-bit
Fluent-bitFluent-bit
Fluent-bit
 
Fluentd and AWS at classmethod
Fluentd and AWS at classmethodFluentd and AWS at classmethod
Fluentd and AWS at classmethod
 
Like loggly using open source
Like loggly using open sourceLike loggly using open source
Like loggly using open source
 
Node.js
Node.jsNode.js
Node.js
 
Writing a fast HTTP parser
Writing a fast HTTP parserWriting a fast HTTP parser
Writing a fast HTTP parser
 

Viewers also liked

JavaOne 2017 - Choosing a NoSQL API and Database to Avoid Tombstones and Drag...
JavaOne 2017 - Choosing a NoSQL API and Database to Avoid Tombstones and Drag...JavaOne 2017 - Choosing a NoSQL API and Database to Avoid Tombstones and Drag...
JavaOne 2017 - Choosing a NoSQL API and Database to Avoid Tombstones and Drag...Leonardo De Moura Rocha Lima
 
Reversing Engineering a Web Application - For fun, behavior and detection
Reversing Engineering a Web Application - For fun, behavior and detectionReversing Engineering a Web Application - For fun, behavior and detection
Reversing Engineering a Web Application - For fun, behavior and detectionRodrigo Montoro
 
Do we need a bigger dev data culture
Do we need a bigger dev data cultureDo we need a bigger dev data culture
Do we need a bigger dev data cultureSimon Dittlmann
 
Docker swarm-mike-goelzer-mv-meetup-45min-workshop 02242016 (1)
Docker swarm-mike-goelzer-mv-meetup-45min-workshop 02242016 (1)Docker swarm-mike-goelzer-mv-meetup-45min-workshop 02242016 (1)
Docker swarm-mike-goelzer-mv-meetup-45min-workshop 02242016 (1)Michelle Antebi
 
Monitoring and tuning your chef server - chef conf talk
Monitoring and tuning your chef server - chef conf talk Monitoring and tuning your chef server - chef conf talk
Monitoring and tuning your chef server - chef conf talk Andrew DuFour
 
Docker security introduction-task-2016
Docker security introduction-task-2016Docker security introduction-task-2016
Docker security introduction-task-2016Ricardo Gerardi
 
IBM Bluemix Nice meetup #5 - 20170504 - Container Service based on Kubernetes
IBM Bluemix Nice meetup #5 - 20170504 - Container Service based on KubernetesIBM Bluemix Nice meetup #5 - 20170504 - Container Service based on Kubernetes
IBM Bluemix Nice meetup #5 - 20170504 - Container Service based on KubernetesIBM France Lab
 
"How overlay networks can make public clouds your global WAN" by Ryan Koop o...
 "How overlay networks can make public clouds your global WAN" by Ryan Koop o... "How overlay networks can make public clouds your global WAN" by Ryan Koop o...
"How overlay networks can make public clouds your global WAN" by Ryan Koop o...Cohesive Networks
 
Retelling nonfiction
Retelling nonfictionRetelling nonfiction
Retelling nonfictionEmily Kissner
 
Evolutions et nouveaux outils SEO
Evolutions et nouveaux outils SEOEvolutions et nouveaux outils SEO
Evolutions et nouveaux outils SEODimitri Brunel
 
Performance monitoring and call tracing in microservice environments
Performance monitoring and call tracing in microservice environmentsPerformance monitoring and call tracing in microservice environments
Performance monitoring and call tracing in microservice environmentsMartin Gutenbrunner
 
VoxxedDays Bucharest 2017 - Powering interactive data analysis with Google Bi...
VoxxedDays Bucharest 2017 - Powering interactive data analysis with Google Bi...VoxxedDays Bucharest 2017 - Powering interactive data analysis with Google Bi...
VoxxedDays Bucharest 2017 - Powering interactive data analysis with Google Bi...Márton Kodok
 
Sitios turísticos de valledupar
Sitios turísticos de valleduparSitios turísticos de valledupar
Sitios turísticos de valleduparelkin
 
CloudStack EU user group - Trillian
CloudStack EU user group - TrillianCloudStack EU user group - Trillian
CloudStack EU user group - TrillianShapeBlue
 
Monitor all the cloud things - security monitoring for everyone
Monitor all the cloud things - security monitoring for everyoneMonitor all the cloud things - security monitoring for everyone
Monitor all the cloud things - security monitoring for everyoneDuncan Godfrey
 
AppSensor - Near Real Time Event Detection and Response
AppSensor - Near Real Time Event Detection and ResponseAppSensor - Near Real Time Event Detection and Response
AppSensor - Near Real Time Event Detection and Responsejtmelton
 
150430 regiosessie corv_almelo
150430 regiosessie corv_almelo150430 regiosessie corv_almelo
150430 regiosessie corv_almeloKING
 
How Cisco Migrated from MapReduce Jobs to Spark Jobs - StampedeCon 2015
How Cisco Migrated from MapReduce Jobs to Spark Jobs - StampedeCon 2015How Cisco Migrated from MapReduce Jobs to Spark Jobs - StampedeCon 2015
How Cisco Migrated from MapReduce Jobs to Spark Jobs - StampedeCon 2015StampedeCon
 

Viewers also liked (20)

Arquitecturas de microservicios - Medianet Software
Arquitecturas de microservicios   -  Medianet SoftwareArquitecturas de microservicios   -  Medianet Software
Arquitecturas de microservicios - Medianet Software
 
JavaOne 2017 - Choosing a NoSQL API and Database to Avoid Tombstones and Drag...
JavaOne 2017 - Choosing a NoSQL API and Database to Avoid Tombstones and Drag...JavaOne 2017 - Choosing a NoSQL API and Database to Avoid Tombstones and Drag...
JavaOne 2017 - Choosing a NoSQL API and Database to Avoid Tombstones and Drag...
 
Reversing Engineering a Web Application - For fun, behavior and detection
Reversing Engineering a Web Application - For fun, behavior and detectionReversing Engineering a Web Application - For fun, behavior and detection
Reversing Engineering a Web Application - For fun, behavior and detection
 
Do we need a bigger dev data culture
Do we need a bigger dev data cultureDo we need a bigger dev data culture
Do we need a bigger dev data culture
 
Docker swarm-mike-goelzer-mv-meetup-45min-workshop 02242016 (1)
Docker swarm-mike-goelzer-mv-meetup-45min-workshop 02242016 (1)Docker swarm-mike-goelzer-mv-meetup-45min-workshop 02242016 (1)
Docker swarm-mike-goelzer-mv-meetup-45min-workshop 02242016 (1)
 
Monitoring and tuning your chef server - chef conf talk
Monitoring and tuning your chef server - chef conf talk Monitoring and tuning your chef server - chef conf talk
Monitoring and tuning your chef server - chef conf talk
 
Docker security introduction-task-2016
Docker security introduction-task-2016Docker security introduction-task-2016
Docker security introduction-task-2016
 
IBM Bluemix Nice meetup #5 - 20170504 - Container Service based on Kubernetes
IBM Bluemix Nice meetup #5 - 20170504 - Container Service based on KubernetesIBM Bluemix Nice meetup #5 - 20170504 - Container Service based on Kubernetes
IBM Bluemix Nice meetup #5 - 20170504 - Container Service based on Kubernetes
 
"How overlay networks can make public clouds your global WAN" by Ryan Koop o...
 "How overlay networks can make public clouds your global WAN" by Ryan Koop o... "How overlay networks can make public clouds your global WAN" by Ryan Koop o...
"How overlay networks can make public clouds your global WAN" by Ryan Koop o...
 
Retelling nonfiction
Retelling nonfictionRetelling nonfiction
Retelling nonfiction
 
Evolutions et nouveaux outils SEO
Evolutions et nouveaux outils SEOEvolutions et nouveaux outils SEO
Evolutions et nouveaux outils SEO
 
Performance monitoring and call tracing in microservice environments
Performance monitoring and call tracing in microservice environmentsPerformance monitoring and call tracing in microservice environments
Performance monitoring and call tracing in microservice environments
 
VoxxedDays Bucharest 2017 - Powering interactive data analysis with Google Bi...
VoxxedDays Bucharest 2017 - Powering interactive data analysis with Google Bi...VoxxedDays Bucharest 2017 - Powering interactive data analysis with Google Bi...
VoxxedDays Bucharest 2017 - Powering interactive data analysis with Google Bi...
 
Sitios turísticos de valledupar
Sitios turísticos de valleduparSitios turísticos de valledupar
Sitios turísticos de valledupar
 
CloudStack EU user group - Trillian
CloudStack EU user group - TrillianCloudStack EU user group - Trillian
CloudStack EU user group - Trillian
 
What is DevOps?
What is DevOps?What is DevOps?
What is DevOps?
 
Monitor all the cloud things - security monitoring for everyone
Monitor all the cloud things - security monitoring for everyoneMonitor all the cloud things - security monitoring for everyone
Monitor all the cloud things - security monitoring for everyone
 
AppSensor - Near Real Time Event Detection and Response
AppSensor - Near Real Time Event Detection and ResponseAppSensor - Near Real Time Event Detection and Response
AppSensor - Near Real Time Event Detection and Response
 
150430 regiosessie corv_almelo
150430 regiosessie corv_almelo150430 regiosessie corv_almelo
150430 regiosessie corv_almelo
 
How Cisco Migrated from MapReduce Jobs to Spark Jobs - StampedeCon 2015
How Cisco Migrated from MapReduce Jobs to Spark Jobs - StampedeCon 2015How Cisco Migrated from MapReduce Jobs to Spark Jobs - StampedeCon 2015
How Cisco Migrated from MapReduce Jobs to Spark Jobs - StampedeCon 2015
 

Similar to Fluentd v1.0 in a nutshell

Fluentd v0.14 Overview
Fluentd v0.14 OverviewFluentd v0.14 Overview
Fluentd v0.14 OverviewN Masahiro
 
Treasure Data Summer Internship 2016
Treasure Data Summer Internship 2016Treasure Data Summer Internship 2016
Treasure Data Summer Internship 2016Yuta Iwama
 
Extending OpenShift Origin: Build Your Own Cartridge with Bill DeCoste of Red...
Extending OpenShift Origin: Build Your Own Cartridge with Bill DeCoste of Red...Extending OpenShift Origin: Build Your Own Cartridge with Bill DeCoste of Red...
Extending OpenShift Origin: Build Your Own Cartridge with Bill DeCoste of Red...OpenShift Origin
 
Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018
Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018
Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018Mandi Walls
 
What's new in kubernetes 1.3?
What's new in kubernetes 1.3?What's new in kubernetes 1.3?
What's new in kubernetes 1.3?Suraj Deshmukh
 
Hogy jussunk ki lezárt hálózatokból?
Hogy jussunk ki lezárt hálózatokból?Hogy jussunk ki lezárt hálózatokból?
Hogy jussunk ki lezárt hálózatokból?hackersuli
 
Kubernetes #1 intro
Kubernetes #1   introKubernetes #1   intro
Kubernetes #1 introTerry Cho
 
Neutrondev ppt
Neutrondev pptNeutrondev ppt
Neutrondev pptmarunewby
 
KACE Agent Architecture and Troubleshooting Overview
KACE Agent Architecture and Troubleshooting OverviewKACE Agent Architecture and Troubleshooting Overview
KACE Agent Architecture and Troubleshooting OverviewDell World
 
Fluentd - RubyKansai 65
Fluentd - RubyKansai 65Fluentd - RubyKansai 65
Fluentd - RubyKansai 65N Masahiro
 
Introduction to Node (15th May 2017)
Introduction to Node (15th May 2017)Introduction to Node (15th May 2017)
Introduction to Node (15th May 2017)Lucas Jellema
 
NGINX Installation and Tuning
NGINX Installation and TuningNGINX Installation and Tuning
NGINX Installation and TuningNGINX, Inc.
 
Provisioning with Puppet
Provisioning with PuppetProvisioning with Puppet
Provisioning with PuppetJoe Ray
 
Monitoring in Big Data Platform - Albert Lewandowski, GetInData
Monitoring in Big Data Platform - Albert Lewandowski, GetInDataMonitoring in Big Data Platform - Albert Lewandowski, GetInData
Monitoring in Big Data Platform - Albert Lewandowski, GetInDataGetInData
 
Chef - Administration for programmers
Chef - Administration for programmersChef - Administration for programmers
Chef - Administration for programmersmrsabo
 
Deep Dive Azure Functions - Global Azure Bootcamp 2019
Deep Dive Azure Functions - Global Azure Bootcamp 2019Deep Dive Azure Functions - Global Azure Bootcamp 2019
Deep Dive Azure Functions - Global Azure Bootcamp 2019Andrea Tosato
 
Apache Street Smarts Presentation (SANS 99)
Apache Street Smarts Presentation (SANS 99)Apache Street Smarts Presentation (SANS 99)
Apache Street Smarts Presentation (SANS 99)Michael Dobe, Ph.D.
 

Similar to Fluentd v1.0 in a nutshell (20)

Fluentd v0.14 Overview
Fluentd v0.14 OverviewFluentd v0.14 Overview
Fluentd v0.14 Overview
 
Treasure Data Summer Internship 2016
Treasure Data Summer Internship 2016Treasure Data Summer Internship 2016
Treasure Data Summer Internship 2016
 
Serverless design with Fn project
Serverless design with Fn projectServerless design with Fn project
Serverless design with Fn project
 
Extending OpenShift Origin: Build Your Own Cartridge with Bill DeCoste of Red...
Extending OpenShift Origin: Build Your Own Cartridge with Bill DeCoste of Red...Extending OpenShift Origin: Build Your Own Cartridge with Bill DeCoste of Red...
Extending OpenShift Origin: Build Your Own Cartridge with Bill DeCoste of Red...
 
Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018
Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018
Habitat talk at CodeMonsters Sofia, Bulgaria Nov 27 2018
 
What's new in kubernetes 1.3?
What's new in kubernetes 1.3?What's new in kubernetes 1.3?
What's new in kubernetes 1.3?
 
Hogy jussunk ki lezárt hálózatokból?
Hogy jussunk ki lezárt hálózatokból?Hogy jussunk ki lezárt hálózatokból?
Hogy jussunk ki lezárt hálózatokból?
 
Kubernetes #1 intro
Kubernetes #1   introKubernetes #1   intro
Kubernetes #1 intro
 
Neutrondev ppt
Neutrondev pptNeutrondev ppt
Neutrondev ppt
 
KACE Agent Architecture and Troubleshooting Overview
KACE Agent Architecture and Troubleshooting OverviewKACE Agent Architecture and Troubleshooting Overview
KACE Agent Architecture and Troubleshooting Overview
 
Fluentd - RubyKansai 65
Fluentd - RubyKansai 65Fluentd - RubyKansai 65
Fluentd - RubyKansai 65
 
NodeJS
NodeJSNodeJS
NodeJS
 
Introduction to Node (15th May 2017)
Introduction to Node (15th May 2017)Introduction to Node (15th May 2017)
Introduction to Node (15th May 2017)
 
NGINX Installation and Tuning
NGINX Installation and TuningNGINX Installation and Tuning
NGINX Installation and Tuning
 
Running php on nginx
Running php on nginxRunning php on nginx
Running php on nginx
 
Provisioning with Puppet
Provisioning with PuppetProvisioning with Puppet
Provisioning with Puppet
 
Monitoring in Big Data Platform - Albert Lewandowski, GetInData
Monitoring in Big Data Platform - Albert Lewandowski, GetInDataMonitoring in Big Data Platform - Albert Lewandowski, GetInData
Monitoring in Big Data Platform - Albert Lewandowski, GetInData
 
Chef - Administration for programmers
Chef - Administration for programmersChef - Administration for programmers
Chef - Administration for programmers
 
Deep Dive Azure Functions - Global Azure Bootcamp 2019
Deep Dive Azure Functions - Global Azure Bootcamp 2019Deep Dive Azure Functions - Global Azure Bootcamp 2019
Deep Dive Azure Functions - Global Azure Bootcamp 2019
 
Apache Street Smarts Presentation (SANS 99)
Apache Street Smarts Presentation (SANS 99)Apache Street Smarts Presentation (SANS 99)
Apache Street Smarts Presentation (SANS 99)
 

More from N Masahiro

Presto changes
Presto changesPresto changes
Presto changesN Masahiro
 
Fluentd and Kafka
Fluentd and KafkaFluentd and Kafka
Fluentd and KafkaN Masahiro
 
Technologies for Data Analytics Platform
Technologies for Data Analytics PlatformTechnologies for Data Analytics Platform
Technologies for Data Analytics PlatformN Masahiro
 
Docker and Fluentd
Docker and FluentdDocker and Fluentd
Docker and FluentdN Masahiro
 
How to create Treasure Data #dotsbigdata
How to create Treasure Data #dotsbigdataHow to create Treasure Data #dotsbigdata
How to create Treasure Data #dotsbigdataN Masahiro
 
Fluentd v0.12 master guide
Fluentd v0.12 master guideFluentd v0.12 master guide
Fluentd v0.12 master guideN Masahiro
 
Fluentd and Embulk Game Server 4
Fluentd and Embulk Game Server 4Fluentd and Embulk Game Server 4
Fluentd and Embulk Game Server 4N Masahiro
 
Treasure Data and AWS - Developers.io 2015
Treasure Data and AWS - Developers.io 2015Treasure Data and AWS - Developers.io 2015
Treasure Data and AWS - Developers.io 2015N Masahiro
 
Fluentd Unified Logging Layer At Fossasia
Fluentd Unified Logging Layer At FossasiaFluentd Unified Logging Layer At Fossasia
Fluentd Unified Logging Layer At FossasiaN Masahiro
 
Treasure Data and OSS
Treasure Data and OSSTreasure Data and OSS
Treasure Data and OSSN Masahiro
 
Fluentd - road to v1 -
Fluentd - road to v1 -Fluentd - road to v1 -
Fluentd - road to v1 -N Masahiro
 
Fluentd: Unified Logging Layer at CWT2014
Fluentd: Unified Logging Layer at CWT2014Fluentd: Unified Logging Layer at CWT2014
Fluentd: Unified Logging Layer at CWT2014N Masahiro
 
SQL for Everything at CWT2014
SQL for Everything at CWT2014SQL for Everything at CWT2014
SQL for Everything at CWT2014N Masahiro
 
Can you say the same words even in oss
Can you say the same words even in ossCan you say the same words even in oss
Can you say the same words even in ossN Masahiro
 
I am learing the programming
I am learing the programmingI am learing the programming
I am learing the programmingN Masahiro
 
Fluentd meetup dive into fluent plugin (outdated)
Fluentd meetup dive into fluent plugin (outdated)Fluentd meetup dive into fluent plugin (outdated)
Fluentd meetup dive into fluent plugin (outdated)N Masahiro
 
D vs OWKN Language at LLnagoya
D vs OWKN Language at LLnagoyaD vs OWKN Language at LLnagoya
D vs OWKN Language at LLnagoyaN Masahiro
 
Final presentation at pfintern
Final presentation at pfinternFinal presentation at pfintern
Final presentation at pfinternN Masahiro
 
Kernel VM 5 LT
Kernel VM 5 LTKernel VM 5 LT
Kernel VM 5 LTN Masahiro
 

More from N Masahiro (20)

Presto changes
Presto changesPresto changes
Presto changes
 
Fluentd and Kafka
Fluentd and KafkaFluentd and Kafka
Fluentd and Kafka
 
Technologies for Data Analytics Platform
Technologies for Data Analytics PlatformTechnologies for Data Analytics Platform
Technologies for Data Analytics Platform
 
Docker and Fluentd
Docker and FluentdDocker and Fluentd
Docker and Fluentd
 
How to create Treasure Data #dotsbigdata
How to create Treasure Data #dotsbigdataHow to create Treasure Data #dotsbigdata
How to create Treasure Data #dotsbigdata
 
Fluentd v0.12 master guide
Fluentd v0.12 master guideFluentd v0.12 master guide
Fluentd v0.12 master guide
 
Fluentd and Embulk Game Server 4
Fluentd and Embulk Game Server 4Fluentd and Embulk Game Server 4
Fluentd and Embulk Game Server 4
 
Treasure Data and AWS - Developers.io 2015
Treasure Data and AWS - Developers.io 2015Treasure Data and AWS - Developers.io 2015
Treasure Data and AWS - Developers.io 2015
 
Fluentd Unified Logging Layer At Fossasia
Fluentd Unified Logging Layer At FossasiaFluentd Unified Logging Layer At Fossasia
Fluentd Unified Logging Layer At Fossasia
 
Treasure Data and OSS
Treasure Data and OSSTreasure Data and OSS
Treasure Data and OSS
 
Fluentd - road to v1 -
Fluentd - road to v1 -Fluentd - road to v1 -
Fluentd - road to v1 -
 
Fluentd: Unified Logging Layer at CWT2014
Fluentd: Unified Logging Layer at CWT2014Fluentd: Unified Logging Layer at CWT2014
Fluentd: Unified Logging Layer at CWT2014
 
SQL for Everything at CWT2014
SQL for Everything at CWT2014SQL for Everything at CWT2014
SQL for Everything at CWT2014
 
Can you say the same words even in oss
Can you say the same words even in ossCan you say the same words even in oss
Can you say the same words even in oss
 
I am learing the programming
I am learing the programmingI am learing the programming
I am learing the programming
 
Fluentd meetup dive into fluent plugin (outdated)
Fluentd meetup dive into fluent plugin (outdated)Fluentd meetup dive into fluent plugin (outdated)
Fluentd meetup dive into fluent plugin (outdated)
 
D vs OWKN Language at LLnagoya
D vs OWKN Language at LLnagoyaD vs OWKN Language at LLnagoya
D vs OWKN Language at LLnagoya
 
Goodbye Doost
Goodbye DoostGoodbye Doost
Goodbye Doost
 
Final presentation at pfintern
Final presentation at pfinternFinal presentation at pfintern
Final presentation at pfintern
 
Kernel VM 5 LT
Kernel VM 5 LTKernel VM 5 LT
Kernel VM 5 LT
 

Recently uploaded

Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfWhere to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfFIDO Alliance
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfFIDO Alliance
 
THE BEST IPTV in GERMANY for 2024: IPTVreel
THE BEST IPTV in  GERMANY for 2024: IPTVreelTHE BEST IPTV in  GERMANY for 2024: IPTVreel
THE BEST IPTV in GERMANY for 2024: IPTVreelreely ones
 
Top 10 Symfony Development Companies 2024
Top 10 Symfony Development Companies 2024Top 10 Symfony Development Companies 2024
Top 10 Symfony Development Companies 2024TopCSSGallery
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityScyllaDB
 
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutesconfluent
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024   Teams Premium - Pretty SecureECS 2024   Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty SecureFemke de Vroome
 
Strategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering TeamsStrategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering TeamsUXDXConf
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxJennifer Lim
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FIDO Alliance
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...FIDO Alliance
 
AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101vincent683379
 
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxDavid Michel
 
Connecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKConnecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKUXDXConf
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?Mark Billinghurst
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераMark Opanasiuk
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessUXDXConf
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlPeter Udo Diehl
 
The UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, OcadoThe UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, OcadoUXDXConf
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsStefano
 

Recently uploaded (20)

Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfWhere to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
THE BEST IPTV in GERMANY for 2024: IPTVreel
THE BEST IPTV in  GERMANY for 2024: IPTVreelTHE BEST IPTV in  GERMANY for 2024: IPTVreel
THE BEST IPTV in GERMANY for 2024: IPTVreel
 
Top 10 Symfony Development Companies 2024
Top 10 Symfony Development Companies 2024Top 10 Symfony Development Companies 2024
Top 10 Symfony Development Companies 2024
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
 
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024   Teams Premium - Pretty SecureECS 2024   Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty Secure
 
Strategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering TeamsStrategic AI Integration in Engineering Teams
Strategic AI Integration in Engineering Teams
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 
AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101
 
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
 
Connecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAKConnecting the Dots in Product Design at KAYAK
Connecting the Dots in Product Design at KAYAK
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
 
The UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, OcadoThe UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, Ocado
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. Startups
 

Fluentd v1.0 in a nutshell

  • 1. Fluentd v1.0 in a nutshell June 1, 2017 Masahiro Nakagawa
  • 2. Fluentd v0.12 • Current stable and widely used on production • Input, Parser, Filter, Formatter, Buffer, Output plugins • Known issues • Event time is second unit • No Windows support • No multi core support • Need to improve plugin API to support more various use cases
  • 3. • Development version of v1 • Latest version is v0.14.17: March 30, 2017 • Implemented New features • New Plugin APIs • Event Time with Nanosecond resolution • ServerEngine based Supervisor • Windows support • Multicore support • New Plugin Helpers & Plugin Storage Fluentd v0.14
  • 4. Fluentd v1 • Stable announcement for APIs / features • No breaking API changes in v1.x • Compatible with v0.12 and v0.14 • exclude v0 config syntax and detach_process • Release plan • Q3, 2017 • Need v0.14 feedback from developers and users
 https://hub.docker.com/r/fluent/fluentd/
  • 5. New Plugin APIs • Input/Output plugin APIs w/ well-controlled lifecycle • stop, shutdown, close, terminate • Integrate all output plugin into Fluent::Plugin::Output • New Buffer API for delayed commit and flexible chunking with metadata • parallel/async "commit" operation for chunks • For high latency case: forward’s at-least-once, issuing job, etc… • Users can choose chunk keys by configuration for dynamic parameters • Compatible w/ v0.12 plugins • compatibility layer for traditional APIs • it will be supported between v1.x versions
  • 6. Router buffer_chunk_limit enqueue: exceed flush_interval or buffer_chunk_limit Key pattern: - BufferedOutput empty string or specified key -ObjectBufferedOutput tag -TimeSlicedOutput time slice emit emit Buffer Queue buffer_queue_limit Output OutputInput / Filter Tag Time Record Chunk Chunk Chunk Chunk Chunk key:foo key:bar key:baz v0.12 buffer design
  • 8. Buffer keys and placeholders • Dynamic parameters for table name, object path and more • We can embed time, tag and any field with placeholder
 
 
 
 
 
 
 <match s3.**> @type s3 aws_key_id "#{ENV['AWS_ACCESS_KEY']}" aws_sec_key "#{ENV['AWS_SECRETA_KEY']}" s3_bucket fluent-plugin-s3 path test/%Y/%m/${tag}/${key}/ <buffer time,tag,key> timekey 3600 </buffer> </match> http://docs.fluentd.org/v0.14/articles/buffer-section for more details time: 2017-06-01 12:00:00 +0700 tag: “test” record: {“key”:”hello”} - Event sample test/2017/6/test/hello/ - Generated “path”
  • 9. Time with nanosecond • For sub-second systems: Elasticsearch, InfluxData, etc… • Fluent::EventTime • behaves as Integer for v0.12’s second unit compatibility • has methods to get sub-second resolution • be serialized into msgpack using Ext type • Fluent::Engine.now now returns EventTime, not Integer • Fluentd core can handle both of Integer and EventTime as time • compatible with older versions and software in eco-system (e.g., fluent-logger, Docker logging driver)
  • 10. ServerEngine based Supervisor • ServerEngine is a framework for building robust server • https://github.com/treasure-data/serverengine • Replacing supervisor process with ServerEngine • it has SocketManager to share listening sockets between 2 or more worker processes • Replacing Fluentd's processing model from fork to spawn • to support Windows environment
  • 11. Windows support • Fluentd and core plugins work on Windows • Windows service registration is also supported • http://docs.fluentd.org/v0.14/articles/install-by-msi • Use HTTP RPC instead of signals • https://github.com/fluent/fluent-plugin-windows-eventlog • We can collect windows eventlog :)
  • 12. Symmetric multi core processing • 2 or more workers share a configuration file • and share listening sockets via PluginHelper • under a supervisor process (ServerEngine) • Multi core scalability for huge traffic • one input plugin for a tcp port, some filters and one (or some) output plugin • buffer paths are managed automatically by Fluentd. Need root_dir and @id parameters
  • 13. Worker0 Supervisor v0.14’s multi process feature grep forward tdlog Worker1 Worker2 grep forward tdlog grep forward tdlog socket
  • 14. Configuration example <system> workers 2 root_dir /var/log/fluentd </system> <source> @type forward </source> <filter pattern> @type grep </filter> <match pattern> @type tdlog @id out_td </match> /var/log/fluentd/worker0/out_td/buffer/buffer.xxx.log /var/log/fluentd/worker0/out_td/buffer/buffer.xxx.log.meta - buf_file’s path is automatically generated worker id root_dir plugin’s @id
  • 15. TLS/Authn/Authz support for forward plugin • Support v1 forward protocol spec • secure-forward is merged into built-in forward • TLS w/ at-least-one semantics • Simple authentication/authorization w/o SSL • Different points • secure-forward uses keep-alive, but forward doesn’t • secure-forward uses thread per connection, but forward uses cool.io, libev based IO. http://www.fluentd.org/blog/fluentd-v0.14.12-has-been-released
  • 16. Plugin Storage & Helpers • Plugin Storage: new plugin type for plugins • provides key-value storage to persistent intermediate status • built-in plugins: in-memory, local file • pluggable: 3rd party plugin to store data into storage • storage-redis • Plugin Helpers: • collections of utility methods for plugins • fully integrated with test drivers to run test code after setup phase of helpers (e.g. test started after created threads)
  • 17. server helper: before def start @loop = Coolio::Loop.new @handler = Coolio::TCPServer.new(@bind, @port, SocketUtil::TcpHandler, log, @delimiter, method(:on_message)) @loop.attach(@handler) @thread = Thread.new(&method(:run)) end def shutdown @loop.watchers.each { |w| w.detach } @loop.stop @handler.close @thread.join end def run @loop.run rescue => e log.error "unexpected error", error: e log.error_backtrace end def on_message(msg, addr) # body end
  • 18. server helper: after def start server_create(:foo_server, @port, bind: @bind) do |data, conn|
 # body
 end end
  • 19. v0.12 plugins ParserInput Buffer Output FormatterFilter “output-ish”“input-ish”
  • 20. v0.14 plugins ParserInput Buffer Output FormatterFilter “output-ish”“input-ish” Storage Helper
  • 21. Other helpers • Timer: one-shot / periodic timer • Event Loop: Low-layer event loop • Socket: TCP/UDP/TLS support • Formatter/Parser: Manage parser/formatter plugins • Chile Process: Manage process for exec like plugin • etc…
  • 22. Is v0.14 stable? • Not yet but it will be stable in this month (I hope!) • Remaining tickets: • Nested record support • Fix core bugs • v0.14 stable means ready for production, not feature freeze. Feature freeze is done by v1 release
  • 23. TODO list for v1 • Counter API to store metrics between processes • https://github.com/fluent/fluentd/tree/counter-api • Optimize performance and remove deprecated code • Migrate more plugin into v0.14 API • Core plugins are migrated • Popular plugins release beta version for v0.14 • Write document: design, plugin, operation, etc…
  • 24. Treasure Agent 3.0 serise (td-agent 3) • fluentd v0.14/v1, Ruby 2.4, and latest core components • Latest beta version is 3.0.1: May 19, 2017 • Environments • Add msi Windows package • Remove CentOS 5, Ubuntu 10.04 support • Beta packages have beed released • http://docs.fluentd.org/v0.14/categories/installation • beta will be removed after v0.14 becomes stable :)