SlideShare a Scribd company logo
Fluency
Yet another fluent logger
Mitsunori Komatsu
About me
• Mitsunori Komatsu, 

Software engineer
• github:komamitsu

msgpack-java

(jackson-dataformat-msgpack),

fluent-logger-java,

digdag,

:
Today’s talk
• What's Fluency?
• Internal design
• Performance and profiling
• Future improvements
Fluentd
What’s Fluency?
• Yet another fluent-logger-java

(https://github.com/komamitsu/fluency)
• 3x 4x faster than fluent-logger using
PackedForward format
• Easy to extend
• Many features

(Async flush, HB, Failover, Ack response)
• Used in LINE, Treasure Data, kafka-fluentd-consumer
Internal design
• Asynchronous flushing
• Buffering PackedForward format
data
Async flushing
Application Fluency Buffer FluentdFlusher
emit(event:A) append(event:A)
emit(event:B) append(event:B)
store events as
MessagePack
flush
send(event:[A, B])
emit(event:C) append(event:C)
flush
invoked
repeatedly
PackedForward
format
close close
force flush
emit(event:D) append(event:D)
send(event:[C,D])
event:C isn’t
ready for flush
event:A,B are
ready for flush
terminate
thread
Async flushing
Application Fluency Buffer FluentdFlusher
emit(event:A) append(event:A)
emit(event:B) append(event:B)
store events as
MessagePack
flush
send(event:[A, B])
emit(event:C) append(event:C)
flush
invoked
repeatedly
PackedForward
format
close close
force flush
emit(event:D) append(event:D)
send(event:[C,D])
event:C isn’t
ready for flush
event:A,B are
ready for flush
terminate
thread
Async flushing
Application Fluency Buffer FluentdFlusher
emit(event:A) append(event:A)
emit(event:B) append(event:B)
store events as
MessagePack
flush
send(event:[A, B])
emit(event:C) append(event:C)
flush
invoked
repeatedly
PackedForward
format
close close
force flush
emit(event:D) append(event:D)
send(event:[C,D])
event:C isn’t
ready for flush
event:A,B are
ready for flush
terminate
thread
Async flushing
Application Fluency Buffer FluentdFlusher
emit(event:A) append(event:A)
emit(event:B) append(event:B)
store events as
MessagePack
flush
send(event:[A, B])
emit(event:C) append(event:C)
flush
invoked
repeatedly
PackedForward
format
close close
force flush
emit(event:D) append(event:D)
send(event:[C,D])
event:C isn’t
ready for flush
event:A,B are
ready for flush
terminate
thread
Async flushing
Application Fluency Buffer FluentdFlusher
emit(event:A) append(event:A)
emit(event:B) append(event:B)
store events as
MessagePack
flush
send(event:[A, B])
emit(event:C) append(event:C)
flush
invoked
repeatedly
PackedForward
format
close close
force flush
emit(event:D) append(event:D)
send(event:[C,D])
event:C isn’t
ready for flush
event:A,B are
ready for flush
terminate
thread
Async flushing
Application Fluency Buffer FluentdFlusher
emit(event:A) append(event:A)
emit(event:B) append(event:B)
store events as
MessagePack
flush
send(event:[A, B])
emit(event:C) append(event:C)
flush
invoked
repeatedly
PackedForward
format
close close
force flush
emit(event:D) append(event:D)
send(event:[C,D])
event:C isn’t
ready for flush
event:A,B are
ready for flush
terminate
thread
Async flushing
Application Fluency Buffer FluentdFlusher
emit(event:A) append(event:A)
emit(event:B) append(event:B)
store events as
MessagePack
flush
send(event:[A, B])
emit(event:C) append(event:C)
flush
invoked
repeatedly
PackedForward
format
close close
force flush
emit(event:D) append(event:D)
send(event:[C,D])
event:C isn’t
ready for flush
event:A,B are
ready for flush
terminate
thread
Async flushing
Application Fluency Buffer FluentdFlusher
emit(event:A) append(event:A)
emit(event:B) append(event:B)
store events as
MessagePack
flush
send(event:[A, B])
emit(event:C) append(event:C)
flush
invoked
repeatedly
PackedForward
format
close close
force flush
emit(event:D) append(event:D)
send(event:[C,D])
event:C isn’t
ready for flush
event:A,B are
ready for flush
terminate
thread
Buffering events
tag: aaa,
time: 100,
{name: a_0}
Buffer
pool
retention buffers
flushable buffers
Fluentd
emit event
acquire buffer
Buffering events
tag: bbb,
time: 101,
{name: b_0}
Buffer
pool
time: 100, {name: a_0}
retention buffers
flushable buffers
tag: aaa
Fluentd
emit event
acquire buffer
Buffering events
tag: ccc,
time: 102,
{name: c_0}
Buffer
pool
time: 100, {name: a_0} time: 101, {name: b_0}
retention buffers
flushable buffers
tag: aaa tag: bbb
Fluentd
emit event
acquire buffer
Buffering events
tag: aaa,
time: 110,
{name: a_1}
Buffer
pool
time: 100, {name: a_0} time: 101, {name: b_0} time: 102, {name: c_0}
retention buffers
flushable buffers
tag: aaa tag: bbb tag: ccc
Fluentd
emit event
Buffering events
tag: aaa,
time: 120,
{name: a_2}
Buffer
pool
time: 100, {name: a_0}
time: 110, {name: a_1}
time: 101, {name: b_0} time: 102, {name: c_0}
retention buffers
flushable buffers
tag: aaa tag: bbb tag: ccc
Fluentd
emit event
expand buffer
Buffering events
tag: bbb,
time: 111,
{name: b_1}
:
:
Buffer
pool
time: 100, {name: a_0}
time: 110, {name: a_1}
time: 120, {name: a_2}
time: 101, {name: b_0} time: 102, {name: c_0}
retention buffers
flushable buffers
tag: aaa tag: bbb tag: ccc
Fluentd
emit event
Buffering events
tag: aaa,
time: 200,
{name: a_0}
Buffer
pool
time: 100, {name: a_0}
time: 110, {name: a_1}
time: 120, {name: a_2}
time: 130, {name: a_3}
time: 101, {name: b_0}
time: 111, {name: b_1}
time: 121, {name: b_2}
time: 131, {name: b_3}
time: 102, {name: c_0}
time: 112, {name: c_1}
retention buffers
flushable buffers
tag: aaa tag: bbb tag: ccc
Fluentd
emit event
move buffers
(zero copy)
tag: aaa
Buffering events
tag: ccc,
time: 122,
{name: c_2}
Buffer
pool
time: 200, {name: a_0} time: 102, {name: c_0}
time: 112, {name: c_1}
retention buffers
flushable buffers
tag: aaa tag: ccc
time: 100, {name: a_0}
time: 110, {name: a_1}
time: 120, {name: a_2}
time: 130, {name: a_3}
Fluentd
tag: bbb
time: 101, {name: b_0}
time: 111, {name: b_1}
time: 121, {name: b_2}
time: 131, {name: b_3}
emit event
expand buffer
send buffer
return buffer
(after sending)
PackedForword
format
Buffering events
tag: bbb,
time: 201,
{name: b_0}
Buffer
pool
time: 200, {name: a_0} time: 102, {name: c_0}
time: 112, {name: c_1}
retention buffers
flushable buffers
tag: aaa tag: ccc
Fluentd
tag: bbb
time: 101, {name: b_0}
time: 111, {name: b_1}
time: 121, {name: b_2}
time: 131, {name: b_3}
time: 122, {name: c_2}
emit event
send buffer
return buffer
(after sending)
acquire buffer
PackedForword
format
Buffering events
tag: aaa,
time: 210,
{name: a_1}
Buffer
pool
time: 200, {name: a_0} time: 102, {name: c_0}
time: 112, {name: c_1}
retention buffers
flushable buffers
tag: aaa tag: ccc
Fluentd
time: 122, {name: c_2}
time: 201, {name: b_0}
tag: bbb
emit event
4x faster than

fluent-logger-java
fluency / fluent-logger-java: m3.large
fluentd: m3.large
See details: https://gist.github.com/komamitsu/c1e4045fe2ddb108cfbf12d5f014b683
But the actual
bottleneck
of this benchmark
might be Fluentd…
Java profiling (1/2)
fluency fluent-logger-java
fluency uses off heap
for buffer pool
Due to synchronization
Java profiling (2/2)
fluency fluent-logger-java
GC Pause: 33ms/s GC Pause: 7ms/s
Finding a bottleneck
fluency
emit (MessagePack
serialization & appending to
buffer) accounts for most of
the duration
String.getBytes in
MessagePack serialization
takes total 18%
ObjectMapper.writeValue
itself takes 40%
Finding a bottleneck
fluency
MessagePackGenerator.
writeStartArray takes 10%
java.nio.channels.SocketCh
annel.write and
java.nio.ByteBuffer.allocate
Direct are both 0-1%
Finding a bottleneck
fluent-logger-java
Sending events via socket
accounts for most of the
duration
Future improvements:
removing ObjectMapper
ObjectMapper
can be removed
to improve
performance?
Hmm. But Fluency
accepts Jackson
Modules…
Future improvements:
reducing use of ObjectMapper
ObjectMapper is needed only for
serialisation of data not array
Thanks!

More Related Content

What's hot

Python高级编程(二)
Python高级编程(二)Python高级编程(二)
Python高级编程(二)
Qiangning Hong
 
How I Learned to Stop Worrying and Love the Cloud - Wesley Beary, Engine Yard
How I Learned to Stop Worrying and Love the Cloud - Wesley Beary, Engine YardHow I Learned to Stop Worrying and Love the Cloud - Wesley Beary, Engine Yard
How I Learned to Stop Worrying and Love the Cloud - Wesley Beary, Engine Yard
SV Ruby on Rails Meetup
 
Video Transcoding at the ABC with Microservices at GOTO Chicago
Video Transcoding at the ABC with Microservices at GOTO ChicagoVideo Transcoding at the ABC with Microservices at GOTO Chicago
Video Transcoding at the ABC with Microservices at GOTO Chicago
Daphne Chong
 
This is not your father's monitoring.
This is not your father's monitoring.This is not your father's monitoring.
This is not your father's monitoring.
Mathias Herberts
 
Device-specific Clang Tooling for Embedded Systems
Device-specific Clang Tooling for Embedded SystemsDevice-specific Clang Tooling for Embedded Systems
Device-specific Clang Tooling for Embedded Systems
emBO_Conference
 
Artimon - Apache Flume (incubating) NYC Meetup 20111108
Artimon - Apache Flume (incubating) NYC Meetup 20111108Artimon - Apache Flume (incubating) NYC Meetup 20111108
Artimon - Apache Flume (incubating) NYC Meetup 20111108
Mathias Herberts
 
Perl at SkyCon'12
Perl at SkyCon'12Perl at SkyCon'12
Perl at SkyCon'12
Tim Bunce
 
InfluxData Platform Future and Vision
InfluxData Platform Future and VisionInfluxData Platform Future and Vision
InfluxData Platform Future and Vision
InfluxData
 
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
Daphne Chong
 
Devel::NYTProf v5 at YAPC::NA 201406
Devel::NYTProf v5 at YAPC::NA 201406Devel::NYTProf v5 at YAPC::NA 201406
Devel::NYTProf v5 at YAPC::NA 201406
Tim Bunce
 
CPAN 模組二三事
CPAN 模組二三事CPAN 模組二三事
CPAN 模組二三事
Lin Yo-An
 
Perl Memory Use 201207 (OUTDATED, see 201209 )
Perl Memory Use 201207 (OUTDATED, see 201209 )Perl Memory Use 201207 (OUTDATED, see 201209 )
Perl Memory Use 201207 (OUTDATED, see 201209 )
Tim Bunce
 
Inside the ABC's new Media Transcoding system, Metro
Inside the ABC's new Media Transcoding system, MetroInside the ABC's new Media Transcoding system, Metro
Inside the ABC's new Media Transcoding system, Metro
Daphne Chong
 
Stream or not to Stream?

Stream or not to Stream?
Stream or not to Stream?

Stream or not to Stream?

Lukasz Byczynski
 
HaskellとDebianの辛くて甘い関係
HaskellとDebianの辛くて甘い関係HaskellとDebianの辛くて甘い関係
HaskellとDebianの辛くて甘い関係
Kiwamu Okabe
 
Perl Memory Use 201209
Perl Memory Use 201209Perl Memory Use 201209
Perl Memory Use 201209
Tim Bunce
 
Redis as a message queue
Redis as a message queueRedis as a message queue
Redis as a message queue
Brandon Lamb
 
Zendcon 2007 Api Design
Zendcon 2007 Api DesignZendcon 2007 Api Design
Zendcon 2007 Api Design
unodelostrece
 
Perl Memory Use - LPW2013
Perl Memory Use - LPW2013Perl Memory Use - LPW2013
Perl Memory Use - LPW2013
Tim Bunce
 
Deferred Gratification
Deferred GratificationDeferred Gratification
Deferred Gratification
Terry Jones
 

What's hot (20)

Python高级编程(二)
Python高级编程(二)Python高级编程(二)
Python高级编程(二)
 
How I Learned to Stop Worrying and Love the Cloud - Wesley Beary, Engine Yard
How I Learned to Stop Worrying and Love the Cloud - Wesley Beary, Engine YardHow I Learned to Stop Worrying and Love the Cloud - Wesley Beary, Engine Yard
How I Learned to Stop Worrying and Love the Cloud - Wesley Beary, Engine Yard
 
Video Transcoding at the ABC with Microservices at GOTO Chicago
Video Transcoding at the ABC with Microservices at GOTO ChicagoVideo Transcoding at the ABC with Microservices at GOTO Chicago
Video Transcoding at the ABC with Microservices at GOTO Chicago
 
This is not your father's monitoring.
This is not your father's monitoring.This is not your father's monitoring.
This is not your father's monitoring.
 
Device-specific Clang Tooling for Embedded Systems
Device-specific Clang Tooling for Embedded SystemsDevice-specific Clang Tooling for Embedded Systems
Device-specific Clang Tooling for Embedded Systems
 
Artimon - Apache Flume (incubating) NYC Meetup 20111108
Artimon - Apache Flume (incubating) NYC Meetup 20111108Artimon - Apache Flume (incubating) NYC Meetup 20111108
Artimon - Apache Flume (incubating) NYC Meetup 20111108
 
Perl at SkyCon'12
Perl at SkyCon'12Perl at SkyCon'12
Perl at SkyCon'12
 
InfluxData Platform Future and Vision
InfluxData Platform Future and VisionInfluxData Platform Future and Vision
InfluxData Platform Future and Vision
 
Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)Video Transcoding at Scale for ABC iview (NDC Sydney)
Video Transcoding at Scale for ABC iview (NDC Sydney)
 
Devel::NYTProf v5 at YAPC::NA 201406
Devel::NYTProf v5 at YAPC::NA 201406Devel::NYTProf v5 at YAPC::NA 201406
Devel::NYTProf v5 at YAPC::NA 201406
 
CPAN 模組二三事
CPAN 模組二三事CPAN 模組二三事
CPAN 模組二三事
 
Perl Memory Use 201207 (OUTDATED, see 201209 )
Perl Memory Use 201207 (OUTDATED, see 201209 )Perl Memory Use 201207 (OUTDATED, see 201209 )
Perl Memory Use 201207 (OUTDATED, see 201209 )
 
Inside the ABC's new Media Transcoding system, Metro
Inside the ABC's new Media Transcoding system, MetroInside the ABC's new Media Transcoding system, Metro
Inside the ABC's new Media Transcoding system, Metro
 
Stream or not to Stream?

Stream or not to Stream?
Stream or not to Stream?

Stream or not to Stream?

 
HaskellとDebianの辛くて甘い関係
HaskellとDebianの辛くて甘い関係HaskellとDebianの辛くて甘い関係
HaskellとDebianの辛くて甘い関係
 
Perl Memory Use 201209
Perl Memory Use 201209Perl Memory Use 201209
Perl Memory Use 201209
 
Redis as a message queue
Redis as a message queueRedis as a message queue
Redis as a message queue
 
Zendcon 2007 Api Design
Zendcon 2007 Api DesignZendcon 2007 Api Design
Zendcon 2007 Api Design
 
Perl Memory Use - LPW2013
Perl Memory Use - LPW2013Perl Memory Use - LPW2013
Perl Memory Use - LPW2013
 
Deferred Gratification
Deferred GratificationDeferred Gratification
Deferred Gratification
 

Similar to Fluency - Yet another fluent logger

Exploiting Multicore CPUs Now: Scalability and Reliability for Off-the-shelf ...
Exploiting Multicore CPUs Now: Scalability and Reliability for Off-the-shelf ...Exploiting Multicore CPUs Now: Scalability and Reliability for Off-the-shelf ...
Exploiting Multicore CPUs Now: Scalability and Reliability for Off-the-shelf ...
Emery Berger
 
Node.js - Advanced Basics
Node.js - Advanced BasicsNode.js - Advanced Basics
Node.js - Advanced Basics
Doug Jones
 
Apache Kafka Demo
Apache Kafka DemoApache Kafka Demo
Apache Kafka Demo
Edward Capriolo
 
Timelapse: interactive record/replay for the web
Timelapse: interactive record/replay for the webTimelapse: interactive record/replay for the web
Timelapse: interactive record/replay for the web
brrian
 
Exactly-Once Made Easy: Transactional Messaging in Apache Pulsar - Pulsar Sum...
Exactly-Once Made Easy: Transactional Messaging in Apache Pulsar - Pulsar Sum...Exactly-Once Made Easy: Transactional Messaging in Apache Pulsar - Pulsar Sum...
Exactly-Once Made Easy: Transactional Messaging in Apache Pulsar - Pulsar Sum...
StreamNative
 
Data Pipeline at Tapad
Data Pipeline at TapadData Pipeline at Tapad
Data Pipeline at Tapad
Toby Matejovsky
 
Message reliability in kafka
Message reliability in kafkaMessage reliability in kafka
Message reliability in kafka
Mihai Ghita
 
Immutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS LambdaImmutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS Lambda
AOE
 
Developing with Cassandra
Developing with CassandraDeveloping with Cassandra
Developing with Cassandra
Sperasoft
 
AWS SDK for PHP のインストールから 始めるクラウドマスターへの道 〜 Promise による非同期オペレーション 〜
AWS SDK for PHP のインストールから 始めるクラウドマスターへの道 〜 Promise による非同期オペレーション 〜 AWS SDK for PHP のインストールから 始めるクラウドマスターへの道 〜 Promise による非同期オペレーション 〜
AWS SDK for PHP のインストールから 始めるクラウドマスターへの道 〜 Promise による非同期オペレーション 〜
崇之 清水
 
Concurrency at the Database Layer
Concurrency at the Database Layer Concurrency at the Database Layer
Concurrency at the Database Layer
mcwilson1
 
The Future of Apache Storm
The Future of Apache StormThe Future of Apache Storm
The Future of Apache Storm
P. Taylor Goetz
 
Parse, scale to millions
Parse, scale to millionsParse, scale to millions
Parse, scale to millions
Florent Vilmart
 
The Nuts and Bolts of Kafka Streams---An Architectural Deep Dive
The Nuts and Bolts of Kafka Streams---An Architectural Deep DiveThe Nuts and Bolts of Kafka Streams---An Architectural Deep Dive
The Nuts and Bolts of Kafka Streams---An Architectural Deep Dive
HostedbyConfluent
 
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Jonathan Dahl
 
Scaling python webapps from 0 to 50 million users - A top-down approach
Scaling python webapps from 0 to 50 million users - A top-down approachScaling python webapps from 0 to 50 million users - A top-down approach
Scaling python webapps from 0 to 50 million users - A top-down approach
Jinal Jhaveri
 
Web development automatisation for fun and profit (Artem Daniliants)
Web development automatisation for fun and profit (Artem Daniliants)Web development automatisation for fun and profit (Artem Daniliants)
Web development automatisation for fun and profit (Artem Daniliants)
LumoSpark
 
Logging for Production Systems in The Container Era
Logging for Production Systems in The Container EraLogging for Production Systems in The Container Era
Logging for Production Systems in The Container Era
Sadayuki Furuhashi
 
Embedding Generic Monadic Transformer into Scala. [Tfp2022]
Embedding Generic Monadic Transformer into Scala. [Tfp2022]Embedding Generic Monadic Transformer into Scala. [Tfp2022]
Embedding Generic Monadic Transformer into Scala. [Tfp2022]
Ruslan Shevchenko
 
Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...
Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...
Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...
Databricks
 

Similar to Fluency - Yet another fluent logger (20)

Exploiting Multicore CPUs Now: Scalability and Reliability for Off-the-shelf ...
Exploiting Multicore CPUs Now: Scalability and Reliability for Off-the-shelf ...Exploiting Multicore CPUs Now: Scalability and Reliability for Off-the-shelf ...
Exploiting Multicore CPUs Now: Scalability and Reliability for Off-the-shelf ...
 
Node.js - Advanced Basics
Node.js - Advanced BasicsNode.js - Advanced Basics
Node.js - Advanced Basics
 
Apache Kafka Demo
Apache Kafka DemoApache Kafka Demo
Apache Kafka Demo
 
Timelapse: interactive record/replay for the web
Timelapse: interactive record/replay for the webTimelapse: interactive record/replay for the web
Timelapse: interactive record/replay for the web
 
Exactly-Once Made Easy: Transactional Messaging in Apache Pulsar - Pulsar Sum...
Exactly-Once Made Easy: Transactional Messaging in Apache Pulsar - Pulsar Sum...Exactly-Once Made Easy: Transactional Messaging in Apache Pulsar - Pulsar Sum...
Exactly-Once Made Easy: Transactional Messaging in Apache Pulsar - Pulsar Sum...
 
Data Pipeline at Tapad
Data Pipeline at TapadData Pipeline at Tapad
Data Pipeline at Tapad
 
Message reliability in kafka
Message reliability in kafkaMessage reliability in kafka
Message reliability in kafka
 
Immutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS LambdaImmutable Deployments with AWS CloudFormation and AWS Lambda
Immutable Deployments with AWS CloudFormation and AWS Lambda
 
Developing with Cassandra
Developing with CassandraDeveloping with Cassandra
Developing with Cassandra
 
AWS SDK for PHP のインストールから 始めるクラウドマスターへの道 〜 Promise による非同期オペレーション 〜
AWS SDK for PHP のインストールから 始めるクラウドマスターへの道 〜 Promise による非同期オペレーション 〜 AWS SDK for PHP のインストールから 始めるクラウドマスターへの道 〜 Promise による非同期オペレーション 〜
AWS SDK for PHP のインストールから 始めるクラウドマスターへの道 〜 Promise による非同期オペレーション 〜
 
Concurrency at the Database Layer
Concurrency at the Database Layer Concurrency at the Database Layer
Concurrency at the Database Layer
 
The Future of Apache Storm
The Future of Apache StormThe Future of Apache Storm
The Future of Apache Storm
 
Parse, scale to millions
Parse, scale to millionsParse, scale to millions
Parse, scale to millions
 
The Nuts and Bolts of Kafka Streams---An Architectural Deep Dive
The Nuts and Bolts of Kafka Streams---An Architectural Deep DiveThe Nuts and Bolts of Kafka Streams---An Architectural Deep Dive
The Nuts and Bolts of Kafka Streams---An Architectural Deep Dive
 
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)Asynchronous Processing with Ruby on Rails (RailsConf 2008)
Asynchronous Processing with Ruby on Rails (RailsConf 2008)
 
Scaling python webapps from 0 to 50 million users - A top-down approach
Scaling python webapps from 0 to 50 million users - A top-down approachScaling python webapps from 0 to 50 million users - A top-down approach
Scaling python webapps from 0 to 50 million users - A top-down approach
 
Web development automatisation for fun and profit (Artem Daniliants)
Web development automatisation for fun and profit (Artem Daniliants)Web development automatisation for fun and profit (Artem Daniliants)
Web development automatisation for fun and profit (Artem Daniliants)
 
Logging for Production Systems in The Container Era
Logging for Production Systems in The Container EraLogging for Production Systems in The Container Era
Logging for Production Systems in The Container Era
 
Embedding Generic Monadic Transformer into Scala. [Tfp2022]
Embedding Generic Monadic Transformer into Scala. [Tfp2022]Embedding Generic Monadic Transformer into Scala. [Tfp2022]
Embedding Generic Monadic Transformer into Scala. [Tfp2022]
 
Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...
Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...
Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail C...
 

Recently uploaded

Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...
Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...
Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...
bhumivarma35300
 
Software development... for all? (keynote at ICSOFT'2024)
Software development... for all? (keynote at ICSOFT'2024)Software development... for all? (keynote at ICSOFT'2024)
Software development... for all? (keynote at ICSOFT'2024)
miso_uam
 
Artificial intelligence in customer services or chatbots
Artificial intelligence  in customer services or chatbotsArtificial intelligence  in customer services or chatbots
Artificial intelligence in customer services or chatbots
kayash1656
 
Odoo E-commerce website development guides
Odoo E-commerce website development guidesOdoo E-commerce website development guides
Odoo E-commerce website development guides
jhkdigitalmarketing
 
Folding Cheat Sheet #7 - seventh in a series
Folding Cheat Sheet #7 - seventh in a seriesFolding Cheat Sheet #7 - seventh in a series
Folding Cheat Sheet #7 - seventh in a series
Philip Schwarz
 
The Ultimate Guide to Phone Spy Apps: Everything You Need to Know
The Ultimate Guide to Phone Spy Apps: Everything You Need to KnowThe Ultimate Guide to Phone Spy Apps: Everything You Need to Know
The Ultimate Guide to Phone Spy Apps: Everything You Need to Know
onemonitarsoftware
 
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
aslasdfmkhan4750
 
Il Data Streaming per un’AI real-time di nuova generazione
Il Data Streaming per un’AI real-time di nuova generazioneIl Data Streaming per un’AI real-time di nuova generazione
Il Data Streaming per un’AI real-time di nuova generazione
confluent
 
AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.
AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.
AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.
Srinivas Dukka
 
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
ashiklo9823
 
ERP Software Solutions Provider in Coimbatore
ERP Software Solutions Provider in CoimbatoreERP Software Solutions Provider in Coimbatore
ERP Software Solutions Provider in Coimbatore
Nextskill Technologies
 
🚂🚘 Premium Girls Call Ranchi 🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
🚂🚘 Premium Girls Call Ranchi  🛵🚡000XX00000 💃 Choose Best And Top Girl Service...🚂🚘 Premium Girls Call Ranchi  🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
🚂🚘 Premium Girls Call Ranchi 🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
bahubalikumar09988
 
Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...
Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...
Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...
rachitkumar09887
 
NYGGS 360: A Complete ERP for Construction Innovation
NYGGS 360: A Complete ERP for Construction InnovationNYGGS 360: A Complete ERP for Construction Innovation
NYGGS 360: A Complete ERP for Construction Innovation
NYGGS Construction ERP Software
 
Top Chinese Government-backed APT Groups
Top Chinese Government-backed APT GroupsTop Chinese Government-backed APT Groups
Top Chinese Government-backed APT Groups
SOCRadar
 
當測試開始左移
當測試開始左移當測試開始左移
當測試開始左移
Jersey (CHE-PING) Su
 
Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...
Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...
Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...
OnePlan Solutions
 
ThaiPy meetup - Indexes and Django
ThaiPy meetup - Indexes and DjangoThaiPy meetup - Indexes and Django
ThaiPy meetup - Indexes and Django
akshesh doshi
 
Russian Girls Call Mumbai 🎈🔥9930687706 🔥💋🎈 Provide Best And Top Girl Service ...
Russian Girls Call Mumbai 🎈🔥9930687706 🔥💋🎈 Provide Best And Top Girl Service ...Russian Girls Call Mumbai 🎈🔥9930687706 🔥💋🎈 Provide Best And Top Girl Service ...
Russian Girls Call Mumbai 🎈🔥9930687706 🔥💋🎈 Provide Best And Top Girl Service ...
shanihomely
 
welcome to presentation on Google Apps
welcome to   presentation on Google Appswelcome to   presentation on Google Apps
welcome to presentation on Google Apps
AsifKarimJim
 

Recently uploaded (20)

Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...
Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...
Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...
 
Software development... for all? (keynote at ICSOFT'2024)
Software development... for all? (keynote at ICSOFT'2024)Software development... for all? (keynote at ICSOFT'2024)
Software development... for all? (keynote at ICSOFT'2024)
 
Artificial intelligence in customer services or chatbots
Artificial intelligence  in customer services or chatbotsArtificial intelligence  in customer services or chatbots
Artificial intelligence in customer services or chatbots
 
Odoo E-commerce website development guides
Odoo E-commerce website development guidesOdoo E-commerce website development guides
Odoo E-commerce website development guides
 
Folding Cheat Sheet #7 - seventh in a series
Folding Cheat Sheet #7 - seventh in a seriesFolding Cheat Sheet #7 - seventh in a series
Folding Cheat Sheet #7 - seventh in a series
 
The Ultimate Guide to Phone Spy Apps: Everything You Need to Know
The Ultimate Guide to Phone Spy Apps: Everything You Need to KnowThe Ultimate Guide to Phone Spy Apps: Everything You Need to Know
The Ultimate Guide to Phone Spy Apps: Everything You Need to Know
 
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
Independent Girls Call ServiCe Hyderabad 0000000000 Tanisha Best High Class H...
 
Il Data Streaming per un’AI real-time di nuova generazione
Il Data Streaming per un’AI real-time di nuova generazioneIl Data Streaming per un’AI real-time di nuova generazione
Il Data Streaming per un’AI real-time di nuova generazione
 
AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.
AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.
AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.
 
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
 
ERP Software Solutions Provider in Coimbatore
ERP Software Solutions Provider in CoimbatoreERP Software Solutions Provider in Coimbatore
ERP Software Solutions Provider in Coimbatore
 
🚂🚘 Premium Girls Call Ranchi 🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
🚂🚘 Premium Girls Call Ranchi  🛵🚡000XX00000 💃 Choose Best And Top Girl Service...🚂🚘 Premium Girls Call Ranchi  🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
🚂🚘 Premium Girls Call Ranchi 🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
 
Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...
Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...
Agra Girls Call Agra 0X0000000X Unlimited Short Providing Girls Service Avail...
 
NYGGS 360: A Complete ERP for Construction Innovation
NYGGS 360: A Complete ERP for Construction InnovationNYGGS 360: A Complete ERP for Construction Innovation
NYGGS 360: A Complete ERP for Construction Innovation
 
Top Chinese Government-backed APT Groups
Top Chinese Government-backed APT GroupsTop Chinese Government-backed APT Groups
Top Chinese Government-backed APT Groups
 
當測試開始左移
當測試開始左移當測試開始左移
當測試開始左移
 
Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...
Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...
Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...
 
ThaiPy meetup - Indexes and Django
ThaiPy meetup - Indexes and DjangoThaiPy meetup - Indexes and Django
ThaiPy meetup - Indexes and Django
 
Russian Girls Call Mumbai 🎈🔥9930687706 🔥💋🎈 Provide Best And Top Girl Service ...
Russian Girls Call Mumbai 🎈🔥9930687706 🔥💋🎈 Provide Best And Top Girl Service ...Russian Girls Call Mumbai 🎈🔥9930687706 🔥💋🎈 Provide Best And Top Girl Service ...
Russian Girls Call Mumbai 🎈🔥9930687706 🔥💋🎈 Provide Best And Top Girl Service ...
 
welcome to presentation on Google Apps
welcome to   presentation on Google Appswelcome to   presentation on Google Apps
welcome to presentation on Google Apps
 

Fluency - Yet another fluent logger

  • 1. Fluency Yet another fluent logger Mitsunori Komatsu
  • 2. About me • Mitsunori Komatsu, 
 Software engineer • github:komamitsu
 msgpack-java
 (jackson-dataformat-msgpack),
 fluent-logger-java,
 digdag,
 :
  • 3. Today’s talk • What's Fluency? • Internal design • Performance and profiling • Future improvements
  • 5. What’s Fluency? • Yet another fluent-logger-java
 (https://github.com/komamitsu/fluency) • 3x 4x faster than fluent-logger using PackedForward format • Easy to extend • Many features
 (Async flush, HB, Failover, Ack response) • Used in LINE, Treasure Data, kafka-fluentd-consumer
  • 6. Internal design • Asynchronous flushing • Buffering PackedForward format data
  • 7. Async flushing Application Fluency Buffer FluentdFlusher emit(event:A) append(event:A) emit(event:B) append(event:B) store events as MessagePack flush send(event:[A, B]) emit(event:C) append(event:C) flush invoked repeatedly PackedForward format close close force flush emit(event:D) append(event:D) send(event:[C,D]) event:C isn’t ready for flush event:A,B are ready for flush terminate thread
  • 8. Async flushing Application Fluency Buffer FluentdFlusher emit(event:A) append(event:A) emit(event:B) append(event:B) store events as MessagePack flush send(event:[A, B]) emit(event:C) append(event:C) flush invoked repeatedly PackedForward format close close force flush emit(event:D) append(event:D) send(event:[C,D]) event:C isn’t ready for flush event:A,B are ready for flush terminate thread
  • 9. Async flushing Application Fluency Buffer FluentdFlusher emit(event:A) append(event:A) emit(event:B) append(event:B) store events as MessagePack flush send(event:[A, B]) emit(event:C) append(event:C) flush invoked repeatedly PackedForward format close close force flush emit(event:D) append(event:D) send(event:[C,D]) event:C isn’t ready for flush event:A,B are ready for flush terminate thread
  • 10. Async flushing Application Fluency Buffer FluentdFlusher emit(event:A) append(event:A) emit(event:B) append(event:B) store events as MessagePack flush send(event:[A, B]) emit(event:C) append(event:C) flush invoked repeatedly PackedForward format close close force flush emit(event:D) append(event:D) send(event:[C,D]) event:C isn’t ready for flush event:A,B are ready for flush terminate thread
  • 11. Async flushing Application Fluency Buffer FluentdFlusher emit(event:A) append(event:A) emit(event:B) append(event:B) store events as MessagePack flush send(event:[A, B]) emit(event:C) append(event:C) flush invoked repeatedly PackedForward format close close force flush emit(event:D) append(event:D) send(event:[C,D]) event:C isn’t ready for flush event:A,B are ready for flush terminate thread
  • 12. Async flushing Application Fluency Buffer FluentdFlusher emit(event:A) append(event:A) emit(event:B) append(event:B) store events as MessagePack flush send(event:[A, B]) emit(event:C) append(event:C) flush invoked repeatedly PackedForward format close close force flush emit(event:D) append(event:D) send(event:[C,D]) event:C isn’t ready for flush event:A,B are ready for flush terminate thread
  • 13. Async flushing Application Fluency Buffer FluentdFlusher emit(event:A) append(event:A) emit(event:B) append(event:B) store events as MessagePack flush send(event:[A, B]) emit(event:C) append(event:C) flush invoked repeatedly PackedForward format close close force flush emit(event:D) append(event:D) send(event:[C,D]) event:C isn’t ready for flush event:A,B are ready for flush terminate thread
  • 14. Async flushing Application Fluency Buffer FluentdFlusher emit(event:A) append(event:A) emit(event:B) append(event:B) store events as MessagePack flush send(event:[A, B]) emit(event:C) append(event:C) flush invoked repeatedly PackedForward format close close force flush emit(event:D) append(event:D) send(event:[C,D]) event:C isn’t ready for flush event:A,B are ready for flush terminate thread
  • 15. Buffering events tag: aaa, time: 100, {name: a_0} Buffer pool retention buffers flushable buffers Fluentd emit event acquire buffer
  • 16. Buffering events tag: bbb, time: 101, {name: b_0} Buffer pool time: 100, {name: a_0} retention buffers flushable buffers tag: aaa Fluentd emit event acquire buffer
  • 17. Buffering events tag: ccc, time: 102, {name: c_0} Buffer pool time: 100, {name: a_0} time: 101, {name: b_0} retention buffers flushable buffers tag: aaa tag: bbb Fluentd emit event acquire buffer
  • 18. Buffering events tag: aaa, time: 110, {name: a_1} Buffer pool time: 100, {name: a_0} time: 101, {name: b_0} time: 102, {name: c_0} retention buffers flushable buffers tag: aaa tag: bbb tag: ccc Fluentd emit event
  • 19. Buffering events tag: aaa, time: 120, {name: a_2} Buffer pool time: 100, {name: a_0} time: 110, {name: a_1} time: 101, {name: b_0} time: 102, {name: c_0} retention buffers flushable buffers tag: aaa tag: bbb tag: ccc Fluentd emit event expand buffer
  • 20. Buffering events tag: bbb, time: 111, {name: b_1} : : Buffer pool time: 100, {name: a_0} time: 110, {name: a_1} time: 120, {name: a_2} time: 101, {name: b_0} time: 102, {name: c_0} retention buffers flushable buffers tag: aaa tag: bbb tag: ccc Fluentd emit event
  • 21. Buffering events tag: aaa, time: 200, {name: a_0} Buffer pool time: 100, {name: a_0} time: 110, {name: a_1} time: 120, {name: a_2} time: 130, {name: a_3} time: 101, {name: b_0} time: 111, {name: b_1} time: 121, {name: b_2} time: 131, {name: b_3} time: 102, {name: c_0} time: 112, {name: c_1} retention buffers flushable buffers tag: aaa tag: bbb tag: ccc Fluentd emit event move buffers (zero copy)
  • 22. tag: aaa Buffering events tag: ccc, time: 122, {name: c_2} Buffer pool time: 200, {name: a_0} time: 102, {name: c_0} time: 112, {name: c_1} retention buffers flushable buffers tag: aaa tag: ccc time: 100, {name: a_0} time: 110, {name: a_1} time: 120, {name: a_2} time: 130, {name: a_3} Fluentd tag: bbb time: 101, {name: b_0} time: 111, {name: b_1} time: 121, {name: b_2} time: 131, {name: b_3} emit event expand buffer send buffer return buffer (after sending) PackedForword format
  • 23. Buffering events tag: bbb, time: 201, {name: b_0} Buffer pool time: 200, {name: a_0} time: 102, {name: c_0} time: 112, {name: c_1} retention buffers flushable buffers tag: aaa tag: ccc Fluentd tag: bbb time: 101, {name: b_0} time: 111, {name: b_1} time: 121, {name: b_2} time: 131, {name: b_3} time: 122, {name: c_2} emit event send buffer return buffer (after sending) acquire buffer PackedForword format
  • 24. Buffering events tag: aaa, time: 210, {name: a_1} Buffer pool time: 200, {name: a_0} time: 102, {name: c_0} time: 112, {name: c_1} retention buffers flushable buffers tag: aaa tag: ccc Fluentd time: 122, {name: c_2} time: 201, {name: b_0} tag: bbb emit event
  • 25. 4x faster than
 fluent-logger-java fluency / fluent-logger-java: m3.large fluentd: m3.large See details: https://gist.github.com/komamitsu/c1e4045fe2ddb108cfbf12d5f014b683 But the actual bottleneck of this benchmark might be Fluentd…
  • 26. Java profiling (1/2) fluency fluent-logger-java fluency uses off heap for buffer pool Due to synchronization
  • 27. Java profiling (2/2) fluency fluent-logger-java GC Pause: 33ms/s GC Pause: 7ms/s
  • 28. Finding a bottleneck fluency emit (MessagePack serialization & appending to buffer) accounts for most of the duration String.getBytes in MessagePack serialization takes total 18% ObjectMapper.writeValue itself takes 40%
  • 29. Finding a bottleneck fluency MessagePackGenerator. writeStartArray takes 10% java.nio.channels.SocketCh annel.write and java.nio.ByteBuffer.allocate Direct are both 0-1%
  • 30. Finding a bottleneck fluent-logger-java Sending events via socket accounts for most of the duration
  • 31. Future improvements: removing ObjectMapper ObjectMapper can be removed to improve performance? Hmm. But Fluency accepts Jackson Modules…
  • 32. Future improvements: reducing use of ObjectMapper ObjectMapper is needed only for serialisation of data not array