Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Fluency
Yet another fluent logger
Mitsunori Komatsu
About me
• Mitsunori Komatsu, 

Software engineer
• github:komamitsu

msgpack-java

(jackson-dataformat-msgpack),

fluent-l...
Today’s talk
• What's Fluency?
• Internal design
• Performance and profiling
• Future improvements
Fluentd
What’s Fluency?
• Yet another fluent-logger-java

(https://github.com/komamitsu/fluency)
• 3x 4x faster than fluent-logger us...
Internal design
• Asynchronous flushing
• Buffering PackedForward format
data
Async flushing
Application Fluency Buffer FluentdFlusher
emit(event:A) append(event:A)
emit(event:B) append(event:B)
store ...
Async flushing
Application Fluency Buffer FluentdFlusher
emit(event:A) append(event:A)
emit(event:B) append(event:B)
store ...
Async flushing
Application Fluency Buffer FluentdFlusher
emit(event:A) append(event:A)
emit(event:B) append(event:B)
store ...
Async flushing
Application Fluency Buffer FluentdFlusher
emit(event:A) append(event:A)
emit(event:B) append(event:B)
store ...
Async flushing
Application Fluency Buffer FluentdFlusher
emit(event:A) append(event:A)
emit(event:B) append(event:B)
store ...
Async flushing
Application Fluency Buffer FluentdFlusher
emit(event:A) append(event:A)
emit(event:B) append(event:B)
store ...
Async flushing
Application Fluency Buffer FluentdFlusher
emit(event:A) append(event:A)
emit(event:B) append(event:B)
store ...
Async flushing
Application Fluency Buffer FluentdFlusher
emit(event:A) append(event:A)
emit(event:B) append(event:B)
store ...
Buffering events
tag: aaa,
time: 100,
{name: a_0}
Buffer
pool
retention buffers
flushable buffers
Fluentd
emit event
acquir...
Buffering events
tag: bbb,
time: 101,
{name: b_0}
Buffer
pool
time: 100, {name: a_0}
retention buffers
flushable buffers
ta...
Buffering events
tag: ccc,
time: 102,
{name: c_0}
Buffer
pool
time: 100, {name: a_0} time: 101, {name: b_0}
retention buff...
Buffering events
tag: aaa,
time: 110,
{name: a_1}
Buffer
pool
time: 100, {name: a_0} time: 101, {name: b_0} time: 102, {na...
Buffering events
tag: aaa,
time: 120,
{name: a_2}
Buffer
pool
time: 100, {name: a_0}
time: 110, {name: a_1}
time: 101, {na...
Buffering events
tag: bbb,
time: 111,
{name: b_1}
:
:
Buffer
pool
time: 100, {name: a_0}
time: 110, {name: a_1}
time: 120,...
Buffering events
tag: aaa,
time: 200,
{name: a_0}
Buffer
pool
time: 100, {name: a_0}
time: 110, {name: a_1}
time: 120, {na...
tag: aaa
Buffering events
tag: ccc,
time: 122,
{name: c_2}
Buffer
pool
time: 200, {name: a_0} time: 102, {name: c_0}
time:...
Buffering events
tag: bbb,
time: 201,
{name: b_0}
Buffer
pool
time: 200, {name: a_0} time: 102, {name: c_0}
time: 112, {na...
Buffering events
tag: aaa,
time: 210,
{name: a_1}
Buffer
pool
time: 200, {name: a_0} time: 102, {name: c_0}
time: 112, {na...
4x faster than

fluent-logger-java
fluency / fluent-logger-java: m3.large
fluentd: m3.large
See details: https://gist.github.c...
Java profiling (1/2)
fluency fluent-logger-java
fluency uses off heap
for buffer pool
Due to synchronization
Java profiling (2/2)
fluency fluent-logger-java
GC Pause: 33ms/s GC Pause: 7ms/s
Finding a bottleneck
fluency
emit (MessagePack
serialization & appending to
buffer) accounts for most of
the duration
Strin...
Finding a bottleneck
fluency
MessagePackGenerator.
writeStartArray takes 10%
java.nio.channels.SocketCh
annel.write and
jav...
Finding a bottleneck
fluent-logger-java
Sending events via socket
accounts for most of the
duration
Future improvements:
removing ObjectMapper
ObjectMapper
can be removed
to improve
performance?
Hmm. But Fluency
accepts Ja...
Future improvements:
reducing use of ObjectMapper
ObjectMapper is needed only for
serialisation of data not array
Thanks!
Upcoming SlideShare
Loading in …5
×

Fluency - Yet another fluent logger

3,440 views

Published on

Fluency is a high performance logger for Fluentd. This presentation is about the internal designs and the result of profiling.

Published in: Software
  • Be the first to comment

Fluency - Yet another fluent logger

  1. 1. Fluency Yet another fluent logger Mitsunori Komatsu
  2. 2. About me • Mitsunori Komatsu, 
 Software engineer • github:komamitsu
 msgpack-java
 (jackson-dataformat-msgpack),
 fluent-logger-java,
 digdag,
 :
  3. 3. Today’s talk • What's Fluency? • Internal design • Performance and profiling • Future improvements
  4. 4. Fluentd
  5. 5. What’s Fluency? • Yet another fluent-logger-java
 (https://github.com/komamitsu/fluency) • 3x 4x faster than fluent-logger using PackedForward format • Easy to extend • Many features
 (Async flush, HB, Failover, Ack response) • Used in LINE, Treasure Data, kafka-fluentd-consumer
  6. 6. Internal design • Asynchronous flushing • Buffering PackedForward format data
  7. 7. Async flushing Application Fluency Buffer FluentdFlusher emit(event:A) append(event:A) emit(event:B) append(event:B) store events as MessagePack flush send(event:[A, B]) emit(event:C) append(event:C) flush invoked repeatedly PackedForward format close close force flush emit(event:D) append(event:D) send(event:[C,D]) event:C isn’t ready for flush event:A,B are ready for flush terminate thread
  8. 8. Async flushing Application Fluency Buffer FluentdFlusher emit(event:A) append(event:A) emit(event:B) append(event:B) store events as MessagePack flush send(event:[A, B]) emit(event:C) append(event:C) flush invoked repeatedly PackedForward format close close force flush emit(event:D) append(event:D) send(event:[C,D]) event:C isn’t ready for flush event:A,B are ready for flush terminate thread
  9. 9. Async flushing Application Fluency Buffer FluentdFlusher emit(event:A) append(event:A) emit(event:B) append(event:B) store events as MessagePack flush send(event:[A, B]) emit(event:C) append(event:C) flush invoked repeatedly PackedForward format close close force flush emit(event:D) append(event:D) send(event:[C,D]) event:C isn’t ready for flush event:A,B are ready for flush terminate thread
  10. 10. Async flushing Application Fluency Buffer FluentdFlusher emit(event:A) append(event:A) emit(event:B) append(event:B) store events as MessagePack flush send(event:[A, B]) emit(event:C) append(event:C) flush invoked repeatedly PackedForward format close close force flush emit(event:D) append(event:D) send(event:[C,D]) event:C isn’t ready for flush event:A,B are ready for flush terminate thread
  11. 11. Async flushing Application Fluency Buffer FluentdFlusher emit(event:A) append(event:A) emit(event:B) append(event:B) store events as MessagePack flush send(event:[A, B]) emit(event:C) append(event:C) flush invoked repeatedly PackedForward format close close force flush emit(event:D) append(event:D) send(event:[C,D]) event:C isn’t ready for flush event:A,B are ready for flush terminate thread
  12. 12. Async flushing Application Fluency Buffer FluentdFlusher emit(event:A) append(event:A) emit(event:B) append(event:B) store events as MessagePack flush send(event:[A, B]) emit(event:C) append(event:C) flush invoked repeatedly PackedForward format close close force flush emit(event:D) append(event:D) send(event:[C,D]) event:C isn’t ready for flush event:A,B are ready for flush terminate thread
  13. 13. Async flushing Application Fluency Buffer FluentdFlusher emit(event:A) append(event:A) emit(event:B) append(event:B) store events as MessagePack flush send(event:[A, B]) emit(event:C) append(event:C) flush invoked repeatedly PackedForward format close close force flush emit(event:D) append(event:D) send(event:[C,D]) event:C isn’t ready for flush event:A,B are ready for flush terminate thread
  14. 14. Async flushing Application Fluency Buffer FluentdFlusher emit(event:A) append(event:A) emit(event:B) append(event:B) store events as MessagePack flush send(event:[A, B]) emit(event:C) append(event:C) flush invoked repeatedly PackedForward format close close force flush emit(event:D) append(event:D) send(event:[C,D]) event:C isn’t ready for flush event:A,B are ready for flush terminate thread
  15. 15. Buffering events tag: aaa, time: 100, {name: a_0} Buffer pool retention buffers flushable buffers Fluentd emit event acquire buffer
  16. 16. Buffering events tag: bbb, time: 101, {name: b_0} Buffer pool time: 100, {name: a_0} retention buffers flushable buffers tag: aaa Fluentd emit event acquire buffer
  17. 17. Buffering events tag: ccc, time: 102, {name: c_0} Buffer pool time: 100, {name: a_0} time: 101, {name: b_0} retention buffers flushable buffers tag: aaa tag: bbb Fluentd emit event acquire buffer
  18. 18. Buffering events tag: aaa, time: 110, {name: a_1} Buffer pool time: 100, {name: a_0} time: 101, {name: b_0} time: 102, {name: c_0} retention buffers flushable buffers tag: aaa tag: bbb tag: ccc Fluentd emit event
  19. 19. Buffering events tag: aaa, time: 120, {name: a_2} Buffer pool time: 100, {name: a_0} time: 110, {name: a_1} time: 101, {name: b_0} time: 102, {name: c_0} retention buffers flushable buffers tag: aaa tag: bbb tag: ccc Fluentd emit event expand buffer
  20. 20. Buffering events tag: bbb, time: 111, {name: b_1} : : Buffer pool time: 100, {name: a_0} time: 110, {name: a_1} time: 120, {name: a_2} time: 101, {name: b_0} time: 102, {name: c_0} retention buffers flushable buffers tag: aaa tag: bbb tag: ccc Fluentd emit event
  21. 21. Buffering events tag: aaa, time: 200, {name: a_0} Buffer pool time: 100, {name: a_0} time: 110, {name: a_1} time: 120, {name: a_2} time: 130, {name: a_3} time: 101, {name: b_0} time: 111, {name: b_1} time: 121, {name: b_2} time: 131, {name: b_3} time: 102, {name: c_0} time: 112, {name: c_1} retention buffers flushable buffers tag: aaa tag: bbb tag: ccc Fluentd emit event move buffers (zero copy)
  22. 22. tag: aaa Buffering events tag: ccc, time: 122, {name: c_2} Buffer pool time: 200, {name: a_0} time: 102, {name: c_0} time: 112, {name: c_1} retention buffers flushable buffers tag: aaa tag: ccc time: 100, {name: a_0} time: 110, {name: a_1} time: 120, {name: a_2} time: 130, {name: a_3} Fluentd tag: bbb time: 101, {name: b_0} time: 111, {name: b_1} time: 121, {name: b_2} time: 131, {name: b_3} emit event expand buffer send buffer return buffer (after sending) PackedForword format
  23. 23. Buffering events tag: bbb, time: 201, {name: b_0} Buffer pool time: 200, {name: a_0} time: 102, {name: c_0} time: 112, {name: c_1} retention buffers flushable buffers tag: aaa tag: ccc Fluentd tag: bbb time: 101, {name: b_0} time: 111, {name: b_1} time: 121, {name: b_2} time: 131, {name: b_3} time: 122, {name: c_2} emit event send buffer return buffer (after sending) acquire buffer PackedForword format
  24. 24. Buffering events tag: aaa, time: 210, {name: a_1} Buffer pool time: 200, {name: a_0} time: 102, {name: c_0} time: 112, {name: c_1} retention buffers flushable buffers tag: aaa tag: ccc Fluentd time: 122, {name: c_2} time: 201, {name: b_0} tag: bbb emit event
  25. 25. 4x faster than
 fluent-logger-java fluency / fluent-logger-java: m3.large fluentd: m3.large See details: https://gist.github.com/komamitsu/c1e4045fe2ddb108cfbf12d5f014b683 But the actual bottleneck of this benchmark might be Fluentd…
  26. 26. Java profiling (1/2) fluency fluent-logger-java fluency uses off heap for buffer pool Due to synchronization
  27. 27. Java profiling (2/2) fluency fluent-logger-java GC Pause: 33ms/s GC Pause: 7ms/s
  28. 28. Finding a bottleneck fluency emit (MessagePack serialization & appending to buffer) accounts for most of the duration String.getBytes in MessagePack serialization takes total 18% ObjectMapper.writeValue itself takes 40%
  29. 29. Finding a bottleneck fluency MessagePackGenerator. writeStartArray takes 10% java.nio.channels.SocketCh annel.write and java.nio.ByteBuffer.allocate Direct are both 0-1%
  30. 30. Finding a bottleneck fluent-logger-java Sending events via socket accounts for most of the duration
  31. 31. Future improvements: removing ObjectMapper ObjectMapper can be removed to improve performance? Hmm. But Fluency accepts Jackson Modules…
  32. 32. Future improvements: reducing use of ObjectMapper ObjectMapper is needed only for serialisation of data not array
  33. 33. Thanks!

×