This document summarizes a talk on hacking the Fluentd log streaming framework. It discusses Fluentd's bootstrap sequence and how it loads plugins. It explains how input plugins pass data to output plugins and how the BufferedOutput plugin works by buffering data to avoid blocking. It cautions that output plugins can block which BufferedOutput avoids, but it can also get stuck if the buffer exceeds capacity. It recommends tuning buffer size and thread counts to improve performance.

1. Fluentd Hacking Guide
(Fluentd ソースコード完全解説)
September 20th, 2014
!
Naotoshi Seo @sonots
DeNA Co., Ltd.
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.

2. 2
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
Who am I?
・Naotoshi Seo @sonots
・DeNA Co, Ltd.
・Infrastructure Engineer
・Fluentd Commiter

3. ⁃ An application (or a framework) to process log streaming
⁃ Powerful plugin architecture (+250 plugins)
⁃ written in Ruby
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.

4. Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
Disclaimer
4
1. I do not explain what is Fluentd, how to use Fluentd
2. I assume audiences have ever used Fluentd, have ever
created Fluentd plugins, have interests inside Fluentd
!
3. Source Codes are from v0.10 branch as of September 2014

5. Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
Topics
5
1. The bootstrap sequence of Fluentd, and how Fluentd loads
plugins
2. How an input plugin passes data to output plugins
3. How BufferedOutput plugin works
4. How Fluentd parses the config file
5. The event-driven programming using cool.io, and effects of
GVL

6. 6
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
Let's Dive into
Source Codes

7. 7
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
vim

8. Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
Topics
8
1. The bootstrap sequence of Fluentd, and how Fluentd loads
plugins
2. How an input plugin passes data to output plugins
3. How BufferedOutput plugin works

9. The Bootstrap Sequence
9
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
Fluent::Supervisor#run_configure
1. require
2. new
3. configure(conf )
Fluent::Supervisor#run_engine
4. start
5. shutdown (if signal received)
Input plugin creates threads on #start
Output plugin does nothing on #start (typically)

10. Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
Topics
10
1. The bootstrap sequence of Fluentd, and how Fluentd loads
plugins
2. How an input plugin passes data to output plugins
3. How BufferedOutput plugin works

11. Dataflow from Input to Output
11
Input Engine Output
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
emit(tag, es)
emit(tag, es)
If an input thread receives data, call Engine.emit

12. CAUTION: Output Plugin
12
Input Engine Output
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
emit(tag, es)
emit(tag, es)
BLOCK!!!
Can not receive new input during blocking
ex) HTTP POST

13. 13
Use BufferedOutput
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
to avoid blocking

14. Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
Topics
14
1. The bootstrap sequence of Fluentd, and how Fluentd loads
plugins
2. How an input plugin passes data to output plugins
3. How BufferedOutput plugin works

15. BufferedOutput Plugin
Input Engine BufferedOutput BasicBuffer
15
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
emit(tag,es)
emit(tag,es)
emit(tag,data)
enqueue
immediately
return!!
Can receive new input. No blocking.

16. BufferedOutput Plugin
16
BufferedOutput BasicBuffer
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
try_flush
(push)
OutputThread
pop
write(chunk)
do some EXPENSIVE things
Run heavy processing in OTHER threads

17. CAUTION: BufferedOutput
17
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
Stuck if enqueued over its capability
!
HOW TO IMPROVE
1. Increase num_threads
2. Enlarge buffer_chunk_limit
3. Set smaller queued_chunk_flush_interval, try_flush_interval
(Secret parameters)

18. 18
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
But,

19. 19
Improving actual
processing throughputs
is most important
fluent-plugin-elasticsearch gets stuck? Then, tune Elasticsearch!
Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.

20. Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
Conclusion
20
1. Output plugin blocks
2. BufferedOutput does not block, but stuck if enqueued
over its capability
• Tune with option parameters such as num_threads
3. In either case, improving actual processing capability
itself is most important!!
4. I am happy if this talk helps your operation, your
plugin development, and contributions