Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Fluentd Hacking Guide at RubyKaigi 2014

67,682 views

Published on

Fluentd Hacking Guide at RubyKaigi 2014

Published in: Engineering
  • Hello! High Quality And Affordable Essays For You. Starting at $4.99 per page - Check our website! https://vk.cc/82gJD2
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Fluentd Hacking Guide at RubyKaigi 2014

  1. 1. Fluentd Hacking Guide (Fluentd ソースコード完全解説) September 20th, 2014 ! Naotoshi Seo @sonots DeNA Co., Ltd. Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
  2. 2. 2 Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Who am I? ・Naotoshi Seo @sonots ・DeNA Co, Ltd. ・Infrastructure Engineer ・Fluentd Commiter
  3. 3. ⁃ An application (or a framework) to process log streaming ⁃ Powerful plugin architecture (+250 plugins) ⁃ written in Ruby Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
  4. 4. Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Disclaimer 4 1. I do not explain what is Fluentd, how to use Fluentd 2. I assume audiences have ever used Fluentd, have ever created Fluentd plugins, have interests inside Fluentd ! 3. Source Codes are from v0.10 branch as of September 2014
  5. 5. Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Topics 5 1. The bootstrap sequence of Fluentd, and how Fluentd loads plugins 2. How an input plugin passes data to output plugins 3. How BufferedOutput plugin works 4. How Fluentd parses the config file 5. The event-driven programming using cool.io, and effects of GVL
  6. 6. 6 Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Let's Dive into Source Codes
  7. 7. 7 Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. vim
  8. 8. Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Topics 8 1. The bootstrap sequence of Fluentd, and how Fluentd loads plugins 2. How an input plugin passes data to output plugins 3. How BufferedOutput plugin works
  9. 9. The Bootstrap Sequence 9 Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Fluent::Supervisor#run_configure 1. require 2. new 3. configure(conf ) Fluent::Supervisor#run_engine 4. start 5. shutdown (if signal received) Input plugin creates threads on #start Output plugin does nothing on #start (typically)
  10. 10. Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Topics 10 1. The bootstrap sequence of Fluentd, and how Fluentd loads plugins 2. How an input plugin passes data to output plugins 3. How BufferedOutput plugin works
  11. 11. Dataflow from Input to Output 11 Input Engine Output Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. emit(tag, es) emit(tag, es) If an input thread receives data, call Engine.emit
  12. 12. CAUTION: Output Plugin 12 Input Engine Output Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. emit(tag, es) emit(tag, es) BLOCK!!! Can not receive new input during blocking ex) HTTP POST
  13. 13. 13 Use BufferedOutput Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. to avoid blocking
  14. 14. Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Topics 14 1. The bootstrap sequence of Fluentd, and how Fluentd loads plugins 2. How an input plugin passes data to output plugins 3. How BufferedOutput plugin works
  15. 15. BufferedOutput Plugin Input Engine BufferedOutput BasicBuffer 15 Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. emit(tag,es) emit(tag,es) emit(tag,data) enqueue immediately return!! Can receive new input. No blocking.
  16. 16. BufferedOutput Plugin 16 BufferedOutput BasicBuffer Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. try_flush (push) OutputThread pop write(chunk) do some EXPENSIVE things Run heavy processing in OTHER threads
  17. 17. CAUTION: BufferedOutput 17 Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Stuck if enqueued over its capability ! HOW TO IMPROVE 1. Increase num_threads 2. Enlarge buffer_chunk_limit 3. Set smaller queued_chunk_flush_interval, try_flush_interval (Secret parameters)
  18. 18. 18 Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. But,
  19. 19. 19 Improving actual processing throughputs is most important fluent-plugin-elasticsearch gets stuck? Then, tune Elasticsearch! Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved.
  20. 20. Copyright (C) 2014 DeNA Co.,Ltd. All Rights Reserved. Conclusion 20 1. Output plugin blocks 2. BufferedOutput does not block, but stuck if enqueued over its capability • Tune with option parameters such as num_threads 3. In either case, improving actual processing capability itself is most important!! 4. I am happy if this talk helps your operation, your plugin development, and contributions

×