Tornado is a Python web framework and asynchronous networking library. It is used by companies like Facebook to build web servers and services that can handle many concurrent connections efficiently. Tornado uses a non-blocking network I/O approach which allows it to scale to large numbers of open connections without performance issues.
Tornado is a Python web framework and asynchronous networking library. It is used by companies like Facebook to build web servers and services that can handle many concurrent connections efficiently. Tornado uses a non-blocking network I/O approach which allows it to scale to large numbers of open connections without performance issues.
Fluentd is a data collector that can unify logging and metrics formats and enable real-time extraction, transformation, and storage of data. It will be used at 10xLab to collect logging data from their Co-Work app and infrastructure components and enable real-time analysis and long-term storage. Fluentd makes it easy to set up log collection pipelines and extend functionality through plugins. 10xLab plans to use Fluentd with Resque to reliably queue and process job data, store logs in S3, analyze logs in Treasure Data, and monitor systems. Fluentd will be installed via AWS cloud-init and managed using Chef.
Fluentd Hacking Guide at RubyKaigi 2014Naotoshi Seo
This document summarizes a talk on hacking the Fluentd log streaming framework. It discusses Fluentd's bootstrap sequence and how it loads plugins. It explains how input plugins pass data to output plugins and how the BufferedOutput plugin works by buffering data to avoid blocking. It cautions that output plugins can block which BufferedOutput avoids, but it can also get stuck if the buffer exceeds capacity. It recommends tuning buffer size and thread counts to improve performance.
Fluentd is an open source data collector that allows flexible data collection, processing, and storage. It collects log data from various sources using input plugins and sends the data to various outputs like files, databases or forward to other Fluentd servers. It uses a pluggable architecture so new input/output plugins can be added through Ruby gems. It provides features like buffering, retries and reliability.
This document discusses middleware in Ruby and provides examples of considerations when writing middleware:
- Middleware should be a long-running daemon process that is compatible across platforms and environments and handles various data formats and traffic volumes.
- Tests must be run on all supported platforms to ensure compatibility as thread and process scheduling differs between operating systems.
- Memory usage and object leaks must be carefully managed in long-running processes to avoid consuming resources over time.
- Performance of JSON parsing/generation should be benchmarked and the most optimized library used to avoid unnecessary CPU usage.
7. 導入事例のご紹介
Fluentd meetup in Japan #1
http://www.zusaar.com/event/193104
Fluentd meetup in Japan #2
http://www.zusaar.com/event/355006
Fluentd Casual Talks
http://atnd.org/events/27808
8. 導入事例のご紹介
Fluentd meetup in Japan #2のスライドから
Fluentdを優しく見守る監視事例
http://www.slideshare.net/GedowFather/fluentd-
meetup-2-fluentd
Fluentd & TreasureDataで
こっそり始めるログ収集
http://www.slideshare.net/baguzy/fluentd-
meetup-2-14073930
