Download as PDF, PPTX
Uploaded byMuga Nishizawa
fluentd -- the missing log collector
Fluentd is an open source log collector that allows flexible collection and routing of log data. It uses JSON format for log messages and supports many input and output plugins. Fluentd can collect logs from files, network services, and applications before routing them to storage and analysis services like MongoDB, HDFS, and Treasure Data. The open source project has grown a large community contributing over 100 plugins to make log collection and processing easier.
More Related Content
![[March sn meetup] apache pulsar + apache nifi for cloud data lake](https://cdn.slidesharecdn.com/ss_thumbnails/marchsnmeetupapachepulsarapachenififorclouddatalake-220311125943-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to fluentd -- the missing log collector
fluentd -- the missing log collector
- 1. Muga Nishizawa Treasure Data,Inc. the missing log collector
- 2. Muga Nishizawa (@muga_nishizawa) ChiefSoftware Architect, Treasure Data
- 3. 3 Treasure Data Overview Founded to deliver big data analytics in days not months without specialist IT resources for one-tenth the cost of other alternatives Service based subscription business model World class open source team • Founded world’s largest Hadoop User Group • Developed Fluentd and MessagePack • Contributed to Memcached, Hibernate, etc. Treasure Data is in production • 60+ customers incl. Fortune 500 companies • 400+ billion records stored Processing 40,000 messages per second
- 4.
- 5.
- 6. > Open sourcedlog collector written in Ruby > Using rubygems ecosystem for plugins In short It’s like syslogd, but uses JSON for log messages
- 7. Make log collectioneasy using Fluentd
- 8.
- 9. Reporting & Monitoring CollectStore Process Visualize
- 10. Collect Store ProcessVisualize easier & shorter time Hadoop / Hive MongoDB Treasure Data Tableau Excel R Reporting & Monitoring
- 11. Collect Store ProcessVisualize easier & shorter timeHow to shorten here? Hadoop / Hive MongoDB Treasure Data Tableau Excel R
- 12. Collect Store ProcessVisualize easier & shorter timeHow to shorten here? Hadoop / Hive MongoDB Treasure Data Tableau Excel R
- 13. Before Fluentd Application ･･･ Server2 Application ･･･ Server3 Application ･･･ Server1 FluentLog Server HighLatency! must wait for a day...
- 14. After Fluentd Application ･･･ Server2 Application ･･･ Server3 Application ･･･ Server1 In streaming! FluentdFluentd Fluentd Fluentd Fluentd
- 15.
- 16.
- 17.
- 18. Why did wedevelop Fluentd?
- 19. Apache App App Other data sources td-agent RDBMS TreasureData columnar data warehouse Query Processing Cluster Query API HIVE, PIG (to be supported) JDBC, REST MAPREDUCE JOBS User td-command BI apps Treasure Data Service Architecture
- 20. Apache App App Other data sources td-agent RDBMS TreasureData columnar data warehouse Query Processing Cluster Query API HIVE, PIG (to be supported) JDBC, REST MAPREDUCE JOBS User td-command BI apps Treasure Data Service Architecture Open Sourced
- 21. writes logs totext files Rails app Google Spreadsheet MySQL MySQL MySQL MySQL writes logs to text files Nightly INSERT hundreds of app servers Daily/Hourly Batch KPI visualizationFeedback rankings Rails app writes logs to text files Rails app - Limited scalability - Fixed schema - Not realtime - Unexpected INSERT latency Example Use Case – MySQL to TD
- 22. hundreds of appservers sends event logs sends event logs sends event logs Rails app td-agent td-agent td-agent Google Spreadsheet Treasure Data MySQL Logs are available after several mins. Daily/Hourly Batch KPI visualizationFeedback rankings Rails app Rails app Unlimited scalability Flexible schema Realtime Less performance impact Example Use Case – MySQL to TD
- 23. td-agent > Open sourceddistribution package of fluentd > ETL part of Treasure Data > Including useful components > ruby, jemalloc, fluentd > 3rd party gems: td, mongo, webhdfs, etc... td plugin is for TD > http://packages.treasure-data.com/
- 24.
- 25.
- 26. Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd Applogs System logs Backend Databases filter / buffer / routing
- 27. Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd Applogs System logs Backend Databases filter / buffer / routing
- 28. Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd Applogs System logs Backend Databases filter / buffer / routing
- 29. Input Plugins OutputPlugins Buffer Plugins (Filter Plugins) Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Databases filter / buffer / routing
- 30. Architecture Buffer OutputInput > Forward >HTTP > File tail > dstat > ... > Forward > File > Amazon S3 > MongoDB > ... > Memory > File Pluggable Pluggable Pluggable
- 31. Architecture Buffer OutputInput > Forward >HTTP > File tail > dstat > ... > Forward > File > Amazon S3 > MongoDB > ... > Memory > File Pluggable Pluggable Pluggable 117 plugins! Contributions by Community
- 32. Input Plugins OutputPlugins 2012-02-04 01:33:51 myapp.buylog { “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” } time tag record JSON log
- 33. > second unit >from data source or adding parsed time Event structure(log message) ✓ Time > for message routing ✓ Tag > JSON format > MessagePack internally > non-unstructured ✓ Record
- 34. in_tail: reads fileand parses lines fluentd apache access.log ✓ read a log file ✓ custom regexp ✓ custom parser in Ruby in_tail
- 35.
- 36. failure handling &retrying fluentd apache access.log buffer ✓ retry automatically ✓ exponential retry wait ✓ persistent on a file in_tail
- 37. out_s3 fluentd apache access.log buffer ✓ retryautomatically ✓ exponential retry wait ✓ persistent on a file Amazon S3 ✓ slice files based on time in_tail 2013-01-01/01/access.log.gz 2013-01-01/02/access.log.gz 2013-01-01/03/access.log.gz ...
- 38. out_hdfs fluentd apache access.log buffer ✓ retryautomatically ✓ exponential retry wait ✓ persistent on a file ✓ slice files based on time in_tail 2013-01-01/01/access.log.gz 2013-01-01/02/access.log.gz 2013-01-01/03/access.log.gz ... HDFS ✓ custom text formater
- 39. routing / copying fluentd apache access.logbuffer ✓ routing based on tags ✓ copy to multiple storages in_tail Amazon S3 Hadoop
- 40. Fluentd # Ruby Fluent.open(“myapp”) Fluent.event(“login”, {“user”=> 38}) #=> 2012-12-11 07:56:01 myapp.login {“user”:38} > Ruby > Java > Perl > PHP > Python > D > Scala > ... Application Time:Tag:Record Client libraries
- 41. # logs froma file <source> type tail path /var/log/httpd.log format apache2 tag web.access </source> # logs from client libraries <source> type forward port 24224 </source> # store logs to MongoDB and S3 <match **> type copy <match> type mongo host mongo.example.com capped capped_size 200m </match> <match> type s3 path archive/ </match> </match> Fluentd
- 42. out_forward fluentd apache access.log buffer ✓ retryautomatically ✓ exponential retry wait ✓ persistent on a file ✓ slice files based on time in_tail 2013-01-01/01/access.log.gz 2013-01-01/02/access.log.gz 2013-01-01/03/access.log.gz ... fluentd fluentd fluentd ✓ automatic fail-over ✓ load balancing
- 43.
- 44. Fluentd - plugindistribution platform $ fluent-gem search -rd fluent-plugin $ fluent-gem install fluent-plugin-mongo
- 45.
- 46. hundreds of appservers sends event logs sends event logs sends event logs Rails app td-agent td-agent td-agent Google Spreadsheet Treasure Data MySQL Logs are available after several mins. Daily/Hourly Batch KPI visualizationFeedback rankings Rails app Rails app Unlimited scalability Flexible schema Realtime Less performance impact Cookpad ✓ Over 100 RoR servers (2012/2/4)
- 47. http://www.slideshare.net/tagomoris/log-analysis-with-hadoop-in-livedoor-2013 NHN Japan by @tagomoris ✓16 nodes ✓ 120,000+ lines/sec ✓ 400Mbps at peak ✓ 1.5+ TB/day (raw) Web Servers Fluentd Cluster Archive Storage (scribed) Fluentd Watchers Graph Tools Notifications (IRC) Hadoop Cluster CDH4 (HDFS, YARN) webhdfs Huahin Manager hive server STREAM Shib ShibUI BATCH SCHEDULED BATCH
- 48. Treasure Data Frontend Job Queue Worker Hadoop Hadoop Fluentd Applicationspush metrics to Fluentd (via local Fluentd) Librato Metrics for realtime analysis Treasure Data for historical analysis Fluentd sums up data minutes (partial aggregation)
- 49. Key to Fluentd’sgrowth is...
- 50.
- 51. Muga Nishizawa Treasure Data,Inc. the missing log collector