Fluentd and AWS at classmethod

2,524 views

Published on

Presented at http://connpass.com/event/5222/

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,524
On SlideShare
0
From Embeds
0
Number of Embeds
1,059
Actions
Shares
0
Downloads
24
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Fluentd and AWS at classmethod

  1. 1. Mar 21, 2014 www.treasuredata.com/ Fluentd & AWS! Masahiro Nakagawa Treasure Data, Inc 1
  2. 2. Who are you? • Masahiro Nakagawa • @repeatedly • Treasure Data, Inc • Senior Software Engineer • Fluentd, td-agent, etc... • Dlang, MessagePack, ... 2
  3. 3. Treasure Data on AWS 4
  4. 4. Frontend Queue Worker Hadoop Fluentd Applications push metrics to Fluentd (via local Fluentd) Librato Metrics for realtime analysis Treasure Data for historical analysis Fluentd sums up data minutes (partial aggregation) Backend overview Impala Presto Hadoop
  5. 5. Used AWS products • RDS • Store service data • Queue / Scheduler • S3 • Columnar storage • EC2 • Clusters: Hadoop,Workers,APIs, etc… 6 Separate Storage and Processor!
  6. 6. Classmethod use case! 7
  7. 7. Fluentd (Treasure Agent) 8
  8. 8. Structured logging Reliable forwarding Pluggable architecture http://fluentd.org/
  9. 9. Collect Store Process Visualize Data source Reporting Monitoring Data Processing
  10. 10. Related Products Store Process Cloudera Horton Works Treasure Data Collect Visualize Tableau Excel R easier & shorter time ???
  11. 11. Before… 12 Application ・・・ Server2 Application ・・・ Server3 Application ・・・ Server1 FluentLog Server High Latency! must wait for a day...
  12. 12. Divide & Conquer & Retry 13 error retry error retry retry retry
  13. 13. After! 14 Application ・・・ Server2 Application ・・・ Server3 Application ・・・ Server1 Fluentd Fluentd Fluentd Fluentd Fluentd In streaming!
  14. 14. Lambda Architecture 15 http://www.drdobbs.com/database/applying-the-big-data-lambda-architectur/240162604
  15. 15. In short • Open sourced log collector written in Ruby • Customization is essential small core + many plugins 16 Fluentd is a robust log collector designed for processing data streams
  16. 16. Core Plugins • Divide & Conquer • Buffring & Retrying • Error handling • Message routing • Parallelize • read / receive data • write / send data 17
  17. 17. M x N → M + N 18 Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Databases buffer / buffer / routing
  18. 18. Pluggable Architecture 19 Buffer Output Input > Forward > HTTP > File tail > dstat > ... > Forward > File > MongoDB > ... > File > Memory Engine Output > rewrite > ... Pluggable Pluggable
  19. 19. Next release 20 • Fluentd v0.10.45 • in_tail supports multiline and * watch • in_exec supports json / msgpack • several fixes • td-agent 1.1.19
  20. 20. AWS use cases 21
  21. 21. Collecting instance logs 22 • A sign of Immutable Infrastructure • Hard to manage state-full instance • Almost instance should be disposable • Excluding DB, Master, etc... • How to manage such instance logs? • Common problem on Cloud environment
  22. 22. • Start Fluentd at launch phase • It is also useful for Docker / other containers • Including metadata or host to identify Collecting using Fluentd 23 Collector Aggregator
  23. 23. AWS Plugins 24 http://fluentd.org/plugin/ • s3 • dynamodb • redshift • rds • elb • cloudwatch • sns • sqs • ses • kinesis (soon!)

×