Your SlideShare is downloading. ×
Fluentd and AWS at classmethod
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Fluentd and AWS at classmethod

1,610
views

Published on

Presented at http://connpass.com/event/5222/

Presented at http://connpass.com/event/5222/

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,610
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
19
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Mar 21, 2014 www.treasuredata.com/ Fluentd & AWS! Masahiro Nakagawa Treasure Data, Inc 1
  • 2. Who are you? • Masahiro Nakagawa • @repeatedly • Treasure Data, Inc • Senior Software Engineer • Fluentd, td-agent, etc... • Dlang, MessagePack, ... 2
  • 3. Treasure Data on AWS 4
  • 4. Frontend Queue Worker Hadoop Fluentd Applications push metrics to Fluentd (via local Fluentd) Librato Metrics for realtime analysis Treasure Data for historical analysis Fluentd sums up data minutes (partial aggregation) Backend overview Impala Presto Hadoop
  • 5. Used AWS products • RDS • Store service data • Queue / Scheduler • S3 • Columnar storage • EC2 • Clusters: Hadoop,Workers,APIs, etc… 6 Separate Storage and Processor!
  • 6. Classmethod use case! 7
  • 7. Fluentd (Treasure Agent) 8
  • 8. Structured logging Reliable forwarding Pluggable architecture http://fluentd.org/
  • 9. Collect Store Process Visualize Data source Reporting Monitoring Data Processing
  • 10. Related Products Store Process Cloudera Horton Works Treasure Data Collect Visualize Tableau Excel R easier & shorter time ???
  • 11. Before… 12 Application ・・・ Server2 Application ・・・ Server3 Application ・・・ Server1 FluentLog Server High Latency! must wait for a day...
  • 12. Divide & Conquer & Retry 13 error retry error retry retry retry
  • 13. After! 14 Application ・・・ Server2 Application ・・・ Server3 Application ・・・ Server1 Fluentd Fluentd Fluentd Fluentd Fluentd In streaming!
  • 14. Lambda Architecture 15 http://www.drdobbs.com/database/applying-the-big-data-lambda-architectur/240162604
  • 15. In short • Open sourced log collector written in Ruby • Customization is essential small core + many plugins 16 Fluentd is a robust log collector designed for processing data streams
  • 16. Core Plugins • Divide & Conquer • Buffring & Retrying • Error handling • Message routing • Parallelize • read / receive data • write / send data 17
  • 17. M x N → M + N 18 Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Databases buffer / buffer / routing
  • 18. Pluggable Architecture 19 Buffer Output Input > Forward > HTTP > File tail > dstat > ... > Forward > File > MongoDB > ... > File > Memory Engine Output > rewrite > ... Pluggable Pluggable
  • 19. Next release 20 • Fluentd v0.10.45 • in_tail supports multiline and * watch • in_exec supports json / msgpack • several fixes • td-agent 1.1.19
  • 20. AWS use cases 21
  • 21. Collecting instance logs 22 • A sign of Immutable Infrastructure • Hard to manage state-full instance • Almost instance should be disposable • Excluding DB, Master, etc... • How to manage such instance logs? • Common problem on Cloud environment
  • 22. • Start Fluentd at launch phase • It is also useful for Docker / other containers • Including metadata or host to identify Collecting using Fluentd 23 Collector Aggregator
  • 23. AWS Plugins 24 http://fluentd.org/plugin/ • s3 • dynamodb • redshift • rds • elb • cloudwatch • sns • sqs • ses • kinesis (soon!)