Your SlideShare is downloading. ×
Apache Flume NG
Apache Flume NG
Apache Flume NG
Apache Flume NG
Apache Flume NG
Apache Flume NG
Apache Flume NG
Apache Flume NG
Apache Flume NG
Apache Flume NG
Apache Flume NG
Apache Flume NG
Apache Flume NG
Apache Flume NG
Apache Flume NG
Apache Flume NG
Apache Flume NG
Apache Flume NG
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Apache Flume NG

2,787

Published on

Talk given by Kai Voigt, Cloudera Inc, at the Hadoop User Group UK meetup on 10 Oct 2012 in London

Talk given by Kai Voigt, Cloudera Inc, at the Hadoop User Group UK meetup on 10 Oct 2012 in London

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,787
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
150
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. APACHE FLUME NG Kai Voigt, Cloudera Inc London, Hadoop User Group, 10 Oct 2012Donnerstag, 11. Oktober 12
  • 2. “ FLUME IS A DISTRIBUTED, RELIABLE, AND AVAILABLE SERVICE FOR EFFICIENTLY COLLECTING, AGGREGATING, AND MOVING LARGE AMOUNTS OF LOG DATA ”Donnerstag, 11. Oktober 12
  • 3. “ FLUME IS A DISTRIBUTED, RELIABLE, AND AVAILABLE SERVICE FOR EFFICIENTLY COLLECTING, AGGREGATING, AND MOVING LARGE AMOUNTS OF LOG DATA ”Donnerstag, 11. Oktober 12
  • 4. httpd /var/log/htaccess Flume HDFSDonnerstag, 11. Oktober 12
  • 5. 5Donnerstag, 11. Oktober 12
  • 6. mysource mysink mychannel myagent.sources = mysource myagent.sinks = mysink myagent.channels = mychannel 6Donnerstag, 11. Oktober 12
  • 7. mysource mysink mychannel myagent.sources.mysource.type = exec myagent.sources.mysource.command = tail -F /var/log/htaccess myagent.sources.mysource.channels = mychannel 7Donnerstag, 11. Oktober 12
  • 8. mysource mysink mychannel myagent.sinks.mysink.type = hdfs myagent.sinks.mysink.hdfs.path = /user/cloudera/htaccess myagent.sinks.mysink.hdfs.fileType = DataStream myagent.sinks.mysink.channel = mychannel 8Donnerstag, 11. Oktober 12
  • 9. mysource mysink mychannel myagent.channels.mychannel.type = memory myagent.channels.mychannel.capacity = 1000 myagent.channels.mychannel.transactionCapactiy = 100 9Donnerstag, 11. Oktober 12
  • 10. $ flume-ng agent --conf-file simple.conf --name myagent $ hadoop fs -ls htaccess -rw-r--r-- 1 cloudera cloudera 1001 2012-09-30 05:58 htaccess/FlumeData.1348999108529 -rw-r--r-- 1 cloudera cloudera 993 2012-09-30 05:58 htaccess/FlumeData.1348999108530 -rw-r--r-- 1 cloudera cloudera 997 2012-09-30 05:59 htaccess/FlumeData.1348999108531 -rw-r--r-- 1 cloudera cloudera 1009 2012-09-30 05:59 htaccess/FlumeData.1348999108532 ... 10Donnerstag, 11. Oktober 12
  • 11. “ FLUME IS A DISTRIBUTED, RELIABLE, AND AVAILABLE SERVICE FOR EFFICIENTLY COLLECTING, AGGREGATING, AND MOVING LARGE AMOUNTS OF LOG DATA ”Donnerstag, 11. Oktober 12
  • 12. MULTI HOP 12Donnerstag, 11. Oktober 12
  • 13. myagent1.sinks = mysink myagent1.sinks.mysink.type = avro myagent1.sinks.mysink.bind = 10.10.10.20 myagent1.sinks.mysink.port = 4141 myagent2.sources = mysource myagent2.sources.mysource.type = avro myagent2.sources.mysource.bind = 10.10.10.10 myagent2.sources.mysource.port = 4141 13Donnerstag, 11. Oktober 12
  • 14. CONSOLIDATION 14Donnerstag, 11. Oktober 12
  • 15. MULTIPLEXING 15Donnerstag, 11. Oktober 12
  • 16. Sources Sinks Channels Avro Avro Memory Exec Logger JDBC NetCat IRC File Sequence Generator File Syslog HBase Scribe 16Donnerstag, 11. Oktober 12
  • 17. DEMO DEMO DEMO DEMO DEMODonnerstag, 11. Oktober 12
  • 18. Thank you! kai@cloudera.com http://www.cloudera.com/Donnerstag, 11. Oktober 12

×