Apache Flume NG
•
2 likes•3,317 views
Talk given by Kai Voigt, Cloudera Inc, at the Hadoop User Group UK meetup on 10 Oct 2012 in London
Recommended
More Related Content
Viewers also liked
Viewers also liked (17)
Similar to Apache Flume NG
Similar to Apache Flume NG (20)
More from huguk
More from huguk (20)
Recently uploaded
Recently uploaded (20)
Apache Flume NG
- 1. APACHE FLUME NG Kai Voigt, Cloudera Inc London, Hadoop User Group, 10 Oct 2012 Donnerstag, 11. Oktober 12
- 2. “ FLUME IS A DISTRIBUTED, RELIABLE, AND AVAILABLE SERVICE FOR EFFICIENTLY COLLECTING, AGGREGATING, AND MOVING LARGE AMOUNTS OF LOG DATA ” Donnerstag, 11. Oktober 12
- 3. “ FLUME IS A DISTRIBUTED, RELIABLE, AND AVAILABLE SERVICE FOR EFFICIENTLY COLLECTING, AGGREGATING, AND MOVING LARGE AMOUNTS OF LOG DATA ” Donnerstag, 11. Oktober 12
- 4. httpd /var/log/htaccess Flume HDFS Donnerstag, 11. Oktober 12
- 6. mysource mysink mychannel myagent.sources = mysource myagent.sinks = mysink myagent.channels = mychannel 6 Donnerstag, 11. Oktober 12
- 7. mysource mysink mychannel myagent.sources.mysource.type = exec myagent.sources.mysource.command = tail -F /var/log/htaccess myagent.sources.mysource.channels = mychannel 7 Donnerstag, 11. Oktober 12
- 8. mysource mysink mychannel myagent.sinks.mysink.type = hdfs myagent.sinks.mysink.hdfs.path = /user/cloudera/htaccess myagent.sinks.mysink.hdfs.fileType = DataStream myagent.sinks.mysink.channel = mychannel 8 Donnerstag, 11. Oktober 12
- 9. mysource mysink mychannel myagent.channels.mychannel.type = memory myagent.channels.mychannel.capacity = 1000 myagent.channels.mychannel.transactionCapactiy = 100 9 Donnerstag, 11. Oktober 12
- 10. $ flume-ng agent --conf-file simple.conf --name myagent $ hadoop fs -ls htaccess -rw-r--r-- 1 cloudera cloudera 1001 2012-09-30 05:58 htaccess/FlumeData.1348999108529 -rw-r--r-- 1 cloudera cloudera 993 2012-09-30 05:58 htaccess/FlumeData.1348999108530 -rw-r--r-- 1 cloudera cloudera 997 2012-09-30 05:59 htaccess/FlumeData.1348999108531 -rw-r--r-- 1 cloudera cloudera 1009 2012-09-30 05:59 htaccess/FlumeData.1348999108532 ... 10 Donnerstag, 11. Oktober 12
- 11. “ FLUME IS A DISTRIBUTED, RELIABLE, AND AVAILABLE SERVICE FOR EFFICIENTLY COLLECTING, AGGREGATING, AND MOVING LARGE AMOUNTS OF LOG DATA ” Donnerstag, 11. Oktober 12
- 12. MULTI HOP 12 Donnerstag, 11. Oktober 12
- 13. myagent1.sinks = mysink myagent1.sinks.mysink.type = avro myagent1.sinks.mysink.bind = 10.10.10.20 myagent1.sinks.mysink.port = 4141 myagent2.sources = mysource myagent2.sources.mysource.type = avro myagent2.sources.mysource.bind = 10.10.10.10 myagent2.sources.mysource.port = 4141 13 Donnerstag, 11. Oktober 12
- 14. CONSOLIDATION 14 Donnerstag, 11. Oktober 12
- 15. MULTIPLEXING 15 Donnerstag, 11. Oktober 12
- 16. Sources Sinks Channels Avro Avro Memory Exec Logger JDBC NetCat IRC File Sequence Generator File Syslog HBase Scribe 16 Donnerstag, 11. Oktober 12
- 17. DEMO DEMO DEMO DEMO DEMO Donnerstag, 11. Oktober 12
- 18. Thank you! kai@cloudera.com http://www.cloudera.com/ Donnerstag, 11. Oktober 12