• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Apache Flume NG
 

Apache Flume NG

on

  • 3,128 views

Talk given by Kai Voigt, Cloudera Inc, at the Hadoop User Group UK meetup on 10 Oct 2012 in London

Talk given by Kai Voigt, Cloudera Inc, at the Hadoop User Group UK meetup on 10 Oct 2012 in London

Statistics

Views

Total Views
3,128
Views on SlideShare
3,013
Embed Views
115

Actions

Likes
1
Downloads
116
Comments
0

1 Embed 115

http://tedwon.com 115

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Apache Flume NG Apache Flume NG Presentation Transcript

    • APACHE FLUME NG Kai Voigt, Cloudera Inc London, Hadoop User Group, 10 Oct 2012Donnerstag, 11. Oktober 12
    • “ FLUME IS A DISTRIBUTED, RELIABLE, AND AVAILABLE SERVICE FOR EFFICIENTLY COLLECTING, AGGREGATING, AND MOVING LARGE AMOUNTS OF LOG DATA ”Donnerstag, 11. Oktober 12
    • “ FLUME IS A DISTRIBUTED, RELIABLE, AND AVAILABLE SERVICE FOR EFFICIENTLY COLLECTING, AGGREGATING, AND MOVING LARGE AMOUNTS OF LOG DATA ”Donnerstag, 11. Oktober 12
    • httpd /var/log/htaccess Flume HDFSDonnerstag, 11. Oktober 12
    • 5Donnerstag, 11. Oktober 12
    • mysource mysink mychannel myagent.sources = mysource myagent.sinks = mysink myagent.channels = mychannel 6Donnerstag, 11. Oktober 12
    • mysource mysink mychannel myagent.sources.mysource.type = exec myagent.sources.mysource.command = tail -F /var/log/htaccess myagent.sources.mysource.channels = mychannel 7Donnerstag, 11. Oktober 12
    • mysource mysink mychannel myagent.sinks.mysink.type = hdfs myagent.sinks.mysink.hdfs.path = /user/cloudera/htaccess myagent.sinks.mysink.hdfs.fileType = DataStream myagent.sinks.mysink.channel = mychannel 8Donnerstag, 11. Oktober 12
    • mysource mysink mychannel myagent.channels.mychannel.type = memory myagent.channels.mychannel.capacity = 1000 myagent.channels.mychannel.transactionCapactiy = 100 9Donnerstag, 11. Oktober 12
    • $ flume-ng agent --conf-file simple.conf --name myagent $ hadoop fs -ls htaccess -rw-r--r-- 1 cloudera cloudera 1001 2012-09-30 05:58 htaccess/FlumeData.1348999108529 -rw-r--r-- 1 cloudera cloudera 993 2012-09-30 05:58 htaccess/FlumeData.1348999108530 -rw-r--r-- 1 cloudera cloudera 997 2012-09-30 05:59 htaccess/FlumeData.1348999108531 -rw-r--r-- 1 cloudera cloudera 1009 2012-09-30 05:59 htaccess/FlumeData.1348999108532 ... 10Donnerstag, 11. Oktober 12
    • “ FLUME IS A DISTRIBUTED, RELIABLE, AND AVAILABLE SERVICE FOR EFFICIENTLY COLLECTING, AGGREGATING, AND MOVING LARGE AMOUNTS OF LOG DATA ”Donnerstag, 11. Oktober 12
    • MULTI HOP 12Donnerstag, 11. Oktober 12
    • myagent1.sinks = mysink myagent1.sinks.mysink.type = avro myagent1.sinks.mysink.bind = 10.10.10.20 myagent1.sinks.mysink.port = 4141 myagent2.sources = mysource myagent2.sources.mysource.type = avro myagent2.sources.mysource.bind = 10.10.10.10 myagent2.sources.mysource.port = 4141 13Donnerstag, 11. Oktober 12
    • CONSOLIDATION 14Donnerstag, 11. Oktober 12
    • MULTIPLEXING 15Donnerstag, 11. Oktober 12
    • Sources Sinks Channels Avro Avro Memory Exec Logger JDBC NetCat IRC File Sequence Generator File Syslog HBase Scribe 16Donnerstag, 11. Oktober 12
    • DEMO DEMO DEMO DEMO DEMODonnerstag, 11. Oktober 12
    • Thank you! kai@cloudera.com http://www.cloudera.com/Donnerstag, 11. Oktober 12