Apache Flume
●

What is it ?

●

How does it work ?

●

Architecture

●

Reliability

www.semtech-solutions.co.nz

info@se...
Flume – What is it ?
●

A data collection service for Hadoop

●

For distributed systems

●

Open source

●

Scaleable

●
...
Flume – How does it work ?
●

Flumes uses agents which have
–

A source

–

Listen for events
● Write events to channel
A ...
Flume – Architecture
●

A single agent showing its parts

●

Generally one agent for a given data type

www.semtech-soluti...
Flume – Architecture
●

Agents can be chained into flows

●

Avro can be used for data serialization

www.semtech-solution...
Flume – Architecture
In complicated flows it may be necessary to think about
●

Event Data Reliability

●

Should we have
...
Flume – Architecture
●

Complex flows may have many links

www.semtech-solutions.co.nz

info@semtech-solutions.co.nz
Contact Us
●

Feel free to contact us at
–

www.semtech-solutions.co.nz

–

info@semtech-solutions.co.nz

●

We offer IT p...
Upcoming SlideShare
Loading in …5
×

Apacheflume 130720235611-phpapp02

343 views
312 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
343
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
11
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Apacheflume 130720235611-phpapp02

  1. 1. Apache Flume ● What is it ? ● How does it work ? ● Architecture ● Reliability www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  2. 2. Flume – What is it ? ● A data collection service for Hadoop ● For distributed systems ● Open source ● Scaleable ● Reliable ● Manageable ● Fault tolerant www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  3. 3. Flume – How does it work ? ● Flumes uses agents which have – A source – Listen for events ● Write events to channel A channel – Queue event data as transactions A sink ● ● ● ● Write event data to target i.e. HDFS Remove event from queue www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  4. 4. Flume – Architecture ● A single agent showing its parts ● Generally one agent for a given data type www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  5. 5. Flume – Architecture ● Agents can be chained into flows ● Avro can be used for data serialization www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  6. 6. Flume – Architecture In complicated flows it may be necessary to think about ● Event Data Reliability ● Should we have – Complete end to end reliability – Send and forget – Or something in between ? www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  7. 7. Flume – Architecture ● Complex flows may have many links www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  8. 8. Contact Us ● Feel free to contact us at – www.semtech-solutions.co.nz – info@semtech-solutions.co.nz ● We offer IT project consultancy ● We are happy to hear about your problems ● You can just pay for those hours that you need ● To solve your problems

×