Apache Flume is a data ingestion system that transports streaming data from sources like web servers and Kafka to centralized data stores like HDFS. An event is the basic unit of data transported by Flume and contains a payload and optional headers. A Flume agent receives data from sources or other agents and forwards it to sinks or other agents. Flume uses sources, channels, and sinks to reliably move data between distributed systems with capabilities like load balancing and failover to prevent data loss.

1. FKafka
360 degree overview
Tanuj Mehta

2. What is Flume?
Apache Flume is a data ingestion mechanism for transporting large amounts of
streaming data such as log data, events etc. from various sources like web servers,
Kafka etc. to a centralized data store like HDFS
Data
Generator
Centralized
storage
HDFS
HBase
FlumeLog/Event data Log/Event data

3. Flume Event
An event is the basic unit of the data transported inside Flume. It contains a
payload of byte array that is to be transported from the source to the destination
accompanied by optional headers
Event
Header Byte Payload

4. Flume Agent
An agent is an independent daemon process (JVM) in Flume. It receives the data
(events) from clients or other agents and forwards it to its next destination (sink or
agent).
Source channel Sink

5. Flume Components
Source Interceptor Selector Channel Sink

6. Multi-hop Flow
Source 1
Source 2
Source 3
Channel 1
Channel 2 Centralized
storage
Channel 3
Fan-in
Fan-out

7. Sink Processor
● Default
● Load Balancing (Multi-Threading)
● Failover

8. Kafka-Flume Integration
Kafka Broker
Topic 1
Topic 2
Flume Agent
Channel Sink
Kafka Channel 1
HBase
SOLR
Kafka Channel 2
HDFS

9. No Data
Loss

10. Data
DUPLICATION

11. Benefits
● Ability to store data to any centralized stores (HDFS, HBase)
● Contextual routing
● Effectively handle peak hour loads
● Allow read and write to operate at different rate
● Reliable, fault tolerant, scalable, manageable, and customizable

12. Q&A