Session on "Building Distributed Data Streaming System" Presented at - https://www.meetup.com/IMC-Pune-Meetup/events/224577931/

1. Proprietary and Confidential 1
Building Distributed Data
Streaming System
AshishTadose
Lead Software Engineer
Big Data Analytics

2. Agenda
• What is stream processing
• Streaming architecture
• Scalable Data Ingestion
• RealTime Streaming Processing system
2

3. What is Streaming Process ?
3

4. In simple words, Streaming is…
4

5. Batch & Streaming processing
Data
Generator
Ingestion
Distributed
File system
Processing Data Store
Batch processing
Data
Generator
Ingestion
Message
Queue
Processing Data Store
Stream Data processing

6. Batch & Streaming processing
6
Data
Generator
Ingestion
Message
Queue
Processing Data Store
Stream Data processing
Distributed
File system
Processing Data Store
Batch processing

7. Batch & Streaming processing
7
Data
Generator
Ingestion
Message
Queue
Processing Data Store
Stream Data processing
Distributed
File system
Processing Data Store
Batch processing

8. Lambda Architecture:Velocity &Volume
8

9. Streaming
Ingestion
Technologies
9

10. Ingestion Ecosystem
• Sources
• Machine data
• External stream & syslogs
• Data Collection
• Flume
• Kafka
• Kinesis
• Confluent
10

11. Flume
• Easier to setup
• Rich set of in-build tools
• No inherent support for data replication
• Nodes works in isolation
• Memory channel vs File Channel 11

12. Kinesis
12

13. Kafka
13
 http://kafka.apache.org/
 Originated at LinkedIn, open sourced in early 2011
 Implemented in Scala, some Java
 9 core committers, plus ~ 20 contributors

14. Why is Kafka so fast?
• Fast writes:
• While Kafka persists all data to disk, essentially all writes go to the
page cache of OS, i.e. RAM.
• Fast reads:
• Very efficient to transfer data from page cache to a network socket
• Linux: sendfile() system call
• Combination of the two = fast Kafka!
• Example (Operations):On a Kafka cluster where the consumers are mostly
caught up you will see no read activity on the disks as they will be serving
data entirely from cache.
14
1
http://kafka.apache.org/documentation.html#persistence

15. Flafka – Flume meets Kafka
15

16. Confluent - Centralized Ingestion with Kafka Pipeline
16

17. Stream
Processing
17

18. RealTime Stream Processing
• Processing system
• Apache Storm
• Apache Samza
• Apache Spark (Streaming)
• Project Apex - DataTorrent
• Storage
• Hive HDFS
• Hbase
• MySql
• Custom
• Access
• Depend of data storage
• Scalable query interface - Kafka 18

19. Streaming Design Patterns
• Micro batching
• Unpredictable incoming data
• Creating multiple streams
• Out of sequence events
• Stream joins
• Top N metrics
• External Lookup
19

20. ThankYou
20