1. Apache Storm Concepts
André Dias
andre.dias@summa.com.br

2. Highlights
● Background
● Storm’s History
● Concepts
● Integrations
● Real Cases
● Q&A

3. Background

4. Background
Big Data V’s

5. Background
Big Data V’s
Volume
Petabytes / Terabytes
Velocity
Real-time / Near Real-time
Variety
Sensors, Blog Posts, Logs, Social Networks...

6. Background
Data Streaming

7. Background
Data Streaming

8. Background
Data Streaming
Now Please… I’
m in traffic…

9. ● Discovery
● Ingest
● Process
● Persist
● Analyze
● Expose
Background
Data Value Chain

10. ● Discovery
● Ingest
● Process
● Persist
● Analyze
● Expose
Background
Data Value Chain (Storm Focus)

11. Background
Data Value Chain
● Discovery
● Ingest
● Process
● Persist
● Analyze
● Expose

12. Background
Data Value Chain (Ingest)
Stream Processing
Data in Motion
Batch Processing

13. ● Data processing architecture
○ Generic
○ Scalable
○ Fault-tolerant (human/hardware)
● Low latency
Background
Lambda Architecture (LA)

14. Background
Lambda Architecture (LA)

15. Storm’s History
Nathan Marz
● Backtype (2008), acquired by Twitter (2011)
● Lambda Architecture Creator
● September, 2011. Storm Creation
● September, 2013. Storm entered to ASF

16. Why Use It?
● Scalable, as Hadoop
● No data loss, reliable
● Fault tolerant
● Language agnostic
● Real-time, real fast

17. Why Use It?
Storm is TLP !!

18. Concepts

19. Concepts
● Tuple
● Streams
● Spouts
● Bolts
● Topologies / Trident API
● Stream Groupings

20. Concepts
Spouts
● Source of Streams
● Data Consumers (Ingestion)
● Emits Tuples

21. Concepts
Bolts
● Units of Work to tuples
● Data streaming logic
● Can emit tuples as well
● Data store integration

22. Concepts
Topology
● Data Streaming Flow Representation
● DAG (Direct Acyclic Graph) of Spouts and
Bolts
● Streaming computation
● Each node as individual task (parallel
execution)
● Stateless

23. Concepts
Trident API
● Abstraction Layer over low-level Storm API
● More Complex Topologies
● Stateful
● Micro-batch
● High-level API (similar to Pig / Cascading -
Hadoop)
● Message processed at least once
(guaranteed)

24. Concepts
Trident API - Micro Batch
● Trident Batches
○ are Ordered

25. Concepts
Trident API - Micro Batch
● Trident Batches
○ can be Partitioned

26. Concepts
Streaming Groups
● Data Flow Control over
Topologies

27. Architecture

28. Architecture
Components - Nimbus
● Master node (similar to JobTracker)
● Monitor and distribute the processing
workload across worker nodes
● Stores all its data into Zookeeper

29. Architecture
Components - Supervisor
● Worker node (similar to TaskTracker)
● Monitor and distribute the processing
workload across worker nodes
● Stores all its data into Zookeeper

30. Architecture
Overview
● Master-slave approach
● Cluster coordination
(Zookeeper)
● Nimbus HA

31. Integrations

32. Real Cases
Collector sensor
information to a
Data Lake
Micro-batch
user contents,
content feeds
and application
logs
Real-time user
music
recommendations

33. Q&A

34. THANK YOU!
Use Storm