An overview of Apache Heron, the open source streaming processing engine developed at Twitter and a core part of the Streamlio solution.

1. Introduction to Heron
Karthik Ramasamy

2. What is Heron
• Streaming data processing engine
• Developed at Twitter to replace Storm
• Contributed to open source in 2016
2

3. Heron Design Goals
3
Efficiency
Reduce resource
consumption
Support for diverse
workloads
Throughput vs latency
sensitive
Support for multiple
semantics
At most once, At least
once, Effectively once
Native Multi-Language
Support
C++, Java, Python
Task Isolation
Ease of debug-ability/
isolation/profiling
Support for back
pressure
Topologies should be self
adjusting
Use of containers
Runs in schedulers -
Kubernetes & DCOS & many
more
Multi-level APIs
Procedural, Functional and
Declarative for diverse
applications
Diverse deployment models
Run as a service or pure library

4. Heron Design Highlights
4
High Throughput and Low
Latency
Performance sensitive
components written in C++
d
Isolation at all Levels
Ease of debugging/resource isolation/
profilingÑ
Extensible Stream Engine
Pluggable higher level API
Initial support for Storm API,
Apache Beam, etc
Multiple Processing Semantics
At most once, At least once and
Exactly once
ddd

5. Heron Terminology
5
Topology
Directed acyclic graph
vertices = computation, and
edges = streams of data tuples
Spouts
Sources of data tuples for the topology
Examples - Pulsar/Kafka/MySQL/Postgres
Bolts
Process incoming tuples, and emit
outgoing tuples
Examples - filtering/aggregation/join/
any function
,
%

6. Heron Architecture
6
Scheduler
Topology 1 Topology 2 Topology N
Topology
Submission

7. Heron Topology Ilustration
7
%
%
%
%
%
Spout 1
Spout 2
Bolt 1
Bolt 2
Bolt 3
Bolt 4
Bolt 5

8. Heron Topology - Physical Execution
8
%
%
%
%
%
Spout 1
Spout 2
Bolt 1
Bolt 2
Bolt 3
Bolt 4
Bolt 5
%%
%%
%%
%%
%%

9. Execution Architecture
9
Topology Master
ZK
Cluster
Stream
Manager
I1 I2 I3 I4
Stream
Manager
I1 I2 I3 I4
Logical Plan,
Physical Plan and
Execution State
Sync Physical Plan
DATA CONTAINER DATA CONTAINER
Metrics
Manager
Metrics
Manager
Metrics
Manager
Health
Manager
MASTER
CONTAINER

10. Topology Master
10
Monitors containers Gateway for metrics Assigns role

11. Stream Manager
11
Routes tuples Implements backpressure ACK management

12. Heron Instance
12
Runs only one task (spout/bolt)
Exposes Heron API
Collects metrics
API
G

13. Heron Instance
13
Stream
Manager
Metrics
Manager
Gateway
Thread
Task Execution
Thread
data-in queue
data-out queue
metrics-out queue

14. Heron Groupings
14
01 02 03 04
Shuffle Grouping
Random distribution of tuples
Fields Grouping
Group tuples by a field or
multiple fields
All Grouping
Replicates tuples to all tasks
Global Grouping
Send the entire stream to one
task
/
.
-
,

15. Writing Heron Topologies
15
Procedural - Low Level API
Directly write your spouts
and bolts
Functional - Mid Level API
Use of maps, flat maps, transform,
windows
Declarative - SQL (coming)
Use of declarative language - specify
what you want, system will figure it
out.
,
%

16. Heron Use Cases
16
Monitoring
Real Time
Machine
Learning
Ads
Real Time Trends
Product Safety
Real Time
Business
Intelligence

17. Example Heron customers
17

18. Heron in Production @ Twitter
• Completely replaced Storm 3 years ago
• 3x reduction in cores and memory
• Significantly reduced operational overhead
• >10x reduction in production incidents
18