This document discusses the Apache Apex stream processing platform. It provides an overview of Apex's architecture, including its native integration with Hadoop YARN and HDFS, its application programming model based on operators and streams, and its support for advanced features like windowing, partitioning, dynamic scaling, fault tolerance, and data processing guarantees. It also shows examples of monitoring dashboards and describes how Apex can be used to build real-time data analytics pipelines.

1. Pramod Immaneni <pramod@datatorrent.com>
PPMC Member, Architect @DataTorrent Inc
Dec 2nd, 2015
Stream Processing Architecture and Applications
Apache Apex (incubating)

2. © 2015 DataTorrent
Apex Platform Overview
2

3. © 2015 DataTorrent
Apache Malhar Library
3

4. © 2015 DataTorrent
Native Hadoop Integration
4
• YARN is
the
resource
manager
• HDFS used
for storing
any
persistent
state

5. © 2015 DataTorrent
Application Programming Model
5
 A Stream is a sequence of data tuples
 An Operator takes one or more input streams, performs computations & emits one or more output streams
• Each Operator is YOUR custom business logic in java, or built-in operator from our open source library
• Operator has many instances that run in parallel and each instance is single-threaded
 Directed Acyclic Graph (DAG) is made up of operators and streams
Directed Acyclic Graph (DAG)
Output StreamTuple Tuple
er
Operator
er
Operator
er
Operator
er
Operator

6. © 2015 DataTorrent
Advanced Windowing Support
6
 Application window
 Sliding window and tumbling window
 Checkpoint window
 No artificial latency

7. © 2015 DataTorrent
Application Specification
7

8. © 2015 DataTorrent
Partitioning and unification
8

9. © 2015 DataTorrent
Advanced Partitioning
9

10. © 2015 DataTorrent
Dynamic Partitioning
10
• Partitioning change while application is running
ᵒ Change number of partitions at runtime based on stats
ᵒ Determine initial number of partitions dynamically
• Kafka operators scale according to number of kafka partitions
ᵒ Supports re-distribution of state when number of partitions change
ᵒ API for custom scaler or partitioner
unifiers not shown
1a 2a
1b 2b
3
2b
1b 2c
3
2a
2d
1a 2b
1b 2c 3b
2a
2d
3a1a

11. © 2015 DataTorrent
How tuples are partitioned
11
• Tuple hashcode and mask used to determine destination partition
ᵒ Mask picks the last n bits of the hashcode of the tuple
ᵒ hashcode method can be overridden
• StreamCodec can be used to specify custom hashcode for tuples
ᵒ Can also be used for specifying custom serialization
tuple: {
Name,
24204842,
San Jose
}
Hashcode:
00101010001
0101
Mask
(0x11)
Partition
00 1
01 2
10 3
11 4

12. © 2015 DataTorrent
Custom partitioning
12
• Custom distribution of tuples
ᵒ E.g.. Broadcast
tuple:{
Name,
24204842,
San Jose
}
Hashcode:
00101010001
0101
Mask
(0x00)
Partition
00 1
00 2
00 3
00 4

13. © 2015 DataTorrent
Fault Tolerance
13
• Operator state is checkpointed to a persistent store
ᵒ Automatically performed by engine, no additional work needed by operator
ᵒ In case of failure operators are restarted from checkpoint state
ᵒ Frequency configurable per operator
ᵒ Asynchronous and distributed by default
ᵒ Default store is HDFS
• Automatic detection and recovery of failed operators
ᵒ Heartbeat mechanism
• Buffering mechanism to ensure replay of data from recovered point so
that there is no loss of data
• Application master state checkpointed

14. © 2015 DataTorrent
Processing Guarantees
14
Atleast once
• On recovery data will be replayed from a previous checkpoint
ᵒ Messages will not be lost
ᵒ Default mechanism and is suitable for most applications
• Can be used in conjunction with following to ensure data is written
once to store in case of fault recovery
ᵒ Transactions with meta information, Rewinding output, Feedback from
external entity, Idempotent operations
Atmost once
• On recovery the latest data is made available to operator
ᵒ Useful in use cases where some data loss is acceptable and latest data is
sufficient
Exactly once
• Operators checkpointed every window
ᵒ Can be combined with transactional mechanisms to ensure end-to-end
exactly once behavior

15. © 2015 DataTorrent
Stream Locality
15
• By default operators are deployed in containers (processes) randomly
on different nodes across the Hadoop cluster
• Custom locality for streams
ᵒ Rack local: Data does not traverse network switches
ᵒ Node local: Data is passed via loopback interface and frees up network
bandwidth
ᵒ Container local: Messages are passed via in memory queues between
operators and does not require serialization
ᵒ Thread local: Messages are passed between operators in a same thread
equivalent to calling a subsequent function on the message

16. © 2015 DataTorrent
Data Processing Pipeline Example
App Builder
16

17. © 2015 DataTorrent
Monitoring Console
Logical View
17

18. © 2015 DataTorrent
Monitoring Console
Physical View
18

19. © 2015 DataTorrent
Real-Time Dashboards
Real Time Visualization
19

20. © 2015 DataTorrent
Resources
20
Apache Apex Community Page - http://apex.incubator.apache.org/

21. © 2015 DataTorrent
We Need Your Vote (Today)
21
Introducing Apache Apex - Not Just Another Stream Processing Platform
Next Gen Big Data Analytics with Apache Apex
Enterprise-grade streaming under 2ms on Hadoop

22. End
22

23. © 2015 DataTorrent
Extra Slides

24. © 2015 DataTorrent
Partitioning and Scaling Out
24
• Operators can be dynamically
scaled
• Flexible Streams split
• Parallel partitioning
• MxN partitioning
• Unifiers

25. © 2015 DataTorrent
Fault Tolerance Overview
25
Stateful Fault Tolerance Processing Semantics Data Locality
 Supported out of the box
– Application state
– Application master state
– No data loss
 Automatic recovery
 Lunch test
 Buffer server
 At least once
 At most once
 Exactly once
 Stream locality for placement of
operators
 Rack local – Distributed
deployment
 Node local – Data does
not traverse NIC
 Container local – Data
doesn’t need to be
serialized
 Thread local – Operators
run in same thread
 Data locality

26. © 2015 DataTorrent
Machine Data Application
Logical View
26

27. © 2015 DataTorrent
Machine Data Application
Physical View
27