- Apache Apex is a platform and framework for building highly scalable and fault-tolerant distributed applications on Hadoop. - It allows developers to build any custom logic as distributed applications and ensures fault tolerance, scalability and data flow. Applications can process streaming or batch data with high throughput and low latency. - Apex applications are composed of operators that perform processing on streams of data tuples. Operators can run in a distributed fashion across a cluster and automatically recover from failures without reprocessing data from the beginning.

1. Pramod Immaneni <pramod@datatorrent.com>
PPMC Member, Senior Architect @DataTorrent Inc
Mar 2nd, 2016
Stream Processing with Apache Apex
Apache Apex (incubating)

2. © 2015 DataTorrent
What is Apex
2
• Platform and framework to build highly scalable and fault-
tolerant distributed applications
• 100% Hadoop native
• Build any custom logic in your application
• Unobtrusive API to facilitate distributed application development
• Runtime engine to ensure fault tolerance, scalability and
data flow
• Process streaming or batch big data
• High throughput and low latency
• Realtime applications

3. © 2015 DataTorrent
Applications on Apex
3
• Distributed processing
• Application logic broken into operators that run in a distributed fashion across your cluster
• Natural programming model
• Code as if you were writing regular Java logic
• Maintain state in your application variables
• Scalable
• Operators can be scaled up or down at runtime according to the load and SLA
• Fault tolerant
• Automatically recover from node outages without having to reprocess from beginning
• State is preserved
• Long running applications
• Operational insight – DataTorrent RTS
• See how each operator is performing and even record data

4. © 2015 DataTorrent
Apex Platform Overview
4

5. © 2015 DataTorrent
Apache Malhar Library
5

6. © 2015 DataTorrent
Native Hadoop Integration
6
• YARN is
the
resource
manager
• HDFS used
for storing
any
persistent
state

7. © 2015 DataTorrent
Application Development Model
7
 A Stream is a sequence of data tuples
 A typical Operator takes one or more input streams, performs computations & emits one or more output streams
• Each Operator is YOUR custom business logic in java, or built-in operator from our open source library
• Operator has many instances that run in parallel and each instance is single-threaded
 Directed Acyclic Graph (DAG) is made up of operators and streams
Directed Acyclic Graph (DAG)
Output
Stream
Tupl
e
Tupl
e
er
Operator
er
Operator
er
Operator
er
Operator
er
Operator
er
Operator

8. © 2015 DataTorrent
Advanced Windowing Support
8
 Application window
 Sliding window and tumbling window
 Checkpoint window
 No artificial latency

9. © 2015 DataTorrent
Application in Java
9

10. © 2015 DataTorrent
Operators
10

11. © 2015 DataTorrent
Operators (contd)
11

12. © 2015 DataTorrent
Partitioning and unification
12

13. © 2015 DataTorrent
Advanced Partitioning
13

14. © 2015 DataTorrent
Dynamic Partitioning
14
• Partitioning change while application is running
ᵒ Change number of partitions at runtime based on stats
ᵒ Determine initial number of partitions dynamically
• Kafka operators scale according to number of kafka partitions
ᵒ Supports re-distribution of state when number of partitions change
ᵒ API for custom scaler or partitioner
unifiers not shown
1a 2a
1b 2b
3
2b
1b 2c
3
2a
2d
1a 2b
1b 2c 3b
2a
2d
3a1a

15. © 2015 DataTorrent
How tuples are partitioned
15
• Tuple hashcode and mask used to determine destination partition
ᵒ Mask picks the last n bits of the hashcode of the tuple
ᵒ hashcode method can be overridden
• StreamCodec can be used to specify custom hashcode for tuples
ᵒ Can also be used for specifying custom serialization
tuple: {
Name,
24204842,
San Jose
}
Hashcode:
00101010001
0101
Mask
(0x11)
Partition
00 1
01 2
10 3
11 4

16. © 2015 DataTorrent
Custom partitioning
16
• Custom distribution of tuples
ᵒ E.g.. Broadcast
tuple:{
Name,
24204842,
San Jose
}
Hashcode:
00101010001
0101
Mask
(0x00)
Partition
00 1
00 2
00 3
00 4

17. © 2015 DataTorrent
Fault Tolerance
17
• Operator state is checkpointed to a persistent store
ᵒ Automatically performed by engine, no additional work needed by operator
ᵒ In case of failure operators are restarted from checkpoint state
ᵒ Frequency configurable per operator
ᵒ Asynchronous and distributed by default
ᵒ Default store is HDFS
• Automatic detection and recovery of failed operators
ᵒ Heartbeat mechanism
• Buffering mechanism to ensure replay of data from recovered point so
that there is no loss of data
• Application master state checkpointed

18. © 2015 DataTorrent
Processing Guarantees
18
Atleast once
• On recovery data will be replayed from a previous checkpoint
ᵒ Messages will not be lost
ᵒ Default mechanism and is suitable for most applications
• Can be used in conjunction with following mechanisms to achieve
exactly-once behavior in fault recovery scenarios
ᵒ Transactions with meta information, Rewinding output, Feedback from
external entity, Idempotent operations
Atmost once
• On recovery the latest data is made available to operator
ᵒ Useful in use cases where some data loss is acceptable and latest data is
sufficient
Windowed Exactly once
• Operators checkpointed every window
ᵒ Can be combined with transactional mechanisms to ensure end-to-end
exactly once behavior

19. © 2015 DataTorrent
Stream Locality
19
• By default operators are deployed in containers (processes) randomly
on different nodes across the Hadoop cluster
• Custom locality for streams
ᵒ Rack local: Data does not traverse network switches
ᵒ Node local: Data is passed via loopback interface and frees up network
bandwidth
ᵒ Container local: Messages are passed via in memory queues between
operators and does not require serialization
ᵒ Thread local: Messages are passed between operators in a same thread
equivalent to calling a subsequent function on the message

20. © 2015 DataTorrent
Data Processing Pipeline Example
App Builder
20

21. © 2015 DataTorrent
Monitoring Console
Logical View
21

22. © 2015 DataTorrent
Monitoring Console
Physical View
22

23. © 2015 DataTorrent
Real-Time Dashboards
Real Time Visualization
23

24. © 2015 DataTorrent
Resources
24
Apache Apex Community Page - http://apex.incubator.apache.org/

25. End
25

26. © 2015 DataTorrent
Extra Slides

27. © 2015 DataTorrent
Application Programming Model
27
 A Stream is a sequence of data tuples
 An Operator takes one or more input streams, performs computations & emits one or more output streams
• Each Operator is YOUR custom business logic in java, or built-in operator from our open source library
• Operator has many instances that run in parallel and each instance is single-threaded
 Directed Acyclic Graph (DAG) is made up of operators and streams
Directed Acyclic Graph (DAG)
Output StreamTuple Tuple
er
Operator
er
Operator
er
Operator
er
Operator

28. © 2015 DataTorrent
Partitioning and Scaling Out
28
• Operators can be dynamically
scaled
• Flexible Streams split
• Parallel partitioning
• MxN partitioning
• Unifiers

29. © 2015 DataTorrent
Fault Tolerance Overview
29
Stateful Fault Tolerance Processing Semantics Data Locality
 Supported out of the box
– Application state
– Application master state
– No data loss
 Automatic recovery
 Lunch test
 Buffer server
 At least once
 At most once
 Exactly once
 Stream locality for placement of
operators
 Rack local – Distributed
deployment
 Node local – Data does
not traverse NIC
 Container local – Data
doesn’t need to be
serialized
 Thread local – Operators
run in same thread
 Data locality

30. © 2015 DataTorrent
Machine Data Application
Logical View
30

31. © 2015 DataTorrent
Machine Data Application
Physical View
31