Speaker - Chinmay Kolhatkar, Software Engineer at DataTorrent

1. Apache Apex as
YARN Application
Chinmay Kolhatkar (chinmay@apache.org)
Mar 22, 2016
Apache Apex Meetup

2. Agenda
• Directed Acyclic Graph
• Apex as a YARN Application
• Application Components of Apex
• Lifecycle of Apex as a YARN Application
Apache Apex Meetup

3. Directed Acyclic Graph (DAG)
• Defines compute stages of streaming application
• Defines tuple flow across Operators via Stream
Compute
1
Apache Apex Meetup
Compute
3
Compute
2
Compute
4

4. DAG Components
• Tuple
● Atomic data that flows over a stream
• Operator
● Basic compute unit per tuple
• Stream
● Connector abstraction between operators
● Tuples flow over this
Operator
1
Operator
2
Apache Apex Meetup
Stream
tuple
3
tuple
1
tuple
2

5. DAG Types
O1 O2
O3
O4
Physical DAG
Apache Apex Meetup
O5
Logical DAG
• Logical Plan
● Logical representation of computation
● Defines operators, streams and dataflow
• Physical Plan
● Deployable plan on cluster
● Contains partition information
of operators
● Has ready-to-deploy serialized operator
instances
O1
P1
O1
P2
O1
P3
O2
P1
O2
P2
O2
P3
U
O3
O4
O5

6. Apex as YARN application
Node
ResourceManager
(AsM + Scheduler)
NM Node NM Node NM
YarnClient
AppMaster
YarnContainer
YarnContainer
YarnContainer
StrAM
(AppMaster)
YarnContainer
StrAMChild
O1 O2
YarnContainer
StrAMChild
O3
DTCLI
StrAMClient
YarnClient
Apache Apex Meetup
ClientRM
Protocol
AMRM
Protocol
ContainerManager
Protocol
ContainerManager
Protocol
ClientRM
Protocol
AMRM
Protocol
ContainerManager
Protocol

7. Application Components of Apex - StrAMClient
• Part of dtcli client interface
• Invoked by “launch” command of dtcli
• Tasks:
● Copy required the application package files into HDFS
● Validate Logical Plan
● Serialize Logical plan to HDFS
● Launch Application Master i.e. StrAM
Apache Apex Meetup

8. Application Components of Apex - StrAM
• Streaming Application Master
• Started by StrAMClient on a YarnContainer
• Tasks:
● Convert logical plan to physical plan
● Serialize operators to HDFS
● Request for resources to ResourceManager
● Start StrAMChild in YarnContainer(s)
● Monitor StrAMChild using ContainerManager protocol
● Generate Application statistics
● Host results on WebService (dtManage)
● Fault Tolerance
● Checkpointing/Committing Application States
● Support Security
● Shutdown Application
Apache Apex Meetup

9. Application Components of Apex - StrAMChild
• Deployed on YarnContainer
• Started by NodeManager as instructed by StrAM
• Instance of StreamingContainer
• Contains Operators (compute-related)
• Contains BufferServer (stream-related)
• Tasks:
● Regularly send heartbeat to StrAM
● Execute commands from StrAM
● Shutdown or Kill self if instructed
● Manage lifecycle of an Operator
● Network communication using BufferServer
Apache Apex Meetup

10. Lifecycle of Apex/YARN Application - Start
Node
ResourceManager
(AsM + Scheduler)
NM Node NM Node NM
DTCLI/
StrAMClient
(YarnClient)
1) Access cluster information
HDFS
3) Submit Application to RM
StrAM
(AppMaster)
4) StrAM Registers with RM
5) StrAM sends heartbeats regularly
6) StrAM request containers with specifications
7) StrAMChild reads
serialized operator
from HDFS
8) StrAMChild starts
operator lifecycle
Apache Apex Meetup
2) Copies files from HDFS
ClientRMProtocol
AMRMProtocol
YarnContainer
StrAMChild
O2
O1
YarnContainer
StrAMChild
O3
YarnContainer
StrAMChild
O4ContainerManager
Protocol
ContainerManager
Protocol

11. Lifecycle of Apex/YARN Application - Running
Node
ResourceManager
(AsM + Scheduler)
NM Node NM Node NM
DTCLI/
StrAMClient
(YarnClient)
HDFS
StrAM
(AppMaster)
Apache Apex Meetup
ClientRMProtocol
AMRMProtocol
YarnContainer
StrAMChild
O2
O1
YarnContainer
StrAMChild
O3
YarnContainer
StrAMChild
O4ContainerManager
Protocol
ContainerManager
Protocol
1) StrAMChild sends
heartbeats
2) StrAMChild sends operator
data
3) StrAM send regular
heartbeats to RM
4) Query status of application

12. Lifecycle of Apex/YARN Application - Shutdown
Node
ResourceManager
(AsM + Scheduler)
NM Node NM Node NM
DTCLI/
StrAMClient
(YarnClient)
HDFS
StrAM
(AppMaster)
Apache Apex Meetup
ClientRMProtocol
AMRMProtocol
YarnContainer
StrAMChild
O2
O1
YarnContainer
StrAMChild
O3
YarnContainer
StrAMChild
O4ContainerManager
Protocol
ContainerManager
Protocol
1) Connect on WebService
REST API
3) Send shutdown signal to
StrAMChild
4) StrAMChild finishes
operator lifecycle
5) Check if all containers are freed
6) StrAM unregisters itself
7) StrAM exits
8) Check if application has
shutdown
2) Send command to StrAM

13. Lifecycle of Apex/YARN Application - Kill
Node
ResourceManager
(AsM + Scheduler)
NM Node NM Node NM
DTCLI/
StrAMClient
(YarnClient)
HDFS
StrAM
(AppMaster)
Apache Apex Meetup
ClientRMProtocol
AMRMProtocol
YarnContainer
StrAMChild
O2
O1
YarnContainer
StrAMChild
O3
YarnContainer
StrAMChild
O4ContainerManager
Protocol
ContainerManager
Protocol
1) Send kill-app command to YARN
2) RM kills all containers

14. Summary – Apex platform
• Enables YARN to be used for Streaming Applications
• Takes care of YARN specific work
• User can focus on business logic defined in Operators
Apache Apex Meetup

15. 15
Apache Apex Meetup

16. Resources
Apache Apex Meetup
• Apache Apex website - http://apex.incubator.apache.org/
• Subscribe - http://apex.incubator.apache.org/community.html
• Download - http://apex.incubator.apache.org/downloads.html
• Twitter - @ApacheApex; Follow - https://twitter.com/apacheapex
• Facebook - https://www.facebook.com/ApacheApex/
• Meetup - http://www.meetup.com/topics/apache-apex
• Startup Program – Free Enterprise License for startups, Universities, Non-Profits

17. Upcoming events...
Apache Apex Meetup
• March 24th 9am PST - Fault Tolerance and Processing Semantics with Apache
Apex
• March 28th 6pm PST - Low-latency ingestion and analytics with Apache Kafka
and Apache Apex (Hadoop)
• ...