VIP Call Girls Service Hitech City Hyderabad Call +91-8250192130
Apex as yarn application
1. Apache Apex as
YARN Application
Chinmay Kolhatkar (chinmay@apache.org)
Mar 22, 2016
Apache Apex Meetup
2. Agenda
• Directed Acyclic Graph
• Apex as a YARN Application
• Application Components of Apex
• Lifecycle of Apex as a YARN Application
Apache Apex Meetup
4. DAG Components
• Tuple
● Atomic data that flows over a stream
• Operator
● Basic compute unit per tuple
• Stream
● Connector abstraction between operators
● Tuples flow over this
Operator
1
Operator
2
Apache Apex Meetup
Stream
tuple
3
tuple
1
tuple
2
5. DAG Types
O1 O2
O3
O4
Physical DAG
Apache Apex Meetup
O5
Logical DAG
• Logical Plan
● Logical representation of computation
● Defines operators, streams and dataflow
• Physical Plan
● Deployable plan on cluster
● Contains partition information
of operators
● Has ready-to-deploy serialized operator
instances
O1
P1
O1
P2
O1
P3
O2
P1
O2
P2
O2
P3
U
O3
O4
O5
7. Application Components of Apex - StrAMClient
• Part of dtcli client interface
• Invoked by “launch” command of dtcli
• Tasks:
● Copy required the application package files into HDFS
● Validate Logical Plan
● Serialize Logical plan to HDFS
● Launch Application Master i.e. StrAM
Apache Apex Meetup
8. Application Components of Apex - StrAM
• Streaming Application Master
• Started by StrAMClient on a YarnContainer
• Tasks:
● Convert logical plan to physical plan
● Serialize operators to HDFS
● Request for resources to ResourceManager
● Start StrAMChild in YarnContainer(s)
● Monitor StrAMChild using ContainerManager protocol
● Generate Application statistics
● Host results on WebService (dtManage)
● Fault Tolerance
● Checkpointing/Committing Application States
● Support Security
● Shutdown Application
Apache Apex Meetup
9. Application Components of Apex - StrAMChild
• Deployed on YarnContainer
• Started by NodeManager as instructed by StrAM
• Instance of StreamingContainer
• Contains Operators (compute-related)
• Contains BufferServer (stream-related)
• Tasks:
● Regularly send heartbeat to StrAM
● Execute commands from StrAM
● Shutdown or Kill self if instructed
● Manage lifecycle of an Operator
● Network communication using BufferServer
Apache Apex Meetup
14. Summary – Apex platform
• Enables YARN to be used for Streaming Applications
• Takes care of YARN specific work
• User can focus on business logic defined in Operators
Apache Apex Meetup