Current AM Components Job Running TA Listener Task Task TaskAttemp Container t Launcher RMAllocato RM r/ NM Scheduler
• TaskAttempt and Container operations are tightly coupled – CLC construction, Container Launch invocation is handled by the TaskAttempt – Container Launch is tied to the TaskAttempt (instead of container size, LocalResources) – Container shutdown.
AM post 3902 Container Job Running JVM Listener Running TA Listener Task Task NM TaskAttemp Container Communicat t or Node RM AMScheduler NM Rack ?
• Container and Node have their own states.• Containers interact with the NodeManager• Tasks interact with the scheduler – which matches containers to task attempts.• Nodes take care of blacklisting – simplifies the scheduler.• Easier to write a custom scheduler.
Current State• Most of the AM functional changes are done. (Cleanup pending)• Task side changes are required• A re-use scheduler needs to be implemented.
Facilitates• Common MapOutputBuffer for maps assigned to the same container.• Merging per-node or per-rack map output• Custom Task Types