The document discusses changes to the ApplicationMaster (AM) component architecture in Hadoop to improve container reuse. It proposes separating container and task operations so that containers can be launched independently of tasks and reused for multiple tasks. This would facilitate features like common map output buffers for tasks in the same container and merging map outputs per node or rack. The current state of the AM changes is described, along with requirements for task side changes and a new reuse scheduler. Potential benefits highlighted include simplifying deployment, monitoring large clusters, integrating with other systems, and reducing costs through high availability and training/support services.

1. MR Container Re-Use
(Not YARN specific)
Siddarth Seth
Member of Technical Staff
© Hortonworks Inc. 2012

2. Current AM Components
Job Running
TA Listener
Task
Task
TaskAttemp Container
t Launcher
RMAllocato
RM r/ NM
Scheduler

3. • TaskAttempt and Container operations are
tightly coupled
– CLC construction, Container Launch
invocation is handled by the TaskAttempt
– Container Launch is tied to the TaskAttempt
(instead of container size, LocalResources)
– Container shutdown.

4. AM post 3902
Container
Job Running JVM
Listener
Running
TA
Listener Task
Task
NM
TaskAttemp
Container Communicat
t
or
Node
RM AMScheduler NM
Rack ?

5. • Container and Node have their own states.
• Containers interact with the NodeManager
• Tasks interact with the scheduler – which
matches containers to task attempts.
• Nodes take care of blacklisting – simplifies
the scheduler.
• Easier to write a custom scheduler.

6. Current State
• Most of the AM functional changes are
done. (Cleanup pending)
• Task side changes are required
• A re-use scheduler needs to be
implemented.

7. Facilitates
• Common MapOutputBuffer for maps
assigned to the same container.
• Merging per-node or per-rack map output
• Custom Task Types

8. Hortonworks Data Platform
• Simplify deployment to get
started quickly and easily
• Monitor, manage any size
cluster with familiar
console and tools
1 • Only platform to include
data integration services
to interact with any data
• Metadata services opens
the platform for integration
with existing applications
• Dependable high
availability architecture
 Reduce risks and cost of adoption
 Lower the total cost to administer and provision • Tested at scale to future
proof your cluster growth
 Integrate with your existing ecosystem
Page 8
© Hortonworks Inc. 2012

9. Hortonworks Training
The expert source for
Apache Hadoop training & certification
Role-based Developer and Administration training
– Coursework built and maintained by the core Apache Hadoop development team.
– The “right” course, with the most extensive and realistic hands-on materials
– Provide an immersive experience into real-world Hadoop scenarios
– Public and Private courses available
Comprehensive Apache Hadoop Certification
– Become a trusted and valuable
Apache Hadoop expert
Page 9
© Hortonworks Inc. 2012

10. Next Steps?
1 Download Hortonworks Data Platform
hortonworks.com/download
2 Use the getting started guide
hortonworks.com/get-started
3 Learn more… get support
Hortonworks Support
• Expert role based training • Full lifecycle technical support
• Course for admins, developers across four service levels
and operators • Delivered by Apache Hadoop
• Certification program Experts/Committers
• Custom onsite options • Forward-compatible
hortonworks.com/training hortonworks.com/support
Page 10
© Hortonworks Inc. 2012

11. Thank You!
Questions & Answers
Follow: @hortonworks
Read: hortonworks.com/blog
Page 11
© Hortonworks Inc. 2012