The document discusses Hadoop MapReduce and YARN. It provides an overview of MapReduce concepts like parallel processing and data locality. An example of vote counting is used to illustrate the MapReduce approach. Key components of MapReduce like Map, Reduce, and YARN are explained. The YARN application workflow is described through 8 steps from client submission to application completion. Hands-on MapReduce programming and learning resources are also mentioned.

1. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING

2. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Agenda for today’s Session
1. What is Hadoop MapReduce?
2. MapReduce In Nutshell
3. Advantages of MapReduce
4. Hadoop MapReduce Approach with an Example
5. Hadoop MapReduce/YARN Components
6. YARN With MapReduce
7. Yarn Application Workflow
8. MapReduce Program with Hands On

3. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Hadoop Components
2 main Hadoop Components
Storage Processing

4. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
MapReduce: Data Processing Using Programming
Big Data
Result
 Hadoop MapReduce is the processing component of Apache
Hadoop
 It processes data parallelly in distributed environment

5. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
MapReduce In Nutshell
MapReduce
FeaturesLarge Scale
Distributed Model
Used in
Function
Design Pattern
Parallel
Programming
A Program Model
Classification
Analytics
Recommendation
Index and Search
Map
Reduce
Classification
Eg: Top N records
Analytics
Eg: Join, Selection
Recommendation
Eg: Sort
Summarization
Eg: Inverted Index
Implemented
Google
Apache Hadoop
HDFS
Pig
Hive
HBase
For

6. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
2 Biggest Advantages of MapReduce

7. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
 Data is processed in parallel
 Processing becomes fast
Advantage 1: Parallel Processing
Slave A
Slave B
Slave C Slave D
Slave E
Master
Data 

8. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
 Moving Data to processing is very costly
 In MapReduce, we move processing to
Data
Advantage 2: Data Locality - Processing to Storage
Slave A
Slave B
Slave C Slave D
Slave E
Data 
Master

9. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Traditional vs MapReduce Way

10. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Election Votes Counting
Election Votes Casting
 Votes is stored at different Booths
 Result Centre has the details of all the Booths
Data 
Booth A
Booth B
Booth C Booth D
Booth E
Result
Centre

11. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Election Votes Counting – Traditional Way
Counting – Traditional Approach
 Votes are moved to Result Centre for
counting
 Moving all the votes to Centre is costly
 Result Centre is over-burdened
 Counting takes time
Data 
Booth A
Booth B
Booth C Booth D
Booth E
Result
Centre

12. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Result
Centre
Data 
Hadoop MapReduce To the Rescue!
Hadoop MapReduce Doesn’t
Follow This Approach
Booth A
Booth B
Booth C Booth D
Booth E

13. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Election Votes Counting – MapReduce Way
Booth A
Booth B
Booth C Booth D
Booth E
Result
Centre
Counting – MapReduce Approach
 Votes are counted at individual booths
 Booth-wise results are sent back to the result
centre
 Final Result is declared easily and quickly using
this way
Votes

14. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
MapReduce In Detail

15. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
MapReduce Way

16. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Anatomy of a MapReduce Program
MapReduce
Map:
Reduce:
(K1, V1) List (K2, V2)
(K2, list (V2)) List (K3, V3)
Key Value

17. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Let us take an example to understand
MapReduce Way

18. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
MapReduce Way – Word Count Process

19. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Executing a MapReduce Program

20. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
MapReduce Using Yarn

21. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
YARN – Moving beyond MapReduce
BATCH
(MapReduce)
INTERACTIVE
(Text)
ONLINE
(HBase)
STREAMING
(Storm, S4, …)
GRAPH
(Giraph)
IN-MEMORY
(Spark)
HPC MPI
(OpenMPI)
OTHER
(Search)
(Weave..)

22. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Hadoop 2.x Daemons

23. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Hadoop 2.x MapReduce Yarn Components
 ApplicationMaster
» One per application
» Short life
» Coordinates and Manages MapReduce Jobs
» Negotiates with Resource Manager to
schedule tasks
» The tasks are started by NodeManager(s)
 Job History Server
» Maintains information about submitted
MapReduce jobs after their ApplicationMaster
terminates
 Client
» Submits a MapReduce Job
 Resource Manager
» Cluster Level resource manager
» Long Life, High Quality Hardware
 Node Manager
» One per Data Node
» Monitors resources on Data Node
 Container
» Created by NM when requested
» Allocates certain amount of resources
(memory, CPU etc.) on a slave node

24. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
YARN Application Workflow in MapReduce

25. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
YARN Workflow
Node Manager
Node Manager
Node Manager
Node Manager
Node Manager
Node Manager
Node Manager
Node Manager
Node Manager
Node Manager
Node Manager
Node Manager
Container 1.2
Container 1.1
Container 2.1
Container 2.2
Container 2.3
App
Master 2
App
Master 1
Scheduler
Applications
Manager (AsM)
Resource
Manager

26. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Application Workflow
Execution Sequence :
1. Client submits an application Client RM NM AM
1

27. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Application Workflow
Execution Sequence :
1. Client submits an application
2. RM allocates a container to start AM
Client RM NM AM
1
2

28. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Application Workflow
Execution Sequence :
1. Client submits an application
2. RM allocates a container to start AM
3. AM registers with RM
Client RM NM AM
1
2
3

29. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Application Workflow
Execution Sequence :
1. Client submits an application
2. RM allocates a container to start AM
3. AM registers with RM
4. AM asks containers from RM
Client RM NM AM
1
2
3
4

30. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Application Workflow
Execution Sequence :
1. Client submits an application
2. RM allocates a container to start AM
3. AM registers with RM
4. AM asks containers from RM
5. AM notifies NM to launch containers
Client RM NM AM
1
2
3
4
5

31. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Application Workflow
Execution Sequence :
1. Client submits an application
2. RM allocates a container to start AM
3. AM registers with RM
4. AM asks containers from RM
5. AM notifies NM to launch containers
6. Application code is executed in container
Client RM NM AM
1
2
3
4
5
6

32. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Application Workflow
Execution Sequence :
1. Client submits an application
2. RM allocates a container to start AM
3. AM registers with RM
4. AM asks containers from RM
5. AM notifies NM to launch containers
6. Application code is executed in container
7. Client contacts RM/AM to monitor application’s status
Client RM NM AM
1
2
3
4
5
7 6

33. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Application Workflow
Execution Sequence :
1. Client submits an application
2. RM allocates a container to start AM
3. AM registers with RM
4. AM asks containers from RM
5. AM notifies NM to launch containers
6. Application code is executed in container
7. Client contacts RM/AM to monitor application’s status
8. AM unregisters with RM
Client RM NM AM
1
2
3
4
5
7
8
6

34. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Learning Resources
 Hadoop Tutorial: www.edureka.co/blog/hadoop-tutorial
 MapReduce Tutorial: www.edureka.co/blog/mapreduce-tutorial
 MapReduce Interview Questions:
www.edureka.co/blog/interview-questions/hadoop-interview-questions-mapreduce

35. www.edureka.co/big-data-and-hadoopEDUREKA HADOOP CERTIFICATION TRAINING
Thank You …
Questions/Queries/Feedback