![Combiner
Combiner
Reducer
(B,1)
(C,1)
(D,1)
(E,1)
(D,1)
(B,1)
(D,1)
(A,1)
(A,1)
(C,1)
(B,1)
(D,1)
(B,2)
(C,1)
(D,2)
(E,1)
(D,2)
(A,2)
(C,1)
(B,1)
(A, [2])
(B, [2,1])
(C, [1,1])
(D, [2,2])
(E, [1])
(A,2)
(B,3)
(C,2)
(D,4)
(E,1)
Shuffle
CombinerMapper
Mapper
B
C
D
E
D
B
D
A
A
C
B
D
Block1Block2
www.edureka.co/big-data-and-hadoopSlide 48](https://image.slidesharecdn.com/module3-bigdataandhadoop-160425063358/75/Hadoop-MapReduce-Framework-48-2048.jpg)
![public static String encrypt(String strToEncrypt, byte[] key)
{
try
{
Cipher cipher = Cipher.getInstance("AES/ECB/PKCS5Padding");
SecretKeySpec secretKey = new SecretKeySpec(key, "AES");
cipher.init(Cipher.ENCRYPT_MODE, secretKey);
String encryptedString = Base64.encodeBase64String(cipher.doFinal(strToEncrypt.getBytes()));
return encryptedString.trim();
}
catch (Exception e)
{
logger.error("Error while encrypting", e);
}
return null;
}
}
www.edureka.co/big-data-and-hadoopSlide 61
DeIdentify MapReduce Code](https://image.slidesharecdn.com/module3-bigdataandhadoop-160425063358/75/Hadoop-MapReduce-Framework-61-2048.jpg)
Uploaded byEdureka!
Hadoop MapReduce Framework
No, combiner and reducer logic cannot be same. Combiner is an optional step that performs local aggregation of the intermediate key-value pairs generated by the mappers. Its goal is to reduce the amount of data transferred from the mapper to the reducer. Reducer performs the final aggregation of the values associated with a particular key. It receives the intermediate outputs from all the mappers, groups them by key, and produces the final output. So while combiner and reducer both perform aggregation, their scopes of operation are different - combiner works locally on mapper output to minimize data transfer, whereas reducer operates globally on all mapper outputs to produce the final output. The logic needs to be optimized for their respective purposes.
TechnologyâŚ
More Related Content
Similar to Hadoop MapReduce Framework
![[BDD 2025 - Artificial Intelligence] Building AI Systems That Users (and Comp...](https://cdn.slidesharecdn.com/ss_thumbnails/ai-buildingaisystemsthatusersandcompanieslove-251124030845-038f7732-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Full-Stack Development] Digital Accessibility: Why Developers nee...](https://cdn.slidesharecdn.com/ss_thumbnails/fs-digitalaccessibilitywhydevelopersneedtoknowandcarein2025-251127011019-0674441d-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Full-Stack Development] PHP in AI Age: The Laravel Way. (Rizqy Hi...](https://cdn.slidesharecdn.com/ss_thumbnails/fs-phpinaiagethelaravelway-251125012602-ef9d330e-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Artificial Intelligence] AI for the Underdogs: Innovation for Sma...](https://cdn.slidesharecdn.com/ss_thumbnails/ai-aifortheunderdogsinnovationforsmallbusinesses-251124030839-72a599a4-thumbnail.jpg?width=640&height=640&fit=bounds)
Hadoop MapReduce Framework
- 1.
- 2. Objectives www.edureka.co/big-data-and-hadoopSlide 2 At theend of this module, you will be able to ďŽ Analyze different use-cases where MapReduce is used ďŽ Differentiate between Traditional way and MapReduce way ďŽ Learn about Hadoop 2.x MapReduce architecture and components ďŽ Understand execution flow of YARN MapReduce application ďŽ Implement basic MapReduce concepts ďŽ Run a MapReduceProgram ďŽ Understand Input Splits concept in MapReduce ďŽ Understand MapReduce Job Submission Flow ďŽ Implement Combiner and Partitioner in MapReduce
- 3. Letâs Revise ďŽ HadoopCluster Configuration ďŽ Data Loading Techniques ďŽ Hadoop ClusterModes Core HDFS core-site.xml hdfs-site.xml yarn-site.xmlYarn mapred-site.xml Map Reduce www.edureka.co/big-data-and-hadoopSlide 3
- 4. Letâs Revise Data Analysis DataLoading Using Pig Using HIVE Using Flume Using Sqoop Using Hadoop Copy Commands HDFS Sqoop Flume â Questions www.edureka.co/big-data-and-hadoopSlide 4
- 5. Annieâs Question Secondary NameNodeis a hot backup for NameNode: Âť TRUE Âť FALSE www.edureka.co/big-data-and-hadoopSlide 5
- 6. Annieâs Answer Ans. FALSE.The Secondary NameNode (SNN) is the most misunderstood component of the HDFS architecture. SNN is not a hot backup for NameNode but a checkpoint backup mechanism enabler in a Hadoop Cluster. www.edureka.co/big-data-and-hadoopSlide 6
- 7. Where MapReduce isUsed? Weather Forecasting ďŽ Problem Statement: Âť De-identify personal health information. HealthCare www.edureka.co/big-data-and-hadoopSlide 8 ďŽ Problem Statement: Âť Finding Maximum temperature recorded in a year.
- 8. The Traditional Way Very Big Data SplitData matches All matches grep grep grep cat grep matches matches matches Split Data Split Data : Split Data www.edureka.co/big-data-and-hadoopSlide 8
- 9. MapReduce Way Very Big Data Split Data All matches SplitData Split Data : Split Data M A P www.edureka.co/big-data-and-hadoopSlide 9 R E D U C E MapReduce Framework
- 10. Why MapReduce? ďŽ Twobiggest Advantages: Âť Taking processing to the data Âť Processing data in parallel a b c Map Task HDFS Block Data Center www.edureka.co/big-data-and-hadoopSlide 10 Rack Node
- 11. Solving the Problemwith MapReduce HDFS Take DB dump in CSV format and copy it on HDFS Store CSV file into HDFS Read CSV file from HDFS matches Reduce Map 0100 1101 1001 0100 1101 1001 0100 1101 Sqoop 1001 www.edureka.co/big-data-and-hadoopSlide 12 Map Logic Reduce Logic
- 12. Node Manager Container Map Task Container Application Master Hadoop2.x MapReduce Architecture Client Job History Server Resource Manager MapReduce Status Job Submission Node Status Resource Request Node Manager Datanode1 www.edureka.co/big-data-and-hadoopSlide 12 Container Reduce Task Datanode2 Namenode
- 13. ďŽ ApplicationMaster www.edureka.co/big-data-and-hadoopSlide 13 ÂťOne per application Âť Short life Âť Coordinates and Manages MapReduce Jobs Âť Negotiates with Resource Manager to schedule tasks Âť The tasks are started by NodeManager(s) ďŽ Job HistoryServer Âť Maintains information about submitted MapReduce jobs after their ApplicationMaster terminates ďŽ Client Âť Submits a MapReduce Job ďŽ Resource Manager Âť Cluster Level resource manager Âť Long Life, High Quality Hardware ďŽ Node Manager Âť One per Data Node Âť Monitors resources on Data Node Hadoop 2.x MapReduce Components ďŽ Container Âť Created by NM when requested Âť Allocates certain amount of resources (memory, CPU etc.) on a slave node
- 14. MapReduce Application Execution www.edureka.co/big-data-and-hadoopSlide14 Executing MapReduce Application on YARN
- 15. YARN MR ApplicationExecution Flow www.edureka.co/big-data-and-hadoopSlide 15 ďŽ MapReduce Job Execution Âť Job Submission Âť Job Initialization Âť Tasks Assignment Âť Memory Assignment Âť StatusUpdates Âť Failure Recovery
- 16. YARN MR ApplicationExecution Flow HDFS Job ObjectApplication Client JVM Client Resource Manager Management Node 1. Run Job 2. Get New Application www.edureka.co/big-data-and-hadoopSlide 16 4. Submit Application 3. Copy Job Resources
- 17. YARN MR ApplicationExecution Flow HDFS Job ObjectApplication Client JVM Client Resource Manager Management Node 1. Run Job 2. Get New Application 4. Submit Application 3. Copy Job Resources Node Manager 5. Start MR AppMaster container 6. Create 9. Start container container www.edureka.co/big-data-and-hadoopSlide 17 7. Get Input Splits MRAppMaster Data Node 8. Request Resources
- 18. YARN MR ApplicationExecution Flow HDFS Job ObjectApplication Client JVM Client Resource Manager Management Node 1. Run Job 2. Get New Application 4. Submit Application 3. Copy Job Resources Node Manager 5. Start MR AppMaster container 6. Create 9. Start container container 8. Request Resources 7. Get Input Splits MRAppMaster Data Node Map/Reduce Task 10. Create Container Task JVM YarnChild 12. Execute 11. Acquire Job Resources www.edureka.co/big-data-and-hadoopSlide 19
- 19. YARN MR ApplicationExecution Flow HDFS Job ObjectApplication Client JVM Client Resource Manager Management Node Node Manager MRAppMaster Data Node Map/Reduce Task YarnChild Task JVM Poll for Status www.edureka.co/big-data-and-hadoopSlide 19 Update Status
- 20. Hadoop 2.x :YARN Workflow Node Manager Node Manager Node Manager Node Manager Node Manager Node Manager Node Manager Container1.2 Node Manager Container1.1 Container2.1 Node Manager Container2.2 Node Manager Container2.3 Node Manager App Master2 Node Manager App Master1 Scheduler Applications Manager(AsM) Resource Manager www.edureka.co/big-data-and-hadoopSlide 20
- 21. Annieâs Question YARN wasdeveloped to overcome the following disadvantage in Hadoop 1.0 MapReduce framework? Âť Single Point Of Failure Of NameNode Âť Only one version can be run in classic MapReduce Âť Too much burden on Job Tracker www.edureka.co/big-data-and-hadoopSlide 21
- 22. Annieâs Answer Too muchburden on Job Tracker www.edureka.co/big-data-and-hadoopSlide 22
- 23. Annieâs Question In YARN,the functionality of JobTracker has been replaced by which of the following YARN features: Âť Job Scheduling Âť TaskMonitoring Âť Resource Management Âť Node management www.edureka.co/big-data-and-hadoopSlide 23
- 24. Annieâs Answer Task Monitoringand Resource Management. The fundamental idea of YARN is to split up the two major functionalities of the JobTracker, i.e. resource management and job scheduling/monitoring, into separate daemons. A global Resource Manager (RM) for resources and per-application ApplicationMaster (AM) for task monitoring. www.edureka.co/big-data-and-hadoopSlide 24
- 25. Annieâs Question In YARN,which of the following daemons takes care of the container and the resource utilization by the applications? Âť Node Manager Âť JobTracker Âť Tasktracker Âť ApplicationMaster www.edureka.co/big-data-and-hadoopSlide 25
- 26.
- 27. Annieâs Question Can werun MRv1 Jobs in a YARN enabled Hadoop Cluster? Âť Yes Âť No www.edureka.co/big-data-and-hadoopSlide 27
- 28. Annieâs Answer Yes. MapReduceon YARN ensures full binary compatibility. These existing applications can run on YARN directly without recompilation. www.edureka.co/big-data-and-hadoopSlide 28
- 29. MapReduce Paradigm The OverallMapReduce Word Count Process Input Splitting Mapping Shuffling Reducing Final Result List(K3,V3) Deer Bear River Dear Bear River Car Car River Deer Car Bear Bear, 2 Car, 3 Deer, 2 River, 2 Deer, 1 Bear, 1 River, 1 Car, 1 Car, 1 River, 1 Deer, 1 Car, 1 Bear, 1 List(K2,V2) K1,V1 Car Car River Deer Car Bear Bear, 2 Car, 3 Deer, 2 River, 2 K2,List(V2) Bear, (1,1) Car, (1,1,1) Deer, (1,1) River, (1,1) www.edureka.co/big-data-and-hadoopSlide 29
- 30. Anatomy of aMapReduce Program MapReduce Map: Reduce: (K1, V1) List (K2, V2) (K2, list (V2)) List (K3, V3) Key Value www.edureka.co/big-data-and-hadoopSlide 30
- 31. Demo of WordCountProgram www.edureka.co/big-data-and-hadoopSlide 31
- 32. Annieâs Question Input tothe mapper is in the form of? Âť A flat file Âť (key, value) pair Âť Only string Âť All the above www.edureka.co/big-data-and-hadoopSlide 32
- 33. Annieâs Answer A Mapperaccepts (key, value) pair as input. www.edureka.co/big-data-and-hadoopSlide 33
- 34.
- 35. Relation Between InputSplits and HDFS Blocks 1 2 3 4 5 6 7 8 9 10 11 ďŽ Logical records do not fit neatly into the HDFS blocks. ďŽ Logical records are lines that cross the boundary of the blocks. ďŽ First split contains line 5 although it spans across blocks. File Lines Block Boundary Block Boundary Block Boundary Block Boundary Split www.edureka.co/big-data-and-hadoopSlide 35 Split Split
- 36. MapReduce Job SubmissionFlow Node 1 Node 2 INPUT DATA www.edureka.co/big-data-and-hadoopSlide 36 ďŽ Input data is distributed to nodes
- 37. MapReduce Job SubmissionFlow Map Node 1 Map Node 2 INPUT DATA www.edureka.co/big-data-and-hadoopSlide 37 ďŽ Input data is distributed to nodes ďŽ Each map task works on a âsplitâ of data
- 38. MapReduce Job SubmissionFlow Map Node 1 Map Node 2 INPUT DATA www.edureka.co/big-data-and-hadoopSlide 38 ďŽ Input data is distributed to nodes ďŽ Each map task works on a âsplitâ of data ďŽ Mapper outputs intermediate data
- 39. MapReduce Job SubmissionFlow Map Node 1 Map Node 2 INPUT DATA Node 1 Node 2 www.edureka.co/big-data-and-hadoopSlide 39 ďŽ Input data is distributed to nodes ďŽ Each map task works on a âsplitâ of data ďŽ Mapper outputs intermediate data ďŽ Data will be copied by the reducer processor once it identifies the respective task using application master for all data the reducer is responsible for
- 40. MapReduce Job SubmissionFlow ďŽ Input data is distributed to nodes ďŽ Each map task works on a âsplitâ of data ďŽ Mapper outputs intermediate data ďŽ Data will be copied by the reducer processor once it identifies the respective task using application master for all data the reducer is responsible for ďŽ Shuffle processor will sort and merge the data for a particular key Map Node 1 Map Node 2 Reduce Reduce INPUT DATA Node 1 Node 2 www.edureka.co/big-data-and-hadoopSlide 40
- 41. MapReduce Job SubmissionFlow Map Node 1 Map Node 2 Reduce Reduce INPUT DATA Node 1 Node 2 www.edureka.co/big-data-and-hadoopSlide 41 ďŽ Input data is distributed to nodes ďŽ Each map task works on a âsplitâ of data ďŽ Mapper outputs intermediate data ďŽ Data will be copied by the reducer processor once it identifies the respective task using application master for all data the reducer is responsible for ďŽ Shuffle processor will sort and merge the data for a particular key ďŽ Reducer output is stored
- 42. Annieâs Question www.edureka.co/big-data-and-hadoopSlide 42 MapReduceprogramming model provides a way for reducers to communicate with each other? Âť Yes, reducers running on the same machine can communicate with each other through shared memory Âť No, each reducer runs independently and in isolation.
- 43. Annieâs Answer www.edureka.co/big-data-and-hadoopSlide 43 Ans.No, reducers run independently and in isolation. Individual tasks do not know the input source. Reducer tasks rely on Hadoop framework to deliver the appropriate input for processing.
- 44. Annieâs Question www.edureka.co/big-data-and-hadoopSlide 44 Whospecify Input Split Information? Âť randomly and decided by name node Âť randomly and decided by job tracker Âť line by Line and decided by Input Splitter Âť we will have to specify explicitly
- 45. Annieâs Answer www.edureka.co/big-data-and-hadoopSlide 45 Ans.The client have to submit the input spit information by specifying the start and end point either in InputFormat Configuration.
- 46. Overview of MapReduce CombinersPartitioners Combiners can be viewed as âmini-reducersâ in the Map phase. Complete view of MapReduce, illustrating combiners and partitioner in addition to Mappers and Reducers MapReduce www.edureka.co/big-data-and-hadoopSlide 46 Partitioners determine which reducer is responsible for a particular key.
- 47. Combiner â LocalReduce Passed workload further to the Reducers Before we distribute the mapper results Mini-Reducers Perform a âLocal Reduceâ COMBINERS www.edureka.co/big-data-and-hadoopSlide 47
- 48. Combiner Combiner Reducer (B,1) (C,1) (D,1) (E,1) (D,1) (B,1) (D,1) (A,1) (A,1) (C,1) (B,1) (D,1) (B,2) (C,1) (D,2) (E,1) (D,2) (A,2) (C,1) (B,1) (A, [2]) (B, [2,1]) (C,[1,1]) (D, [2,2]) (E, [1]) (A,2) (B,3) (C,2) (D,4) (E,1) Shuffle CombinerMapper Mapper B C D E D B D A A C B D Block1Block2 www.edureka.co/big-data-and-hadoopSlide 48
- 49. Annieâs Question www.edureka.co/big-data-and-hadoopSlide 49 Combinerworks at? Âť Mapper Level Âť Patitioner Level Âť Reducer Level Âť All the above
- 50. Annieâs Answer www.edureka.co/big-data-and-hadoopSlide 50 Ans.Mapper level as Combiner works on the output data from Mapper.
- 51. Annieâs Question www.edureka.co/big-data-and-hadoopSlide 51 Combinercan be considered as: Âť Semi Partitioner Âť Semi Reducer Âť Semi Shuffler Âť Major Reducer
- 52. Annieâs Answer www.edureka.co/big-data-and-hadoopSlide 52 Ans.Semi Reducer. Combiner works on the Mapper output and lessen the burden on Reducer.
- 53. Partitioner â RedirectingOutput from Mapper Map Map Map Reducer Reducer Reducer Partitioner Partitioner Partitioner www.edureka.co/big-data-and-hadoopSlide 53
- 54. Demo: Combiner andPartitioner www.edureka.co/big-data-and-hadoopSlide 54 Demo: Combiner and Partitioner MR Code
- 55. Annieâs Question www.edureka.co/big-data-and-hadoopSlide 55 Canwe use same logic for combiner and reducer? Âť No, they are separate entities. Âť Yes, only if reducer and combiner logic are commutative and associative and both of them are of same data types.
- 56. Annieâs Answer www.edureka.co/big-data-and-hadoopSlide 56 Ans.Yes, you can use same logic if Reducer and Combiner logic are both commutative and associative and both of them are of same data types.
- 57. Annieâs Question www.edureka.co/big-data-and-hadoopSlide 57 Canwe change the format of output key class and output value class? Âť TRUE Âť FALSE
- 58.
- 59.
- 60. Revisit De-identification Architecture HDFS TakingDB dump in CSV format and ingest into HDFS Store De-identified CSV file into HDFS De-identify columns based on configurations matches Map Task 1 Map Task 2 . . Read CSV file from HDFS Reduce Task 1 0100 1101 1001 0100 1101 1001 0100 1101 Reduce Task 2 . . Sqoop 1001 www.edureka.co/big-data-and-hadoopSlide 61
- 61. public static Stringencrypt(String strToEncrypt, byte[] key) { try { Cipher cipher = Cipher.getInstance("AES/ECB/PKCS5Padding"); SecretKeySpec secretKey = new SecretKeySpec(key, "AES"); cipher.init(Cipher.ENCRYPT_MODE, secretKey); String encryptedString = Base64.encodeBase64String(cipher.doFinal(strToEncrypt.getBytes())); return encryptedString.trim(); } catch (Exception e) { logger.error("Error while encrypting", e); } return null; } } www.edureka.co/big-data-and-hadoopSlide 61 DeIdentify MapReduce Code
- 62. Demo of DeIdentifyProgram www.edureka.co/big-data-and-hadoopSlide 62
- 63.
- 64. Demo of WeatherDataProgram www.edureka.co/big-data-and-hadoopSlide 64
- 65. Assignment Write MapReduce codefor WordCount on your own and run it on Edureka VM Download all the MapReduce codes from LMS and import them in your Eclipse IDE and execute them Try Maximum Temperature problem in MapReduce Try Hot and Cold day problem in MapReduce www.edureka.co/big-data-and-hadoopSlide 65
- 66. Watch video âRunningMapReduce Programâ under Module-3 of your LMS Attempt the Word Count , Patents,& Alphabets assignment using the items present in the LMS under the tab Module 3 Review the Interview Questions for MapReduce http://www.edureka.in/blog/hadoop-interview-questions-mapreduce/ Review the Next Generation MapReduce (MRv2 or YARN) http://www.edureka.in/blog/apache-hadoop-2-0-and-yarn/ http://www.edureka.in/blog/hadoop-2-0-setting-up-a-single-node-cluster-in-15-minutes/ Setup the CDH4 Hadoop development environment using the documents present in the LMS http://blog.cloudera.com/blog/2013/08/how-to-use-eclipse-with-mapreduce-in-clouderas-quickstart-vm/ www.edureka.co/big-data-and-hadoopSlide 66 Pre-work
- 67. Agenda for NextClass www.edureka.co/big-data-and-hadoopSlide 67 ďŽ Map and Reduce Side Join ďŽCounters ďŽ DistributedCache ďŽ Custom Input Format ďŽ Sequence Input Format ďŽMRUnit
- 68. Your feedback isimportant to us, be it a compliment, a suggestion or a complaint. It helps us to make the course better! Please spare few minutes to take the survey after the webinar. www.edureka.co/big-data-and-hadoopSlide 68 Survey