Hadoop is an open-source software framework for distributed storage and processing of large datasets across clusters of computers. It uses MapReduce as a programming model and HDFS for storage. MapReduce allows for massively parallel processing of large datasets by breaking jobs into smaller tasks that can be run in parallel on multiple machines. HDFS stores very large files across machines in a distributed file system for fault tolerance.

1. Knowledge ShareHadoopYu ZhaoPlatform

2. HadoopWhat is HadoopThe Apache Hadoop project develops open-source software for reliable, scalable, distributed computing.Hadoop IncludesMapReduceHDFSHadoop Common

3. HadoopPositionAPPLICATIONHADOOPOSOSOSOSHOSTHOSTHOSTHOST

4. MapReduceA simple programming model that applies to many large-scale computing problemsHide messy details in MapReduce runtime library:automatic parallelizationload balancingnetwork and disk transfer optimizationhandling of machine failuresrobustness

5. MapReduceTypical problem solved by MapReduceRead a lot of dataMap: extract something you care about from each recordShuffle and SortReduce: aggregate, summarize, filter, or transformWrite the results

6. MapReduceOutline stays the same, map and reduce change to fit the problemMore specifically…Programmer specifies two primary methods:map: (K1, V1) -> list(K2, V2) reduce: (K2, list(V2)) -> list(K3, V3)

7. MapReduceExample- word countCounting the number of occurrences of each word in a large collection of documents. Key:“document1”Value:“to be or not to be”MAPKey Value“to” “1”“be” “1”“or” “1”“not” ”1”“to” ”1”“be” ”1”Key Value “be” “1”“1”“not” “1”“or” “1”“to” “1””1”Key Value“be” “2”“not” “1”“or” “1”“to” “2”REDUCESHUFFLE&SORT

8. MapReduceExample- word countPseudo-codeMap(String key, String value):// key: document name// value: document contentsfor each word w in value:EmitIntermediate(w, “1”);Reduce(String key, Iterator values):// key: a word// values: a list of countsint result = 0;for each v in values:result += ParseInt(v);Emit(AsString(result));

9. MapReduceShuffle and Sort

10. HDFSDesign Very large files Streaming data access Commodity hardwareIgnores Low-latency data access Lots of small files Multiple writers, arbitrary file modifications

11. HDFSConceptsBlocksNamenodes and Datanodes

12. HDFSNetwork Topology and HadoopNetwork is represented as a tree and the distance between two nodes is the sum of their distances to their closest common ancestor.Distances computeProcesses on the same nodeDifferent nodes on the same rackNodes on different racks in the same data centerNodes in different data centers

13. HDFSNetwork Topology and Hadoop

14. HDFSData Flow- read

15. HDFSData Flow- write

16. SetupRequired SoftwareJavaTM 1.6.x, preferably from Sun, must be installed.sshCygwin - Required on windows for shell support in addition to the required software above.

17. SetupConfigureSetup passphraselesssshConfiguration Files core-site.xml hdfs-site.xml mapred-site.xml masters slaves

18. ReferencesHadoop: The Definitive Guide, Tom WhiteMapReduce: Simplified Data Processing on Large Clusters, Jeffrey Dean and Sanjay GhemawatExperiences with MapReduce, an Abstraction for Large-Scale Computation, Jeff Dean