Download as PDF, PPTX
More Related Content
Hadoop
- 1. Hadoop Presented by Rajesh Piryani SouthAsian University
- 3.
- 4. Hadoop • It isopen source software framework • Licensed under Apache V2 License • Created by Doug Cutting and Mike Cafarella in 2005 • Doug, who was working at Yahoo at the time, named it after his son's toy elephant • Derived from Google Map Reduce and Google File System • Written in Java Programming Language
- 5. 0 100 200 300 400 500 600 700 800 900 1000 966 848 715 619 434 364 269 227 Amount of StoredData By Sector (in Petabytes, 2009) 1 zettabyte? = 1 million petabytes = 1 billion terabytes = 1 trillion gigabytes Why Hadoop? 5 Sources: "Big Data: The Next Frontier for Innovation, Competition and Productivity." US Bureau of Labor Statistics | McKinsley Global Institute Analysis Petabytes Mars Earth 35ZB = enough data to fill a stack of DVDs reaching halfway to Mars If you like analogies…
- 6. Why Hadoop? • Needto process 100TB datasets On 1 node: – scanning @ 50MB/s = 23 days • On 1000 node cluster: – scanning @ 50MB/s = 33 min • Need Efficient, Reliable and Usable framework
- 7. Distributed File System(DFS) • classical model of a file system distributed across multiple machines • allows access to files located on another remote host as though working on the actual host computer. • multiple users on multiple machines to share files and storage resources.
- 8. Distributed File System(DFS) • one or more central servers store files that can be accessed, – with proper authorization rights, by any number of remote clients in the network. • facilities for transparent replication and fault tolerance. • DFS Operation should be fast to increase the performance of System – Operation: open, close, read, write file, send and receive file/object
- 9. Hadoop Distributed FileSystem (HDFS) • Type of distributed file system • Originally built as infrastructure for the Apache Nutch Web Search Engine Project • Significant Difference over other Distributed File System – High Fault Tolerance – High Throughput – Easy Deployment on Low Cost Hardware • Suitable for application that process massive data
- 10. HDFS Architecture • Master-SlaveArchitecture • HDFS Master “NameNode” – Manages all file system metadata (hostname of datanode, block node) • Transactions are logged, merged at startup – Controls read/write access to files – Mapping of block to DataNode – Manages block replication • HDFS Slaves “DataNodes” – Notifies NameNode about block-IDs it has – Serve read/write requests from clients – Perform block create and replication tasks upon instruction by NameNode
- 11.
- 12. HDFS Architecture 12 NameNode BackupNode DataNodeDataNode DataNode DataNode DataNode (heartbeat, balancing, replication, etc.) nodes write to local disk namespace backups
- 13. Getting Files FromHDFS 13 NameNode BackupNode Giant File 110010101001 010100101010 011001010100 101010010101 001100101010 010101001010 100110010101 001010100101 HDFS Client return locations of blocks of file DataNode DataNode DataNode DataNode DataNode stream blocks from data nodes
- 14. Failure types: Diskerrors and failures DataNode failures Switch/Rack failures NameNode failures Datacenter failures Failures, Failures, Failures • HDFS was designed with the expectation that failures (both hardware and software) would occur frequently 14 NameNode DataNode
- 15. Fault Tolerance (DataNode Failure) 15 NameNodeBackupNode DataNode DataNode DataNode DataNode DataNode NameNode detects DataNode lossBlocks are auto-replicated on remaining nodes to satisfy replication factor DataNodeDataNode DataNode
- 16. Fault Tolerance (NameNode Failure) 16 NameNodeBackupNode DataNode DataNode DataNode DataNode DataNode Not an epic failure, because you have the BackupNode NameNode loss requires manual intervention Automatic failover is in the works
- 17. Live Horizontal Scalingand Rebalancing 17 NameNode BackupNode DataNode DataNode NameNode detects new DataNode is added to cluster DataNodeDataNode DataNode Blocks are re-balanced and re-distributed DataNode DataNodeDataNode
- 18. • Highly scalable –1000s of nodes and massive (100s of TB) files – Large block sizes to maximize sequential I/O performance • No use of mirroring or RAID. – Reduce cost – Use one mechanism (triply replicated blocks) to deal with a wide variety of failure types rather than multiple different mechanisms HDFS Summary 18 Why?
- 19. Hadoop MapReduce (MR) •Programming framework (library and runtime) for analyzing data sets stored in HDFS • MapReduce jobs are composed of two functions: • User only writes the Map and Reduce functions 19 map() reduce() sub-divide & conquer combine & reduce cardinality
- 20. Essentially, it’s… 1. Takea large problem and divide it into sub-problems 2. Perform the same function on all sub-problems 3. Combine the output from all sub-problems 20 DoWork() DoWork() DoWork() … … … Output MAPREDUCE Hadoop MapReduce (MR)
- 21. MapReduce Layer HDFS Layer hadoop-namenode hadoop- datanode1 hadoop- datanode2 hadoop- datanode3 hadoop- datanode4 MapReduce Components 21 JobTracker TaskTracker TaskTrackerTaskTracker TaskTracker Temporary data stored to local file system JobTracker controls and heartbeats TaskTracker nodes TaskTrackers store temp data Master Slaves - Coordinates all M/R tasks & events - Manages job queues and scheduling - Maintains and Controls TaskTrackers - Moves/restarts map/reduce tasks if needed Execute individual map and reduce tasks as assigned by JobTracker (in separate JVM) DataNode DataNode DataNode DataNode NameNode MapReduce Layer HDFS Layer
- 22. Job Submission 22 JobTracker TaskTracker TaskTrackerTaskTracker TaskTracker Temporary data stored to local file system map()’s are assigned to TaskTrackers (HDFS DataNode locality aware) Submit jobs to JobTracker MR Client jobs get queued Mapper Mapper Mapper Mapper mappers spawned in separate JVM and execute mappers store temp results reduce phase begins Reducer Reducer Reducer Reducer
- 23. Map tasks MapReduce Visualized:Map Phase 23 53705 $65 53705 $30 53705 $15 54235 $75 54235 $22 02115 $15 02115 $15 44313 $10 44313 $25 44313 $55 5 53705 $15 6 44313 $10 5 53705 $65 0 54235 $22 9 02115 $15 6 44313 $25 3 10025 $95 8 44313 $55 2 53705 $30 1 02115 $15 4 54235 $75 7 10025 $60 Mapper Mapper 4 54235 $75 7 10025 $60 2 53705 $30 1 02115 $15 10025 $60 5 53705 $65 0 54235 $22 5 53705 $15 6 44313 $10 3 10025 $95 8 44313 $55 9 02115 $15 6 44313 $25 10025 $95 Get sum sales grouped by zipCode DataNode3DataNode2DataNode1 Blocks of the Sales file in HDFS Group By Group By (custId, zipCode, amount) One output bucket per reduce task
- 24. Reducer Reducer Reduce tasks ReducerReduce Phase 53705 $65 54235$75 54235 $22 10025 $95 44313 $55 10025 $60 Mapper 53705 $30 53705 $15 02115 $15 02115 $15 44313 $10 44313 $25 Mapper 53705 $65 53705 $30 53705 $15 44313 $10 44313 $25 10025 $95 44313 $55 10025 $60 54235 $75 54235 $22 02115 $15 02115 $15 Sort Sort Sort 53705 $65 53705 $30 53705 $15 44313 $10 44313 $25 44313 $55 10025 $95 10025 $60 54235 $75 54235 $22 02115 $15 02115 $15 SUM SUM SUM 10025 $155 44313 $90 53705 $110 54235 $97 02115 $30 Shuffle
- 25. Dealing With Failures •Like HDFS, MapReduce framework designed to be highly fault tolerant • Worker (Map or Reduce) failures – Detected by periodic Master pings – Map or Reduce jobs that fail are reset and then given to a different node – If a node failure occurs after the Map job has completed, the job is redone and all Reduce jobs are notified • Master failure – If the master fails for any reason the entire computation is redone 25
- 26. Application • Hadoop isused in wide area of application. Some Examples are – Search (Yahoo! , Amazon, Zvents) – Log Processing (Facebook,Yahoo! ,ContextWeb, Joost , Last.fm) – Recommendation System (Facebook) – Data WareHousing (Facebook , AOL) – Video and Image Analysis (New York Times, Eysalike)
- 27.
- 28. References • Meet Hadoop!Open Source Grid Computing by Devraj Das • The Hadoop Distributed File System: Architecture and Design by Dhruba Borthakur • Hadoop Design And Architecture http://hadoop.apache.org/docs/r0.18.0/hdfs_design.pdf • An Introduction to Hadoop Distributed File System http://www.ibm.com/developerworks/library/wa- introhdfs/ • Big Data What’s the Big Deal? By David J. DeWitt and Rimma Nehme. Microsoft Jim Gray System Lab
- 29. Thank You forYour Patience