Hadoop

Overview on HADOOP Distributed Computing ,[object Object],[object Object],[object Object],2/7/2011

Dealing with lots of Data ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],2/7/2011

Hadoop, Why? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],2/7/2011

HDFS splits user data across servers in a cluster. It uses replication to ensure that even multiple node failures will not cause data loss. 2/7/2011

Goals of HDFS ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],2/7/2011

Secondary NameNode Client HDFS Architecture NameNode DataNodes 1. filename 2. BlckId, DataNodes o 3.Read data Cluster Membership Cluster Membership NameNode : Maps a file to a file-id and list of MapNodes DataNode : Maps a block-id to a physical location on disk SecondaryNameNode: Periodic merge of Transaction log 2/7/2011

MapReduce: Programming Model How now Brown cow How does It work now brown 1 cow 1 does 1 How 2 it 1 now 2 work 1 M M M M R R <How,1> <now,1> <brown,1> <cow,1> <How,1> <does,1> <it,1> <work,1> <now,1> <How,1 1> <now,1 1> <brown,1> <cow,1> <does,1> <it,1> <work,1> Input Output Map Reduce MapReduce Framework 2/7/2011

MapReduce: Programming Model ,[object Object],[object Object],[object Object],[object Object],2/7/2011

MapReduce Benefits ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],2/7/2011

MapReduce Examples ,[object Object],Map doc Reduce <word,3> <word,1> <word,1> <word,1> Runtime System <word,1,1,1> 2/7/2011

A Brief History ,[object Object],[object Object],[object Object],[object Object],[object Object],2/7/2011

MapReduce Execution Overview ,[object Object],User Program Input Data Shard 0 Shard 1 Shard 2 Shard 3 Shard 4 Shard 5 Shard 6 * Shards are typically 16-64mb in size 2/7/2011

MapReduce Execution Overview ,[object Object],User Program Master Workers Workers Workers Workers Workers 2/7/2011

MapReduce Resources ,[object Object],[object Object],[object Object],Master Idle Worker Message(Do_map_task) 2/7/2011

MapReduce Resources ,[object Object],[object Object],Map worker Shard 0 Key/value pairs 2/7/2011

MapReduce Execution Overview ,[object Object],Master Map worker Disk locations Local Storage 2/7/2011

MapReduce Execution Overview ,[object Object],Master Reduce worker Disk locations remote Storage 2/7/2011

MapReduce Execution Overview ,[object Object],Reduce worker Sorts data Partition Output file 2/7/2011

MapReduce Execution Overview ,[object Object],wakeup User Program Master Output files 2/7/2011

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],2/7/2011

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],2/7/2011

ZooKeeper ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],2/7/2011

Some more projects… ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],2/7/2011

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],2/7/2011

Hadoop

Recommended

Recommended

More Related Content

What's hot

What's hot (18)

Similar to Hadoop

Similar to Hadoop (20)

Recently uploaded

Recently uploaded (20)

Hadoop