Hadoop has a master/slave architecture. The master node runs the NameNode, JobTracker, and optionally SecondaryNameNode. The NameNode stores metadata about data locations. DataNodes on slave nodes store the actual data blocks. The JobTracker schedules jobs, assigning tasks to TaskTrackers on slaves which perform the work. The SecondaryNameNode assists the NameNode in the event of failures. MapReduce jobs split files into blocks, map tasks process the blocks in parallel on slaves, and reduce tasks consolidate the results.

1. Hadoop
Edit by Cassell Hsu
2013.04.19

2. Hadoop
Slave
Master Slave
Slave

3. Hadoop
• Master
• NameNode
• JobTracker
• SecondaryNameNode

4. NameNode
• Where is NameNode?
• Master
• HDFS(Hadoop Distributed File System)
• What is NameNode?
• 資料之位置資訊 ？？
• 資料之屬性

5. NameNode
• 位置資訊 ?
• 所有資料皆存放在 – DataNode

6. DataNode
• What is DataNode
• 存放資料
• Where is DataNode
• HDFS
• Slaves (and Master)

7. DataNode
User NameNode DataNode
A A
128 64Mb 64Mb
Mb B B
64Mb 64Mb
DataNode
Check hdfs-site.xml

8. Hadoop
Slaves
Master
DataNode
NameNode DataNode
DataNode
DataNode

9. Hadoop
• Master
• NameNode
• JobTracker
• SecondaryNameNode

10. JobTracker
• What is JobTracker?
• 排程工作 誰來工作？
• Where is JobTracker?
• Master

11. JobTracker & TaskTracker
JobTracker TaskTracker
Where Master Slaves
What 排程工作 執行工作

12. Hadoop
Slaves
Master
DataNode
TaskTracker
NameNode DataNode
JobTracker

13. Hadoop
• Master
• NameNode
• JobTracker
• SecondaryNameNode

14. SecondaryNameNode
• What is SecondaryNameNode?
• NameNode 發生錯誤時補救
• Where is SecondaryNameNode?
• Master

15. Hadoop
Slaves
Master
DataNode
TaskTracker
NameNode DataNode
JobTracker
SecondaryNode

16. MapReduce
NameNode Reduce
Slave1 A1
HDFS Final Result
Result1
User Master
Result2
Task Task
A Slave2 A2

17. MapReduce
• 檔案切割
• Hadoop 上區塊切割
• 程式指定