The document discusses the key components of Hadoop including the NameNode, DataNodes, JobTracker, TaskTracker, and SecondaryNameNode. The NameNode stores metadata about data locations and attributes and is located on the master node. DataNodes store the actual data blocks across slave nodes. The JobTracker schedules jobs and is also located on the master node, while TaskTrackers execute tasks on slave nodes. The SecondaryNameNode assists if the NameNode fails. Hadoop uses a master/slave architecture with the NameNode and JobTracker on the master and DataNodes and TaskTrackers on slave nodes. It also utilizes MapReduce for parallel processing of large datasets split across blocks in HDFS.

1. Hadoop
Edit by Cassell Hsu
2013.04.19

2. Hadoop
Master
Slave
Slave
Slave

3. Hadoop
• Master
• NameNode
• JobTracker
• SecondaryNameNode

4. NameNode
• Where is NameNode?
• Master
• HDFS(Hadoop Distributed File
System)
• What is NameNode?
• 資料之位置資訊
• 資料之屬性
？？

5. NameNode
• 位置資訊?
• 所有資料皆存放在 – DataNode

6. DataNode
• What is DataNode
• 存放資料
• Where is DataNode
• HDFS
• Slaves (and Master)

7. User
DataNode
NameNode
DataNode
DataNode
12
8M
b
B
64Mb
A
64Mb
Check hdfs-site.xml
B
64Mb
A
64Mb

8. Hadoop
DataNodeNameNode
Master
DataNode
DataNode
DataNode
Slaves

9. Hadoop
• Master
• NameNode
• JobTracker
• SecondaryNameNode

10. JobTracker
• What is JobTracker?
• 排程工作
• Where is JobTracker?
• Master
誰來工作？

11. JobTracker &
TaskTrackerJobTracker TaskTracker
Where Master Slaves
What 排程工作 執行工作

12. Hadoop
DataNodeNameNode
Master
DataNode
Slaves
JobTracker
TaskTracker

13. Hadoop
• Master
• NameNode
• JobTracker
• SecondaryNameNode

14. SecondaryNameNod
e
• What is SecondaryNameNode?
• NameNode發生錯誤時補救
• Where is SecondaryNameNode?
• Master

15. Hadoop
DataNodeNameNode
Master
DataNode
Slaves
JobTracker
TaskTracker
SecondaryNode

16. MapReduce
User Master
Slave2
Slave1
A
A1
A2
Task
NameNode
Task
Result
1
Result
2
Reduc
e
Final
Result
HDFS

17. MapReduce
• 檔案切割
• Hadoop上區塊切割
• 程式指定