The document discusses the Hadoop Distributed File System (HDFS), which was created by Doug Cutting to address the need for large-scale data processing. HDFS is designed for streaming data across commodity hardware and uses a master/slave architecture with one NameNode master and multiple DataNodes. The NameNode manages the file system namespace and regulates access to files by clients via the DataNodes, which store data blocks and ensure replication for fault tolerance.

1. Anand L. Kulkarni.
Hadoop Distributed File System
A Presentation By ,

2. 28 August 2015Hadoop Distributed File System 2
 Need for large data processing –
 Challenges at large scale –
 What is Distributed File System(DFS)?

3.  “Framework for running [distributed]
applications on large cluster built of commodity
hardware“ .
- From Hadoop Wiki.
 Originally created by Doug Cutting .
 Named the project after his son’s name.
 Inspired by Google’s architecture: Map Reduce
and GFS
28 August 2015Hadoop Distributed File System 3

4.  The name “Hadoop” has now evolved to cover a
family of products, but at its core, it’s essentially just
the
 - MapReduce programming paradigm and
 - A distributed file system(HDFS).
28 August 2015Hadoop Distributed File System 4

5. 28 August 2015Hadoop Distributed File System 5

6. 28 August 2015Hadoop Distributed File System 6
 Master/slave architecture
 Fault tolerant via replication .
 Optimized for larger files.
 Hardware failures assumed in
design.
Name Node
(Master)
(Slaves)

7. 28 August 2015Hadoop Distributed File System 7
 Written in Java.
 Focus on streaming data
(High throughput > low-latency)
 Designed to run on commodity hardware
 HDFS is a File System, not a DBMS.

8. Block Data Node
Name
Node
Checkpoint
Node
Backup
Node
28 August 2015Hadoop Distributed File System 8

9. 28 August 2015Hadoop Distributed File System 9
Name Node Backup Node
Data Node Data Node Data Node Data NodeData Node
( Replication, Heartbeats,
balancing )
(Namespace backups)
(Namespace , Metadata
operations)
(Writes to local disks)

10. 28 August 2015Hadoop Distributed File System 10
Name Node Backup Node
10010011001
01001010100
10101010101
00101010010
10101010100
10101010101
010101
File
HDFS
Client
Data Node Data Node Data Node Data NodeData Node
( File locations, block size, file system
operations )
(Data transfer)

11. Data Node Data Node Data Node Data NodeData Node
28 August 2015Hadoop Distributed File System 11
Name Node Backup Node
10010011001
01001010100
10101010101
00101010010
10101010100
10101010101
010101
File
HDFS
Client

12. 28 August 2015Hadoop Distributed File System 12
Data Node Data Node Data Node Data NodeData Node
Name Node Backup Node
10010011001
01001010100
10101010101
00101010010
10101010100
10101010101
010101
File
HDFS
Client
(Return locations of blocks for
a file.)

13. 28 August 2015Hadoop Distributed File System 13
 The Files system namespace
 Replica management
 Replica Selection
 Safe mode

14. 28 August 2015Hadoop Distributed File System 14
 The Persistence Of File System Metadata
 Robustness
 Space Reclamation-
◦ File Deletes And Undeletes
◦ Decrease Replication Factor

15. 28 August 2015Hadoop Distributed File System 15
 Name Node Recovery.
 Data Node Recovery.
 Metadata Disk Failure.

16. 28 August 2015Hadoop Distributed File System 16
Name Node Backup Node
Data Node Data Node Data Node
Data Node
Data Node

17. 28 August 2015Hadoop Distributed File System 17
Data Node Data Node Data Node Data NodeData Node
Name Node Backup Node

18. 28 August 2015Hadoop Distributed File System 18
Scalability of Name node.
Automation of Name node recovery.

19. 28 August 2015Hadoop Distributed File System 19