Hadoop Distributed File System (HDFS) is an open-source software framework that provides distributed storage and processing of large datasets across clusters of commodity hardware. HDFS has two main components - HDFS for storage and MapReduce for distributed processing. HDFS uses a master-slave architecture with a NameNode master and DataNodes slaves. The NameNode manages the file system namespace and metadata, while DataNodes store data blocks and report to the NameNode.

1. Hadoop Distributed File System
Made By:
Ameya Vijay Gokhale
14070121505
B.Tech (IT) 2014-18

2. INTRODUCTION
 Open-Source Software Framework (Apache Licensed)
 Based on Google File System (GFS)
 Two Main Components:
- HDFS
- MapReduce
 Master – Slave Architecture

3. HDFS ARCHITECTURE

4. COMPONENTS
 NameNode
 DataNode
 Secondary NameNode

5. NameNode
 Master
 Single Point of Contact
 Manages File System NameSpace and Metadata
 Metadata includes:
- List Of Files
- List of Blocks for each file
- List of DataNode for each Block
- File Attributes
 Maintain Transaction Log
 Only One Per Cluster

6. Secondary NameNode
 Maintains a copy of NameNode metadata
 Periodically merge the namespace image with the edit
log, if edit log becomes too large
 runs on a different machine than the namenode
 Gap between Primary and Secondary NameNode

7. DataNode
 Slaves
 Location where Data is Stored.
 Stores data in Local File system with it’s metadata like
CRC.
 Periodically sends a report of all existing blocks to the
NameNode
 Can be thousands in a cluster.

8. Data Block
 Data is stored in Blocks
 64MB / 128 MB per block
 Files are split and are stored on DataNodes
 Large Block size for Minimum Disk Seek Times
 Eg: Assuming 10 ms of seek time, and 100 MB/s as disk
transfer rate, if block size if 100 MB, then seek time is
1% of transfer time which is small enough to ignore.

9. WORKING

10. WORKING

11. WORKING

12. ADVANTAGES
 Flexible
 Fault Tolerant
 Scalable
 Performs computation across several hosts
 Locality of Data