A Hadoop Distributed File System (HDFS) is designed to store very large data sets reliably and to stream those data sets at high bandwidth to user applications. By distributing storage and computation across many servers, the resource can grow with demand while remaining economical at every size. An important characteristic of Hadoop is the partition-ing of data and computation across many (thousands) of hosts, and executing application computations in parallel close to their data. A Hadoop cluster scales computation capacity, storage capacity and IO bandwidth by simply add-ing commodity servers. Hadoop Distributed File System (HDFS) are the most common file system deployed in large scale distributed systems such as Face book, Google and Yahoo today.

1. ISSN: XXXX-XXXX Volume X, Issue X, Month Year
Significance of HADOOP Distributed File System
Vivekanand. S. Reshmi
Dept of Computer Science and Engineering
BTL Institute of Technology
Bangalore, India
Ravikumar.reshmi@gmail.com
Abstract:
A Hadoop Distributed File System (HDFS) is designed to
store very large data sets reliably and to stream those data
sets at high bandwidth to user applications. By distributing
storage and computation across many servers, the resource
can grow with demand while remaining economical at every
size. An important characteristic of Hadoop is the partition-
ing of data and computation across many (thousands) of
hosts, and executing application computations in parallel
close to their data. A Hadoop cluster scales computation
capacity, storage capacity and IO bandwidth by simply add-
ing commodity servers. Hadoop Distributed File System
(HDFS) are the most common file system deployed in large
scale distributed systems such as Face book, Google and
Yahoo today.
Introduction
The Hadoop platform [1][5]provides both hadoop distributed
file system (HDFS) and computational capabilities (Map
Reduce).[2] Hadoop is an Apache project all components are
available via the Apache open source license. The newest
Hadoop versions are capable of storing petabytes of da-
ta.HDFS stores file system metadata and application data
separately As in GFS. It is designed to run on clusters of
commodity hardware. HDFS relaxes a few requirements to
enable streaming access to the file system data. Hadoop is a
Distributed parallel fault tolerant file system. It is designed
to reliably store very large files across machines in a large
cluster. It is inspired by the Google
File System. Hadoop DFS stores each file as a sequence of
blocks; all blocks in a file except the last block are the same
size. Blocks belonging to a file are replicated for fault toler-
ance. The block size and replication factor are configurable
per file. In a distributed system even if we decide to deploy
dedicated high performance machines which are really cost-
ly, faults or disruptions are not frequent. So forerunners like
Google decided to use commodity hardware which is ubiq-
uitous and very cost effective , but to use such hardware they
have to make a design choice of treating faults or disruptions
as regular situation and system should able to recover from
such failures. Hadoop developed on similar design choices
to handle faults. So comparing luster, pvfs which system
assumes faults are infrequent and needs manual intervention
to ensure continued services on other hand Hadoop turns out
to be very robust and fault tolerant option. Hadoop ensures
that few failures in the system won’t disrupt continued ser-
vice of data through automatic replication and transfer of
responsibilities from failed machines to live machines in
Hadoop farm transparently. Though it’s mentioned that GFS
has same capabilities since its not available to other compa-
nies those capabilities cannot be availed.
A. Meaning of hadoop
Hadoop is an Open Source implementation of a
large-scale batch processing system. Hadoop is a top-level
Apache project being built and used by a global community
of contributors, written in the Java programming language. It
provides a distributed file system and a framework for the
analysis and transformation of very large data sets using the
Map Reduce paradigm. Hadoop framework is written in
Java, it allows developers to deploy custom- written pro-
grams coded in Java or any other language to process data in
a parallel fashion across hundreds or thousands of commodi-
ty servers An important characteristic of Hadoop is the parti-
tioning of data and computation across many (thousands) of
hosts, and executing application computations in parallel
close to their data.

2. International Journal of Innovatory research in Engineering and Technology - IJIRET
ISSN: XXXX-XXXX Volume X, Issue X, Month Year 22
Fig1.Hadoop systems[6]
The table 1 shows the components of hadoop. Hadoop is an
Apache project; all components are available via the Apache
open source license. Yahoo! has developed and contributed
to 80% of the core of Hadoop (HDFS and Map Reduce).
HBase was originally developed at
Table1 .Hadoop project components [4]
Power set, now a department at Microsoft.
Hive was originated and developed at Facebook. Pig,
Zookeeper, and Chukwa were originated and developed at
Yahoo! Avro was originated at Yahoo! and is being co-
developed with Cloudera. HDFS is the file system compo-
nent of Hadoop. While the interface to HDFS is patterned
after the UNIX file system, faithfulness to standards was
sacrificed in favor of improved performance for the applica-
tions at hand.
B.HDFS architecture
The Hadoop Distributed File System [3] is a distributed file
system designed to run on commodity hardware. It has many
similarities with existing distributed file systems. However,
the differences from other distributed file systems are signif-
icant. HDFS is highly fault-tolerant and is designed to be
deployed on low-cost hardware. HDFS provides high
throughput access to application data and is suitable for ap-
plications that have large data sets. HDFS stores file system
metadata and application data separately. A normal file sys-
tem is separated into several pieces called blocks, which are
the smallest units that can be read or written.
Normally the default size is a few kilobytes. HDFS also has
blocks, but of a much larger size, 64 MB bytes default. The
reason for that is to minimize the costs of seeks for finding
the start of the block. With the abstraction of blocks it is
possible to create files that are larger than any single disk in
the network. HDFS architecture consists of NameNode,
DataNode, and HDFS Client.
Fig2: HDFS architecture[3]
C. Name node
The HDFS namespace is a hierarchy of files and directories.
Files and directories are represented on the NameNode[3] by
inodes, which record attributes like permissions, modifica-
tion and access times, namespace and disk space quotas. The
file content is split into large blocks and each block of the
file is independently replicated at multiple DataNodes. The
NameNode maintains the namespace tree and the mapping
of file blocks to DataNodes. An HDFS client wanting to read
a file first contacts the NameNode for the locations of data
blocks comprising the file and then reads block contents
from the DataNode closest to the client. When writing data,
the client requests the NameNode to nominate a suite of
three DataNodes to host the block replicas. The client then
writes data to the DataNodes in a pipeline fashion. The cur-
rent design has a single NameNode for each cluster. The
cluster can have thousands of DataNodes and tens of thou-
sands of HDFS clients per cluster, as each DataNode may
execute multiple application tasks concurrently. HDFS keeps
the entire namespace in RAM. The inode data and the list of
blocks belonging to each file comprise the metadata of the
name system called the image. The persistent record of the

3. International Journal of Innovatory research in Engineering and Technology - IJIRET
ISSN: XXXX-XXXX Volume X, Issue X, Month Year 23
image stored in the local host’s native files system is called a
checkpoint. The NameNode also stores the modification log
of the image called the journal in the local host’s native file
system. For improved durability, redundant copies of the
checkpoint and journal can be made at other servers. During
restarts the NameNode restores the namespace by reading
the namespace and replaying the journal. The locations of
block replicas may change over time and are not part of the
persistent checkpoint.
D.DATA NODE
Each block replica on a DataNode is represented by two files
in the local host’s native file system. The first file contains
the data itself and the second file is block’s metadata includ-
ing checksums for the block data and the block’s generation
stamp. The size of the data file equals the actual length of
the block and does not require extra space to round it up to
the nominal block size as in traditional file systems. Thus, if
a block is half full it needs only half of the space of the full
block on the local drive. During startup each DataNode con-
nects to the NameNode and performs a handshake. The pur-
pose of the handshake is to verify the namespace ID and the
software version of the DataNode. If either does not match
that of the NameNode the DataNode automatically shuts
down. The namespace ID is assigned to the file system in-
stance when it is formatted. The namespace ID is persistent-
ly stored on all nodes of the cluster. Nodes with a different
namespace ID will not be able to join the cluster, thus pre-
serving the integrity of the file system. The consistency of
software versions is important because incompatible version
may cause data corruption or loss, and on large clusters of
thousands of machines it is easy to overlook nodes that did
not shut down properly prior to the software upgrade or were
not available during the upgrade. A DataNode that is newly
initialized and without any namespace ID is permitted to join
the cluster and receive the cluster’s namespace ID. After the
handshake the DataNode registers with the NameNode.
DataNodes persistently store their unique storage IDs. The
storage ID is an internal identifier of the DataNode, which
makes it recognizable even if it is restarted with a different
IP address or port. The storage ID is assigned to the
DataNode when it registers with the NameNode for the first
time and never changes after that. A DataNode identifies
block replicas in its possession to the NameNode by sending
a block report. A block report contains the block id, the gen-
eration stamp and the length for each block replica the serv-
er hosts. The first block report is sent immediately after the
DataNode registration. Subsequent block reports are sent
every hour and provide the NameNode with an up-todate
view of where block replicas are located on the cluster.
D.HDFC CLIENT
User applications access the file system using the HDFS
client, a code library that exports the HDFS file system inter-
face. HDFS supports operations to read, write and delete
files, and operations to create and delete directories. When
an application reads a file, the HDFS client first asks the
NameNode for the list of DataNodes that host replicas of the
blocks of the file. It then contacts a DataNode directly and
requests the transfer of the desired block. When a client
writes, it first asks the NameNode to choose DataNodes to
host replicas of the first block of the file. When the first
block is filled, the client requests new DataNodes to be cho-
sen to host replicas of the next block. The interactions
among the client, the NameNode and the DataNodes are
illustrated in Fig.2
HDFS cluster has a single name node that manages the file
system namespace. The current limitation that a cluster can
contain only a single name node results in the following is-
sues:
1. Scalability:
Name node maintains the entire file system metadata in
memory. The size of the metadata is limited by the physical
memory available on the node. To address these issues one
encourages larger block sizes, creating a smaller number of
larger files and using tools like the hadoop archive (har).
2. Isolation:
No isolation for a multi‐tenant environment. An experi-
mental client application that puts high load on the central
name node can impact a production application.
3. Availability:
While the design does not prevent building a failover mech-
anism, when a failure occurs the entire namespace and hence
the entire cluster is down.
E.ADVANTAGE
1. Distribute data and computation. The computation
local to data prevents the network overload.
2. Simple programming model. The end user pro-
grammer only writes map-reduce tasks.
3. HDFS store large amount of information.
4. HDFS is simple and robust coherency model.
5. Data will be written to the HDFS once and then read
several times.
6. Fault tolerance by detecting faults and applying
quick, automatic recovery.
7. Ability to rapidly process large amounts of data in
parallel
8. Can be offered as an on-demand service, for exam-
ple as part of Amazon’s EC2 cluster computing service
E.LIMITATIONS
1. Rough manner:- Hadoop Map-reduce and HDFS
are rough in manner. Because the software under active de-
velopment.
2. Programming model is very restrictive:- Lack of
central data can be preventive.
3. Still single master which requires care and may
limit scaling.
4. Managing job flow isn’t trivial when intermediate
data should be kept.
5.Cluster management is hard:- In the cluster, opera-
tions like debugging, distributing software, collection logs
etc. are too hard.
F.CONCLUSION
We have seen the components of hadoop and the hadoop
distributed file system in brief. As compare to other file sys-

4. International Journal of Innovatory research in Engineering and Technology - IJIRET
ISSN: XXXX-XXXX Volume X, Issue X, Month Year 24
tem HDFS is a highly fault tolerance system. HDFS was its
single NameNode which handles all metadata operations.
G. REFERANCES
[1] Apache Hadoop. http://hadoop.apache.org/
[2] S.Ghemawat, H Gobioff, S. Leung. “The Google file
system,” In Proc. of ACM Symposium on Operating Sys-
tems Principles, Lake George, NY, Oct 2003, pp. 29–43.
[3]Konstantin Shvachko, et al. “The Hadoop Distributed File
System,” Mass Storage Systems and Technologies (MSST),
IEEE 26th Symposium on
IEEE,2010,http://storageconference.org/2010/Papers/MSST/
Shvachko.pdf.
[4] P.H. Carns, W. B. Ligon III, R. B. Ross, and R. Thakur.
“PVFS: A parallel file system for Linux clusters,” in Proc. of
4th Annual Linux Showcase and Conference, 2000, pp. 317–
327.
[5] J. Venner, Pro Hadoop. Apress, June 22, 2009.
[6].http://hadoop.apache.org/docs/r0.20.0/hdfs_design.html.