This presentation is based on a project for installing Apache Hadoop on a single node cluster along with Apache Hive for processing of structured data.

1. PROJECT ON
HADOOP
Created by –
Ronit Gaikwad – 1280017
Manan Kapadia -- 1280026
Jainesh Pathak – 1280041

2. INTRODUCTION
▪ Apache Hadoop is an open source framework for distributed storage and processing
of large sets of data on commodity hardware.
▪ It is a set of algorithms (an open-source software framework written in Java)
for distributed storage and distributed processing of very large data sets (Big Data)
on computer clusters built from commodity hardware.
▪ Instead of relying on expensive, proprietary hardware and different systems to store
and process data, Hadoop enables distributed parallel processing of huge amounts
of data across inexpensive, industry-standard servers that both store and process
the data, and can scale without limits.

3. KEY FEATURES
▪ Scalable.
▪ Cost effective.
▪ Flexible.
▪ Fast.
▪ Resilient to failure.
▪ Distributed Metadata.
▪ Low Latency.
▪ High Availability.

4. Hadoop Distributed File System (HDFS)
▪ The Hadoop Distributed File System (HDFS) is a distributed file system designed to
run on commodity hardware.
▪ HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware.
▪ HDFS provides high throughput access to application data and is suitable for
applications that have large data sets.

5. ARCHITECTURE

6. ARCHITECTURE
▪ HDFS has a master/slave architecture. An HDFS cluster consists of a single
NameNode, a master server that manages the file system namespace and regulates
access to files by clients.
▪ In addition, there are a number of DataNodes, usually one per node in the cluster,
which manage storage attached to the nodes that they run on.
▪ The NameNode executes file system namespace operations like opening, closing,
and renaming files and directories. It also determines the mapping of blocks to
DataNodes.
▪ The DataNodes are responsible for serving read and write requests from the file
system’s clients. The DataNodes also perform block creation, deletion, and
replication upon instruction from the NameNode.

7. MAPREDUCE
▪ MapReduce is a framework for writing applications that process large amounts of
structured and unstructured data in parallel across a cluster of thousands of
machines, in a reliable and fault-tolerant manner.
▪ A MapReduce job splits a large data set into independent chunks and organizes
them into key, value pairs for parallel processing.
▪ The Map function divides the input into ranges by the InputFormat and creates a
map task for each range in the input.
▪ The Reduce function then collects the various results and combines them to answer
the larger problem the master node was trying to solve.

8. APACHE HIVE
▪ The Apache Hive data warehouse software facilitates querying and managing large
datasets residing in distributed storage.
▪ Hive provides a mechanism to project structure onto this data and query the data
using a SQL-like language called HiveQL [with schema on read and transparently
converts queries to map/reduce.
▪ Apache Hive supports analysis of large datasets stored in Hadoop's HDFS and
compatible file systems such as Amazon S3 file system.

15. ADVANTAGES
 Storage of unstructured data such as audio files, video files, image files, pdf files
etc. into the HDFS which is achieved using commands
▪ The stored data can also be accessed and downloaded from the Web User Interface
for further processing on the clients machine.
▪ The user can also view all the information about the hadoop environment from the
web user interface.
▪ Complex mathematical calculations can be performed within seconds by typing in
the proper command and by using the concept of Mapreduce.

16. DISADVANTAGES
▪ The hadoop software is installed on commodity hardware. So there is no guarantee
that the hardware will work perfectly at all times which may lead to failing of the
hadoop cluster during critical moments.
▪ The hadoop software should only be used for very large and complex queries to get
the real power of hadoop in processing as when simple queries are used it leads to a
large amount of overhead which leads to degradation of performance.

17. CONCLUSION
▪ For big data to deliver on the promise of its vast potential, however, technology must
be in place to enable organizations to capture and store massive amounts of
unstructured data in its native format.
▪ That’s where Hadoop has become one of the enabling data processing technologies
for big data analytics.
▪ Hadoop allows companies to store and manage far larger volumes of structured and
unstructured data than can be managed affordably by today's relational database
management systems.

18. THANK YOU