This document provides an introduction to Hadoop and big data concepts. It discusses what big data is, the four V's of big data (volume, velocity, variety, and veracity), different data types (structured, semi-structured, unstructured), how data is generated, and the Apache Hadoop framework. It also covers core Hadoop components like HDFS, YARN, and MapReduce, common Hadoop users, the difference between Hadoop and RDBMS systems, Hadoop cluster modes, the Hadoop ecosystem, HDFS daemons and architecture, and basic Hadoop commands.

1. Introduction to
Hadoop

2. Big Data
Overview
• Big Data is a collection of data that
is huge in volume yet growing
exponentially with time.
• It is a data with so large size and
complexity that none of
traditional data management tools
can store it or process it efficiently.
• It cannot be processed using
traditional computing techniques.
• Big data is also a data but
with huge size.

3. The 4 V’s in Big Data
• Volume of Big Data
• The volume of data refers to the size of the data sets that need to be analyzed and processed, which are now frequently
larger than terabytes and petabytes. In other words, this means that the data sets in Big Data are too large to process
with a regular laptop or desktop processor. An example of a high-volume data set would be all credit card transactions
on a day within Europe.
• Velocity of Big Data
• Velocity refers to the speed with which data is generated. An example of a data that is generated with high velocity
would be Twitter messages or Facebook posts.
• Variety of Big Data
• Variety makes Big Data really big. Big Data comes from a great variety of sources and generally is one out of three
types: structured, semi structured and unstructured data. An example of high variety data sets would be the CCTV audio
and video files that are generated at various locations in a city.
• Veracity of Big Data
• Veracity refers to the quality of the data that is being analyzed. High veracity data has many records that are valuable to
analyze and that contribute in a meaningful way to the overall results. Low veracity data, on the other hand, contains a
high percentage of meaningless data. The non-valuable in these data sets is referred to as noise. An example of a high
veracity data set would be data from a medical experiment or trial.

4. Types of Big
Data
• Structured data:-Structured data is data that has been
predefined and formatted to a set structure before being
placed in data storage, The best example of structured
data is the relational database: the data has been formatted
into precisely defined fields, such as credit card numbers or
address, in order to be easily queried with SQL.
• Semi-Structured :-Semi-structured data is a form
of structured data that does not obey the tabular structure
of data models associated with relational databases or other
forms of data tables, but nonetheless contains tags or other
markers to separate semantic elements and enforce
hierarchies of records and fields within the data.
• Unstructured data:-Unstructured data is data stored in its
native format and not processed until it is used, which is
known as schema-on-read. It comes in a myriad of file
formats, including email, social media posts, presentations,
chats, IoT sensor data, and satellite imagery.

5. How Big Data
is generated
• The bulk of big data
generated comes from
three primary sources:
social data,
machine data and
transactional data.

6. Apache Hadoop
Framework
• Apache Hadoop is a collection of
open-source software utilities that
facilitates using a network of
many computers to solve
problems involving massive
amounts of data and
computation. It provides a
software framework for distributed
storage and processing of big
data using the MapReduce
programming model.

7. Core components of
Hadoop
Hadoop Distributed File
System (HDFS): the storage
system for Hadoop spread
out over multiple machines
as a means to reduce cost
and increase reliability.
MapReduce engine: the
algorithm that filters, sorts
and then uses the database
input in some way.

8. Some
Hadoop
Users

9. Difference
Between
Hadoop and
RDBMS

10. Cluster Modes in Hadoop
• Hadoop Mainly works on 3
different Modes:
1.Standalone Mode
2.Pseudo-distributed Mode
3.Fully-Distributed Mode

11. Hadoop Ecosystem
Introduction: Hadoop Ecosystem is a platform or a suite which provides
various services to solve the big data problems. It includes Apache projects
and various commercial tools and solutions. There are four major elements
of Hadoop i.e., HDFS, MapReduce, YARN, and Hadoop Common. Most
of the tools or solutions are used to supplement or support these major
elements. All these tools work collectively to provide services such as
absorption, analysis, storage and maintenance of data etc.
Following are the components that collectively form a Hadoop ecosystem:
•HDFS: Hadoop Distributed File System
•YARN: Yet Another Resource Negotiator
•MapReduce: Programming based Data Processing
•Spark: In-Memory data processing
•PIG, HIVE: Query based processing of data services
•HBase: NoSQL Database
•Mahout, Spark MLLib: Machine Learning algorithm libraries
•Solar, Lucene: Searching and Indexing
•Zookeeper: Managing cluster
•Oozie: Job Scheduling

12. HDFS Daemons
and MapReduce
Daemons

13. Hadoop Cluster Architecture
• A hadoop cluster architecture
consists of a data centre, rack and the
node that actually executes the jobs.
Data centre consists of the racks and
racks consists of nodes. A medium to
large cluster consists of a two or three
level hadoop cluster architecture that
is built with rack mounted servers.
Every rack of servers is interconnected
through 1 gigabyte of Ethernet (1
GigE). Each rack level switch in a
hadoop cluster is connected to a
cluster level switch which are in turn
connected to other cluster level
switches or they uplink to other
switching infrastructure.

14. Hadoop Distributed File
System
• The Hadoop Distributed File System (HDFS) is the
primary data storage system used
by Hadoop applications. It employs a NameNode and
DataNode architecture to implement a distributed file
system that provides high-performance access to data
across highly scalable Hadoop clusters.
• With the Hadoop Distributed File system, the data is
written once on the server and subsequently read and
re-used many times thereafter.
• The NameNode also manages access to the files,
including reads, writes, creates, deletes and replication
of data blocks across different data nodes

15. HDFS Components
• Hadoop cluster consists of three components -
• Master Node – Master node in a hadoop cluster is responsible for storing data in HDFS and
executing parallel computation the stored data using MapReduce. Master Node has 3 nodes –
NameNode, Secondary NameNode and JobTracker. JobTracker monitors the parallel processing of
data using MapReduce while the NameNode handles the data storage function with HDFS.
NameNode keeps a track of all the information on files (i.e. the metadata on files) such as the
access time of the file, which user is accessing a file on current time and which file is saved in which
hadoop cluster. The secondary NameNode keeps a backup of the NameNode data.
• Slave/Worker Node- This component in a hadoop cluster is responsible for storing the data and
performing computations. Every slave/worker node runs both a TaskTracker and a DataNode
service to communicate with the Master node in the cluster. The DataNode service is secondary to
the NameNode and the TaskTracker service is secondary to the JobTracker.
• Client Nodes – Client node has hadoop installed with all the required cluster configuration settings
and is responsible for loading all the data into the hadoop cluster. Client node submits mapreduce
jobs describing on how data needs to be processed and then the output is retrieved by the client
node once the job processing is completed.

16. HDFS Architecture
• Hadoop follows a master slave architecture
design for data storage and distributed data
processing using HDFS and MapReduce
respectively. The master node for data
storage is hadoop HDFS is the NameNode
and the master node for parallel processing
of data using Hadoop MapReduce is the Job
Tracker. The slave nodes in the hadoop
architecture are the other machines in the
Hadoop cluster which store data and perform
complex computations. Every slave node has
a Task Tracker daemon and a DataNode that
synchronizes the processes with the Job
Tracker and NameNode respectively. In
Hadoop architectural implementation the
master or slave systems can be setup in the
cloud or on-premise.

17. HDFS Read File
• Step 1: The client opens the file it wishes to read by calling open() on the File
System Object(which for HDFS is an instance of Distributed File System).
• Step 2: Distributed File System( DFS) calls the name node, using remote
procedure calls (RPCs), to determine the locations of the first few blocks in the
file. For each block, the name node returns the addresses of the data nodes that
have a copy of that block. The DFS returns an FSDataInputStream to the client
for it to read data from. FSDataInputStream in turn wraps a DFSInputStream,
which manages the data node and name node I/O.
• Step 3: The client then calls read() on the stream. DFSInputStream, which has
stored the info node addresses for the primary few blocks within the file, then
connects to the primary (closest) data node for the primary block in the file.
• Step 4: Data is streamed from the data node back to the client, which calls
read() repeatedly on the stream.
• Step 5: When the end of the block is reached, DFSInputStream will close the
connection to the data node, then finds the best data node for the next block.
This happens transparently to the client, which from its point of view is simply
reading an endless stream. Blocks are read as, with the DFSInputStream
opening new connections to data nodes because the client reads through the
stream. It will also call the name node to retrieve the data node locations for the
next batch of blocks as needed.
• Step 6: When the client has finished reading the file, a function is called, close()
on the FSDataInputStream.

18. HDFS Write File
• Step 1: The client creates the file by calling create() on
DistributedFileSystem(DFS).
• Step 2: DFS makes an RPC call to the name node to create a new file in the file
system’s namespace, with no blocks associated with it. The name node performs
various checks to make sure the file doesn’t already exist and that the client has
the right permissions to create the file. If these checks pass, the name node
prepares a record of the new file; otherwise, the file can’t be created and therefore
the client is thrown an error i.e. IOException. The DFS returns an
FSDataOutputStream for the client to start out writing data to.
• Step 3: Because the client writes data, the DFSOutputStream splits it into packets,
which it writes to an indoor queue called the info queue. The data queue is
consumed by the DataStreamer, which is liable for asking the name node to
allocate new blocks by picking an inventory of suitable data nodes to store the
replicas. The list of data nodes forms a pipeline, and here we’ll assume the
replication level is three, so there are three nodes in the pipeline. The
DataStreamer streams the packets to the primary data node within the pipeline,
which stores each packet and forwards it to the second data node within the
pipeline.
• Step 4: Similarly, the second data node stores the packet and forwards it to the
third (and last) data node in the pipeline.
• Step 5: The DFSOutputStream sustains an internal queue of packets that are
waiting to be acknowledged by data nodes, called an “ack queue”.
• Step 6: This action sends up all the remaining packets to the data node pipeline
and waits for acknowledgments before connecting to the name node to signal
whether the file is complete or not.

19. Some Basic
Hadoop
Commands

20. • cat
HDFS Command that reads a file on HDFS and prints the content of that file to the
standard output.
Command: hdfs dfs –cat /new_file/test
• text
HDFS Command that takes a source file and outputs the file in text format.
Command: hdfs dfs –text /new_file/test
• copyFromLocal
HDFS Command to copy the file from a Local file system to HDFS.
Command: hdfs dfs –copyFromLocal /home/file/test /new_file
• copyToLocal
HDFS Command to copy the file from HDFS to Local File System.
Command: hdfs dfs –copyToLocal /file/test /home/file

21. • put
HDFS Command to copy single source or multiple sources from local file system to the destination file system.
Command: hdfs dfs –put /home/file/test /user
• get
HDFS Command to copy files from hdfs to the local file system.
Command: hdfs dfs –get /user/test /home/file
• count
HDFS Command to count the number of directories, files, and bytes under the paths that match the specified file pattern.
Command: hdfs dfs –count /user
• rm
HDFS Command to remove the file from HDFS.
Command: hdfs dfs –rm /new_file/test
• rm -r
HDFS Command to remove the entire directory and all of its content from HDFS.
Command: hdfs dfs -rm -r /new_file

22. • cp
HDFS Command to copy files from source to destination. This command allows multiple
sources as well, in which case the destination must be a directory.
Command: hdfs dfs -cp /user/hadoop/file1 /user/hadoop/file2
• mv
HDFS Command to move files from source to destination. This command allows multiple
sources as well, in which case the destination needs to be a directory.
Command: hdfs dfs -mv /user/hadoop/file1 /user/hadoop/file2
• rmdir
HDFS Command to remove the directory.
Command: hdfs dfs –rmdir /user/Hadoop
• help
HDFS Command that displays help for given command or all commands if none is
specified.
Command: hdfs dfs -help