Below are the topics covered in this tutorial: Hadoop Architecture Secondary NameNode & Checkpointing NameNode Availability NameNode Failover Mechanism HDFS HA Architecture Backup

1. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
Webpage

2. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
Agenda
❖ Hadoop Architecture
❖ Secondary NameNode & Checkpointing
❖ NameNode Availability
❖ NameNode Failover Mechanism
❖ HDFS HA Architecture
❖ Backup
❖ Security

3. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
Edureka’s Hadoop Admin Course Curriculum
❖ Understanding Big Data and Hadoop
❖ Hadoop Architecture and Cluster setup
❖ Hadoop cluster Administration & Understanding MapReduce
❖ Backup, Recovery and Maintenance
❖ Hadoop 2.0 Cluster: Planning and Management
❖ Setting up Hadoop 2.X with High Availability and upgrading
Hadoop
❖ Project: Cloudera manager and Cluster setup, Overview on
Kerberos

4. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
HDFS Architecture

5. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
HDFS Architecture
NameNode Metadata
DataNode1 DataNode2 DataNode3
client
NameNode manages all the data nodes and
maintain all the metadata information
NameNode receives heartbeat and block
report from all the DataNodes
Clients first contacts the NameNode for file
metadata & then perform actual file I/O
directly with the DataNodes
Secondary
NameNode

6. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
Hadoop Admin Job Responsibilities
Responsible for implementation and
support of the Enterprise Hadoop
environment
Responsible for implementation and
support of the Enterprise Hadoop
environment
Need to implement concepts of Hadoop
eco system such as YARN, MapReduce,
HDFS, HBase, Zookeeper, Pig and Hive
Manage, monitor and analyze Hadoop File
System & Log files. Also responsible for
Security measures

7. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
Secondary NameNode
and
Checkpointing

8. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
Secondary NameNode & Checkpointing
Secondary
NameNode
NameNode
editLog editLog
fsImage fsImage
editLog
(new)
FsImage
(final)
First time copy
Temporary
During checkpoint
➢ Checkpointing is a process of combining
edit logs with FsImage
➢ Secondary NameNode takes over the
responsibility of checkpointing, therefore,
making NameNode more available
➢ Allows faster Failover as it prevents edit
logs from getting too huge
➢ Checkpointing happens periodically
(default: 1 hour)

9. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
Secondary NameNode & Checkpointing
local directory where the
temporary edits to be
merge are stored
number of seconds
between two periodic
checkpoints.
Checkpointing Related properties in hdfs-site.xml:

10. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
Secondary NameNode & Checkpointing
Manual or Forced Checkpointing:
1. Save the latest metadata to FsImage on the Master Node:
➢ hdfs dfsadmin -safemode enter
➢ hdfs dfsadmin -saveNamespace
➢ hdfs dfsadmin -safemode leave
2. Run the manual checkpointing on the Secondary NameNode:
➢ hdfs secondarynamenode –checkpoint force

11. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
NameNode Availability
(Single Point of Failure)

12. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
NameNode Availability
Availability of NameNode means we need
NameNode to be always up and running or
available for executing any Hadoop jobs
Secondary
NameNode NameNode
DN1 DN2 DN3
In a standard HDFS configuration, NameNode
becomes a Single Point of Failure i.e. once
NameNode crashes whole cluster becomes
unavailable
Planned event:
Maintenance work like software or
hardware upgradation
Unplanned Event:
NameNode crashes because of
Hardware Failure

13. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
NameNode Failover

14. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
NameNode Failover
New NameNode loads the file
system namespace image into
memory
1
It replays all the or edits
transactions in the edit log to
catch up to the most recent
state of the NameSystem
2
Leaves the safe mode
once it has received
enough block reports from
the datanodes
3

15. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
HDFS HA Architecture
(Faster Failover)

16. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
HDFS HA Architecture
Two NameNodes running at
the same time:
➢ Active NameNode
➢ Standby NameNode
DN1 DN2 DN3
I will take
CONTROL
CRASH !
In case of a NameNode
(active) failover, other
NameNode (Standby) takes
over responsibility

17. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
Shared Storage

18. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
Standby
NameNode
Shared storage device that provide
access to both the NameNode for
storing namespace modification
Group of separate lightweight
daemons that logs record of any
namespace modification
HDFS HA Architecture
Active
NameNode
Shared Storage
sync sync
Active NameNode and Standby NameNode keep their state in sync with each other using shared storage
Quorum Journal Nodes
NFS
(Network File System)
Shared Storage
Implementation
1 2

19. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
Shared Storage Implementation
using Journal Nodes

20. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
HDFS HA Architecture

21. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
HDFS HA Architecture

22. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
Backup

23. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
Backup and Recovery
Some Useful Commands:
To check Status of Cluster and details of data
nodes
1 hadoop dfsadmin -report
To check the list of files/directories on HDFS2 hadoop fs –ls <dir_path>
➢ To check block information
➢ To check blocks and files information
3
hdfs fsck <path> -blocks
hdfs fsck <path> -files

24. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
Backup

25. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
Backup
S O L U T I O N F O R D A T A B A C K U P
Distributed Copy Using Flume
➢ hadoop distcp hdsf://<source NN> hdfs://<target NN>
distcp
Cluster 1 Cluster 2
Cluster 1 Cluster 2
Flume
Parallel Data
Ingestion
➢ Ingesting Data Using Flume

26. www.edureka.co/hadoop-adminEDUREKA HADOOP ADMINISTRATION CERTIFICATION TRAINING
• Hadoop Administration with Edureka
M A S T E R H A D O O P A D M I N I S T R AT I O N W I T H E D U R E K A
Go to www.edureka.co/hadoop-admin

27. EDUREKA ANGULAR 2 CERTIFICATION TRAINING www.edureka.co/angular-js
Thank You …
Questions/Queries/Feedback