The document provides an overview of Hadoop2 and describes its key components HDFS, YARN, and improvements over earlier versions. HDFS introduces federation and high availability to address limitations of single NameNode architecture. YARN improves on MapReduce by separating cluster resource management from application execution through a ResourceManager and per-application ApplicationMasters for better scalability and utilization.

1. Hadoop2 Overview
- Gagan Agrawal

2. Agenda
●
HDFS
●
●
●
HDFS Federation
HDFS High Availability
Map Reduce
●
YARN Architecture

3. HDFS

4. "Classical" HDFS

5. HDFS Main Limitations
●
Single NameNode that
●
●
●
Performs all metadata Operations
●
●
Keeps all metadata in RAM
Becomes a Single Point Of Failure (SPOF)
Number of files limited by the amount of RAM
of one machine
More RAM means also heavier GC

6. HDFS Main Limitations
●
Stats based on Yahoo! Cluster
●
●
An average file = 1.5 blocks (block size = 128 MB)
An average file = 600 bytes of metadata (1 file and
2 blocks objects)
●
100M files = 60GB of metadata
●
1 GB of metadata = 1 PB physical storage

7. HDFS Main Limitations
●
Read/Write operations throughput limited by
one machine
●
MapReduce tasks are also HDFS clients
●
Internal load increases as the cluster grows
●
●
Block Reports and heartbeats from DataNodes
Bigger snapshots transferred from/to Secondary
NameNode

8. HDFS Main Improvement
●
Introduce multiple NameNodes
●
HDFS Federation
●
HDFS High Availability (HA)

9. HDFS Federation

10. HDFS Federation

11. HDFS Federation

12. HDFS Federation

13. The Client View of Cluster
●
●
●
There are multiple namespaces and
corresponding NameNodes
Client can use any of them to compose its own
view of HDFS
Idea is much like Linux /etc/fstab file

14. Client Side Mount Tables

15. core-site.xml

16. Deploying HDFS Federation
●
Deploying HDFS Federation
●
●
NameNodes can be added/removed at any
time
●
●
Use for small(+isolation) and large(+scalability)
cluster
Cluster does have to be restarted
Add Federation to the existing cluster
●
Do not format existing NameNode
●
Newly added NameNode will be "empty"

17. HDFS Federation Benefits
●
Namespace scalability
●
Performance
●
Isolation

18. HDFS High Availability

19. HDFS High Availability
●
Introduces a pair of redundant NameNodes
●
●
One Active and one Standby
If the Active NameNode crashes or is
intentionally stopped,
●
Then the Standby takes over quickly

20. NameNode Responsibilities
●
●
The Active NameNode is responsible for all
client operations
The Standby NameNode is watchful
●
Maintains enough state to provide a fast
failover
●
Does also checkpointing (disappearance of
Secondary NN)

21. Synchronizing the state of metadata
●
●
●
The Standby must keep its state as up-to-date
as possible
Edit logs - two alternative ways of sharing them
● Shared storage using NFS
● Quorum-based storage (recommended)
Block locations
● DataNodes send block reports to both
NameNodes

22. Shared Storage Using NFS

24. hdfs-site.xml
<property>
<name>dfs.nameservices</name>
<value>mycluster</value>
</property>
<property>
<name>dfs.ha.namenodes.mycluster</name>
<value>nn1,nn2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.mycluster.nn1</name>
<value>machine1.example.com:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.mycluster.nn2</name>
<value>machine2.example.com:8020</value>
</property>

25. Supported Failover Modes
●
●
Manual fail-over
● Initiated by administrator by a dedicated
command
Automatic fail-over
● Detected by a fault in the active NN
● Implemented via Zookeeper

26. Manual Failover
●
Initiate a failover from the first NN to the second
one
hdfs haadmin -failover <tobestandbyNN>
<tobeactiveNN>

27. Automatic Failover

28. The Split Brain Scenario
●
A potential scenario when two NameNodes
●
●
●
Both think they are active
Both make conflicting changes to the namespace
Example: The ZKFC crashes, but its NN is still
running

29. Fencing
●
Ensure that the previous Active NameNode is
no longer able to make any changes to the
system metadata

30. Fencing with Shared Edit File
●
sshfence
●
●
ssh into the active namenode host and kill the
process listening on the service port
shell
●
Alows to perform arbitrarily complex fencing using
–
–
–
IPMI (Intelligent Platform Management Interface)
NFS filer level fencing
or other vendor specific functions

31. Fencing with Quorum-based storage
●
Only a single NN can be a writer to
JournalNodes at a time
●
●
●
Whenever NN becomes active, it generated an
“epoch number”
NN that has a higher “epoch number” can be a
writer
This prevents from corrupting the file system
metadata

32. HDFS Highly-Available Federated
Cluster
●
Any combination is possible
●
Federation without HA
●
HA without Federation
●
HA with Federation
●
e.g. NameNode HA for HBase, but not for the
others

33. HDFS Federation + HA

34. YARN
(YET ANOTHER RESOURCE NEGOTIATOR)

35. WHY YARN?

36. Hadoop Map Reduce Classic
●
Job Tracker
●
●
Manages cluster resources and job scheduling
Task Tracker
●
Per-node agent
●
Manage tasks

37. Hadoop Map Reduce Classic

38. Job Tracker Resposibilities
●
●
Manages the computational resources (map
and reduce slots)
Schedules all user jobs
● Schedules all tasks that belongs to a job
● Monitors tasks executions
● Restarts failed and speculatively runs slow
tasks
● Calculates job counters totals

39. MapReduce Classic: Limitations
●
Scalability
●
●
●
Maximum Cluster size – 4,000 nodes
Maximum concurrent tasks – 40,000
Availability
●
Job Tracker failure kills all queued and running jobs

40. MapReduce Classic: Limitations
●
Hard partition of resources into map and reduce
slots
●
●
Low resource utilization
Lacks support for alternate paradigms and
services

41. Poor Cluster Resources Utilization

42. Poor Cluster Resources Utilization

43. Job Tracker Redesign Idea
●
●
Reduce responsibilities of JobTracker
● Separate cluster resource management from
job coordination
● Use slaves (many of them!) to manage jobs
life-cycle
Scale to (at least) 10K nodes, 10K jobs, 100K
tasks

44. YARN : Hadoop as Next-Gen
Platform

45. YARN : Taking Hadoop Beyond
Batch
●
●
●
Store ALL DATA in one place
Interact with that data in MULTIPLE WAYS
With Predictable Performance and Quality of
Service

46. YARN : Concepts
●
Application
●
●
●
Application is a job submitted to the framework
Example – Map Reduce Job
Container
●
●
Basic unit of allocation
Fine-grained resource allocation across multiple
resource type (memory, cpu, disk, network, etc.)
–
–
●
container_0 = 2GB, 1 CPU
container_1 = 1GB, 6 CPU
Replaces the fixed map/reduce slots

47. YARN Architecture

48. YARN Architectue
●
Resource Manager
●
●
●
Global Resource Scheduler
Hierarchial queues
Node Manager
●
●
Manages the life-cycle of container
●
●
Per-machine agent
Container resource monitoring
Application Master
●
Per-application
●
Manages application scheduling and task execution
●
E.g MapReduce Application Master

49. YARN Architecture

50. YARN Design Consideration
●
Split up the two major functions of JobTracker
●
Cluster resource management
●
Application life-cycle management

51. 5 Key Benefits of YARN
●
●
●
●
●
Scale
New Programming Models and Services
Improved cluster utilization
Agility
Beyond Java

52. Thank You