Your SlideShare is downloading. ×
Ambari Meetup: NameNode HA
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Ambari Meetup: NameNode HA

1,086
views

Published on

Published in: Technology, Business

0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,086
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
78
Comments
0
Likes
7
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. © Hortonworks Inc. 2013 HDFS NameNode HA Jing Zhao Hortonworks Page 1
  • 2. © Hortonworks Inc. 2013 Background and Motivation • Prior to Hadoop 2.0 – One NameNode for metadata management – Single point of failure • HDFS High Availability – Two NameNodes in the same cluster – Active NameNode: responsive for all client operations – Standby NameNode: maintain enough state to provide a fast failover • Shared storage – Active NN writes edit log – Standby NN reads edit log and applies to its own namespace – During failover, Standby NN reads all the edits and transitions to Active state Page 2 Architecting the Future of Big Data
  • 3. © Hortonworks Inc. 2013 Quorum Journal Management • Before QJM – NFS directory as shared storage – Limitation: custom hardware, complicated deployment… • QJM – Quorum commits of edit log to separate daemons (JournalNodes) – Paxos-like protocol to guarantee correctness • JournalNodes – A successful edit log commit must write to a majority of JNs – Lightweight, can be co-located on machines for NN, JT, or RM – Recommended number of JNs: 3 or 5 Page 3 Architecting the Future of Big Data
  • 4. © Hortonworks Inc. 2013 Page 4 Architecting the Future of Big Data DataNode DataNode DataNode . . . . . Standby NameNode Active NameNode Shared Edits JournalNode JournalNode JournalNode Block Reports to Active & Standby Write Read
  • 5. © Hortonworks Inc. 2013 Automatic Failover • Manual failover – Administrative commands • Use ZKFailoverController for Automatic Failover – ZKFC: a ZooKeeper client that monitors and manages the NN state – Health monitoring – ZooKeeper session management – ZooKeeper-base election Page 5 Architecting the Future of Big Data
  • 6. © Hortonworks Inc. 2013 Page 6 Architecting the Future of Big Data DataNode DataNode DataNode . . . . . Standby NameNode Write Active NameNode Shared Edits JournalNode JournalNode JournalNode ZKFC ZKFC ZooKeeper Service Cluster Monitor and try to take active lock Monitor liveness and health. Manage HA state. Write Read Monitor liveness and health. Manage HA state. Monitor and maintain active lock