Your SlideShare is downloading. ×
0
© Hortonworks Inc. 2013
HDFS NameNode HA
Jing Zhao
Hortonworks
Page 1
© Hortonworks Inc. 2013
Background and Motivation
• Prior to Hadoop 2.0
– One NameNode for metadata management
– Single po...
© Hortonworks Inc. 2013
Quorum Journal Management
• Before QJM
– NFS directory as shared storage
– Limitation: custom hard...
© Hortonworks Inc. 2013
Page 4
Architecting the Future of Big Data
DataNode DataNode DataNode
. . . . .
Standby
NameNode
A...
© Hortonworks Inc. 2013
Automatic Failover
• Manual failover
– Administrative commands
• Use ZKFailoverController for Auto...
© Hortonworks Inc. 2013
Page 6
Architecting the Future of Big Data
DataNode DataNode DataNode
. . . . .
Standby
NameNode
W...
Upcoming SlideShare
Loading in...5
×

Ambari Meetup: NameNode HA

1,201

Published on

Published in: Technology, Business

Transcript of "Ambari Meetup: NameNode HA"

  1. 1. © Hortonworks Inc. 2013 HDFS NameNode HA Jing Zhao Hortonworks Page 1
  2. 2. © Hortonworks Inc. 2013 Background and Motivation • Prior to Hadoop 2.0 – One NameNode for metadata management – Single point of failure • HDFS High Availability – Two NameNodes in the same cluster – Active NameNode: responsive for all client operations – Standby NameNode: maintain enough state to provide a fast failover • Shared storage – Active NN writes edit log – Standby NN reads edit log and applies to its own namespace – During failover, Standby NN reads all the edits and transitions to Active state Page 2 Architecting the Future of Big Data
  3. 3. © Hortonworks Inc. 2013 Quorum Journal Management • Before QJM – NFS directory as shared storage – Limitation: custom hardware, complicated deployment… • QJM – Quorum commits of edit log to separate daemons (JournalNodes) – Paxos-like protocol to guarantee correctness • JournalNodes – A successful edit log commit must write to a majority of JNs – Lightweight, can be co-located on machines for NN, JT, or RM – Recommended number of JNs: 3 or 5 Page 3 Architecting the Future of Big Data
  4. 4. © Hortonworks Inc. 2013 Page 4 Architecting the Future of Big Data DataNode DataNode DataNode . . . . . Standby NameNode Active NameNode Shared Edits JournalNode JournalNode JournalNode Block Reports to Active & Standby Write Read
  5. 5. © Hortonworks Inc. 2013 Automatic Failover • Manual failover – Administrative commands • Use ZKFailoverController for Automatic Failover – ZKFC: a ZooKeeper client that monitors and manages the NN state – Health monitoring – ZooKeeper session management – ZooKeeper-base election Page 5 Architecting the Future of Big Data
  6. 6. © Hortonworks Inc. 2013 Page 6 Architecting the Future of Big Data DataNode DataNode DataNode . . . . . Standby NameNode Write Active NameNode Shared Edits JournalNode JournalNode JournalNode ZKFC ZKFC ZooKeeper Service Cluster Monitor and try to take active lock Monitor liveness and health. Manage HA state. Write Read Monitor liveness and health. Manage HA state. Monitor and maintain active lock
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×