HMS: Scalable    Configuration Management System     for Hadoop               Kan Zhang (IBM)               Eric Yang (IBM...
Motivation■    Goal: managing Hadoop stack in a data center     –  Multiple clusters, 10,000+ nodes, inter-cluster operati...
HMS Approach     Controller                                          Agent                            ZooKeeper       Stor...
ZooKeeper                                        /                            /dir1                /dir2                  ...
Leveraging ZooKeeper■    Storing cluster state     –  Each cluster node is represented by a znode in ZooKeeper     –  Node...
HMS Overview                  ZooKeeper                                cmd-                               queue           ...
Design Implications■    All cluster state is stored in ZooKeeper     –  Built-in fault-tolerance and HA     –  Cluster and...
Node List    {        "@url":"http://10.0.1.201:4080/v1/nodes/manifest/test",        "roles":[          {"@name":"namenode...
Package Manifest{    "@url": "http://10.0.1.201:4080/v1/software/stack/hadoop-1.0.3",    "@name": "hadoop",    "@version":...
Configuration Plan{    “@url”: ”http://host/config/hadoop-1.0.3”,    “@roles”: [      { “role” : “namenode”, “actions” : [...
Start NameNode{    “@url”: ”http://host/config/hadoop-1.0.3”,    “@roles”: [      { “role” : “namenode”, “actions” : [ … ]...
Compiled Plan{     "startTime":"Thu Jun 07 13:11:29 {PDT 2012",                                        "action":{     "sta...
Start JobTracker{    “@url”: ”http://host/config/hadoop-1.0.3”,    “@roles”: [      { “role” : “namenode”, “actions” : [ …...
Compiled Plan 2{                                                {                                                         ...
Node State[zk: localhost:2181(CONNECTED) 5] get /hms/clusters/my-test-cluster/bdvm021.svl.ibm.com{  "states":[    { ... },...
Q&A•  HMS prototype is available on GitHub     https://github.com/macroadster/hms•  Credits     Kan Zhang (kzhang@apache.o...
Upcoming SlideShare
Loading in …5
×

HMS: Scalable Configuration Management System for Hadoop

2,984 views

Published on

Published in: Technology, Business
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,984
On SlideShare
0
From Embeds
0
Number of Embeds
69
Actions
Shares
0
Downloads
0
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

HMS: Scalable Configuration Management System for Hadoop

  1. 1. HMS: Scalable Configuration Management System for Hadoop Kan Zhang (IBM) Eric Yang (IBM)June 5, 2012 © 2011 IBM Corporation
  2. 2. Motivation■  Goal: managing Hadoop stack in a data center –  Multiple clusters, 10,000+ nodes, inter-cluster operations■  Scalability –  Traditional client/server architecture is cumbersome to scale –  Need to keep server states in sync, load balance client requests –  Fault tolerance adds further complexity■  Real-time interaction and feedback –  No poll or push, interact via asynchronous notification –  Visibility into cluster state is a major pain point for sys admins■  Cross-node ordering dependency –  Example: start JobTracker after NameNode is running –  Simple to specify and efficient to implement2 © 2011 IBM Corporation
  3. 3. HMS Approach Controller Agent ZooKeeper Storage ■  ZooKeeper plays a central role –  Fault-tolerant and scalable storage –  Asynchronous messaging service3 © 2011 IBM Corporation
  4. 4. ZooKeeper / /dir1 /dir2 Watch Ephemeral Queue •  A hierarchical namespace of znodes for storing data •  Sequential znodes for message queuing •  Watches for asynchronous notification •  Ephemeral znodes for failure detection4 © 2011 IBM Corporation
  5. 5. Leveraging ZooKeeper■  Storing cluster state –  Each cluster node is represented by a znode in ZooKeeper –  Node state is stored in its corresponding znode■  Storing system state –  Any state needed for failure recovery is persisted in ZooKeeper –  Server failures are detected via ephemeral nodes■  Distributed orchestration –  Messages are exchanged asynchronously via ZooKeeper queues –  Notifications are triggered by watches on queues■  Cross-node dependency –  Node states are stored in ZooKeeper and accessible to all –  Watches are registered on state znodes to get notified of state change5 © 2011 IBM Corporation
  6. 6. HMS Overview ZooKeeper cmd- queue cmd1 cmdStatus Status- watch queue action- action1 agent NameNode queue Controller1 worklog /hms clusters cluster1 Controller2 action- action1 agent JobTracker queue Client worklog cluster2 Controller1 live- controllers Controller26 © 2011 IBM Corporation
  7. 7. Design Implications■  All cluster state is stored in ZooKeeper –  Built-in fault-tolerance and HA –  Cluster and command status at your finger tips■  Controllers and agents don’t interact directly –  All communications are via ZooKeeper async notifications –  Good for scalability and fault-isolation■  Controllers and agents are stateless –  Controllers can be replicated for load balancing –  Controller failures are automatically detected and handled■  Dependency specified in terms of node states –  Actions come and go, but their effects are captured in node states –  Node state changes will trigger dependent actions to be re-evaluated 7 © 2011 IBM Corporation
  8. 8. Node List { "@url":"http://10.0.1.201:4080/v1/nodes/manifest/test", "roles":[ {"@name":"namenode","host":"host1"}, {"@name":"jobtracker","host":"host2"}, {...} ] } Roles map to hostnames8 © 2011 IBM Corporation
  9. 9. Package Manifest{ "@url": "http://10.0.1.201:4080/v1/software/stack/hadoop-1.0.3", "@name": "hadoop", "@version": "1.0.3", "roles": [ { "@name": "namenode", Define software stack by roles "package": [ { "name": "http://.../hadoop-1.0.3-1.x86_64.rpm" } ] }, { "@name": "datanode", "package": [ { "name": "http://.../hadoop-1.0.3-1.x86_64.rpm" Software download URL } ] } ]}9 © 2011 IBM Corporation
  10. 10. Configuration Plan{ “@url”: ”http://host/config/hadoop-1.0.3”, “@roles”: [ { “role” : “namenode”, “actions” : [ … ] }, { “role” : “jobtracker”, “actions” : [ … ] }, { “role” : “datanode” , “actions”, : [ … ] } ]} Run a list of scripts to configure NameNode10 © 2011 IBM Corporation
  11. 11. Start NameNode{ “@url”: ”http://host/config/hadoop-1.0.3”, “@roles”: [ { “role” : “namenode”, “actions” : [ … ] }, { “role” : “jobtracker”, “actions” : [ … ] }, { “role” : “datanode” , “actions”, : [ … ] } ] {} "@type": "scriptAction", "expectedResults": { "type": "DAEMON", "name": "hadoop-namenode", Define expected "status": "STARTED" result }, "script": "/usr/sbin/hadoop-setup-hdfs.sh", "parameters": [ "--format", "--hdfs-user=hdfs", "--mapreduce-user=mapred", "--namenode-host=${namenode}" ] }, Run a script to setup HDFS on namenode11 © 2011 IBM Corporation
  12. 12. Compiled Plan{ "startTime":"Thu Jun 07 13:11:29 {PDT 2012", "action":{ "status":"SUCCEEDED", "@action":"DaemonAction", "daemonName":"hadoop-namenode", "clusterName":"my-test-cluster", "actionId":1, "cmdPath":"/cmdqueue/cmd-0000000000", "actionEntries":[ "actionType":"start", "expectedResults":[ { ... }, { "name":"hadoop-namenode", { ... } "type":"DAEMON", "status":"STARTED" ], } ], "completedActions":6, "role":"namenode" }, "totalActions":6, "hostStatus":[ { "endTime":"Thu Jun 07 13:12:07 PDT 2012" "host":"bdvm021.svl.ibm.com", Host Status "status":"SUCCEEDED"} } ] },12 © 2011 IBM Corporation
  13. 13. Start JobTracker{ “@url”: ”http://host/config/hadoop-1.0.3”, “@roles”: [ { “role” : “namenode”, “actions” : [ … ] }, { “role” : “jobtracker”, “actions” : [ … ] }, { “role” : “datanode” , “actions”, : [ … ] } ] { "@type": "daemonAction",} "actionType": "start", "dependencies": { "states": { Dependency "type": "DAEMON", of namenode "name": "hadoop-namenode", "status": "STARTED" started }, "roles": "namenode" }, "expectedResults": { "type": "DAEMON", "name": "hadoop-jobtracker", "status": "STARTED" }, "daemon": "hadoop-jobtracker" }, Start JobTracker13 © 2011 IBM Corporation
  14. 14. Compiled Plan 2{ { "action":{ "startTime":"Thu Jun 07 13:11:29 PDT 2012", "@action":"DaemonAction", "daemonName":"hadoop-jobtracker", "status":"SUCCEEDED", "actionId":5, "cmdPath":"/cmdqueue/cmd-0000000000", "clusterName":"my-test-cluster", "actionType":"start", "dependencies":[ "actionEntries":[ { "roles":[ { ... }, "namenode" ], { ... } "hosts":[ "/clusters/my-test-cluster/bdvm022.svl.ibm.com" ], ], "states":[ "completedActions":6, { "name":"hadoop-namenode", "totalActions":6, "type":"DAEMON", "status":"STARTED" "endTime":"Thu Jun 07 13:12:07 PDT 2012" } ]} } ], "role":"jobtracker" }, }, Translated Dependency14 © 2011 IBM Corporation
  15. 15. Node State[zk: localhost:2181(CONNECTED) 5] get /hms/clusters/my-test-cluster/bdvm021.svl.ibm.com{ "states":[ { ... }, { … }, { "name":"hadoop-namenode", "type":"DAEMON", "status":"STARTED" } ]} State updates based on status reported by Agent 15 © 2011 IBM Corporation
  16. 16. Q&A•  HMS prototype is available on GitHub https://github.com/macroadster/hms•  Credits Kan Zhang (kzhang@apache.org) Eric Yang (eyang@apache.org) Jagane Sundar (jagane@apache.org)16 © 2011 IBM Corporation

×