HMS: Scalable Configuration Management System for Hadoop

Motivation■  Goal: managing Hadoop stack in a data center –  Multiple clusters, 10,000+ nodes, inter-cluster operations■  Scalability –  Traditional client/server architecture is cumbersome to scale –  Need to keep server states in sync, load balance client requests –  Fault tolerance adds further complexity■  Real-time interaction and feedback –  No poll or push, interact via asynchronous notification –  Visibility into cluster state is a major pain point for sys admins■  Cross-node ordering dependency –  Example: start JobTracker after NameNode is running –  Simple to specify and efficient to implement2 © 2011 IBM Corporation

ZooKeeper / /dir1 /dir2 Watch Ephemeral Queue •  A hierarchical namespace of znodes for storing data •  Sequential znodes for message queuing •  Watches for asynchronous notification •  Ephemeral znodes for failure detection4 © 2011 IBM Corporation

Leveraging ZooKeeper■  Storing cluster state –  Each cluster node is represented by a znode in ZooKeeper –  Node state is stored in its corresponding znode■  Storing system state –  Any state needed for failure recovery is persisted in ZooKeeper –  Server failures are detected via ephemeral nodes■  Distributed orchestration –  Messages are exchanged asynchronously via ZooKeeper queues –  Notifications are triggered by watches on queues■  Cross-node dependency –  Node states are stored in ZooKeeper and accessible to all –  Watches are registered on state znodes to get notified of state change5 © 2011 IBM Corporation

HMS Overview ZooKeeper cmdqueue cmd1 cmdStatus Status- watch queue action- action1 agent NameNode queue Controller1 worklog /hms clusters cluster1 Controller2 action- action1 agent JobTracker queue Client worklog cluster2 Controller1 live- controllers Controller26 © 2011 IBM Corporation

Design Implications■  All cluster state is stored in ZooKeeper –  Built-in fault-tolerance and HA –  Cluster and command status at your finger tips■  Controllers and agents don’t interact directly –  All communications are via ZooKeeper async notifications –  Good for scalability and fault-isolation■  Controllers and agents are stateless –  Controllers can be replicated for load balancing –  Controller failures are automatically detected and handled■  Dependency specified in terms of node states –  Actions come and go, but their effects are captured in node states –  Node state changes will trigger dependent actions to be re-evaluated 7 © 2011 IBM Corporation

Node List { "@url":"http://10.0.1.201:4080/v1/nodes/manifest/test", "roles":[ {"@name":"namenode","host":"host1"}, {"@name":"jobtracker","host":"host2"}, {...} ] } Roles map to hostnames8 © 2011 IBM Corporation

Package Manifest{ "@url": "http://10.0.1.201:4080/v1/software/stack/hadoop-1.0.3", "@name": "hadoop", "@version": "1.0.3", "roles": [ { "@name": "namenode", Define software stack by roles "package": [ { "name": "http://.../hadoop-1.0.3-1.x86_64.rpm" } ] }, { "@name": "datanode", "package": [ { "name": "http://.../hadoop-1.0.3-1.x86_64.rpm" Software download URL } ] } ]}9 © 2011 IBM Corporation

Configuration Plan{ “@url”: ”http://host/config/hadoop-1.0.3”, “@roles”: [ { “role” : “namenode”, “actions” : [ … ] }, { “role” : “jobtracker”, “actions” : [ … ] }, { “role” : “datanode” , “actions”, : [ … ] } ]} Run a list of scripts to configure NameNode10 © 2011 IBM Corporation

Start NameNode{ “@url”: ”http://host/config/hadoop-1.0.3”, “@roles”: [ { “role” : “namenode”, “actions” : [ … ] }, { “role” : “jobtracker”, “actions” : [ … ] }, { “role” : “datanode” , “actions”, : [ … ] } ] {} "@type": "scriptAction", "expectedResults": { "type": "DAEMON", "name": "hadoop-namenode", Define expected "status": "STARTED" result }, "script": "/usr/sbin/hadoop-setup-hdfs.sh", "parameters": [ "--format", "--hdfs-user=hdfs", "--mapreduce-user=mapred", "--namenode-host=${namenode}" ] }, Run a script to setup HDFS on namenode11 © 2011 IBM Corporation

Compiled Plan{ "startTime":"Thu Jun 07 13:11:29 {PDT 2012", "action":{ "status":"SUCCEEDED", "@action":"DaemonAction", "daemonName":"hadoop-namenode", "clusterName":"my-test-cluster", "actionId":1, "cmdPath":"/cmdqueue/cmd-0000000000", "actionEntries":[ "actionType":"start", "expectedResults":[ { ... }, { "name":"hadoop-namenode", { ... } "type":"DAEMON", "status":"STARTED" ], } ], "completedActions":6, "role":"namenode" }, "totalActions":6, "hostStatus":[ { "endTime":"Thu Jun 07 13:12:07 PDT 2012" "host":"bdvm021.svl.ibm.com", Host Status "status":"SUCCEEDED"} } ] },12 © 2011 IBM Corporation

Start JobTracker{ “@url”: ”http://host/config/hadoop-1.0.3”, “@roles”: [ { “role” : “namenode”, “actions” : [ … ] }, { “role” : “jobtracker”, “actions” : [ … ] }, { “role” : “datanode” , “actions”, : [ … ] } ] { "@type": "daemonAction",} "actionType": "start", "dependencies": { "states": { Dependency "type": "DAEMON", of namenode "name": "hadoop-namenode", "status": "STARTED" started }, "roles": "namenode" }, "expectedResults": { "type": "DAEMON", "name": "hadoop-jobtracker", "status": "STARTED" }, "daemon": "hadoop-jobtracker" }, Start JobTracker13 © 2011 IBM Corporation

Compiled Plan 2{ { "action":{ "startTime":"Thu Jun 07 13:11:29 PDT 2012", "@action":"DaemonAction", "daemonName":"hadoop-jobtracker", "status":"SUCCEEDED", "actionId":5, "cmdPath":"/cmdqueue/cmd-0000000000", "clusterName":"my-test-cluster", "actionType":"start", "dependencies":[ "actionEntries":[ { "roles":[ { ... }, "namenode" ], { ... } "hosts":[ "/clusters/my-test-cluster/bdvm022.svl.ibm.com" ], ], "states":[ "completedActions":6, { "name":"hadoop-namenode", "totalActions":6, "type":"DAEMON", "status":"STARTED" "endTime":"Thu Jun 07 13:12:07 PDT 2012" } ]} } ], "role":"jobtracker" }, }, Translated Dependency14 © 2011 IBM Corporation

Node State[zk: localhost:2181(CONNECTED) 5] get /hms/clusters/my-test-cluster/bdvm021.svl.ibm.com{ "states":[ { ... }, { … }, { "name":"hadoop-namenode", "type":"DAEMON", "status":"STARTED" } ]} State updates based on status reported by Agent 15 © 2011 IBM Corporation

Q&A•  HMS prototype is available on GitHub https://github.com/macroadster/hms•  Credits Kan Zhang (kzhang@apache.org) Eric Yang (eyang@apache.org) Jagane Sundar (jagane@apache.org)16 © 2011 IBM Corporation

HMS: Scalable Configuration Management System for Hadoop

by Hadoop_Summit on Jun 20, 2012

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 22

Statistics