Whirr dev-up-puppetconf2011

Clustered Services withApache Whirr: From Dev Up @jclouds

agendaintroconceptsjcloudswhirr

introAdrian Cole (@jclouds) founded jclouds march 2009 member of the Apache Whirr project chief evangelist at Cloudsoft

so you want to build a cluster.. Web REST HTable Browser ClientPublic UI RPC UI RPC UI RPC HTTP Master RegionServer RegionServer RestServer listen listen listen listenPrivate

First, you need to analyzethe roles you need in thecluster hadoop- hadoop- hbase- zookeeper namenode datanode master hadoop- hadoop- hbase- jobtracker tasktracker regionserver

Next, organize andunderstand the topology hadoop- tasktracker hadoop- zookeeper namenode hadoop- datanode hadoop- hbase- jobtracker master hbase- regionserver

Then you need to developthe workflow provisionbootstrap install conﬁgureoperate manage

provisionPublic hbase-master hbase-regionserver hbase-regionserver hbase-restserver Ubuntu 10.04 Ubuntu 10.04 Ubuntu 10.04 Ubuntu 10.04 8GB RAM 8GB RAM 8GB RAM 2GB RAM 8 Cores 8 Cores 8 Cores 1 CPUPrivate

installPublic hbase-master hbase-regionserver hbase-regionserver hbase-restserver login: hbase login: hbase login: hbase login: hbase java: 1.6.0_16 java: 1.6.0_16 java: 1.6.0_16 java: 1.6.0_16 hbase: 0.90.1 hbase: 0.90.1 hbase: 0.90.1 hbase: 0.90.1Private

configurePublic UI RPC UI RPC UI RPC HTTP Master RegionServer RegionServer RestServer listen listen listen listenPrivate

managePublic UI RPC UI RPC UI RPC Master #2 Master RegionServer listen listen listenPrivate

you can make a clustermanager with jclouds provisionbootstrap install conﬁgureoperate manage

Portable APIs Compute LoadBalancer BlobStore TableProvider APIsDriver-ArchitectureTested Providers!

jclouds conceptsTemplates abstractly describe nodes:What OS to use, what versions...What kind of hardware, what features are needed...Groups organize identically conﬁgured nodes

bootstrapcreateNodesInGroup minRam(8192) hbase-master 1 osFamily(UBUNTU) runScript(nodeInstall) hbase-regionserver 6 proxied hbase- proxied proxied master proxied hbase- proxied regionserver

operationsOperate based on predicates runScriptOnNodesMatching group: hbase state: running proxied proxied proxied proxied proxied

java overview github jclouds/jclouds// initcontext = new ComputeServiceContextFactory(). createContext("gogrid", apikey, secret, singleton(new SshjSshClientModule()));compute = context.getComputeService();// create a couple nodes and open a couple portsnodes = compute.createNodesInGroup(“hbase-master”, 1, runScript(install). inboundPorts(60000, 60010));// gather my ip addressesfor (NodeMetadata node : nodes) { node.getPublicAddresses(); node.getPrivateAddresses();}

Whirr can build your cluster!

whirr apachespec = new ClusterSpec();spec.setProvider(“cloudservers-uk”);spec.setIdentity(apikey);spec.setCredential(secret);spec.setClusterName(“hbase”);spec.setInstanceTemplates(ImmutableList.of( new InstanceTemplate(1,”hbase-master”), new InstanceTemplate(6,”hbase-regionserver”)));cluster = new ClusterController().launchCluster(spec); $ whirr launch-cluster --cluster-name=hbase --instance-templates=1 hbase-master,6 hbase-regionserver

Whirr internalsBootstrapClusterAction creates cluster and runs bootstrap script for all rolesConfigureClusterAction configures firewall and runs configure script for all rolesDestroyClusterAction destroys the nodes in the cluster

makes yourcluster sparkle!

you can weave in puppetmanifest just like other roles puppet-logrotate munin::host munin::clientmodules come from git or

example conﬁg

Puppet-Whirr implementationbeforeBootstrap install all modules for roles prefixed with puppet:beforeConfigure apply all manifests w/attribs, then puppet applyNotes! tar.gz files are staged to a BlobStore instead of scp’ing

Special Thanks!Alex Heneveld: Garrett Honeycutt: Chad Metcalf: design + puppety advice concept + integration elaboration

What’s next?• More services• VBox Support• Chef Solo/Puppet Master integration• Cluster healing

Questions? github jclouds-examples whirr.apache.org @jcloudsadrian@cloudsoftcorp.com

Whirr dev-up-puppetconf2011

by Puppet Labs

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Whirr dev-up-puppetconf2011 Presentation Transcript