Whirr dev-up-puppetconf2011
Upcoming SlideShare
Loading in...5

Like this? Share it with your network


Whirr dev-up-puppetconf2011



Adrian Cole

Adrian Cole



Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • \n
  • \n
  • Podling Project Management Committee \njclouds focuses on cloud storage and infrastructure portability\nwhirr focuses on portable management of clustered services\n
  • The UI ports are for web browsers to see the status pages of HBase (master and regionservers have this available). The RPC port uses native (Hadoop) RPC to talk between client and servers. The HTTP/REST exposes a similar API as the RPC does, and is used by clients wanting to tunnel through a firewall for example, or implement in non-Java languages. Internally the same ports are used to communicate between servers, i.e. RPC native calls. Default ports are: Master (60000 for RPC and 60010 for UI) and RegionServer (60020 RPC, 60030 UI) but can be changed. REST uses 8080 by default.\n
  • \n
  • \n
  • \n
  • get a hold of a few machines with the right base operating system, role based templates can help selecting the appropriate features of the hardware.\n
  • \n
  • \n
  • \n
  • \n
  • Abstractions\nBlobStore ( atmos, azure, rackspace, s3 )\nCompute ( ec2, rackspace, vcloud, gogrid...)\nClojure bindings\nStuff (stubs/filesystem providers, plugins...)\n
  • \n
  • \n
  • \n
  • notice you don’t even have to choose an operating system\n\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Whirr dev-up-puppetconf2011 Presentation Transcript

  • 1. Clustered Services withApache Whirr: From Dev Up @jclouds
  • 2. agendaintroconceptsjcloudswhirr
  • 3. introAdrian Cole (@jclouds) founded jclouds march 2009 member of the Apache Whirr project chief evangelist at Cloudsoft
  • 4. so you want to build a cluster.. Web REST HTable Browser ClientPublic UI RPC UI RPC UI RPC HTTP Master RegionServer RegionServer RestServer listen listen listen listenPrivate
  • 5. First, you need to analyzethe roles you need in thecluster hadoop- hadoop- hbase- zookeeper namenode datanode master hadoop- hadoop- hbase- jobtracker tasktracker regionserver
  • 6. Next, organize andunderstand the topology hadoop- tasktracker hadoop- zookeeper namenode hadoop- datanode hadoop- hbase- jobtracker master hbase- regionserver
  • 7. Then you need to developthe workflow provisionbootstrap install configureoperate manage
  • 8. provisionPublic hbase-master hbase-regionserver hbase-regionserver hbase-restserver Ubuntu 10.04 Ubuntu 10.04 Ubuntu 10.04 Ubuntu 10.04 8GB RAM 8GB RAM 8GB RAM 2GB RAM 8 Cores 8 Cores 8 Cores 1 CPUPrivate
  • 9. installPublic hbase-master hbase-regionserver hbase-regionserver hbase-restserver login: hbase login: hbase login: hbase login: hbase java: 1.6.0_16 java: 1.6.0_16 java: 1.6.0_16 java: 1.6.0_16 hbase: 0.90.1 hbase: 0.90.1 hbase: 0.90.1 hbase: 0.90.1Private
  • 10. configurePublic UI RPC UI RPC UI RPC HTTP Master RegionServer RegionServer RestServer listen listen listen listenPrivate
  • 11. managePublic UI RPC UI RPC UI RPC Master #2 Master RegionServer listen listen listenPrivate
  • 12. you can make a clustermanager with jclouds provisionbootstrap install configureoperate manage
  • 13. Portable APIs Compute LoadBalancer BlobStore TableProvider APIsDriver-ArchitectureTested Providers!
  • 14. jclouds conceptsTemplates abstractly describe nodes:What OS to use, what versions...What kind of hardware, what features are needed...Groups organize identically configured nodes
  • 15. bootstrapcreateNodesInGroup minRam(8192) hbase-master 1 osFamily(UBUNTU) runScript(nodeInstall) hbase-regionserver 6 proxied hbase- proxied proxied master proxied hbase- proxied regionserver
  • 16. operationsOperate based on predicates runScriptOnNodesMatching group: hbase state: running proxied proxied proxied proxied proxied
  • 17. java overview github jclouds/jclouds// initcontext = new ComputeServiceContextFactory(). createContext("gogrid", apikey, secret, singleton(new SshjSshClientModule()));compute = context.getComputeService();// create a couple nodes and open a couple portsnodes = compute.createNodesInGroup(“hbase-master”, 1, runScript(install). inboundPorts(60000, 60010));// gather my ip addressesfor (NodeMetadata node : nodes) { node.getPublicAddresses(); node.getPrivateAddresses();}
  • 18. Whirr can build your cluster!
  • 19. whirr apachespec = new ClusterSpec();spec.setProvider(“cloudservers-uk”);spec.setIdentity(apikey);spec.setCredential(secret);spec.setClusterName(“hbase”);spec.setInstanceTemplates(ImmutableList.of( new InstanceTemplate(1,”hbase-master”), new InstanceTemplate(6,”hbase-regionserver”)));cluster = new ClusterController().launchCluster(spec); $ whirr launch-cluster --cluster-name=hbase --instance-templates=1 hbase-master,6 hbase-regionserver
  • 20. Whirr internalsBootstrapClusterAction creates cluster and runs bootstrap script for all rolesConfigureClusterAction configures firewall and runs configure script for all rolesDestroyClusterAction destroys the nodes in the cluster
  • 21. makes yourcluster sparkle!
  • 22. you can weave in puppetmanifest just like other roles puppet-logrotate munin::host munin::clientmodules come from git or
  • 23. example config
  • 24. Puppet-Whirr implementationbeforeBootstrap install all modules for roles prefixed with puppet:beforeConfigure apply all manifests w/attribs, then puppet applyNotes! tar.gz files are staged to a BlobStore instead of scp’ing
  • 25. Special Thanks!Alex Heneveld: Garrett Honeycutt: Chad Metcalf: design + puppety advice concept + integration elaboration
  • 26. What’s next?• More services• VBox Support• Chef Solo/Puppet Master integration• Cluster healing
  • 27. Questions? github jclouds-examples whirr.apache.org @jcloudsadrian@cloudsoftcorp.com