Whirr dev-up-puppetconf2011


Published on

Adrian Cole

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • Podling Project Management Committee \njclouds focuses on cloud storage and infrastructure portability\nwhirr focuses on portable management of clustered services\n
  • The UI ports are for web browsers to see the status pages of HBase (master and regionservers have this available). The RPC port uses native (Hadoop) RPC to talk between client and servers. The HTTP/REST exposes a similar API as the RPC does, and is used by clients wanting to tunnel through a firewall for example, or implement in non-Java languages. Internally the same ports are used to communicate between servers, i.e. RPC native calls. Default ports are: Master (60000 for RPC and 60010 for UI) and RegionServer (60020 RPC, 60030 UI) but can be changed. REST uses 8080 by default.\n
  • \n
  • \n
  • \n
  • get a hold of a few machines with the right base operating system, role based templates can help selecting the appropriate features of the hardware.\n
  • \n
  • \n
  • \n
  • \n
  • Abstractions\nBlobStore ( atmos, azure, rackspace, s3 )\nCompute ( ec2, rackspace, vcloud, gogrid...)\nClojure bindings\nStuff (stubs/filesystem providers, plugins...)\n
  • \n
  • \n
  • \n
  • notice you don’t even have to choose an operating system\n\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Whirr dev-up-puppetconf2011

    1. 1. Clustered Services withApache Whirr: From Dev Up @jclouds
    2. 2. agendaintroconceptsjcloudswhirr
    3. 3. introAdrian Cole (@jclouds) founded jclouds march 2009 member of the Apache Whirr project chief evangelist at Cloudsoft
    4. 4. so you want to build a cluster.. Web REST HTable Browser ClientPublic UI RPC UI RPC UI RPC HTTP Master RegionServer RegionServer RestServer listen listen listen listenPrivate
    5. 5. First, you need to analyzethe roles you need in thecluster hadoop- hadoop- hbase- zookeeper namenode datanode master hadoop- hadoop- hbase- jobtracker tasktracker regionserver
    6. 6. Next, organize andunderstand the topology hadoop- tasktracker hadoop- zookeeper namenode hadoop- datanode hadoop- hbase- jobtracker master hbase- regionserver
    7. 7. Then you need to developthe workflow provisionbootstrap install configureoperate manage
    8. 8. provisionPublic hbase-master hbase-regionserver hbase-regionserver hbase-restserver Ubuntu 10.04 Ubuntu 10.04 Ubuntu 10.04 Ubuntu 10.04 8GB RAM 8GB RAM 8GB RAM 2GB RAM 8 Cores 8 Cores 8 Cores 1 CPUPrivate
    9. 9. installPublic hbase-master hbase-regionserver hbase-regionserver hbase-restserver login: hbase login: hbase login: hbase login: hbase java: 1.6.0_16 java: 1.6.0_16 java: 1.6.0_16 java: 1.6.0_16 hbase: 0.90.1 hbase: 0.90.1 hbase: 0.90.1 hbase: 0.90.1Private
    10. 10. configurePublic UI RPC UI RPC UI RPC HTTP Master RegionServer RegionServer RestServer listen listen listen listenPrivate
    11. 11. managePublic UI RPC UI RPC UI RPC Master #2 Master RegionServer listen listen listenPrivate
    12. 12. you can make a clustermanager with jclouds provisionbootstrap install configureoperate manage
    13. 13. Portable APIs Compute LoadBalancer BlobStore TableProvider APIsDriver-ArchitectureTested Providers!
    14. 14. jclouds conceptsTemplates abstractly describe nodes:What OS to use, what versions...What kind of hardware, what features are needed...Groups organize identically configured nodes
    15. 15. bootstrapcreateNodesInGroup minRam(8192) hbase-master 1 osFamily(UBUNTU) runScript(nodeInstall) hbase-regionserver 6 proxied hbase- proxied proxied master proxied hbase- proxied regionserver
    16. 16. operationsOperate based on predicates runScriptOnNodesMatching group: hbase state: running proxied proxied proxied proxied proxied
    17. 17. java overview github jclouds/jclouds// initcontext = new ComputeServiceContextFactory(). createContext("gogrid", apikey, secret, singleton(new SshjSshClientModule()));compute = context.getComputeService();// create a couple nodes and open a couple portsnodes = compute.createNodesInGroup(“hbase-master”, 1, runScript(install). inboundPorts(60000, 60010));// gather my ip addressesfor (NodeMetadata node : nodes) { node.getPublicAddresses(); node.getPrivateAddresses();}
    18. 18. Whirr can build your cluster!
    19. 19. whirr apachespec = new ClusterSpec();spec.setProvider(“cloudservers-uk”);spec.setIdentity(apikey);spec.setCredential(secret);spec.setClusterName(“hbase”);spec.setInstanceTemplates(ImmutableList.of( new InstanceTemplate(1,”hbase-master”), new InstanceTemplate(6,”hbase-regionserver”)));cluster = new ClusterController().launchCluster(spec); $ whirr launch-cluster --cluster-name=hbase --instance-templates=1 hbase-master,6 hbase-regionserver
    20. 20. Whirr internalsBootstrapClusterAction creates cluster and runs bootstrap script for all rolesConfigureClusterAction configures firewall and runs configure script for all rolesDestroyClusterAction destroys the nodes in the cluster
    21. 21. makes yourcluster sparkle!
    22. 22. you can weave in puppetmanifest just like other roles puppet-logrotate munin::host munin::clientmodules come from git or
    23. 23. example config
    24. 24. Puppet-Whirr implementationbeforeBootstrap install all modules for roles prefixed with puppet:beforeConfigure apply all manifests w/attribs, then puppet applyNotes! tar.gz files are staged to a BlobStore instead of scp’ing
    25. 25. Special Thanks!Alex Heneveld: Garrett Honeycutt: Chad Metcalf: design + puppety advice concept + integration elaboration
    26. 26. What’s next?• More services• VBox Support• Chef Solo/Puppet Master integration• Cluster healing
    27. 27. Questions? github jclouds-examples whirr.apache.org @jcloudsadrian@cloudsoftcorp.com