Clustered Services withApache Whirr: From Dev Up @jclouds
agendaintroconceptsjcloudswhirr
introAdrian Cole (@jclouds)  founded jclouds march 2009     member of the Apache Whirr project        chief evangelist at ...
so you want to build a                 cluster..                               Web                                        ...
First, you need to analyzethe roles you need in thecluster   hadoop-     hadoop-                     hbase-               ...
Next, organize andunderstand the topology                                  hadoop-                                tasktrac...
Then you need to developthe workflow            provisionbootstrap         install                        configureoperate ...
provisionPublic            hbase-master   hbase-regionserver   hbase-regionserver   hbase-restserver          Ubuntu 10.04...
installPublic           hbase-master    hbase-regionserver   hbase-regionserver   hbase-restserver          login: hbase  ...
configurePublic          UI            RPC   UI            RPC   UI            RPC     HTTP               Master         R...
managePublic          UI            RPC     UI            RPC   UI            RPC           Master #2                 Mast...
you can make a clustermanager with jclouds            provisionbootstrap         install                        configureop...
Portable APIs        Compute       LoadBalancer        BlobStore        TableProvider APIsDriver-ArchitectureTested Provid...
jclouds conceptsTemplates abstractly describe nodes:What OS to use, what versions...What kind of hardware, what features a...
bootstrapcreateNodesInGroup                          minRam(8192)   hbase-master       1   osFamily(UBUNTU)               ...
operationsOperate based on predicates            runScriptOnNodesMatching                group: hbase          state: runn...
java overview                            github jclouds/jclouds// initcontext = new ComputeServiceContextFactory().       ...
Whirr can build your cluster!
whirr                                                                  apachespec = new ClusterSpec();spec.setProvider(“cl...
Whirr internalsBootstrapClusterAction               creates cluster and runs bootstrap script for all rolesConfigureCluste...
makes yourcluster sparkle!
you can weave in puppetmanifest just like other roles puppet-logrotate   munin::host   munin::clientmodules come from git or
example config
Puppet-Whirr implementationbeforeBootstrap             install all modules for roles prefixed with puppet:beforeConfigure ...
Special Thanks!Alex Heneveld:     Garrett Honeycutt:   Chad Metcalf:    design +        puppety advice        concept +  i...
What’s next?• More services• VBox Support• Chef Solo/Puppet Master integration• Cluster healing
Questions? github jclouds-examples    whirr.apache.org     @jcloudsadrian@cloudsoftcorp.com
Upcoming SlideShare
Loading in...5
×

Whirr dev-up-puppetconf2011

1,182

Published on

Adrian Cole

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,182
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
33
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • Podling Project Management Committee \njclouds focuses on cloud storage and infrastructure portability\nwhirr focuses on portable management of clustered services\n
  • The UI ports are for web browsers to see the status pages of HBase (master and regionservers have this available). The RPC port uses native (Hadoop) RPC to talk between client and servers. The HTTP/REST exposes a similar API as the RPC does, and is used by clients wanting to tunnel through a firewall for example, or implement in non-Java languages. Internally the same ports are used to communicate between servers, i.e. RPC native calls. Default ports are: Master (60000 for RPC and 60010 for UI) and RegionServer (60020 RPC, 60030 UI) but can be changed. REST uses 8080 by default.\n
  • \n
  • \n
  • \n
  • get a hold of a few machines with the right base operating system, role based templates can help selecting the appropriate features of the hardware.\n
  • \n
  • \n
  • \n
  • \n
  • Abstractions\nBlobStore ( atmos, azure, rackspace, s3 )\nCompute ( ec2, rackspace, vcloud, gogrid...)\nClojure bindings\nStuff (stubs/filesystem providers, plugins...)\n
  • \n
  • \n
  • \n
  • notice you don’t even have to choose an operating system\n\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Whirr dev-up-puppetconf2011

    1. 1. Clustered Services withApache Whirr: From Dev Up @jclouds
    2. 2. agendaintroconceptsjcloudswhirr
    3. 3. introAdrian Cole (@jclouds) founded jclouds march 2009 member of the Apache Whirr project chief evangelist at Cloudsoft
    4. 4. so you want to build a cluster.. Web REST HTable Browser ClientPublic UI RPC UI RPC UI RPC HTTP Master RegionServer RegionServer RestServer listen listen listen listenPrivate
    5. 5. First, you need to analyzethe roles you need in thecluster hadoop- hadoop- hbase- zookeeper namenode datanode master hadoop- hadoop- hbase- jobtracker tasktracker regionserver
    6. 6. Next, organize andunderstand the topology hadoop- tasktracker hadoop- zookeeper namenode hadoop- datanode hadoop- hbase- jobtracker master hbase- regionserver
    7. 7. Then you need to developthe workflow provisionbootstrap install configureoperate manage
    8. 8. provisionPublic hbase-master hbase-regionserver hbase-regionserver hbase-restserver Ubuntu 10.04 Ubuntu 10.04 Ubuntu 10.04 Ubuntu 10.04 8GB RAM 8GB RAM 8GB RAM 2GB RAM 8 Cores 8 Cores 8 Cores 1 CPUPrivate
    9. 9. installPublic hbase-master hbase-regionserver hbase-regionserver hbase-restserver login: hbase login: hbase login: hbase login: hbase java: 1.6.0_16 java: 1.6.0_16 java: 1.6.0_16 java: 1.6.0_16 hbase: 0.90.1 hbase: 0.90.1 hbase: 0.90.1 hbase: 0.90.1Private
    10. 10. configurePublic UI RPC UI RPC UI RPC HTTP Master RegionServer RegionServer RestServer listen listen listen listenPrivate
    11. 11. managePublic UI RPC UI RPC UI RPC Master #2 Master RegionServer listen listen listenPrivate
    12. 12. you can make a clustermanager with jclouds provisionbootstrap install configureoperate manage
    13. 13. Portable APIs Compute LoadBalancer BlobStore TableProvider APIsDriver-ArchitectureTested Providers!
    14. 14. jclouds conceptsTemplates abstractly describe nodes:What OS to use, what versions...What kind of hardware, what features are needed...Groups organize identically configured nodes
    15. 15. bootstrapcreateNodesInGroup minRam(8192) hbase-master 1 osFamily(UBUNTU) runScript(nodeInstall) hbase-regionserver 6 proxied hbase- proxied proxied master proxied hbase- proxied regionserver
    16. 16. operationsOperate based on predicates runScriptOnNodesMatching group: hbase state: running proxied proxied proxied proxied proxied
    17. 17. java overview github jclouds/jclouds// initcontext = new ComputeServiceContextFactory(). createContext("gogrid", apikey, secret, singleton(new SshjSshClientModule()));compute = context.getComputeService();// create a couple nodes and open a couple portsnodes = compute.createNodesInGroup(“hbase-master”, 1, runScript(install). inboundPorts(60000, 60010));// gather my ip addressesfor (NodeMetadata node : nodes) { node.getPublicAddresses(); node.getPrivateAddresses();}
    18. 18. Whirr can build your cluster!
    19. 19. whirr apachespec = new ClusterSpec();spec.setProvider(“cloudservers-uk”);spec.setIdentity(apikey);spec.setCredential(secret);spec.setClusterName(“hbase”);spec.setInstanceTemplates(ImmutableList.of( new InstanceTemplate(1,”hbase-master”), new InstanceTemplate(6,”hbase-regionserver”)));cluster = new ClusterController().launchCluster(spec); $ whirr launch-cluster --cluster-name=hbase --instance-templates=1 hbase-master,6 hbase-regionserver
    20. 20. Whirr internalsBootstrapClusterAction creates cluster and runs bootstrap script for all rolesConfigureClusterAction configures firewall and runs configure script for all rolesDestroyClusterAction destroys the nodes in the cluster
    21. 21. makes yourcluster sparkle!
    22. 22. you can weave in puppetmanifest just like other roles puppet-logrotate munin::host munin::clientmodules come from git or
    23. 23. example config
    24. 24. Puppet-Whirr implementationbeforeBootstrap install all modules for roles prefixed with puppet:beforeConfigure apply all manifests w/attribs, then puppet applyNotes! tar.gz files are staged to a BlobStore instead of scp’ing
    25. 25. Special Thanks!Alex Heneveld: Garrett Honeycutt: Chad Metcalf: design + puppety advice concept + integration elaboration
    26. 26. What’s next?• More services• VBox Support• Chef Solo/Puppet Master integration• Cluster healing
    27. 27. Questions? github jclouds-examples whirr.apache.org @jcloudsadrian@cloudsoftcorp.com
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×