Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Clustered Services withApache Whirr: From Dev Up @jclouds
agendaintroconceptsjcloudswhirr
introAdrian Cole (@jclouds)  founded jclouds march 2009     member of the Apache Whirr project        chief evangelist at ...
so you want to build a                 cluster..                               Web                                        ...
First, you need to analyzethe roles you need in thecluster   hadoop-     hadoop-                     hbase-               ...
Next, organize andunderstand the topology                                  hadoop-                                tasktrac...
Then you need to developthe workflow            provisionbootstrap         install                        configureoperate ...
provisionPublic            hbase-master   hbase-regionserver   hbase-regionserver   hbase-restserver          Ubuntu 10.04...
installPublic           hbase-master    hbase-regionserver   hbase-regionserver   hbase-restserver          login: hbase  ...
configurePublic          UI            RPC   UI            RPC   UI            RPC     HTTP               Master         R...
managePublic          UI            RPC     UI            RPC   UI            RPC           Master #2                 Mast...
you can make a clustermanager with jclouds            provisionbootstrap         install                        configureop...
Portable APIs        Compute       LoadBalancer        BlobStore        TableProvider APIsDriver-ArchitectureTested Provid...
jclouds conceptsTemplates abstractly describe nodes:What OS to use, what versions...What kind of hardware, what features a...
bootstrapcreateNodesInGroup                          minRam(8192)   hbase-master       1   osFamily(UBUNTU)               ...
operationsOperate based on predicates            runScriptOnNodesMatching                group: hbase          state: runn...
java overview                            github jclouds/jclouds// initcontext = new ComputeServiceContextFactory().       ...
Whirr can build your cluster!
whirr                                                                  apachespec = new ClusterSpec();spec.setProvider(“cl...
Whirr internalsBootstrapClusterAction               creates cluster and runs bootstrap script for all rolesConfigureCluste...
makes yourcluster sparkle!
you can weave in puppetmanifest just like other roles puppet-logrotate   munin::host   munin::clientmodules come from git or
example config
Puppet-Whirr implementationbeforeBootstrap             install all modules for roles prefixed with puppet:beforeConfigure ...
Special Thanks!Alex Heneveld:     Garrett Honeycutt:   Chad Metcalf:    design +        puppety advice        concept +  i...
What’s next?• More services• VBox Support• Chef Solo/Puppet Master integration• Cluster healing
Questions? github jclouds-examples    whirr.apache.org     @jcloudsadrian@cloudsoftcorp.com
Upcoming SlideShare
Loading in …5
×

Whirr dev-up-puppetconf2011

1,610 views

Published on

Adrian Cole

Published in: Technology
  • Be the first to comment

Whirr dev-up-puppetconf2011

  1. 1. Clustered Services withApache Whirr: From Dev Up @jclouds
  2. 2. agendaintroconceptsjcloudswhirr
  3. 3. introAdrian Cole (@jclouds) founded jclouds march 2009 member of the Apache Whirr project chief evangelist at Cloudsoft
  4. 4. so you want to build a cluster.. Web REST HTable Browser ClientPublic UI RPC UI RPC UI RPC HTTP Master RegionServer RegionServer RestServer listen listen listen listenPrivate
  5. 5. First, you need to analyzethe roles you need in thecluster hadoop- hadoop- hbase- zookeeper namenode datanode master hadoop- hadoop- hbase- jobtracker tasktracker regionserver
  6. 6. Next, organize andunderstand the topology hadoop- tasktracker hadoop- zookeeper namenode hadoop- datanode hadoop- hbase- jobtracker master hbase- regionserver
  7. 7. Then you need to developthe workflow provisionbootstrap install configureoperate manage
  8. 8. provisionPublic hbase-master hbase-regionserver hbase-regionserver hbase-restserver Ubuntu 10.04 Ubuntu 10.04 Ubuntu 10.04 Ubuntu 10.04 8GB RAM 8GB RAM 8GB RAM 2GB RAM 8 Cores 8 Cores 8 Cores 1 CPUPrivate
  9. 9. installPublic hbase-master hbase-regionserver hbase-regionserver hbase-restserver login: hbase login: hbase login: hbase login: hbase java: 1.6.0_16 java: 1.6.0_16 java: 1.6.0_16 java: 1.6.0_16 hbase: 0.90.1 hbase: 0.90.1 hbase: 0.90.1 hbase: 0.90.1Private
  10. 10. configurePublic UI RPC UI RPC UI RPC HTTP Master RegionServer RegionServer RestServer listen listen listen listenPrivate
  11. 11. managePublic UI RPC UI RPC UI RPC Master #2 Master RegionServer listen listen listenPrivate
  12. 12. you can make a clustermanager with jclouds provisionbootstrap install configureoperate manage
  13. 13. Portable APIs Compute LoadBalancer BlobStore TableProvider APIsDriver-ArchitectureTested Providers!
  14. 14. jclouds conceptsTemplates abstractly describe nodes:What OS to use, what versions...What kind of hardware, what features are needed...Groups organize identically configured nodes
  15. 15. bootstrapcreateNodesInGroup minRam(8192) hbase-master 1 osFamily(UBUNTU) runScript(nodeInstall) hbase-regionserver 6 proxied hbase- proxied proxied master proxied hbase- proxied regionserver
  16. 16. operationsOperate based on predicates runScriptOnNodesMatching group: hbase state: running proxied proxied proxied proxied proxied
  17. 17. java overview github jclouds/jclouds// initcontext = new ComputeServiceContextFactory(). createContext("gogrid", apikey, secret, singleton(new SshjSshClientModule()));compute = context.getComputeService();// create a couple nodes and open a couple portsnodes = compute.createNodesInGroup(“hbase-master”, 1, runScript(install). inboundPorts(60000, 60010));// gather my ip addressesfor (NodeMetadata node : nodes) { node.getPublicAddresses(); node.getPrivateAddresses();}
  18. 18. Whirr can build your cluster!
  19. 19. whirr apachespec = new ClusterSpec();spec.setProvider(“cloudservers-uk”);spec.setIdentity(apikey);spec.setCredential(secret);spec.setClusterName(“hbase”);spec.setInstanceTemplates(ImmutableList.of( new InstanceTemplate(1,”hbase-master”), new InstanceTemplate(6,”hbase-regionserver”)));cluster = new ClusterController().launchCluster(spec); $ whirr launch-cluster --cluster-name=hbase --instance-templates=1 hbase-master,6 hbase-regionserver
  20. 20. Whirr internalsBootstrapClusterAction creates cluster and runs bootstrap script for all rolesConfigureClusterAction configures firewall and runs configure script for all rolesDestroyClusterAction destroys the nodes in the cluster
  21. 21. makes yourcluster sparkle!
  22. 22. you can weave in puppetmanifest just like other roles puppet-logrotate munin::host munin::clientmodules come from git or
  23. 23. example config
  24. 24. Puppet-Whirr implementationbeforeBootstrap install all modules for roles prefixed with puppet:beforeConfigure apply all manifests w/attribs, then puppet applyNotes! tar.gz files are staged to a BlobStore instead of scp’ing
  25. 25. Special Thanks!Alex Heneveld: Garrett Honeycutt: Chad Metcalf: design + puppety advice concept + integration elaboration
  26. 26. What’s next?• More services• VBox Support• Chef Solo/Puppet Master integration• Cluster healing
  27. 27. Questions? github jclouds-examples whirr.apache.org @jcloudsadrian@cloudsoftcorp.com

×