Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

CloudStack, jclouds and Whirr!


Published on

Three Apache Java cloud projects, working together!

Published in: Technology, Education
  • Be the first to comment

CloudStack, jclouds and Whirr!

  1. 1. Apache Java Cloud Projects! Andrew Bayer Bay Area Apache CloudStack Users Group August 7, 2013
  2. 2. Who am I? ● Andrew Bayer ● Build and Tools Architect at Cloudera ● User of CloudStack ● VP, Apache Whirr ● PPMC, Apache jclouds ● Bad at designing slide decks
  3. 3. What am I here to talk about? How the Apache Java-based cloud projects play together! CloudStack (of course!) jclouds Whirr
  4. 4. Hold on a sec, gotta start the demo! (it takes a while, y'see)
  5. 5. What's CloudStack? ...shouldn't you know that already? Well, ok. “The project develops open source software for deploying public and private Infrastructure-as-a- Service (IaaS) clouds.”
  6. 6. What can you do with CloudStack? Run a cloud? Seriously, what can't you do with CloudStack?
  7. 7. What's jclouds? Java library for abstracting and interacting more directly with a plethora of compute, blobstore (i.e., S3 or Swift) and other cloud-related APIs and providers. High-level abstractions allow you to use the same code to work with different APIs/providers. Java interfaces directly to the various APIs let you drill down and get more specific control.
  8. 8. What can you do with jclouds? Cloud automation in your Java/JVM application Provision instances Use object stores Control your cloud Write a management layer on top of one or more clouds (hey, I did that!)
  9. 9. What's Whirr? A set of Java libraries (built on top of jclouds) for provisioning and deploying distributed systems. Cloud-neutral, or bring your own node Common service API with service-specific details for configuring/provisioning Smart defaults so you can get going easily, but easy to override
  10. 10. What can you do with Whirr? Set up an Apache Hadoop cluster on your favorite cloud in 5 minutes of work. Deploy a distributed application onto existing physical hosts (yes, Whirr works with bare metal!) Write your own service to deploy your own app Integrate all the above into a Java/JVM app
  11. 11. All at Apache!
  12. 12. CloudStack and jclouds jclouds has support for the full CloudStack API User, Domain and Root APIs (if something's missing, open a JIRA and we'll add it!) jclouds compute abstraction allows you to use the same code with multiple clouds Or you can drill down to the full CloudStack-specific API and do, well, anything!
  13. 13. CloudStack and jclouds CloudStack is a first class API in jclouds Right up there with EC2, OpenStack, and (soon) Google Compute Provisioning, security groups, image creation – one abstraction works with all these clouds
  14. 14. jclouds: Testing, Testing, Testing Full battery of API tests Unit tests to verify correctness of API calls Live tests exercise every API call against an actual CloudStack setup
  15. 15. Wait, All the APIs? You mean the Root Admin APIs too? Creating/deleting zones, acounts, etc? Yup. But isn't that risky to run against a live CloudStack setup? Ah, I've got an answer for that!
  16. 16. So how do you do that safely? The CloudStack simulator, of course! Build CloudStack, run the simulator (and the marvin scripts to configure it, etc) Point jclouds' tests at the simulator Test everything! Break nothing!
  17. 17. A bit more detail... You do need to grab the root admin API creds, and create domain admin/normal users as well Not all tests will actually work – anything that expects to SSH into a VM or attach a disk, etc We'll be marking simulator-friendly tests so you can run “mvn clean install -Plive -Pcloudstack-simulator” and just run those tests
  18. 18. So what's using jclouds and CloudStack now? A good number of tools, like... - The Jenkins jclouds plugin (for provisioning Jenkins slaves on the fly) - Pallet (config management like Chef/Puppet, in Clojure) - CloudCat (Cloudera's internal cloud management app) ...and of course, Whirr!
  19. 19. Is Whirr like Chef or Puppet too? Short answer? No. Chef/Puppet/Pallet/etc are great at getting nodes to a target state, initially and/or regularly But they're not good at dealing with a set of distributed apps that need to coordinate with each other while getting set up Whirr, on the other hand, is.
  20. 20. Deployment Orchestration Engine! That's the term I use to describe Whirr Coordinating installation and configuration, across hosts/instances, in pre-defined stages
  21. 21. What services does Whirr support? Among others: Hadoop HBase ZooKeeper Cassandra Chef Puppet Solr Kerberos and more...
  22. 22. My personal favorite: whirr-cm (I'm admittedly a bit biased) That's Whirr support for Cloudera Manager Deploys a fully configured CDH (Cloudera's Distribution for Apache Hadoop) cluster, including Cloudera Impala and Search, using Cloudera Manager All in one command!
  23. 23. Cloudera Manager I should probably mention a bit more about CM. It manages, reports, configures, monitors, deploys the complete CDH Hadoop stack, with lots of handy features and integrations, all exposed via a great web UI and a REST API (which has publicly available Java and Python libraries as well).
  24. 24. whirr-cm works everywhere ...well, nearly everywhere. Since it's jclouds-powered, it works with any cloud supported by jclouds. But it does require certain things like reverse-resolvable public IPs, as do some other Whirr services.
  25. 25. Running whirr Set up a config file for your cluster – see the recipes dir for examples For CloudStack, you have to specify “jclouds.endpoint=http://your.server/client/api” Run “bin/whirr launch-cluster –config path/to/” to provision/deploy Run “bin/whirr destroy-cluster –config path/to/” to destroy
  26. 26. Oh yeah, that demo... It's whirr-cm! Let's see how it's doing...
  27. 27. Another caveat Whirr 0.8.2 (the latest release) has some problems with CloudStack Namely, security groups/firewalls and keypairs These will be fixed in 0.9.0, coming out this fall Patch already available - WHIRR-725
  28. 28. How do I get whirr-cm? First option: download or build Whirr, and then download or build whirr-cm – see Second option: install Whirr from the CDH deb/rpm packages – CDH 4.3.0 uses whirr-cm 1.2.
  29. 29. Links! ● CloudStack - ● jclouds - ● Whirr - ● whirr-cm - ● Installing Whirr from CDH ● CloudCat - (for CloudStack-specific API use with jclouds)
  30. 30. Thanks for listening!