Apache Java Cloud Projects!
Bay Area Apache CloudStack Users Group
August 7, 2013
Who am I?
● Andrew Bayer
● Build and Tools Architect at Cloudera
● User of CloudStack
● VP, Apache Whirr
● PPMC, Apache jclouds
● Bad at designing slide decks
What am I here to talk about?
How the Apache Java-based cloud
projects play together!
Hold on a sec, gotta start the demo!
(it takes a while, y'see)
...shouldn't you know that already?
“The project develops open source software for
deploying public and private Infrastructure-as-a-
Service (IaaS) clouds.”
What can you do with CloudStack?
Run a cloud?
Seriously, what can't you do with CloudStack?
Java library for abstracting and interacting
more directly with a plethora of
compute, blobstore (i.e., S3 or Swift)
and other cloud-related APIs and
High-level abstractions allow you to use the same
code to work with different APIs/providers.
Java interfaces directly to the various APIs let you
drill down and get more specific control.
What can you do with jclouds?
Cloud automation in your Java/JVM application
Use object stores
Control your cloud
Write a management layer on top of one
or more clouds (hey, I did that!)
A set of Java libraries (built on top of
jclouds) for provisioning and deploying
Cloud-neutral, or bring your own node
Common service API with service-specific details for
Smart defaults so you can get going easily, but easy
What can you do with Whirr?
Set up an Apache Hadoop cluster on your
favorite cloud in 5 minutes of work.
Deploy a distributed application onto existing
physical hosts (yes, Whirr works with bare
Write your own service to deploy your own
Integrate all the above into a Java/JVM app
CloudStack and jclouds
jclouds has support for the full CloudStack API
User, Domain and Root APIs
(if something's missing, open a JIRA and we'll add it!)
jclouds compute abstraction allows you to use the
same code with multiple clouds
Or you can drill down to the full CloudStack-specific
API and do, well, anything!
CloudStack and jclouds
CloudStack is a first class API in jclouds
Right up there with EC2, OpenStack, and (soon)
Provisioning, security groups, image creation – one
abstraction works with all these clouds
jclouds: Testing, Testing, Testing
Full battery of API tests
Unit tests to verify correctness of API calls
Live tests exercise every API call against
an actual CloudStack setup
Wait, All the APIs?
You mean the Root Admin APIs too?
Creating/deleting zones, acounts, etc?
But isn't that risky to run against a live CloudStack
Ah, I've got an answer for that!
So how do you do that safely?
The CloudStack simulator, of course!
Build CloudStack, run the simulator
(and the marvin scripts to configure it, etc)
Point jclouds' tests at the simulator
Test everything! Break nothing!
A bit more detail...
You do need to grab the root admin API
creds, and create domain admin/normal
users as well
Not all tests will actually work – anything
that expects to SSH into a VM or attach
a disk, etc
We'll be marking simulator-friendly tests
so you can run “mvn clean install -Plive
-Pcloudstack-simulator” and just run those
So what's using jclouds and
A good number of tools, like...
- The Jenkins jclouds plugin
(for provisioning Jenkins slaves on the fly)
(config management like Chef/Puppet, in Clojure)
(Cloudera's internal cloud management app)
...and of course, Whirr!
Is Whirr like Chef or Puppet too?
Short answer? No.
Chef/Puppet/Pallet/etc are great at getting
nodes to a target state, initially and/or
But they're not good at dealing with a set of
distributed apps that need to coordinate
with each other while getting set up
Whirr, on the other hand, is.
Deployment Orchestration Engine!
That's the term I use to describe Whirr
Coordinating installation and
configuration, across hosts/instances, in
What services does Whirr support?
My personal favorite: whirr-cm
(I'm admittedly a bit biased)
That's Whirr support for Cloudera Manager
Deploys a fully configured CDH
(Cloudera's Distribution for Apache
Hadoop) cluster, including Cloudera
Impala and Search, using Cloudera
All in one command!
I should probably mention a bit more about CM.
It manages, reports, configures, monitors, deploys
the complete CDH Hadoop stack, with lots of
handy features and integrations, all exposed via a
great web UI and a REST API (which has publicly
available Java and Python libraries as well).
whirr-cm works everywhere
...well, nearly everywhere.
Since it's jclouds-powered, it works with
any cloud supported by jclouds.
But it does require certain things like
reverse-resolvable public IPs, as do
some other Whirr services.
Set up a config file for your cluster – see the
recipes dir for examples
For CloudStack, you have to specify
Run “bin/whirr launch-cluster –config path/to/your.properties” to
Run “bin/whirr destroy-cluster –config path/to/your.properties” to
Oh yeah, that demo...
Let's see how it's doing...
Whirr 0.8.2 (the latest release) has some
problems with CloudStack
Namely, security groups/firewalls and keypairs
These will be fixed in 0.9.0, coming out this fall
Patch already available - WHIRR-725
How do I get whirr-cm?
First option: download or build Whirr, and
then download or build whirr-cm – see
Second option: install Whirr from the CDH
deb/rpm packages – CDH 4.3.0 uses
● CloudStack - http://cloudstack.apache.org/
● jclouds - http://jclouds.incubator.apache.org/
● Whirr - http://whirr.apache.org/
● whirr-cm - https://github.com/cloudera/whirr-cm
● Installing Whirr from CDH
● CloudCat - https://github.com/abayer/cloudcat/
(for CloudStack-specific API use with jclouds)