Three Apache Java cloud projects, working together!

1. Apache Java Cloud Projects!
Andrew Bayer
Bay Area Apache CloudStack Users Group
August 7, 2013

2. Who am I?
● Andrew Bayer
● Build and Tools Architect at Cloudera
● User of CloudStack
● VP, Apache Whirr
● PPMC, Apache jclouds
● Bad at designing slide decks

3. What am I here to talk about?
How the Apache Java-based cloud
projects play together!
CloudStack
(of course!)
jclouds
Whirr

4. Hold on a sec, gotta start the demo!
(it takes a while, y'see)

5. What's CloudStack?
...shouldn't you know that already?
Well, ok.
“The project develops open source software for
deploying public and private Infrastructure-as-a-
Service (IaaS) clouds.”

6. What can you do with CloudStack?
Run a cloud?
Seriously, what can't you do with CloudStack?

7. What's jclouds?
Java library for abstracting and interacting
more directly with a plethora of
compute, blobstore (i.e., S3 or Swift)
and other cloud-related APIs and
providers.
High-level abstractions allow you to use the same
code to work with different APIs/providers.
Java interfaces directly to the various APIs let you
drill down and get more specific control.

8. What can you do with jclouds?
Cloud automation in your Java/JVM application
Provision instances
Use object stores
Control your cloud
Write a management layer on top of one
or more clouds (hey, I did that!)

9. What's Whirr?
A set of Java libraries (built on top of
jclouds) for provisioning and deploying
distributed systems.
Cloud-neutral, or bring your own node
Common service API with service-specific details for
configuring/provisioning
Smart defaults so you can get going easily, but easy
to override

10. What can you do with Whirr?
Set up an Apache Hadoop cluster on your
favorite cloud in 5 minutes of work.
Deploy a distributed application onto existing
physical hosts (yes, Whirr works with bare
metal!)
Write your own service to deploy your own
app
Integrate all the above into a Java/JVM app

11. All at Apache!

12. CloudStack and jclouds
jclouds has support for the full CloudStack API
User, Domain and Root APIs
(if something's missing, open a JIRA and we'll add it!)
jclouds compute abstraction allows you to use the
same code with multiple clouds
Or you can drill down to the full CloudStack-specific
API and do, well, anything!

13. CloudStack and jclouds
CloudStack is a first class API in jclouds
Right up there with EC2, OpenStack, and (soon)
Google Compute
Provisioning, security groups, image creation – one
abstraction works with all these clouds

14. jclouds: Testing, Testing, Testing
Full battery of API tests
Unit tests to verify correctness of API calls
Live tests exercise every API call against
an actual CloudStack setup

15. Wait, All the APIs?
You mean the Root Admin APIs too?
Creating/deleting zones, acounts, etc?
Yup.
But isn't that risky to run against a live CloudStack
setup?
Ah, I've got an answer for that!

16. So how do you do that safely?
The CloudStack simulator, of course!
Build CloudStack, run the simulator
(and the marvin scripts to configure it, etc)
Point jclouds' tests at the simulator
Test everything! Break nothing!

17. A bit more detail...
You do need to grab the root admin API
creds, and create domain admin/normal
users as well
Not all tests will actually work – anything
that expects to SSH into a VM or attach
a disk, etc
We'll be marking simulator-friendly tests
so you can run “mvn clean install -Plive
-Pcloudstack-simulator” and just run those
tests

18. So what's using jclouds and
CloudStack now?
A good number of tools, like...
- The Jenkins jclouds plugin
(for provisioning Jenkins slaves on the fly)
- Pallet
(config management like Chef/Puppet, in Clojure)
- CloudCat
(Cloudera's internal cloud management app)
...and of course, Whirr!

19. Is Whirr like Chef or Puppet too?
Short answer? No.
Chef/Puppet/Pallet/etc are great at getting
nodes to a target state, initially and/or
regularly
But they're not good at dealing with a set of
distributed apps that need to coordinate
with each other while getting set up
Whirr, on the other hand, is.

20. Deployment Orchestration Engine!
That's the term I use to describe Whirr
Coordinating installation and
configuration, across hosts/instances, in
pre-defined stages

21. What services does Whirr support?
Among others:
Hadoop
HBase
ZooKeeper
Cassandra
Chef
Puppet
Solr
Kerberos
and more...

22. My personal favorite: whirr-cm
(I'm admittedly a bit biased)
That's Whirr support for Cloudera Manager
Deploys a fully configured CDH
(Cloudera's Distribution for Apache
Hadoop) cluster, including Cloudera
Impala and Search, using Cloudera
Manager
All in one command!

23. Cloudera Manager
I should probably mention a bit more about CM.
It manages, reports, configures, monitors, deploys
the complete CDH Hadoop stack, with lots of
handy features and integrations, all exposed via a
great web UI and a REST API (which has publicly
available Java and Python libraries as well).

24. whirr-cm works everywhere
...well, nearly everywhere.
Since it's jclouds-powered, it works with
any cloud supported by jclouds.
But it does require certain things like
reverse-resolvable public IPs, as do
some other Whirr services.

25. Running whirr
Set up a config file for your cluster – see the
recipes dir for examples
For CloudStack, you have to specify
“jclouds.endpoint=http://your.server/client/api”
Run “bin/whirr launch-cluster –config path/to/your.properties” to
provision/deploy
Run “bin/whirr destroy-cluster –config path/to/your.properties” to
destroy

26. Oh yeah, that demo...
It's whirr-cm!
Let's see how it's doing...

27. Another caveat
Whirr 0.8.2 (the latest release) has some
problems with CloudStack
Namely, security groups/firewalls and keypairs
These will be fixed in 0.9.0, coming out this fall
Patch already available - WHIRR-725

28. How do I get whirr-cm?
First option: download or build Whirr, and
then download or build whirr-cm – see
https://github.com/cloudera/whirr-cm
Second option: install Whirr from the CDH
deb/rpm packages – CDH 4.3.0 uses
whirr-cm 1.2.

29. Links!
● CloudStack - http://cloudstack.apache.org/
● jclouds - http://jclouds.incubator.apache.org/
● Whirr - http://whirr.apache.org/
● whirr-cm - https://github.com/cloudera/whirr-cm
● Installing Whirr from CDH
● CloudCat - https://github.com/abayer/cloudcat/
(for CloudStack-specific API use with jclouds)

30. Thanks for listening!