The document discusses integrating Java and Clojure with cloud platforms, specifically focusing on cloud storage, blobstores, and provisioning. It highlights the benefits and challenges of using key-value storage, discusses various cloud vendors, and introduces jclouds as a solution for managing cloud complexity and portability between services. The presentation also provides code examples for provisioning and utilizing blobstores in both Java and Clojure, along with considerations for effective cloud management.

1. connecting java (and clojure) to the cloud
adrian@opscode.com
@jclouds
http://blog.jclouds.org
Monday, July 12, 2010 1

2. Agenda
• Motivation
• jclouds
• BlobStores
• Provisioning
Monday, July 12, 2010 2

3. Motivation
• Cloud Storage
• Vendors
• Differences
Monday, July 12, 2010 3

4. what is key value storage?
global name space
key, value with metadata
http accessible
sites on demand
unlimited scaling
Monday, July 12, 2010 4

5. key value storage concepts
GLOBAL NAME SPACE
KEY/ VALUE
Monday, July 12, 2010 5

6. THE VENDORS
Monday, July 12, 2010 6

7. They aren’t the same
Monday, July 12, 2010 7

8. • FILE SIZE
• RESUMABILITY
Monday, July 12, 2010 8

9. CONTENT DELIVERY NETWORK
REPLICATION
SLA
Monday, July 12, 2010 9

10. Consistency Model
Monday, July 12, 2010 10

11. AUTHORIZATION
Monday, July 12, 2010 11

12. api complexity
Monday, July 12, 2010 12

13. CODE AND SIGN THE HTTP REQUEST
PUT /adriansmovies/sushi.avi HTTP/1.1
PUT /sushi.avi HTTP/1.1 Host: <account>.blob.core.windows.net
Host: adriansmovies.s3.amazonaws.com Content-Length: 734859264
Content-Length: 734859264 Date: Wed, 01 Mar 2006 12:00:00 GMT
Date: Wed, 01 Mar 2006 12:00:00 GMT Authorization: SharedKey <app>:signature
Authorization: signature x-ms-meta-Chef: Kawasaki
x-amz-meta-Chef: Kawasaki
POST /namespace/adriansmovies/sushi.avi HTTP/1.1
PUT /<api version>/<account>/
Content-Length: 734859264
adriansmovies/sushi.avi HTTP/1.1
Date: Wed, 01 Mar 2006 12:00:00 GMT
Host: storage.clouddrive.com
x-emc-uid: <uid>
Transfer-Encoding: chunked
x-emc-signature: signature
X-Auth-Token: session-token
x-emc-meta: Chef=Kawasaki
X-Object-Meta-Chef: Kawasaki
Monday, July 12, 2010 13

14. CODE AND SIGN THE HTTP REQUEST
GET /ws/IMFS/GetStorageNodeExtended.ashx?&fileOverwrite=true&ipRestricted=true&destFolderPath= adriansmovies&sizeBytes=
734859264&firstByteExpiration=6000&lastByteExpiration=259200&sessionToken=session-token HTTP/1.1
POST /Upload.ashx?uploadToken=from_above&destFolderPath=adriansmovies HTTP/1.1
Host: from_above
Content-Length: 734859382
Content-Type=multipart/form-data; boundary=--jclouds--
Authorization=Basic GpjbG9=
----jclouds--
Content-Disposition: form-data; name="sushi.avi"; filename="sushi.avi"
Content-Type: application/octetstring
...
PUT /ws/Metadata/SetMetadata.ashx?&path=Folders/adriansmovies/sushi.avi&sessionToken=session-token&metadata=Chef:Kawasaki HTTP/1.1
Monday, July 12, 2010 14

15. CODE AND SIGN THE HTTP REQUEST
POST /<api version>/containers/id_of_ adriansmovies/contents HTTP/1.1
Content-Length: 734859382
Content-Type=multipart/form-data; boundary=--jclouds--
Authorization=Basic GpjbG9=
----jclouds--
Content-Disposition: form-data; name="sushi.avi"; filename="sushi.avi"
Content-Type: application/octetstring
...
PUT /<api version>/files/from_above/metadata/Chef HTTP/1.1
Content-Length: 8
Content-Type: text/plain
Authorization: Basic GpjbG9=
Kawasaki
Monday, July 12, 2010 15

16. do you want to
• Deal with Errors
• Deal with Concurrency
• Deal with Cloud Complexity
Monday, July 12, 2010 16

17. jclouds
open source
feels like java (and clojure)
portability between clouds
deal with web complexity
unit testability
thread-safe and scalable
Monday, July 12, 2010 17

18. Tools we provide
• Abstractions
• BlobStore ( atmos, azure, rackspace, s3 )
• Compute ( vcloud, ec2, gogrid, ibmdev,
rackspace, rimu )
• Clojure bindings
• Third-party library integration
Monday, July 12, 2010 18

19. Alternatives to jclouds
• Roll-your-own
• Jersey, RESTEasy
• EC2-based cloud apis
• typica, jets3t
• Dasein Cloud API
• Service provided SDKs
Monday, July 12, 2010 19

20. BlobStore
• Java Code
• Clojure Code
Monday, July 12, 2010 20

21. java
// init
context = new BlobStoreContextFactory().createContext(
"s3",
accesskeyid,
secretaccesskey);
blobStore = context.getBlobStore();
// create container
blobStore.createContainerInLocation(null, "mycontainer");
// add blob
blob = blobStore.newBlob("test");
blob.setPayload("testdata");
blobStore.putBlob(containerName, blob);
Monday, July 12, 2010 21

22. commons vfs
vfs > open blobstore://user:key@cloudfiles/mycontainer
Opened blobstore://cloudfiles/mycontainer/
Current folder is blobstore://cloudfiles/mycontainer/
vfs > ls
Contents of blobstore://cloudfiles/mycontainer/
README.txt
0 Folder(s), 1 File(s)
vfs > close
Monday, July 12, 2010 22

23. clojure
(ns demo
(:use org.jclouds.blobstore)
(:use clojure.contrib.pprint)
)
(def blobstore (blobstore-context service account secret ))
(pprint (containers blobstore))
(pprint (blobs blobstore "mycontainer" ))
Monday, July 12, 2010 23

24. Provisioning
• The Good, the Bad, and the Ugly
• Java Code
• Clojure Code
Monday, July 12, 2010 24

25. The Good
provisioning (and re-provisioning) is cheap
APIs = automation
tools exist
Monday, July 12, 2010 25

26. The Bad
forgetting to turn things off
licensing
erratic service quality
Monday, July 12, 2010 26

27. The Ugly
cloud apis are sometimes unreliable
apis are very different across clouds
features are very different across clouds
accidental complexity
Monday, July 12, 2010 27

28. Things to consider when provisioning
Can you create an image?
Can you push credentials or files?
Do you need to VPN in?
How is storage provisioned?
How close are your dependencies?
Monday, July 12, 2010 28

29. Java Code
Monday, July 12, 2010 29

30. jclouds github jclouds/jclouds
service = new ComputeServiceContextFactory().createContext(
“rimuhosting”, user, password ).getComputeService();
template = service.templateBuilder().any().biggest().build();
template.getOptions().installPrivateKey(privateRSA)
.authorizePublicKey(publicRSA)
.runScript(installGemsAndRunChef);
nodes = service.runNodesWithTag(“webserver”, 5, template);
Monday, July 12, 2010 30

31. dasein sourceforge dasein-cloud
CloudProvider provider = providerClass.newInstance();
ProviderContext context = new ProviderContext();
context.setAccountNumber(accountNumber);
context.setAccessPublic(apiKeyBytes);
context.setAccessPrivate(privateKeyBytes);
provider.connect(context);
ServerServices services = provider.getServerServices();
server = services.launch(imageId, size,
dataCenterId, serverName,
keypairOrPassword, vlan,
analytics, firewalls);
Monday, July 12, 2010 31

32. whirr github tomwhite/whirr
ServiceSpec serviceSpec = new ServiceSpec();
serviceSpec.setProvider("gogrid");
serviceSpec.setAccount(account);
serviceSpec.setKey(key);
serviceSpec.setSecretKeyFile(secretKeyFile);
serviceSpec.setClusterName(clusterName);
service = new HadoopService(serviceSpec);
ClusterSpec clusterSpec = new ClusterSpec(
new InstanceTemplate(1, HadoopService.MASTER_ROLE),
new InstanceTemplate(1, HadoopService.WORKER_ROLE));
cluster = service.launchCluster(clusterSpec);
proxy = new HadoopProxy(serviceSpec, cluster);
proxy.start();
Monday, July 12, 2010 32

33. jclouds-chef github jclouds/jclouds
context = ChefContextFactory.createContext(server, identity, key);
rsaPrivateKey = context.getApi().createClient(clientName);
cookbooks = context.getApi().listCookbooks();
ChefAsyncClient nonBlockingClient = context.getAsyncApi();
nonBlockingClient.uploadCookbook(“apache2”,
new File(“/cookbooks/apache2.tgz”));
Monday, July 12, 2010 33

34. (code clojure)
Monday, July 12, 2010 34

35. jclouds github jclouds/jclouds
(def service
(compute-service “ec2” account key :ssh :log4j))
(with-compute-service [service]
(def template
(build-template :run-script bootstrap))
(def node
(run-node "couchdb" template))
(create-volume :node node :size 250))
Monday, July 12, 2010 35

36. crane github clj-sys/crane
(def hadoop-config (conf "/path/to/conf.clj"))
(def compute (ec2 (creds "/path/to/creds.clj")))
(launch-hadoop-cluster compute hadoop-config)
Monday, July 12, 2010 36

37. pallet github hugoduncan/pallet
(defnode webserver []
:bootstrap [(public-dns-if-no-nameserver)
(automated-admin-user)]
:configure [(chef)])
(with-compute-service [service]
(converge {webserver 3})
(cook webserver "/cookbooks/apache-chef-demo"))
Monday, July 12, 2010 37

38. Questions?
adrian@opscode.org
@jclouds
http://blog.jclouds.org
Monday, July 12, 2010 38