2011-11-01 | 04:20 PM - 05:10 PM Key/value stores are the most common storage offerings in the cloud today. We'll introduce storage concepts and differences between cloud storage providers including Amazon S3, Rackspace Cloud Files & Microsoft Azure Blob Service. We'll demonstrate how jclouds BlobStore helps Java and Clojure developers avoid lock-in and increase testability without restricting access to provider-specific features

1. Adrian Cole / Cloudsoft
Cloud Storage Simplified
Tuesday, November 1, 11

2. Adrian Cole (@jclouds)
founded jclouds march 2009
chief evangelist at Cloudsoft
Tuesday, November 1, 11

3. Agenda
• intro
• concepts
• jclouds
• code
• awkward silence (or Q/A)
Tuesday, November 1, 11

4. Blob Storage
Set<String> containers = namespacesInMyAccount;
Map<String, InputStream> keyValues = contentsOfContainer;
4
Tuesday, November 1, 11

5. Blob Storage
adrian@googlestorage
Love Letters
Movies
Tron putBlob
The One Shrek
Goonies The Blob
3d = true
url = http://disney.go.com/tron
5
Tuesday, November 1, 11

6. Can I use
BlobStores
portably?
6
Tuesday, November 1, 11

7. are they close enough?
7
Tuesday, November 1, 11

8. who builds and runs
your service?
8
Tuesday, November 1, 11

9. 9
Tuesday, November 1, 11

10. tenancy
10
Tuesday, November 1, 11

11. size and
resumability
11
Tuesday, November 1, 11

12. How many copies?
12
Tuesday, November 1, 11

13. Consistency model?
13
Tuesday, November 1, 11

14. Are you experienced? Is your product?
14
Tuesday, November 1, 11

15. 15
Tuesday, November 1, 11

16. CODE AND SIGN THE HTTP REQUEST
PUT /adriansmovies/sushi.avi HTTP/1.1
PUT /sushi.avi HTTP/1.1 Host: <account>.blob.core.windows.net
Host: adriansmovies.s3.amazonaws.com Content-Length: 734859264
Content-Length: 734859264 Date: Wed, 01 Mar 2006 12:00:00 GMT
Date: Wed, 01 Mar 2006 12:00:00 GMT Authorization: SharedKey <app>:signature
Authorization: signature x-ms-meta-Chef: Kawasaki
x-amz-meta-Chef: Kawasaki
POST /namespace/adriansmovies/sushi.avi HTTP/1.1
PUT /<api version>/<account>/
Content-Length: 734859264
adriansmovies/sushi.avi HTTP/1.1
Date: Wed, 01 Mar 2006 12:00:00 GMT
Host: storage.clouddrive.com
x-emc-uid: <uid>
Transfer-Encoding: chunked
x-emc-signature: signature
X-Auth-Token: session-token
x-emc-meta: Chef=Kawasaki
X-Object-Meta-Chef: Kawasaki
16
Tuesday, November 1, 11

17. CODE AND SIGN THE HTTP REQUEST
GET /ws/IMFS/GetStorageNodeExtended.ashx?&fileOverwrite=true&ipRestricted=true&destFolderPath= adriansmovies&sizeBytes=
734859264&firstByteExpiration=6000&lastByteExpiration=259200&sessionToken=session-token HTTP/1.1
POST /Upload.ashx?uploadToken=from_above&destFolderPath=adriansmovies HTTP/1.1
Host: from_above
Content-Length: 734859382
Content-Type=multipart/form-data; boundary=--jclouds--
Authorization=Basic GpjbG9=
----jclouds--
Content-Disposition: form-data; name="sushi.avi"; filename="sushi.avi"
Content-Type: application/octetstring
...
PUT /ws/Metadata/SetMetadata.ashx?&path=Folders/adriansmovies/sushi.avi&sessionToken=session-token&metadata=Chef:Kawasaki HTTP/1.1
17
Tuesday, November 1, 11

18. CODE AND SIGN THE HTTP REQUEST
POST /<api version>/containers/id_of_ adriansmovies/contents HTTP/1.1
Content-Length: 734859382
Content-Type=multipart/form-data; boundary=--jclouds--
Authorization=Basic GpjbG9=
----jclouds--
Content-Disposition: form-data; name="sushi.avi"; filename="sushi.avi"
Content-Type: application/octetstring
...
PUT /<api version>/files/from_above/metadata/Chef HTTP/1.1
Content-Length: 8
Content-Type: text/plain
Authorization: Basic GpjbG9=
Kawasaki
18
Tuesday, November 1, 11

19. POST https://vcloud.safesecureweb.com/api/v0.8/vdc/1/action/instantiateVAppTemplate HTTP/1.1
do you want to
Accept: application/vnd.vmware.vcloud.vApp+xml
Content-Type: application/vnd.vmware.vcloud.instantiateVAppTemplateParams+xml
<InstantiateVAppTemplateParams xmlns="http://www.vmware.com/vcloud/v0.8"
xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
name="my-vapp"
xsi:schemaLocation="http://www.vmware.com/vcloud/v0.8 http://vcloud.safesecureweb.com/ns/vcloud.xsd">
<VAppTemplate
href="https://vcloud.safesecureweb.com/api/v0.8/vAppTemplate/3" />
<InstantiationParams>
<VirtualHardwareSection xmlns:q1="http://www.vmware.com/vcloud/v0.8">
<Item xmlns="http://schemas.dmtf.org/ovf/envelope/1">
<InstanceID
xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">1</InstanceID>
<ResourceType
xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">3</ResourceType>
<VirtualQuantity
GET https://api.gogrid.com/api/grid/server/add?
</Item>
xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">1</VirtualQuantity>
<Item xmlns="http://schemas.dmtf.org/ovf/envelope/1">
v=1.5&name=serverName&server.ram=memory&image=img55&ip=127.0.0.1&isSandbox=true&descri
<InstanceID
xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">2</InstanceID>
<ResourceType
ption=fooy&sig=3f446f171455fbb5574aecff4997b273&api_key=foo HTTP/1.1
xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">4</ResourceType>
<VirtualQuantity
xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">512</VirtualQuantity>
</Item>
<Item xmlns="http://schemas.dmtf.org/ovf/envelope/1">
<InstanceID
xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">9</InstanceID>
<ResourceType
xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">17</ResourceType>
<VirtualQuantity
xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">1024</VirtualQuantity>
</Item>
</VirtualHardwareSection>
<NetworkConfigSection>
<NetworkConfig name="my-vapp">
<Features>
<FenceMode>allowInOut</FenceMode>
</Features>
<NetworkAssociation href="https://vcloud.safesecureweb.com/network/1990" />
</NetworkConfig>
</NetworkConfigSection>
</InstantiationParams>
</InstantiateVAppTemplateParams>
Deal with Errors
POST https://rimuhosting.com/r/orders/new-vps HTTP/1.1
Accept: Accept: application/json
Content-Type: application/json
{"request":{"instantiation_options":{"distro":"lenny","domain_name":"test.ivan.api.com"},"pricing_plan_code":"MIRO1B","meta_data":[]}}
Deal with Concurrency
POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Version=2010-06-15&Action=RunInstances&ImageId=ami-
voo&MinCount=1&MaxCount=5&KernelId=kernelId&Monitoring.Enabled=true&SecurityGroup.
Deal with Complexity
1=group1&SecurityGroup.2=group2&Placement.AvailabilityZone=eu-west-1a
POST http://serverManagementUrl/servers?format=json HTTP/1.1
Accept: application/json
Content-Type: application/json
{"server":{"name":"ralphie","imageId":2,"flavorId":1,"sharedIpGroupId":2,"addresses":{"public":["127.0.0.1"]}}}
POST https://www-180.ibm.com/cloud/enterprise/beta/api/rest/20090403/instances HTTP/1.1
Accept: Accept: application/json
Content-Type: application/x-www-form-urlencoded
location=location&imageID=22&name=name&instanceType=instanceType&ip=1&publicKey=MOO&volumeID=2&oss.storage.id.0.mnt-
%2Fmnt&insight_admin_password=myPassword1&db2_admin_password=myPassword2&report_user_password=myPassword3
19
Tuesday, November 1, 11

20. open source
feels like java (and clojure)
portability between clouds
deal with web complexity
unit testability
thread-safe and scalable
20
Tuesday, November 1, 11

21. Portable APIs
BlobStore LoadBalancer
Compute Table
Provider-Specific Hooks
Embeddable
Over 30 Tested Providers!
21
Tuesday, November 1, 11

22. jclouds locations
Location helps
normalize placement
across resource types
All top-level resources listAssignableLocations
have a location US
Location metadata is SG
IE
extensible
22
Tuesday, November 1, 11

23. jclouds modularity
jclouds-blobstore
APIs are software focused s3
Providers are offering aws-s3
scality-ring walrus
focused googlestorage
scaleup-storage
API + location + defaults =
Provider eucaluptus-partnercloud-s3
23
Tuesday, November 1, 11

24. Alternatives to jclouds
Roll-your-own
• RESTEasy, Jersey
Cloud-specific SDKs
• typica, jets3t, Amazon SDK, vCloud SDK
Dasein Cloud API
24
Tuesday, November 1, 11

25. BlobStore
25
Tuesday, November 1, 11

26. java overview github jclouds/jclouds
// init
context = new BlobStoreContextFactory().createContext("s3",
accesskeyid,
secret);
blobStore = context.getBlobStore();
// create container
blobStore.createContainerInLocation(null, “adriansmovies”);
// add blob
blob = blobStore.blobBuilder("sushi.avi").payload(file).build();
blobStore.putBlob(“adriansmovies”, blob);
26
Tuesday, November 1, 11

27. clojure overview github jclouds/jclouds
(use 'org.jclouds.blobstore2)
(def *blobstore* (blobstore “azureblob” account key))
(create-container *blobstore* “movies”)
(put-blob *blobstore* “movies”
(blob “tron.mp4“ :payload tron-file))
27
Tuesday, November 1, 11

28. Who’s coding BlobStore?
28
Tuesday, November 1, 11

29. Filesystem github jclouds/jclouds
props.setProperty(FilesystemConstants.PROPERTY_BASEDIR, “basedir”);
blobStore = new BlobStoreContextFactory().
createContext("filesystem", props);
(blobStore "filesystem" “” “”
FilesystemConstants/PROPERTY_BASEDIR “basedir”);
29
Tuesday, November 1, 11

30. BlobRequestSigner github jclouds/jclouds
HttpRequest request = context.getSigner().
signGetBlob(“adriansmovies”,
"sushi.avi");
(let [request (sign-blob-request "adriansmovies"
"sushi.avi" {:method :get})])
30
Tuesday, November 1, 11

31. What’s next?
• ACL support
• OutputStream returns
• Public/Private pathing
31
Tuesday, November 1, 11

32. Questions?
github jclouds-examples
@jclouds
adrian@cloudsoftcorp.com
32
Tuesday, November 1, 11