Adrian Cole / Cloudsoft        Cloud Storage SimplifiedTuesday, November 1, 11
Adrian Cole (@jclouds)    founded jclouds march 2009    chief evangelist at CloudsoftTuesday, November 1, 11
Agenda    •   intro    •   concepts    •   jclouds    •   code    •   awkward silence (or Q/A)Tuesday, November 1, 11
Blob Storage        Set<String> containers = namespacesInMyAccount;        Map<String, InputStream> keyValues = contentsOf...
Blob Storage                                                     adrian@googlestorage                                     ...
Can I use BlobStores portably?                          6Tuesday, November 1, 11
are they close enough?                              7Tuesday, November 1, 11
who builds and runs           your service?                                 8Tuesday, November 1, 11
9Tuesday, November 1, 11
tenancy                          10Tuesday, November 1, 11
size and                          resumability                                   11Tuesday, November 1, 11
How many copies?                                             12Tuesday, November 1, 11
Consistency model?                                               13Tuesday, November 1, 11
Are you experienced?      Is your product?                                    14Tuesday, November 1, 11
15Tuesday, November 1, 11
CODE AND SIGN THE HTTP REQUEST                                                                         PUT /adriansmovies/...
CODE AND SIGN THE HTTP REQUESTGET /ws/IMFS/GetStorageNodeExtended.ashx?&fileOverwrite=true&ipRestricted=true&destFolderPath...
CODE AND SIGN THE HTTP REQUEST                          POST /<api version>/containers/id_of_ adriansmovies/contents HTTP/...
POST https://vcloud.safesecureweb.com/api/v0.8/vdc/1/action/instantiateVAppTemplate HTTP/1.1    do you want to            ...
open source   feels like java (and clojure)   portability between clouds   deal with web complexity   unit testability   t...
Portable APIs               BlobStore          LoadBalancer               Compute            Table       Provider-Specific...
jclouds locations     Location helps     normalize placement     across resource types     All top-level resources   listA...
jclouds modularity                                          jclouds-blobstore    APIs are software focused                ...
Alternatives to jclouds           Roll-your-own              • RESTEasy, Jersey           Cloud-specific SDKs              ...
BlobStore                          25Tuesday, November 1, 11
java overview                        github jclouds/jclouds // init context = new BlobStoreContextFactory().createContext(...
clojure overview                 github jclouds/jclouds (use org.jclouds.blobstore2) (def *blobstore* (blobstore “azureblo...
Who’s coding BlobStore?                            28Tuesday, November 1, 11
Filesystem                                  github jclouds/jclouds props.setProperty(FilesystemConstants.PROPERTY_BASEDIR,...
BlobRequestSigner                    github jclouds/jclouds HttpRequest request = context.getSigner().                    ...
What’s next?                • ACL support                • OutputStream returns                • Public/Private pathing   ...
Questions?                          github jclouds-examples                             @jclouds                      adri...
Upcoming SlideShare
Loading in...5
×

Big Data & Cloud | Cloud Storage Simplified | Adrian Cole

893

Published on

2011-11-01 | 04:20 PM - 05:10 PM
Key/value stores are the most common storage offerings in the cloud today. We'll introduce storage concepts and differences between cloud storage providers including Amazon S3, Rackspace Cloud Files & Microsoft Azure Blob Service. We'll demonstrate how jclouds BlobStore helps Java and Clojure developers avoid lock-in and increase testability without restricting access to provider-specific features

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
893
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
21
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Big Data & Cloud | Cloud Storage Simplified | Adrian Cole

  1. 1. Adrian Cole / Cloudsoft Cloud Storage SimplifiedTuesday, November 1, 11
  2. 2. Adrian Cole (@jclouds) founded jclouds march 2009 chief evangelist at CloudsoftTuesday, November 1, 11
  3. 3. Agenda • intro • concepts • jclouds • code • awkward silence (or Q/A)Tuesday, November 1, 11
  4. 4. Blob Storage Set<String> containers = namespacesInMyAccount; Map<String, InputStream> keyValues = contentsOfContainer; 4Tuesday, November 1, 11
  5. 5. Blob Storage adrian@googlestorage Love Letters Movies Tron putBlob The One Shrek Goonies The Blob 3d = true url = http://disney.go.com/tron 5Tuesday, November 1, 11
  6. 6. Can I use BlobStores portably? 6Tuesday, November 1, 11
  7. 7. are they close enough? 7Tuesday, November 1, 11
  8. 8. who builds and runs your service? 8Tuesday, November 1, 11
  9. 9. 9Tuesday, November 1, 11
  10. 10. tenancy 10Tuesday, November 1, 11
  11. 11. size and resumability 11Tuesday, November 1, 11
  12. 12. How many copies? 12Tuesday, November 1, 11
  13. 13. Consistency model? 13Tuesday, November 1, 11
  14. 14. Are you experienced? Is your product? 14Tuesday, November 1, 11
  15. 15. 15Tuesday, November 1, 11
  16. 16. CODE AND SIGN THE HTTP REQUEST PUT /adriansmovies/sushi.avi HTTP/1.1 PUT /sushi.avi HTTP/1.1 Host: <account>.blob.core.windows.net Host: adriansmovies.s3.amazonaws.com Content-Length: 734859264 Content-Length: 734859264 Date: Wed, 01 Mar 2006 12:00:00 GMT Date: Wed, 01 Mar 2006 12:00:00 GMT Authorization: SharedKey <app>:signature Authorization: signature x-ms-meta-Chef: Kawasaki x-amz-meta-Chef: Kawasaki POST /namespace/adriansmovies/sushi.avi HTTP/1.1 PUT /<api version>/<account>/ Content-Length: 734859264 adriansmovies/sushi.avi HTTP/1.1 Date: Wed, 01 Mar 2006 12:00:00 GMT Host: storage.clouddrive.com x-emc-uid: <uid> Transfer-Encoding: chunked x-emc-signature: signature X-Auth-Token: session-token x-emc-meta: Chef=Kawasaki X-Object-Meta-Chef: Kawasaki 16Tuesday, November 1, 11
  17. 17. CODE AND SIGN THE HTTP REQUESTGET /ws/IMFS/GetStorageNodeExtended.ashx?&fileOverwrite=true&ipRestricted=true&destFolderPath= adriansmovies&sizeBytes=734859264&firstByteExpiration=6000&lastByteExpiration=259200&sessionToken=session-token HTTP/1.1POST /Upload.ashx?uploadToken=from_above&destFolderPath=adriansmovies HTTP/1.1Host: from_aboveContent-Length: 734859382Content-Type=multipart/form-data; boundary=--jclouds--Authorization=Basic GpjbG9=----jclouds--Content-Disposition: form-data; name="sushi.avi"; filename="sushi.avi"Content-Type: application/octetstring...PUT /ws/Metadata/SetMetadata.ashx?&path=Folders/adriansmovies/sushi.avi&sessionToken=session-token&metadata=Chef:Kawasaki HTTP/1.1 17Tuesday, November 1, 11
  18. 18. CODE AND SIGN THE HTTP REQUEST POST /<api version>/containers/id_of_ adriansmovies/contents HTTP/1.1 Content-Length: 734859382 Content-Type=multipart/form-data; boundary=--jclouds-- Authorization=Basic GpjbG9= ----jclouds-- Content-Disposition: form-data; name="sushi.avi"; filename="sushi.avi" Content-Type: application/octetstring ... PUT /<api version>/files/from_above/metadata/Chef HTTP/1.1 Content-Length: 8 Content-Type: text/plain Authorization: Basic GpjbG9= Kawasaki 18Tuesday, November 1, 11
  19. 19. POST https://vcloud.safesecureweb.com/api/v0.8/vdc/1/action/instantiateVAppTemplate HTTP/1.1 do you want to Accept: application/vnd.vmware.vcloud.vApp+xml Content-Type: application/vnd.vmware.vcloud.instantiateVAppTemplateParams+xml <InstantiateVAppTemplateParams xmlns="http://www.vmware.com/vcloud/v0.8" xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="my-vapp" xsi:schemaLocation="http://www.vmware.com/vcloud/v0.8 http://vcloud.safesecureweb.com/ns/vcloud.xsd"> <VAppTemplate href="https://vcloud.safesecureweb.com/api/v0.8/vAppTemplate/3" /> <InstantiationParams> <VirtualHardwareSection xmlns:q1="http://www.vmware.com/vcloud/v0.8"> <Item xmlns="http://schemas.dmtf.org/ovf/envelope/1"> <InstanceID xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">1</InstanceID> <ResourceType xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">3</ResourceType> <VirtualQuantity GET https://api.gogrid.com/api/grid/server/add? </Item> xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">1</VirtualQuantity> <Item xmlns="http://schemas.dmtf.org/ovf/envelope/1"> v=1.5&name=serverName&server.ram=memory&image=img55&ip=127.0.0.1&isSandbox=true&descri <InstanceID xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">2</InstanceID> <ResourceType ption=fooy&sig=3f446f171455fbb5574aecff4997b273&api_key=foo HTTP/1.1 xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">4</ResourceType> <VirtualQuantity xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">512</VirtualQuantity> </Item> <Item xmlns="http://schemas.dmtf.org/ovf/envelope/1"> <InstanceID xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">9</InstanceID> <ResourceType xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">17</ResourceType> <VirtualQuantity xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">1024</VirtualQuantity> </Item> </VirtualHardwareSection> <NetworkConfigSection> <NetworkConfig name="my-vapp"> <Features> <FenceMode>allowInOut</FenceMode> </Features> <NetworkAssociation href="https://vcloud.safesecureweb.com/network/1990" /> </NetworkConfig> </NetworkConfigSection> </InstantiationParams> </InstantiateVAppTemplateParams> Deal with Errors POST https://rimuhosting.com/r/orders/new-vps HTTP/1.1 Accept: Accept: application/json Content-Type: application/json {"request":{"instantiation_options":{"distro":"lenny","domain_name":"test.ivan.api.com"},"pricing_plan_code":"MIRO1B","meta_data":[]}} Deal with Concurrency POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 Content-Type: application/x-www-form-urlencoded Version=2010-06-15&Action=RunInstances&ImageId=ami- voo&MinCount=1&MaxCount=5&KernelId=kernelId&Monitoring.Enabled=true&SecurityGroup. Deal with Complexity 1=group1&SecurityGroup.2=group2&Placement.AvailabilityZone=eu-west-1a POST http://serverManagementUrl/servers?format=json HTTP/1.1 Accept: application/json Content-Type: application/json {"server":{"name":"ralphie","imageId":2,"flavorId":1,"sharedIpGroupId":2,"addresses":{"public":["127.0.0.1"]}}} POST https://www-180.ibm.com/cloud/enterprise/beta/api/rest/20090403/instances HTTP/1.1 Accept: Accept: application/json Content-Type: application/x-www-form-urlencoded location=location&imageID=22&name=name&instanceType=instanceType&ip=1&publicKey=MOO&volumeID=2&oss.storage.id.0.mnt- %2Fmnt&insight_admin_password=myPassword1&db2_admin_password=myPassword2&report_user_password=myPassword3 19Tuesday, November 1, 11
  20. 20. open source feels like java (and clojure) portability between clouds deal with web complexity unit testability thread-safe and scalable 20Tuesday, November 1, 11
  21. 21. Portable APIs BlobStore LoadBalancer Compute Table Provider-Specific Hooks Embeddable Over 30 Tested Providers! 21Tuesday, November 1, 11
  22. 22. jclouds locations Location helps normalize placement across resource types All top-level resources listAssignableLocations have a location US Location metadata is SG IE extensible 22Tuesday, November 1, 11
  23. 23. jclouds modularity jclouds-blobstore APIs are software focused s3 Providers are offering aws-s3 scality-ring walrus focused googlestorage scaleup-storage API + location + defaults = Provider eucaluptus-partnercloud-s3 23Tuesday, November 1, 11
  24. 24. Alternatives to jclouds Roll-your-own • RESTEasy, Jersey Cloud-specific SDKs • typica, jets3t, Amazon SDK, vCloud SDK Dasein Cloud API 24Tuesday, November 1, 11
  25. 25. BlobStore 25Tuesday, November 1, 11
  26. 26. java overview github jclouds/jclouds // init context = new BlobStoreContextFactory().createContext("s3", accesskeyid, secret); blobStore = context.getBlobStore(); // create container blobStore.createContainerInLocation(null, “adriansmovies”); // add blob blob = blobStore.blobBuilder("sushi.avi").payload(file).build(); blobStore.putBlob(“adriansmovies”, blob); 26Tuesday, November 1, 11
  27. 27. clojure overview github jclouds/jclouds (use org.jclouds.blobstore2) (def *blobstore* (blobstore “azureblob” account key)) (create-container *blobstore* “movies”) (put-blob *blobstore* “movies” (blob “tron.mp4“ :payload tron-file)) 27Tuesday, November 1, 11
  28. 28. Who’s coding BlobStore? 28Tuesday, November 1, 11
  29. 29. Filesystem github jclouds/jclouds props.setProperty(FilesystemConstants.PROPERTY_BASEDIR, “basedir”); blobStore = new BlobStoreContextFactory(). createContext("filesystem", props); (blobStore "filesystem" “” “” FilesystemConstants/PROPERTY_BASEDIR “basedir”); 29Tuesday, November 1, 11
  30. 30. BlobRequestSigner github jclouds/jclouds HttpRequest request = context.getSigner(). signGetBlob(“adriansmovies”, "sushi.avi"); (let [request (sign-blob-request "adriansmovies" "sushi.avi" {:method :get})]) 30Tuesday, November 1, 11
  31. 31. What’s next? • ACL support • OutputStream returns • Public/Private pathing 31Tuesday, November 1, 11
  32. 32. Questions? github jclouds-examples @jclouds adrian@cloudsoftcorp.com 32Tuesday, November 1, 11
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×