• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Big Data & Cloud | Cloud Storage Simplified | Adrian Cole
 

Big Data & Cloud | Cloud Storage Simplified | Adrian Cole

on

  • 1,088 views

2011-11-01 | 04:20 PM - 05:10 PM...

2011-11-01 | 04:20 PM - 05:10 PM
Key/value stores are the most common storage offerings in the cloud today. We'll introduce storage concepts and differences between cloud storage providers including Amazon S3, Rackspace Cloud Files & Microsoft Azure Blob Service. We'll demonstrate how jclouds BlobStore helps Java and Clojure developers avoid lock-in and increase testability without restricting access to provider-specific features

Statistics

Views

Total Views
1,088
Views on SlideShare
1,079
Embed Views
9

Actions

Likes
0
Downloads
16
Comments
0

1 Embed 9

http://lanyrd.com 9

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Big Data & Cloud | Cloud Storage Simplified | Adrian Cole Big Data & Cloud | Cloud Storage Simplified | Adrian Cole Presentation Transcript

    • Adrian Cole / Cloudsoft Cloud Storage SimplifiedTuesday, November 1, 11
    • Adrian Cole (@jclouds) founded jclouds march 2009 chief evangelist at CloudsoftTuesday, November 1, 11
    • Agenda • intro • concepts • jclouds • code • awkward silence (or Q/A)Tuesday, November 1, 11
    • Blob Storage Set<String> containers = namespacesInMyAccount; Map<String, InputStream> keyValues = contentsOfContainer; 4Tuesday, November 1, 11
    • Blob Storage adrian@googlestorage Love Letters Movies Tron putBlob The One Shrek Goonies The Blob 3d = true url = http://disney.go.com/tron 5Tuesday, November 1, 11
    • Can I use BlobStores portably? 6Tuesday, November 1, 11
    • are they close enough? 7Tuesday, November 1, 11
    • who builds and runs your service? 8Tuesday, November 1, 11
    • 9Tuesday, November 1, 11
    • tenancy 10Tuesday, November 1, 11
    • size and resumability 11Tuesday, November 1, 11
    • How many copies? 12Tuesday, November 1, 11
    • Consistency model? 13Tuesday, November 1, 11
    • Are you experienced? Is your product? 14Tuesday, November 1, 11
    • 15Tuesday, November 1, 11
    • CODE AND SIGN THE HTTP REQUEST PUT /adriansmovies/sushi.avi HTTP/1.1 PUT /sushi.avi HTTP/1.1 Host: <account>.blob.core.windows.net Host: adriansmovies.s3.amazonaws.com Content-Length: 734859264 Content-Length: 734859264 Date: Wed, 01 Mar 2006 12:00:00 GMT Date: Wed, 01 Mar 2006 12:00:00 GMT Authorization: SharedKey <app>:signature Authorization: signature x-ms-meta-Chef: Kawasaki x-amz-meta-Chef: Kawasaki POST /namespace/adriansmovies/sushi.avi HTTP/1.1 PUT /<api version>/<account>/ Content-Length: 734859264 adriansmovies/sushi.avi HTTP/1.1 Date: Wed, 01 Mar 2006 12:00:00 GMT Host: storage.clouddrive.com x-emc-uid: <uid> Transfer-Encoding: chunked x-emc-signature: signature X-Auth-Token: session-token x-emc-meta: Chef=Kawasaki X-Object-Meta-Chef: Kawasaki 16Tuesday, November 1, 11
    • CODE AND SIGN THE HTTP REQUESTGET /ws/IMFS/GetStorageNodeExtended.ashx?&fileOverwrite=true&ipRestricted=true&destFolderPath= adriansmovies&sizeBytes=734859264&firstByteExpiration=6000&lastByteExpiration=259200&sessionToken=session-token HTTP/1.1POST /Upload.ashx?uploadToken=from_above&destFolderPath=adriansmovies HTTP/1.1Host: from_aboveContent-Length: 734859382Content-Type=multipart/form-data; boundary=--jclouds--Authorization=Basic GpjbG9=----jclouds--Content-Disposition: form-data; name="sushi.avi"; filename="sushi.avi"Content-Type: application/octetstring...PUT /ws/Metadata/SetMetadata.ashx?&path=Folders/adriansmovies/sushi.avi&sessionToken=session-token&metadata=Chef:Kawasaki HTTP/1.1 17Tuesday, November 1, 11
    • CODE AND SIGN THE HTTP REQUEST POST /<api version>/containers/id_of_ adriansmovies/contents HTTP/1.1 Content-Length: 734859382 Content-Type=multipart/form-data; boundary=--jclouds-- Authorization=Basic GpjbG9= ----jclouds-- Content-Disposition: form-data; name="sushi.avi"; filename="sushi.avi" Content-Type: application/octetstring ... PUT /<api version>/files/from_above/metadata/Chef HTTP/1.1 Content-Length: 8 Content-Type: text/plain Authorization: Basic GpjbG9= Kawasaki 18Tuesday, November 1, 11
    • POST https://vcloud.safesecureweb.com/api/v0.8/vdc/1/action/instantiateVAppTemplate HTTP/1.1 do you want to Accept: application/vnd.vmware.vcloud.vApp+xml Content-Type: application/vnd.vmware.vcloud.instantiateVAppTemplateParams+xml <InstantiateVAppTemplateParams xmlns="http://www.vmware.com/vcloud/v0.8" xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="my-vapp" xsi:schemaLocation="http://www.vmware.com/vcloud/v0.8 http://vcloud.safesecureweb.com/ns/vcloud.xsd"> <VAppTemplate href="https://vcloud.safesecureweb.com/api/v0.8/vAppTemplate/3" /> <InstantiationParams> <VirtualHardwareSection xmlns:q1="http://www.vmware.com/vcloud/v0.8"> <Item xmlns="http://schemas.dmtf.org/ovf/envelope/1"> <InstanceID xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">1</InstanceID> <ResourceType xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">3</ResourceType> <VirtualQuantity GET https://api.gogrid.com/api/grid/server/add? </Item> xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">1</VirtualQuantity> <Item xmlns="http://schemas.dmtf.org/ovf/envelope/1"> v=1.5&name=serverName&server.ram=memory&image=img55&ip=127.0.0.1&isSandbox=true&descri <InstanceID xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">2</InstanceID> <ResourceType ption=fooy&sig=3f446f171455fbb5574aecff4997b273&api_key=foo HTTP/1.1 xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">4</ResourceType> <VirtualQuantity xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">512</VirtualQuantity> </Item> <Item xmlns="http://schemas.dmtf.org/ovf/envelope/1"> <InstanceID xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">9</InstanceID> <ResourceType xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">17</ResourceType> <VirtualQuantity xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">1024</VirtualQuantity> </Item> </VirtualHardwareSection> <NetworkConfigSection> <NetworkConfig name="my-vapp"> <Features> <FenceMode>allowInOut</FenceMode> </Features> <NetworkAssociation href="https://vcloud.safesecureweb.com/network/1990" /> </NetworkConfig> </NetworkConfigSection> </InstantiationParams> </InstantiateVAppTemplateParams> Deal with Errors POST https://rimuhosting.com/r/orders/new-vps HTTP/1.1 Accept: Accept: application/json Content-Type: application/json {"request":{"instantiation_options":{"distro":"lenny","domain_name":"test.ivan.api.com"},"pricing_plan_code":"MIRO1B","meta_data":[]}} Deal with Concurrency POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 Content-Type: application/x-www-form-urlencoded Version=2010-06-15&Action=RunInstances&ImageId=ami- voo&MinCount=1&MaxCount=5&KernelId=kernelId&Monitoring.Enabled=true&SecurityGroup. Deal with Complexity 1=group1&SecurityGroup.2=group2&Placement.AvailabilityZone=eu-west-1a POST http://serverManagementUrl/servers?format=json HTTP/1.1 Accept: application/json Content-Type: application/json {"server":{"name":"ralphie","imageId":2,"flavorId":1,"sharedIpGroupId":2,"addresses":{"public":["127.0.0.1"]}}} POST https://www-180.ibm.com/cloud/enterprise/beta/api/rest/20090403/instances HTTP/1.1 Accept: Accept: application/json Content-Type: application/x-www-form-urlencoded location=location&imageID=22&name=name&instanceType=instanceType&ip=1&publicKey=MOO&volumeID=2&oss.storage.id.0.mnt- %2Fmnt&insight_admin_password=myPassword1&db2_admin_password=myPassword2&report_user_password=myPassword3 19Tuesday, November 1, 11
    • open source feels like java (and clojure) portability between clouds deal with web complexity unit testability thread-safe and scalable 20Tuesday, November 1, 11
    • Portable APIs BlobStore LoadBalancer Compute Table Provider-Specific Hooks Embeddable Over 30 Tested Providers! 21Tuesday, November 1, 11
    • jclouds locations Location helps normalize placement across resource types All top-level resources listAssignableLocations have a location US Location metadata is SG IE extensible 22Tuesday, November 1, 11
    • jclouds modularity jclouds-blobstore APIs are software focused s3 Providers are offering aws-s3 scality-ring walrus focused googlestorage scaleup-storage API + location + defaults = Provider eucaluptus-partnercloud-s3 23Tuesday, November 1, 11
    • Alternatives to jclouds Roll-your-own • RESTEasy, Jersey Cloud-specific SDKs • typica, jets3t, Amazon SDK, vCloud SDK Dasein Cloud API 24Tuesday, November 1, 11
    • BlobStore 25Tuesday, November 1, 11
    • java overview github jclouds/jclouds // init context = new BlobStoreContextFactory().createContext("s3", accesskeyid, secret); blobStore = context.getBlobStore(); // create container blobStore.createContainerInLocation(null, “adriansmovies”); // add blob blob = blobStore.blobBuilder("sushi.avi").payload(file).build(); blobStore.putBlob(“adriansmovies”, blob); 26Tuesday, November 1, 11
    • clojure overview github jclouds/jclouds (use org.jclouds.blobstore2) (def *blobstore* (blobstore “azureblob” account key)) (create-container *blobstore* “movies”) (put-blob *blobstore* “movies” (blob “tron.mp4“ :payload tron-file)) 27Tuesday, November 1, 11
    • Who’s coding BlobStore? 28Tuesday, November 1, 11
    • Filesystem github jclouds/jclouds props.setProperty(FilesystemConstants.PROPERTY_BASEDIR, “basedir”); blobStore = new BlobStoreContextFactory(). createContext("filesystem", props); (blobStore "filesystem" “” “” FilesystemConstants/PROPERTY_BASEDIR “basedir”); 29Tuesday, November 1, 11
    • BlobRequestSigner github jclouds/jclouds HttpRequest request = context.getSigner(). signGetBlob(“adriansmovies”, "sushi.avi"); (let [request (sign-blob-request "adriansmovies" "sushi.avi" {:method :get})]) 30Tuesday, November 1, 11
    • What’s next? • ACL support • OutputStream returns • Public/Private pathing 31Tuesday, November 1, 11
    • Questions? github jclouds-examples @jclouds adrian@cloudsoftcorp.com 32Tuesday, November 1, 11