Big Data & Cloud | Cloud Storage Simplified | Adrian Cole

Adrian Cole / Cloudsoft Cloud Storage SimplifiedTuesday, November 1, 11

Adrian Cole (@jclouds) founded jclouds march 2009 chief evangelist at CloudsoftTuesday, November 1, 11

Agenda • intro • concepts • jclouds • code • awkward silence (or Q/A)Tuesday, November 1, 11

Blob Storage Set<String> containers = namespacesInMyAccount; Map<String, InputStream> keyValues = contentsOfContainer; 4Tuesday, November 1, 11

Blob Storage adrian@googlestorage Love Letters Movies Tron putBlob The One Shrek Goonies The Blob 3d = true url = http://disney.go.com/tron 5Tuesday, November 1, 11

Can I use BlobStores portably? 6Tuesday, November 1, 11

are they close enough? 7Tuesday, November 1, 11

who builds and runs your service? 8Tuesday, November 1, 11

9Tuesday, November 1, 11

tenancy 10Tuesday, November 1, 11

size and resumability 11Tuesday, November 1, 11

How many copies? 12Tuesday, November 1, 11

Consistency model? 13Tuesday, November 1, 11

Are you experienced? Is your product? 14Tuesday, November 1, 11

15Tuesday, November 1, 11

CODE AND SIGN THE HTTP REQUEST PUT /adriansmovies/sushi.avi HTTP/1.1 PUT /sushi.avi HTTP/1.1 Host: <account>.blob.core.windows.net Host: adriansmovies.s3.amazonaws.com Content-Length: 734859264 Content-Length: 734859264 Date: Wed, 01 Mar 2006 12:00:00 GMT Date: Wed, 01 Mar 2006 12:00:00 GMT Authorization: SharedKey <app>:signature Authorization: signature x-ms-meta-Chef: Kawasaki x-amz-meta-Chef: Kawasaki POST /namespace/adriansmovies/sushi.avi HTTP/1.1 PUT /<api version>/<account>/ Content-Length: 734859264 adriansmovies/sushi.avi HTTP/1.1 Date: Wed, 01 Mar 2006 12:00:00 GMT Host: storage.clouddrive.com x-emc-uid: <uid> Transfer-Encoding: chunked x-emc-signature: signature X-Auth-Token: session-token x-emc-meta: Chef=Kawasaki X-Object-Meta-Chef: Kawasaki 16Tuesday, November 1, 11

CODE AND SIGN THE HTTP REQUESTGET /ws/IMFS/GetStorageNodeExtended.ashx?&fileOverwrite=true&ipRestricted=true&destFolderPath= adriansmovies&sizeBytes=734859264&firstByteExpiration=6000&lastByteExpiration=259200&sessionToken=session-token HTTP/1.1POST /Upload.ashx?uploadToken=from_above&destFolderPath=adriansmovies HTTP/1.1Host: from_aboveContent-Length: 734859382Content-Type=multipart/form-data; boundary=--jclouds--Authorization=Basic GpjbG9=----jclouds--Content-Disposition: form-data; name="sushi.avi"; filename="sushi.avi"Content-Type: application/octetstring...PUT /ws/Metadata/SetMetadata.ashx?&path=Folders/adriansmovies/sushi.avi&sessionToken=session-token&metadata=Chef:Kawasaki HTTP/1.1 17Tuesday, November 1, 11

CODE AND SIGN THE HTTP REQUEST POST /<api version>/containers/id_of_ adriansmovies/contents HTTP/1.1 Content-Length: 734859382 Content-Type=multipart/form-data; boundary=--jclouds-- Authorization=Basic GpjbG9= ----jclouds-- Content-Disposition: form-data; name="sushi.avi"; ﬁlename="sushi.avi" Content-Type: application/octetstring ... PUT /<api version>/ﬁles/from_above/metadata/Chef HTTP/1.1 Content-Length: 8 Content-Type: text/plain Authorization: Basic GpjbG9= Kawasaki 18Tuesday, November 1, 11

POST https://vcloud.safesecureweb.com/api/v0.8/vdc/1/action/instantiateVAppTemplate HTTP/1.1 do you want to Accept: application/vnd.vmware.vcloud.vApp+xml Content-Type: application/vnd.vmware.vcloud.instantiateVAppTemplateParams+xml <InstantiateVAppTemplateParams xmlns="http://www.vmware.com/vcloud/v0.8" xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" name="my-vapp" xsi:schemaLocation="http://www.vmware.com/vcloud/v0.8 http://vcloud.safesecureweb.com/ns/vcloud.xsd"> <VAppTemplate href="https://vcloud.safesecureweb.com/api/v0.8/vAppTemplate/3" /> <InstantiationParams> <VirtualHardwareSection xmlns:q1="http://www.vmware.com/vcloud/v0.8"> <Item xmlns="http://schemas.dmtf.org/ovf/envelope/1"> <InstanceID xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">1</InstanceID> <ResourceType xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">3</ResourceType> <VirtualQuantity GET https://api.gogrid.com/api/grid/server/add? </Item> xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">1</VirtualQuantity> <Item xmlns="http://schemas.dmtf.org/ovf/envelope/1"> v=1.5&name=serverName&server.ram=memory&image=img55&ip=127.0.0.1&isSandbox=true&descri <InstanceID xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">2</InstanceID> <ResourceType ption=fooy&sig=3f446f171455fbb5574aecff4997b273&api_key=foo HTTP/1.1 xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">4</ResourceType> <VirtualQuantity xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">512</VirtualQuantity> </Item> <Item xmlns="http://schemas.dmtf.org/ovf/envelope/1"> <InstanceID xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">9</InstanceID> <ResourceType xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">17</ResourceType> <VirtualQuantity xmlns="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">1024</VirtualQuantity> </Item> </VirtualHardwareSection> <NetworkConfigSection> <NetworkConfig name="my-vapp"> <Features> <FenceMode>allowInOut</FenceMode> </Features> <NetworkAssociation href="https://vcloud.safesecureweb.com/network/1990" /> </NetworkConfig> </NetworkConfigSection> </InstantiationParams> </InstantiateVAppTemplateParams> Deal with Errors POST https://rimuhosting.com/r/orders/new-vps HTTP/1.1 Accept: Accept: application/json Content-Type: application/json {"request":{"instantiation_options":{"distro":"lenny","domain_name":"test.ivan.api.com"},"pricing_plan_code":"MIRO1B","meta_data":[]}} Deal with Concurrency POST https://ec2.eu-west-1.amazonaws.com/ HTTP/1.1 Content-Type: application/x-www-form-urlencoded Version=2010-06-15&Action=RunInstances&ImageId=ami- voo&MinCount=1&MaxCount=5&KernelId=kernelId&Monitoring.Enabled=true&SecurityGroup. Deal with Complexity 1=group1&SecurityGroup.2=group2&Placement.AvailabilityZone=eu-west-1a POST http://serverManagementUrl/servers?format=json HTTP/1.1 Accept: application/json Content-Type: application/json {"server":{"name":"ralphie","imageId":2,"flavorId":1,"sharedIpGroupId":2,"addresses":{"public":["127.0.0.1"]}}} POST https://www-180.ibm.com/cloud/enterprise/beta/api/rest/20090403/instances HTTP/1.1 Accept: Accept: application/json Content-Type: application/x-www-form-urlencoded location=location&imageID=22&name=name&instanceType=instanceType&ip=1&publicKey=MOO&volumeID=2&oss.storage.id.0.mnt- %2Fmnt&insight_admin_password=myPassword1&db2_admin_password=myPassword2&report_user_password=myPassword3 19Tuesday, November 1, 11

open source feels like java (and clojure) portability between clouds deal with web complexity unit testability thread-safe and scalable 20Tuesday, November 1, 11

Portable APIs BlobStore LoadBalancer Compute Table Provider-Specific Hooks Embeddable Over 30 Tested Providers! 21Tuesday, November 1, 11

jclouds locations Location helps normalize placement across resource types All top-level resources listAssignableLocations have a location US Location metadata is SG IE extensible 22Tuesday, November 1, 11

jclouds modularity jclouds-blobstore APIs are software focused s3 Providers are offering aws-s3 scality-ring walrus focused googlestorage scaleup-storage API + location + defaults = Provider eucaluptus-partnercloud-s3 23Tuesday, November 1, 11

Alternatives to jclouds Roll-your-own • RESTEasy, Jersey Cloud-speciﬁc SDKs • typica, jets3t, Amazon SDK, vCloud SDK Dasein Cloud API 24Tuesday, November 1, 11

BlobStore 25Tuesday, November 1, 11

java overview github jclouds/jclouds // init context = new BlobStoreContextFactory().createContext("s3", accesskeyid, secret); blobStore = context.getBlobStore(); // create container blobStore.createContainerInLocation(null, “adriansmovies”); // add blob blob = blobStore.blobBuilder("sushi.avi").payload(file).build(); blobStore.putBlob(“adriansmovies”, blob); 26Tuesday, November 1, 11

clojure overview github jclouds/jclouds (use org.jclouds.blobstore2) (def *blobstore* (blobstore “azureblob” account key)) (create-container *blobstore* “movies”) (put-blob *blobstore* “movies” (blob “tron.mp4“ :payload tron-file)) 27Tuesday, November 1, 11

Who’s coding BlobStore? 28Tuesday, November 1, 11

Filesystem github jclouds/jclouds props.setProperty(FilesystemConstants.PROPERTY_BASEDIR, “basedir”); blobStore = new BlobStoreContextFactory(). createContext("filesystem", props); (blobStore "filesystem" “” “” FilesystemConstants/PROPERTY_BASEDIR “basedir”); 29Tuesday, November 1, 11

BlobRequestSigner github jclouds/jclouds HttpRequest request = context.getSigner(). signGetBlob(“adriansmovies”, "sushi.avi"); (let [request (sign-blob-request "adriansmovies" "sushi.avi" {:method :get})]) 30Tuesday, November 1, 11

What’s next? • ACL support • OutputStream returns • Public/Private pathing 31Tuesday, November 1, 11

Questions? github jclouds-examples @jclouds adrian@cloudsoftcorp.com 32Tuesday, November 1, 11

Big Data & Cloud | Cloud Storage Simplified | Adrian Cole

by JAX London

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 8

Statistics