JClouds at San Francisco Java User Group


Published on

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

JClouds at San Francisco Java User Group

  1. 1. connecting java (and clojure) to the cloud adrian@opscode.com @jclouds http://blog.jclouds.org
  2. 2. Agenda • Motivation • jclouds • BlobStores • Provisioning
  3. 3. Motivation • Cloud Storage • Vendors • Differences
  4. 4. what is key value storage? global name space key, value with metadata http accessible sites on demand unlimited scaling Global namespace == think gmail account... separates you from others and must be unique Inter-site copying Distribution publishing
  5. 5. key value storage concepts GLOBAL NAME SPACE KEY/ VALUE azure scopes container names to accounts nirvanix scopes based on application and then subaccount
  6. 6. THE VENDORS Rackspace: cloud files Amazon web services: S3 Windows Azure: Blob Service Nirvaniz: Storage Delivery Network Mezeo: Cloud Storage Platform limelight customers: Disney, MSNBC, NetFlix, Microsoft Xbox, and Amazon Video on Demand azure containers can have metadata, rackspace doesn’t support acls
  7. 7. They aren’t the same
  8. 8. • FILE SIZE • RESUMABILITY nirvanix max 256G; no support for range mezeo; max is limited to filesystem (exabytes) azure; max is account size (50GB)
  9. 9. CONTENT DELIVERY NETWORK REPLICATION SLA nirvanix 5 nodes under one namespace 3 locations; west socal, central, dallas, jersey, frank, tokyo cloudlayer/cloudfiles single location both with CDN 21 location unsure azure/s3 eu/us write ( multiple namespace) 7 node CDN cloudfront nirvanix - 99.9 on single 99.99 99.999 <- first sla
  10. 10. Consistency Model How long can the gaps be between updates and re-reads? How does CDN affect this? Nirvanix, azure, cloudlayer, rackspace <- immediate for local - delay on remote S3 eventual consistency
  11. 11. AUTHORIZATION Can you setup policies for who can access files. delegated billing? nirvanix storage and usage limits per subaccount, fine grained access within application
  12. 12. api complexity nirvanix - rest and soap - cloudnas nfs and cifs - physical cache on cloudnas mezeo - webdav and rest cloudfiles - rest s3 and azure - rest and soap
  13. 13. CODE AND SIGN THE HTTP REQUEST PUT /adriansmovies/sushi.avi HTTP/1.1 PUT /sushi.avi HTTP/1.1 Host: <account>.blob.core.windows.net Host: adriansmovies.s3.amazonaws.com Content-Length: 734859264 Content-Length: 734859264 Date: Wed, 01 Mar 2006 12:00:00 GMT Date: Wed, 01 Mar 2006 12:00:00 GMT Authorization: SharedKey <app>:signature Authorization: signature x-ms-meta-Chef: Kawasaki x-amz-meta-Chef: Kawasaki POST /namespace/adriansmovies/sushi.avi HTTP/1.1 PUT /<api version>/<account>/ Content-Length: 734859264 adriansmovies/sushi.avi HTTP/1.1 Date: Wed, 01 Mar 2006 12:00:00 GMT Host: storage.clouddrive.com x-emc-uid: <uid> Transfer-Encoding: chunked x-emc-signature: signature X-Auth-Token: session-token x-emc-meta: Chef=Kawasaki X-Object-Meta-Chef: Kawasaki PUT for overwrites
  14. 14. CODE AND SIGN THE HTTP REQUEST GET /ws/IMFS/GetStorageNodeExtended.ashx?&fileOverwrite=true&ipRestricted=true&destFolderPath= adriansmovies&sizeBytes= 734859264&firstByteExpiration=6000&lastByteExpiration=259200&sessionToken=session-token HTTP/1.1 POST /Upload.ashx?uploadToken=from_above&destFolderPath=adriansmovies HTTP/1.1 Host: from_above Content-Length: 734859382 Content-Type=multipart/form-data; boundary=--jclouds-- Authorization=Basic GpjbG9= ----jclouds-- Content-Disposition: form-data; name="sushi.avi"; filename="sushi.avi" Content-Type: application/octetstring ... PUT /ws/Metadata/SetMetadata.ashx?&path=Folders/adriansmovies/sushi.avi&sessionToken=session-token&metadata=Chef:Kawasaki HTTP/1.1 overwrite optional
  15. 15. CODE AND SIGN THE HTTP REQUEST POST /<api version>/containers/id_of_ adriansmovies/contents HTTP/1.1 Content-Length: 734859382 Content-Type=multipart/form-data; boundary=--jclouds-- Authorization=Basic GpjbG9= ----jclouds-- Content-Disposition: form-data; name="sushi.avi"; filename="sushi.avi" Content-Type: application/octetstring ... PUT /<api version>/files/from_above/metadata/Chef HTTP/1.1 Content-Length: 8 Content-Type: text/plain Authorization: Basic GpjbG9= Kawasaki 118 bytes overhead for mezeo form
  16. 16. do you want to • Deal with Errors • Deal with Concurrency • Deal with Cloud Complexity Deal with Errors: How about retries? XML implies encoding problems... Deal with Concurrency: Google app engine factor. Deal with Cloud Complexity: Server outages/errors, Upgrades to Cloud APIs, Eventual Consistency, Changing endpoints, Changing signing methods, Chunking large values
  17. 17. jclouds open source feels like java (and clojure) portability between clouds deal with web complexity unit testability thread-safe and scalable works in google app engine high performance thread safe enterprise ready
  18. 18. Tools we provide • Abstractions • BlobStore ( atmos, azure, rackspace, s3 ) • Compute ( vcloud, ec2, gogrid, ibmdev, rackspace, rimu ) • Clojure bindings • Third-party library integration
  19. 19. Alternatives to jclouds • Roll-your-own • Jersey, RESTEasy • EC2-based cloud apis • typica, jets3t • Dasein Cloud API • Service provided SDKs
  20. 20. BlobStore • Java Code • Clojure Code
  21. 21. java // init context = new BlobStoreContextFactory().createContext( "s3", accesskeyid, secretaccesskey); blobStore = context.getBlobStore(); // create container blobStore.createContainerInLocation(null, "mycontainer"); // add blob blob = blobStore.newBlob("test"); blob.setPayload("testdata"); blobStore.putBlob(containerName, blob);
  22. 22. commons vfs vfs > open blobstore://user:key@cloudfiles/mycontainer Opened blobstore://cloudfiles/mycontainer/ Current folder is blobstore://cloudfiles/mycontainer/ vfs > ls Contents of blobstore://cloudfiles/mycontainer/ README.txt 0 Folder(s), 1 File(s) vfs > close
  23. 23. clojure (ns demo   (:use org.jclouds.blobstore)   (:use clojure.contrib.pprint) )  (def blobstore (blobstore-context service account secret ))  (pprint (containers blobstore))  (pprint (blobs blobstore "mycontainer" ))
  24. 24. Provisioning • The Good, the Bad, and the Ugly • Java Code • Clojure Code
  25. 25. The Good provisioning (and re-provisioning) is cheap APIs = automation tools exist .10c/hr for a virtual machine, more for physical, billed for what you use APIs are ways developers can use and automate these services most cloud vendors offer SDKs to their services
  26. 26. The Bad forgetting to turn things off licensing erratic service quality licensing is a challenge from cost control, and also products you use may not be cloud friendly sometimes your VMs can disappear, and performance isn’t always consistent
  27. 27. The Ugly cloud apis are sometimes unreliable apis are very different across clouds features are very different across clouds accidental complexity cloud apis - can throw errors or have frequent upgrades features - build or buy load balancers, volumes complexity - needing to manually replicate images across regions, image conversion, polling
  28. 28. Things to consider when provisioning Can you create an image? Can you push credentials or files? Do you need to VPN in? How is storage provisioned? How close are your dependencies?
  29. 29. Java Code
  30. 30. jclouds github jclouds/jclouds service = new ComputeServiceContextFactory().createContext( “rimuhosting”, user, password ).getComputeService(); template = service.templateBuilder().any().biggest().build(); template.getOptions().installPrivateKey(privateRSA) .authorizePublicKey(publicRSA) .runScript(installGemsAndRunChef); nodes = service.runNodesWithTag(“webserver”, 5, template); focused on semantic portability across clouds I want an image running ubuntu and don’t want to know the id absolute portability where possible, but expose vendor apis where needed
  31. 31. dasein sourceforge dasein-cloud CloudProvider provider = providerClass.newInstance(); ProviderContext context = new ProviderContext(); context.setAccountNumber(accountNumber); context.setAccessPublic(apiKeyBytes); context.setAccessPrivate(privateKeyBytes); provider.connect(context); ServerServices services = provider.getServerServices(); server = services.launch(imageId, size, dataCenterId, serverName, keypairOrPassword, vlan, analytics, firewalls); focused on service portability under the enstratus platform richer service support
  32. 32. whirr github tomwhite/whirr ServiceSpec serviceSpec = new ServiceSpec(); serviceSpec.setProvider("gogrid"); serviceSpec.setAccount(account); serviceSpec.setKey(key); serviceSpec.setSecretKeyFile(secretKeyFile); serviceSpec.setClusterName(clusterName); service = new HadoopService(serviceSpec); ClusterSpec clusterSpec = new ClusterSpec( new InstanceTemplate(1, HadoopService.MASTER_ROLE), new InstanceTemplate(1, HadoopService.WORKER_ROLE)); cluster = service.launchCluster(clusterSpec); proxy = new HadoopProxy(serviceSpec, cluster); proxy.start(); multi-language service management now has zookeeper and hadoop support
  33. 33. jclouds-chef github jclouds/jclouds context = ChefContextFactory.createContext(server, identity, key); rsaPrivateKey = context.getApi().createClient(clientName); cookbooks = context.getApi().listCookbooks(); ChefAsyncClient nonBlockingClient = context.getAsyncApi(); nonBlockingClient.uploadCookbook(“apache2”, new File(“/cookbooks/apache2.tgz”));
  34. 34. (code clojure)
  35. 35. jclouds github jclouds/jclouds (def service (compute-service “ec2” account key :ssh :log4j)) (with-compute-service [service] (def template (build-template :run-script bootstrap)) (def node (run-node "couchdb" template)) (create-volume :node node :size 250)) mix/in proprietary and portable features
  36. 36. crane github clj-sys/crane (def hadoop-config (conf "/path/to/conf.clj")) (def compute (ec2 (creds "/path/to/creds.clj"))) (launch-hadoop-cluster compute hadoop-config) focus on aws and hadoop, but in the future will focus on clusters and multi-cloud
  37. 37. pallet github hugoduncan/pallet (defnode webserver []     :bootstrap [(public-dns-if-no-nameserver)               (automated-admin-user)]   :configure [(chef)]) (with-compute-service [service]   (converge {webserver 3})   (cook webserver "/cookbooks/apache-chef-demo")) serverless and agentless = uses ssh removes bootstrap complexity, like sudo, gems, etc. kick off processes like chef, or provision stacks such as hudson
  38. 38. Questions? adrian@opscode.org @jclouds http://blog.jclouds.org