Scientific Cloud Computing: Present & Future

420 views

Published on

Over the past five years, cloud computing has gone from a curiosity to
core scientific technology. The cloud's relative simplicity, instant
availability, and reasonable cost have made it attractive to
scientists, especially in domains relatively new to large scale data
analysis. This trend will continue into the foreseeable future,
challenging resource providers to adapt their services, to provide
easy federation with other providers, and to accommodate many
different scientific disciplines. For developers of cloud services,
there are also many challenges. Efficient access to, and the curation
of large data sets remain largely unsolved problems. Image
management also raises new issues, especially if these images are to
be shared and trusted. This presentation reviews the current status
of cloud computing and presents some ideas on how the upcoming
challenges might be met.

Presented at CNAF in Bologna, Italy by Charles Loomis in May 2013.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
420
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
21
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Scientific Cloud Computing: Present & Future

  1. 1. Scientific Cloud Computing: Present & FutureCharles (Cal) Loomis (CNRS/LAL & SixSq Sàrl)INFN CNAF, Bologna, Italy (22 May 2013)
  2. 2. 2Cloud Marketing“Cloud” is currently very trendy, used everywhere Many definitions that are often incompatible Used (often) to market pre-existing (non-cloud) softwareCommodityComputing(Sun)UtilityComputing(IBM,HP,…)AmazonEC2AmazonEBSMature VirtualizationSimple APIsExcess Capacity
  3. 3. 3In two pages NIST defines: Essential characteristics Deployment models Service modelsWhat is a Cloud?http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf
  4. 4. 4On-demand self-service No human interventionBroad network access Fast, reliable remote accessRapid elasticity Scale based on app. needsResource pooling Multi-tenant sharingMeasured service Direct or indirect economicmodel with measured useEssential Characteristicshttp://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf
  5. 5. 5Private Single administrative domain,limited number of usersCommunity Different administrative domainswith common interests & proc.Public People outside of institute’sadministrative domainHybrid Federation via combination ofother deployment modelsDeployment Modelshttp://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf
  6. 6. 6Software as a Service (SaaS) Direct (scalable) hosting of enduser applicationsPlatform as a Service (PaaS) Framework and infrastructurefor creating web applicationsInfrastructure as a Service (IaaS) Access to remote virtualmachines with root accessService Modelshttp://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf
  7. 7. 7Advantages No software installation Universally accessibleDisadvantages Questions about data access,ownership, reliability, etc. Integration of services & noveluses of data are (often) difficultTrends Social scientific computing Service APIs to allowintegration  PaaSSoftware as a Service (SaaS)
  8. 8. 8Advantages Programmers take advantage ofintegrated load balancing,automatic failover, etc.Disadvantages Restricted number of languages Applications strongly locked to aparticular providerTrends Dearth of “pure” PaaS offers Encroachment from both SaaSand IaaS sidesPlatform as a Service (PaaS)
  9. 9. 9Advantages Customized environment with“root” access Easy access to scalableresourcesDisadvantages Variety of APIs and interfaces VM image creation is difficultand time-consumingTrends Lots of specialized cloudproviders appearing Orchestration pushing intoPaaS spaceInfrastructure as a Service (IaaS)
  10. 10. 10Focus: Infrastructure as a Service
  11. 11. 11State of the ArtCommercial Provider: Amazon Web Services (AWS) Leading and largest IaaS service provider Improving and adding new services at a phenomenal rate Almost all IaaS providers use AWS-like service semantics, butdifferentiate based on price, SLAs, location, etc.Commercial Cloud Distribution: VM-ware VM-ware: extremely good and complete, but very expensive Provide ESXi virtual machine host for freeOpen Source Cloud Distributions Essentially none in 2004; now easily a dozen different distributions StratusLab, WNoDeS, …, OpenStack, OpenNebula, CloudStack Very different levels of maturity, stability, scalability, etc.
  12. 12. 12Why are cloud technologies useful?For (scientific) users Custom environment: no rewriting or porting applications to fit into aresource provider’s environment Simple access: most providers use a REST or RPC API allowingsimple access from all programming languages Reasonable cost: only pay for what resources are used, especiallyattractive for individuals/groups that do not have large, existinghardware investment
  13. 13. 13Why are cloud technologies useful?Separation of responsibilities Hardware / Services / Platforms / Users People at each layer can focus on their responsibilities with minimalinteractions with people in the other layers.Resource Providers Better utilization of shared resource because wider range ofapplications (and disciplines) can use the cloud With hybrid cloud infrastructures, providers can outsource excessdemand to other providers
  14. 14. 14Trend for Scientific ComputingWill we all just be users of the Amazon cloud?Pendulum swinging towards large data centers with “fat” machines These can offer elastic cloud services at a reasonable price With scientific clouds there is low barrier to entry and users canmaintain administrative control of services and data Providing shared resource between scientific disciplines much easierbecause of virtualizationMigration will be gradual…
  15. 15. 15Overcoming inertia…Users  How to use virtual machines to get my work done?  How to structure, store, access, and protect data?  Realize shared infrastructures with customized env. are possibleApplication Developers  How to use cloud techniques to improve my applications?  … and my development workflows?  Applications can be services (with assoc. pluses and minuses)Data Centers  Reuse existing (commodity) hardware investments  Take advantage of (and train) existing system administrators  How to manage/use a (private, community, public) cloud?Significant benefits from cloud even without large scale elasticity!
  16. 16. 16Challenges
  17. 17. 17ElasticityCan we have infinite elasticity with limited resources?“Local” solutions Economic models to avoid hitting infrastructure limits? Spot instances and/or different service classes? Aside: IPv6 addressing is necessary for large (scientific) cloudsFederated solutions Hybrid (scale-out) infrastructures? Higher-level brokers or orchestrators?Cannot have elasticity without some kind of accounting!
  18. 18. 18Data Management (Legal)Transfer and treatment of data across borders Differing legal protections in different jurisdictions Legal constraints for data locality (banking, medical data) Unclear responsibilities for data: guardian, custodian, owner, etc. Europe working hard to come to a consistent legal frameworkProtection of data in the cloud Consistent access controls for all data locations Guarantees about data protection from cloud provider personnel Reliability of the provider’s storage Knowledge about provider’s policies for data protection
  19. 19. 19Data Management (Technical)Efficient exploitation of large datasets Need significant computing next to storage, AND/OR High bandwidth remote accessLocality matters Sometimes it is inconvenient to transfer raw data away from instrument Clouds can be used to reduce data locally before transfer“Open Data” requirements Cloud may help meet such requirements Does not remove need for well defined dataset metadata and format Need long-term funding for the curation of those datasets
  20. 20. 20SecurityHow to maintain the security of a cloud infrastructure?Shifting some responsibility to users: Users have root access and must secure services within their VMs Users have less security experience  need for education & help Dynamic network configurations can help improve securityChanging expectations from administrators: Leave firewall policies to users running VMs Should not expect to run security software inside of VMs Need to enhance monitoring to discover abnormal behavior
  21. 21. 21Image ManagementImage metadata What does an image contain (OS, services, configuration, etc.)? What versions of the kernel, software, etc. are included? Who is responsible and/or supports a given image? How do I identify a given image?Creating machine images How can an image be created for multiple clouds? What do I have to do to create a secure machine image?Sharing images How can I make my images available to others? Can I parameterize my images to make them useful to more people? Can the images be transported and used efficiently?
  22. 22. 22Vendor Lock-in vs. FederationCommon API Fully interoperable API avoids duplication in cloud control software Has very limited impact on applications and services in the cloud Current (quasi-) standards: EC2, OCCI, CIMI, CDMI, …Common Semantics Semantics determine how apps and services operate in the cloud For IaaS, cloud providers have a broadly similar semantics, but… File-based and block storage is one difference.Contextualization Can a user run the same validated image on all clouds? Can a user share the same parameterized image with others? Neglected issue in standardization; CloudInit becoming de facto std.
  23. 23. 23Solutions
  24. 24. 24StratusLab HistoryInformal collaboration to investigaterunning grid services on AmazonEC2 (2007)StratusLab Project (6/2010 to5/2012) co-funded by EC with6 partners from 5 countriesOpen collaborationto continue thedevelopment and support ofthe StratusLab softwareWebsite: http://stratuslab.euTwitter: @StratusLabSupport: support@stratuslab.euSource: http://github.com/StratusLabIdentified need for opensource cloud distribution.Production dist. with academic& commercial deployments.
  25. 25. 25StratusLabComplete Infrastructure as a Service cloud distribution Developed within EU project, software maintained by partners Focus: Simple to install and simple to useServices Compute: Virtual machine management (currently uses OpenNebula) Storage: Volume-based storage service Network: Simple configuration for public, local, and private VM access Image mgt.: Complete system for trusted sharing of VM images Tools (python CLI) and APIs (Libcloud) to facilitate use of cloud Tools to facilitate installation of services
  26. 26. 26SlipStreamCloud orchestrator and deployment engine Facilitates testing, deployment, and maintenance of complex systems Transparent accessto multiple cloudinfrastructures Allows automateddeployment of systemsin one or more clouds
  27. 27. 27Image Management
  28. 28. 28Image ManagementImage metadata What does an image contain (OS, services, configuration, etc.)? What versions of the kernel, software, etc. are included? Who is responsible and/or supports a given image? How do I identify a given image?Creating machine images How can an image be created for multiple clouds? What do I have to do to create a secure machine image?Sharing images How can I make my images available to others? Can I parameterize my images to make them useful to more people? Can the images be transported and used efficiently?
  29. 29. 29Image Management Actors
  30. 30. 30MarketplacePriorities Mechanism for sharing and trusting images Possible to distribute fixed, read-only data sets as well Split the storage of image metadata and image contents Define roles for creator, user, administrator, and validatorImplementation Marketplace API: Proprietary REST API for create, read, search Marketplace acts as image registry and handles only metadata Image contents can be located on any public (web) server ‘Private’ images can also be held in cloud storage
  31. 31. 31Marketplace: Key service in larger ecosystemTrust Marketplace metadata plays key role in providing information about theimage contents and provenanceFactories stratus-create-image facilitates customization of images VirtualBox and other local virtual tools Bitnami and many similar services for build of standard OS services Quattor and other fabric management can be usedTransport (& Storage) StratusLab uses simple HTTP(S) for image transport Could imagine using ftp, gridftp, bittorrent, etc. vm-caster/catcher in EGI federated cloud task force
  32. 32. 32Image Handling Workflow
  33. 33. 33Cloud Federation
  34. 34. 34StratusLab Federated Cloud InfrastructureFeatures Two sites operating (LAL and GRNET) for ~3 years Common user authentication Ability to use the same images across resources StratusLab client allows easy switching between sites StratusLab Libcloud binding allows common view of both sitesNeed to go further… To sites running different cloud software Helix Nebula and EGI Fed. Cloud Task Force active in this area
  35. 35. 35Federation ModelsTransparent Federation Site operators “outsource” to other providers Completely transparent to end users Difficult to achieve in practice because of data protection concerns andnetwork access/performanceBrokered Federation Variety of different cloud infrastructures are visible to users Users choose to place virtual machines in particular locations Simple clients can handle federation if differences are small Orchestrators are needed for larger differences between cloudsBoth Helix Nebula and EGI take the brokered approach
  36. 36. 36Vendor Lock-in vs. FederationCommon API — minor issue Fully interoperable API avoids duplication in cloud control software Has very limited impact on applications and services in the cloud Current (quasi-) standards: EC2, OCCI, CIMI, CDMI, …Common Semantics — important issue Semantics determine how apps and services operate in the cloud For IaaS, cloud providers have a broadly similar semantics, but… File-based and block storage is one difference.Contextualization — critical issue Can a user run the same validated image on all clouds? Can a user share the same parameterized image with others? Neglected in standardization but CloudInit becoming de facto std.
  37. 37. 37“Our” ContributionsStratusLab Adopt CIMI as the standard interface to services Provide Libcloud (python) driver for StratusLab Provide EC2, OCCI, etc. as adaptors to CIMI interface Move to CloudInit for contextualization framework Flexible authentication system to support different methodsSlipStream Chosen as main interface to supported clouds in Helix Nebula Already supports multi-cloud deployments Support for all clouds within Helix Nebula
  38. 38. 38Running Clouds in Production
  39. 39. 39Cloud Experience at LALPrivate cloud for laboratory services Works well, plan to migrate all services including grid worker nodesand experiment-specific servers Services switched to VMs without users being aware of change Very different way of working, need to change administrator habits Have seen some stability issues related to SL6 kernel/virtualizationPublic cloud open to university Very positive reaction to cloud; LAL resources nearly 100% used Variety of disciplines: biology, software eng., statistics, astrophysics,bioinformatics, … After initial introduction, users require only low level of support Other labs offering StratusLab training without our direct involvement
  40. 40. 40Summary
  41. 41. 41SummaryCloud for scientific computing Already a common tool for many scientific disciplines Cloud technologies will become more pervasive with time Associated swing back to larger, centralized data centersMany challenges for cloud Elasticity with limited resources Data management (legal & technical) Security Image management — unique holistic approach from StratusLab Federation — brokered federation with StratusLab & SlipStreamLAL cloud experience Very positive feedback from both administrators and users
  42. 42. 42Questions and DiscussionWebsite: http://stratuslab.eu/Twitter: @StratusLabSupport: support@stratuslab.euSource: http://github.com/StratusLabSixSq: http://sixsq.comSlipStream: https://github.com/organizations/slipstream
  43. 43. http://stratuslab.eu/Copyright © 2013, Members of the StratusLab collaboration.This work is licensed under the Creative Commons Attribution 3.0Unported License (http://creativecommons.org/licenses/by/3.0/).

×