Your SlideShare is downloading. ×
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply



Published on

Published in: Technology, Business

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide
  • Enabling migration between commercial and academic resources 100 node STAR run
  • What applications can use cloud computing and which ones cannot? What are the problems in practice? 2) NOT SOMETHING WE CAN RUN A SIMULATION FOR
  • Transcript

    • 1. Cloud Computing with Nimbus April 2009 Kate Keahey ( University of Chicago Argonne National Laboratory
    • 2. Cloud Computing IaaS Infrastructure-as-a-Service PaaS Platform-as-a-Service SaaS Software-as-a-Service elasticity computing on demand capital expense operational expense
    • 3. Cloud Computing for Science
      • Environment
      • Resource control
    • 4. “Workspaces”
      • Dynamically provisioned environments
        • Environment control
        • Resource control
      • Implementations
        • Via leasing hardware platforms: reimaging, configuration management, dynamic accounts…
        • Via virtualization: VM deployment
    • 5. A Brief History of Nimbus Research on agreement-based services Xen released First WSRF Workspace Service release EC2 gateway available Support for EC2 interfaces 2003 2009 2006 EC2 goes online First STAR production run on EC2 Nimbus Cloud comes online Context Broker release
    • 6. Nimbus Goals
      • Allow providers to build clouds
        • Private clouds (privacy, expense considerations)
        • Workspace Service: open source EC2 implementation
      • Allow users to use cloud computing
        • Do whatever it takes to enable scientists to use IaaS
        • Context Broker: turnkey virtual clusters
        • IaaS Gateway: interoperability
      • Allow developers to experiment with Nimbus
        • For research or usability/performance improvements
        • Community extensions and contributions
    • 7. Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node VWS Service The Workspace Service
    • 8. The Workspace Service Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node The workspace service publishes information about each workspace Users can find out information about their workspace (e.g. what IP the workspace was bound to) Users can interact directly with their workspaces the same way the would with a physical machine. VWS Service
    • 9. Workspace Service: Interfaces and Clients
      • Web Services based
      • Web Service Resource Framework (WSRF)
        • WS + state management (WS-Notification)
      • Elastic Computing Cloud (EC2)
        • Compatible with EC2 clients
        • Supported: ec2-describe-images, ec2-run-instances, ec2-describe-instances, ec2-terminate-instances, ec2-reboot-instances, ec2-add-keypair, ec2-delete-keypair
        • Unsupported: availability zones, security groups, elastic IP assignment, REST
    • 10. Workspace Service: Security
      • GSI authentication and authorization
        • PKI-based
        • VOMS, Shibboleth (via GridShib), custom PDPs
      • Secure access to VMs
        • EC2 key generation or accessed from .ssh
      • Validating images and image data
        • Extensions from Vienna University of Technology
        • Paper: Descher et al., Retaining Data Control in Infrastructure Clouds, ARES (the International Dependability Conference), 2009.
    • 11. Workspace Service: Networking
      • Network configuration
        • External: public IPs or private IPs (via VPN)
        • Internal: private network via a local cluster network
      • Each VM can specify multiple NICs mixing private and public networks (WSRF only)
        • E.g., cluster worker nodes on a private network, headnode on both public and private network
    • 12. Workspace Components workspace control workspace resource manager workspace pilot workspace client workspace service EC2 WSRF OpenNebula Project
      • See papers at:
      • “ Simple Leases with Workspace Pilot” (EuroPar08)
      • “ Combining Batch Execution and Leasing Using
      • Virtual Machines” (HPDC08),
    • 13. Cloud Capabilities workspace control workspace resource manager workspace pilot workspace client cloud client storage service workspace service EC2 WSRF
    • 14. The IaaS Gateway IaaS gateway EC2 potentially other providers workspace control workspace resource manager workspace pilot workspace client cloud client storage service workspace service EC2 WSRF
    • 15. Cloud Computing Ecosystem User Environments Appliance Providers Marketplaces, commercial providers, Virtual Organizations Appliance management software Deployment Orchestrator VMM/DataCenter/IaaS User Environments VMM/DataCenter/IaaS
    • 16. Turnkey Virtual Clusters
      • Turnkey, tightly-coupled cluster
        • Shared trust/security context
        • Shared configuration/context information
      MPI IP1 HK1 IP1 IP2 IP3 HK1 HK2 HK3 Context Broker IP2 HK2 IP1 IP2 IP3 HK1 HK2 HK3 IP3 HK3 IP1 IP2 IP3 HK1 HK2 HK3
    • 17. Context Broker Goals
      • Can work with every appliance
        • Appliance schema, can be implemented in terms of many configuration systems
      • Can work with every cloud provider
        • Simple and minimal conditions on generic context delivery
      • Can work across multiple cloud providers, in a distributed environment
    • 18. Context Broker Status
      • Releases
        • In alpha since 08/07, first release 06/08, update 01/09
      • Used to contextualize cluster composed of 100s of virtual nodes for multiple apps
      • Contextualized images on workspace marketplace
      • Working with rPath to make contextualization easier for the user
        • Discussing OVF extensions
      Paper: Keahey&Freeman, Contextualization: Providing One-Click Virtual Clusters, eScience 2008
    • 19. End of Nimbus Tour workspace control workspace resource manager workspace pilot workspace service workspace client cloud client IaaS gateway context broker context client EC2 potentially other providers storage service EC2 WSRF
    • 20. Science Clouds
      • Goals
        • Enable scientific projects to experiment with IaaS clouds
        • Evolve software in response to the needs of scientific projects
        • A laboratory for exploration of cloud interoperability issues
      • Participants
        • University of Chicago (since 03/08, 16 nodes), University of Florida (05/08, 16-32 nodes, access via VPN), Masaryk University, Brno, Czech Republic (08/08), Wispy @ Purdue (09/08)
        • In progress: Grid5K, Vrije, others
        • Using EC2 for large runs
      • Simple governance model, access given to any scientific project
    • 21. Who Runs on Nimbus at UC? 100+ DNs projects ranging across Science, CS, education, build&test…
    • 22. STAR
      • STAR: a nuclear physics experiment studies fundamental properties of nuclear matter
      • Computations require complex and consistently configured environments
      • Requirements
        • A virtual OSG STAR cluster: OSG headnode (gridmapfiles, host certificates, NFS, Torque), worker nodes: SL4 + STAR
        • From Science Clouds to EC2 runs
        • One-click virtual cluster deployment: Context Broker
      • Producing just-in-time results for Quark Matter conference:
      • Work by Jerome Lauret, Doug Olson, Leve Hajdu, Lidia Didenko at BNL
    • 23. Alice HEP Experiment at CERN
      • Collaboration with CERNVM project
      • HPCwire article
    • 24. Sky Computing U of Florida U of Chicago ViNE router ViNE router ViNE router Purdue
    • 25. Sky Computing
      • Papers:
        • “ Sky Computing”, by K. Keahey, A. Matsunaga, M. Tsugawa, J. Fortes. Submitted to IEEE Internet Computing.
        • “ CloudBLAST: Combining MapReduce and Virtualization on Distributed Resources for Bioinformatics Applications” by A. Matsunaga, M. Tsugawa and J. Fortes. eScience 2008.
      U of Florida U of Chicago Purdue Hadoop cloud
    • 26. Open Source IaaS Implementations
      • OpenNebula
        • Open source datacenter implementation
        • University of Madrid, I. Llorente & team, 03/2008
      • Eucalyptus
        • Open source implementation of EC2
        • UCSB, R. Wolski & team, 06/2008
      • Cloud-enabled Nimrod-G
        • Open source implementation of EC2
        • Monash University, MeSsAGE Lab, 01/2009
      • Industry efforts
        • openQRM, Enomalism
    • 27. Friends and Family
      • Committers: Kate Keahey & Tim Freeman (ANL/UC), Ian Gable (UVIC)
      • A lot of help from the community, see:
      • Collaborations:
        • Cumulus: S3 implementation (Globus team)
        • EBS: IU project
        • Appliance management: rPath, Bcfg2 project, CohesiveFT
        • Virtual network overlays: University of Florida
        • Security (research): Vienna University of Technology
    • 28. IaaS Clouds vs Grids
      • Grid computing
        • Assumption: site retains control over resources
        • Remote interfaces to local site mechanisms
        • Tradeoff: difficult to provide the right environments and control but easy to deploy
      • Cloud computing
        • Assumption: a user gets a “lease” on a remote resource that it gets to control
        • Enabled by virtauliaztion (Xen)
        • Tradeoff: eanbles a larger class of applications but hard to deploy
        • Raises issues: e.g., site licenses? Configuration support?
      • Towards “sky computing”
        • I can now trust a remote resource: I configured it myself
        • Cloud computing + virtual networks
        • Local distributed environment
    • 29. Parting Thoughts
      • Science-driven cloud computing
      • Importance of open source
        • Drive requirements into the infrastructure, customize
        • Drive the development of standards
      • Cloud computing for the user
        • Combine with what we have (grid computing)
        • Explore new potential
      • Future directions
        • Creating the ecosystem, working out the issues, e.g. licensing, appliance support
        • Interoperability and standards
        • Service Levels