Building Private Clouds for HPC with OpenNebula: Reference Deployments & Lessons Learned

  • 1,771 views
Uploaded on

 

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,771
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
36
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Building Private Clouds for HPC with OpenNebula
  • 2. Building Private Clouds for HPC with OpenNebula: Reference Deployments & Lessons Learned
    Cloud Computing in a Nutshell
    HPC & IaaS Clouds: Two+ Basic Approaches
    Sample Deployments: CERN & Fermilab
    Lessons Learned
    A Third Approach: Grid & Cloud Computing
  • 3. Cloud Computing Disciplines
    An Introduction to Cloud Computing
    Software as a Service
    Platform as a Service
    Infrastructure as a
    Service
    Physical Infrastructure
  • 4. Infrastructure as a Service (IaaS)
    An Introduction to Cloud Computing
    Public Cloud
    • Simple Web Interface
    • 5. Raw Infrastructure Resources
    • 6. Pay-as-you-go (On-demand access)
    • 7. Elastic & “infinite” Capacity
  • Infrastructure as a Service (IaaS)
    An Introduction to Cloud Computing
    Public Cloud
    • Simple Web Interface
    • 8. Raw Infrastructure Resources
    • 9. Pay-as-you-go (On-demand access)
    • 10. Elastic & “infinite” Capacity
    Private Cloud
    A “Public Cloud behind the firewall”
    • Simplify internal operations
    • 11. Dynamic allocation of resources
    • 12. Higher utilization & operational savings
    • 13. Security concerns
  • Infrastructure as a Service (IaaS)
    An Introduction to Cloud Computing
    Public Cloud
    • Simple Web Interface
    • 14. Raw Infrastructure Resources
    • 15. Pay-as-you-go (On-demand access)
    • 16. Elastic & “infinite” Capacity
    Private Cloud
    A “Public Cloud behind the firewall”
    • Simplify internal operations
    • 17. Dynamic allocation of resources
    • 18. Higher utilization & operational savings
    • 19. Security concerns
    Hybrid Cloud
    • Suplement the capacity of the Private Cloud
    • 20. Utility Computing dream made a reality!
  • High Performance & IaaS Cloud Computing
    The pre-cloud time
    Access
    Grid Middleware
    LRMS (LSF, PBS, SGE…)
    The pre-virtualization/cloud
    times
    Provision
  • 21. High Performance & IaaS Cloud Computing
    Two Basic Approaches: AnInfrastructureTool
    • Traditional interfaces
    • 22. Grid Integration
    • 23. Batch Processing
    • 24. Custom Execution Environments
    Access
    Grid Middleware
    LRMS (LSF, PBS, SGE…)
    Service-Provisioning
    Decoupling
    Service
    Virtual WorkerNodes
    • Consolidation of WNs
    • 25. Simplified management
    • 26. Physical – Virtual WNs
    • 27. Dynamic capacity partitioning
    Provision
  • 28. High Performance & IaaS Cloud Computing
    Two Basic Approaches: A ProvisioningTool
    • Simple Provisioning Interface
    • 29. Raw/Appliance VMs
    Access
    IaaS Interface
    PilotJobs, SSH…
    • Custom access to capacity
    • 30. Not only batch workloads
    • 31. Not only scientific workloads
    Service
    • Improve utilization
    • 32. Reduced service management
    • 33. Cost efficiency
    Provision
  • 34. High Performance & Cloud Computing
    Usinghybridcloudcomputing
    Provision
    External Providers
    Physical Resources
  • 42. Examples of HPC Cloud Deployments
    TheCERN’slxcloud
    Goal & Characteristics
    Virtualization of Batch Nodes: limted life-time, dynamic capacity
    Golden Nodes + CERNVM
    EC2 Query – OpenNebula API
    Some Technical Notes
    Custom Network Integration MAC/IP pinning
    Disk I/O Tunning
    Fast boot of VMs: Bittorrent + LVM snapshots
    OpenNebula tests up to 16000 VMs (great feedback!)
  • 43. Examples of HPC Cloud Deployments
    Fermicloud
    Goal & Characteristics
    Scientific stakeholders get access to VMs (no sysadmin needed)
    Developers & Integrators of new Grid applications
    Improve utilization of power, cooling and humans
    Legacy applications
    Some Technical Notes
    VMs access Fermilab Networking and Storage Services
    Using KVM
    OpenNebula + X509 support (contributed back!)
    Other Areas: HA, Batch queues look-ahead, cluster on-demand…
  • 44. Lessons Learned
    HPC & Cloud Computing
    Do not hide the Complexity, Architect the Cloud!
    • Let IT staff adapt/desgin/develop their cloud deployments
    • 45. Design to Automate
    • 46. Scale your deployments: Storage
    Be Interoperable, Have a Rich set of API’s & Interfaces
    • Be prepared for completely different uses cases (don’t look at the service)
    • 47. Be prepared to deal with different solutions (storage, hypervisors, networks)
    • 48. Be prepared to interoperate at different levels (API’s, Cloud services)
    And for HPC Cloud builders…
    • Scientists want it their way (VM images, do not put access restrictions)
    • 49. Deal with tunning : CPU & I/O (image formats, hypervisor I/O policies)
  • A Third Approach: Grid and Clouds
    TheStratusLab Project
    Clouds
    Grids
    Usage
    • Raw infrastructure
    • 50. Elasticity & Pay-per-use
    • 51. Simple Web Interface
    Achievements
    • Agile Infrastructures
    • 52. IT another Utility
    But…
    • Interoperability
    • 53. Federation
    Usage
    • Job Processing
    • 54. Big Batch System
    • 55. File Sharing Services
    Achievements
    • Federation of Resources
    • 56. VO Concept
    But…
    • User experience
    • 57. Complexity
    Resource Sharing
    Resource Management
    Scientific Applications
    Customize Environments
    Flexibility & Simplicity
    Uniform Security
  • 58. A Third Approach: Grid & Clouds
    TheStratusLab Project
    GridServices
    MarketPlace
    • Federation facilities
    • 59. Security
    • 60. Sharing existing VM images
    Multi Cloud
    Cloud/GridSites
    GridServices
    Cloud API
    GridServices
    Cloud API
    Cloud Bursting
    • Needs to consider Image & VLANs
    • 61. Needs to consider placement
    ExternalClouds
  • 62. Building Private Clouds for HPC with OpenNebula: Reference Deployments & Lessons Learned
    @ruben_s_montero
    blog.opennebula.org
    The research leading to these results has received funding from the European Union's Seventh Framework Programme ([FP7/2007-2013] ) under grant agreement n°261552 (StratusLab Project)
    © OpenNebula Project. Creative Commons Attribution-NonCommercial-ShareAlike License