Cloud Computing Architecture with Open Nebula - HPC Cloud Use Cases - NASA Ames 2012

NASA Ames NASA Advanced Supercomputing (NAS) Division California, May 24th, 2012 Cloud Computing Architecture with OpenNebula HPC Cloud Use Cases Ignacio M. Llorente Project Director© OpenNebula Project. Creative Commons Attribution-NonCommercial-ShareAlike License

ContentsCloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! This presentation is about: •  Cloud case studies in HPC •  Private cloud computing using OpenNebula •  The anatomy of a Private Cloud •  OpenNebula innovative features for HPCCloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 2/26

What is OpenNebula?Open Cloud Solution for Building and Managing Virtualized Data Centers! Public Private HybridCloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 3/26

The HPC Cloud Use CaseThe Pre-cloud Era! Access Grid Middleware LRMS (LSF, PBS, SGE…) ProvisionCloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 4/26

The HPC Cloud Use CaseOpenNebula as an Infrastructure Tool! Access Service/Provisioning Decoupling! •  Common interfaces •  Grid integration Grid Middleware LRMS (LSF, PBS, SGE…) •  Custom environments Service •  Dynamic elasticity Virtual Worker Nodes •  Consolidation of WNs •  Simpliﬁed management Provision •  Physical – Virtual WNs •  Dynamic capacity partitioning •  Faster upgradesCloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 5/26

The HPC Cloud Use CaseOpenNebula as an Provisioning Tool! •  Simple Provisioning Interface Access •  Raw/Appliance VMs IaaS Interface Pilot Jobs, SSH… Service •  Dynamic scalable computing •  Custom access to capacity •  Not only batch workloads •  Not only scientiﬁc workloads Provision •  Improve utilization •  Reduced service management •  Cost eﬃciencyCloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 6/26

The HPC Cloud Use CaseExamples: CERN’s lxcloud! http://blog.opennebula.org/?p=620 Goal •  Virtualized HTC Batch Nodes: limited life-time, dynamic capacity •  EC2 Query: CernVM Deployment Notes •  Custom network integration MAC/IP pinning •  Fast image distribution & boot: BitTorrent + LVM snapshots •  OpenNebula tests up to 20,000 VMs (great feedback!)Cloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 7/26

The HPC Cloud Use CaseExamples: FermiCloud! http://www-fermicloud.fnal.gov/ Goal •  Scientiﬁc stakeholders get access to on-demand VMs •  Developers & integrators of new Grid applications •  MPI and legacy applications Deployment Notes •  VMs access Fermilab Networking and Storage Services •  OpenNebula + X509 support (contributed back!) •  Other areas: HA, Batch queues look-ahead, cluster on-demand…Cloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 8/26

The HPC Cloud Use CaseExamples: SARA! https://www.cloud.sara.nl/ Goal •  HPC clusters on-demand: Self service, elastic and fully configurable HPC systems •  Offer production infrastructure for several engineering and scientific communities: Bioinformatics, ecology… Deployment Notes •  Low latency network for HPC •  Extensions in accounting, GUI, firewalling… (contributed back!)Cloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 9/26

The HPC Cloud Use CaseOne of Our Main User Communities! Supercomputing Centers Research Centers Distributed Computing InfrastructuresCloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 10/26

The Anatomy of the CloudDifferent Perspectives of the Cloud! Cloud UserCloud Provider Cloud IntegratorCloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 11/26

Using the CloudHow Can I Create and Manage my Virtual Infrastructure?!… standards (de facto and de jure) Cloud APIs to leverage existing ecosystems and ensure portability across providers, and self-service portal …. " Use AWS existing tools like ElasticFox •  Manage images, " Execute complete networks and computing cluster instances " Manage persistent scientiﬁc data " Catalog with pre-deﬁned environments " Sharing between users or groupsCloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 12/26

Building the CloudWhat are the Main Components to Build a Cloud Infrastructure?! Instance Networks Internet •  Guests •  Public and private networks " Fair share of resources " Place VM close to data Front-end Hosts •  Authentication •  Multiple hypervisors •  Authorization •  Up to 500 hosts •  ACLs, roles, •  Automatic failover and HA groups… •  Automatic resource allocation •  Accounting •  Logging •  Resource quotas Service Networks Datastores •  Monitoring, control… •  VM image storage •  Live migration… •  Multiple datastores •  Storage access… •  Heterogeneous conﬁgurations •  Shared or non-sharedCloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 13/26

Building the CloudBroad Commodity and Enterprise Platform Support! Instance Network •  VLAN per user (layer2) Internet •  Open vSwitch, 802.1q •  Ebtables Front-end Hosts •  X509, LDAP, ssh keys •  VMware •  ACLs, roles, groups… •  Xen •  KVM Datastores Service Network •  NAS: NFS, Gluster, GlusterFS… •  Ganglia/Nagios •  SAN: Fibre Channel, iSCSI, LVM… •  Additional monitor agents •  DFS: Bit Torrent, ssh…Cloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 14/26

Innovative FeaturesClustering the Physical Resources! Clusters •  Pools of hosts that share datastores and networks •  Group different configurations for performance and security " Different Hw for different workloads profiles (HPC vs HTC) " Isolation and security levels Multiple Datastores per Cluster •  Balance I/O operations between storage servers •  Define different SLA policies (e.g. backup) and performance features for different VM types or usersCloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 15/26

Innovative FeaturesCentralized Management of Multiple OpenNebula Instances (Zones)! " Full Isolation Clouds Federation of of performance-sensitive •  Multi-tier architecture •  Scalability applications oZones Server •  Isolation " Isolation and security •  Portal •  Multiple-site support levels Cloud •  Cloud API (EC2, OCCI) " Multi-tier Consumer •  Global AuthN " Scalability " Multiple-siteCloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 16/26

Innovative FeaturesOn-demand Provision of Virtual Data Centers!" Virtual Private Cloud Computing with Provide VOs or Departments ● their own isolated cloud Typical scenario in large organizations and cloud providers compartment" ● Associated provision of fully- On-demand to clusters with specific Hw and isolated VDC with full configurable and Sw profiles, or control and capacity to administer its security levels users and resourcesCloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 17/26

Innovative FeaturesHybrid Cloud Computing! Hybrid ● Extension of the local private infrastructure with resources from remote clouds ● Cloudbursting to meet peak or ﬂuctuating demands Cloud User " HTC workloads " Managed by the admin Cloud ProviderCloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 18/26

Operating the CloudHow Can I Operate my Cloud Infrastructure?!… programming APIs (create new tools and integrate), web interfaces (simplify operation), and command lined interface (create scripts)… " Automatic/Elastic Management of Computing Service CLI API ●  XML-RPC ●  OCA (Ruby, Java, Python)Cloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 19/26

Integrating the CloudThe Ecosystem! … truly open (fully open-source, Apache license) and adaptable (modular and extensible)… because no two data centers are the same Standards Adapters CDMI OVF CIMI Virtualization Drivers Conﬁguration StorageCloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 20/26

About the OpenNebula ProjectA Project Aimed at Building the Industry Standard Open Cloud Management Tool! Third party scalability • Develop & innovate tests: 16,000 VMs • Support the community • Collaborate Commercial Support TP v1.0 v1.2 v1.4 v2.0 v2.2 V3.0 V3.2 V3.42005 2008 2009 2010 2011 2012dsa group doing research… 4,000 downloads/ month European FundingCloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 21/26

About the OpenNebula ProjectA Quickly Growing Community More than Doubling Each Year! 25,200 Downloads ●  900 downloads in the last week 12,828 ●  Linux distro and code repo 4,8611,865 140% Annual Growth 2008 2009 2010 2011 Site Visits 579,571 ●  15,300 visits and 194,000 page 254,186 views in the last week 111,54135,842 150% Annual Growth 2008 2009 2010 2011Mails 4,341 ●  800 registered users at present 2,293 1,141 227 170% Annual Growth Date: November 7th, 2011 2008 2009 2010 2011Cloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 22/26

About the OpenNebula ProjectOpen Cloud Enabler in the ICT Industry! Enabling Hosting Companies and Telcos to Offer Cloud Services Enabling Technology Companies to Offer Cloud Products Enabling Service Companies to Offer Cloud Consulting and IntegrationCloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 23/26

About the OpenNebula ProjectOpen Cloud Enabler for Building and Operating Virtualized Data Centers! Industry Supercomputing Centers Research CentersCloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 24/26

About the OpenNebula ProjectOpen Cloud Enabler for Building and Research and Innovation! Distributed Computing Infrastructures Research Projects 16,000 VMs!Cloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 25/26

Questions?We Will Be Happy to Answer Any Question !Cloud Computing Architecture with OpenNebula - HPC Cloud Use Cases! 26/26

http://blog.opennebula.org	1919
http://blog.cloudplan.org	707
http://4973727423887134232_38b62c36e6cbbc4052db5743c6258cb60002f8df.blogspot.com	39
http://ignaciollorente.ulitzer.com	25
http://eclipse.sys-con.com	17
http://virtualization.sys-con.com	16
http://www.linkedin.com	12
http://translate.googleusercontent.com	8
http://sap.sys-con.com	7
http://opennebula.ulitzer.com	6
http://cloudcomputing.sys-con.com	5
http://opensource.ulitzer.com	3
http://cloudonomics.ulitzer.com	3
http://www.sys-con.com	3
http://infrastructure20.ulitzer.com	3
http://python.sys-con.com	3
http://ignaciollorente.sys-con.com	2
http://javacloud.ulitzer.com	2
http://opensource.sys-con.com	2
http://ciocto.ulitzer.com	1
http://127.0.0.1	1
http://seoautomated.com	1

Cloud Computing Architecture with Open Nebula - HPC Cloud Use Cases - NASA Ames 2012

by Ignacio M. Llorente

Accessibility

Categories

Upload Details

Usage Rights

22 Embeds 2,785

Statistics

Cloud Computing Architecture with Open Nebula - HPC Cloud Use Cases - NASA Ames 2012 Presentation Transcript