This document discusses bringing private cloud computing to high-performance computing (HPC) and science. It covers the private HPC cloud use case, main challenges, and case studies from several research centers that have implemented private HPC clouds using OpenNebula. Some key challenges discussed are flexible application definitions, resource management at scale, performance considerations, and hybrid cloud deployments. Private cloud trends from industry are also briefly summarized.
Utilizing Public AND Private Clouds with Bright Cluster ManagerIan Lumb
Slides corresponding to a webinar (http://hubs.ly/y0F-j80) given on March 25, 2015 for Bright Computing.
Key takeaways:
How Bright Cluster Manager allows you to seamlessly make use public clouds like Amazon Web Services (AWS)
How Bright Cluster Manager allows you to rapidly deploy a private cloud based on OpenStack
The recording (http://hubs.ly/y0F-j80) includes live-product demonstrations using Bright Cluster Manager.
Regarding Clouds, Mainframes, and Desktops … and LinuxRobert Sutor
In this talk, I'll focus on three areas of great opportunity as well as challenge for Linux: the accelerating market for cloud computing, Linux as a significant operating system for mainframes, and the hope for Linux on the desktop.
Reference architecture with MIRANTIS OPENSTACK PLATFORM.The changes that are going on in IT with disruptions from technology, business and culture and so IT to solve the issues has to change from moving from traditional models to broker provider model.
Utilizing Public AND Private Clouds with Bright Cluster ManagerIan Lumb
Slides corresponding to a webinar (http://hubs.ly/y0F-j80) given on March 25, 2015 for Bright Computing.
Key takeaways:
How Bright Cluster Manager allows you to seamlessly make use public clouds like Amazon Web Services (AWS)
How Bright Cluster Manager allows you to rapidly deploy a private cloud based on OpenStack
The recording (http://hubs.ly/y0F-j80) includes live-product demonstrations using Bright Cluster Manager.
Regarding Clouds, Mainframes, and Desktops … and LinuxRobert Sutor
In this talk, I'll focus on three areas of great opportunity as well as challenge for Linux: the accelerating market for cloud computing, Linux as a significant operating system for mainframes, and the hope for Linux on the desktop.
Reference architecture with MIRANTIS OPENSTACK PLATFORM.The changes that are going on in IT with disruptions from technology, business and culture and so IT to solve the issues has to change from moving from traditional models to broker provider model.
Get the inside scoop on what Citrix and Nutanix are up to. In this session, you will discover the collaborative projects and visions that we are most excited about.
Watch the session on SynergyTV and follow along with these slides. http://live.citrixsynergy.com/2016/player/ondemandplayer.php?presentation_id=f2ededd4-64a0-47aa-bbbd-208d62128038
Comparing open source private cloud platformsOSCON Byrum
Private cloud computing has become an integral part of global business. While each platform provides a way for virtual machines to be deployed, implementations vary widely. It can be difficult to determine which features are right for your needs. This session will discuss the top open source private cloud platforms and provide analysis on which one is the best fit for you.
Deep dive: Citrix CloudPlatform for Infrastructure as a ServiceCitrix
Citrix innovation continues to advance our cloud management technology at a rapid rate to keep pace with emerging enterprise customer needs. Learn about the core capabilities and newest innovations for Citrix CloudPlatform, which is powering the world's largest clouds today. CloudPlatform provides the latest and most advanced open source software platform to build highly scalable and reliable cloud computing environments. You’ll also see why Citrix cloud solutions differ from VMware and OpenStack offerings.
The enterprise datacenter is undergoing a massive transformation. Is your organization prepared for what’s next?
Share in proven strategies for delivering frictionless IT services while retaining the precise control your business needs.
See the groundbreaking new Nutanix platform capabilities that tear down IT silos and unify the technology stack.
Engage with peers on best practices in virtualization, application design and cloud technologies
Build clouds the way some of the world’s biggest public and private clouds are built—using CloudStack. This 60-minute webinar with the Cloudstack team will help you gain a better understanding of the CloudStack architecture and feature set.
OpenStack: Everything You Need To Know to Get Started (ATO2014)Mark Voelker
Slides for my talk at All Things Open 2014
OpenStack is widely recognized as a leading open source cloud computing platform and has attracted plenty of attention from developers, end users, IT companies, and media. As OpenStack continue to gain adoption, the audience of potential users continues to expand. Whether you’re building a public cloud service or private clouds for e-commerce, video/collaboration apps, sceintific research, NFV, or are simply looking for a more elastic model of infrastructure, OpenStack is an option to consider. This talk will serve as an extensive introduction for newcomers to OpenStack. We’ll discuss both the software itself and the makeup of the community of developers and users around it. We’ll learn how to contribute to OpenStack, who’s using it today, different deployment scenarios and use cases, and provide both online and local resources for learning more. We’ll also provide an introduction to incubated components, underpinning pieces, and pointers to installers and service providers who can help you get started.
Which cloud-based infrastructure service is right for you? CDW compares Microsoft Azure, HP Helion, VMware vCloud Air, IBM SoftLayer and Amazon Web Services.
Get the inside scoop on what Citrix and Nutanix are up to. In this session, you will discover the collaborative projects and visions that we are most excited about.
Watch the session on SynergyTV and follow along with these slides. http://live.citrixsynergy.com/2016/player/ondemandplayer.php?presentation_id=f2ededd4-64a0-47aa-bbbd-208d62128038
Comparing open source private cloud platformsOSCON Byrum
Private cloud computing has become an integral part of global business. While each platform provides a way for virtual machines to be deployed, implementations vary widely. It can be difficult to determine which features are right for your needs. This session will discuss the top open source private cloud platforms and provide analysis on which one is the best fit for you.
Deep dive: Citrix CloudPlatform for Infrastructure as a ServiceCitrix
Citrix innovation continues to advance our cloud management technology at a rapid rate to keep pace with emerging enterprise customer needs. Learn about the core capabilities and newest innovations for Citrix CloudPlatform, which is powering the world's largest clouds today. CloudPlatform provides the latest and most advanced open source software platform to build highly scalable and reliable cloud computing environments. You’ll also see why Citrix cloud solutions differ from VMware and OpenStack offerings.
The enterprise datacenter is undergoing a massive transformation. Is your organization prepared for what’s next?
Share in proven strategies for delivering frictionless IT services while retaining the precise control your business needs.
See the groundbreaking new Nutanix platform capabilities that tear down IT silos and unify the technology stack.
Engage with peers on best practices in virtualization, application design and cloud technologies
Build clouds the way some of the world’s biggest public and private clouds are built—using CloudStack. This 60-minute webinar with the Cloudstack team will help you gain a better understanding of the CloudStack architecture and feature set.
OpenStack: Everything You Need To Know to Get Started (ATO2014)Mark Voelker
Slides for my talk at All Things Open 2014
OpenStack is widely recognized as a leading open source cloud computing platform and has attracted plenty of attention from developers, end users, IT companies, and media. As OpenStack continue to gain adoption, the audience of potential users continues to expand. Whether you’re building a public cloud service or private clouds for e-commerce, video/collaboration apps, sceintific research, NFV, or are simply looking for a more elastic model of infrastructure, OpenStack is an option to consider. This talk will serve as an extensive introduction for newcomers to OpenStack. We’ll discuss both the software itself and the makeup of the community of developers and users around it. We’ll learn how to contribute to OpenStack, who’s using it today, different deployment scenarios and use cases, and provide both online and local resources for learning more. We’ll also provide an introduction to incubated components, underpinning pieces, and pointers to installers and service providers who can help you get started.
Which cloud-based infrastructure service is right for you? CDW compares Microsoft Azure, HP Helion, VMware vCloud Air, IBM SoftLayer and Amazon Web Services.
Bringing Private Cloud Computing to HPC and Science - Berkeley Lab - July 2014 OpenNebula Project
Berkeley Lab – Computing Sciences Seminar
HPC-optimized clouds provide access to flexible and elastic scientific and technical computing to solve complex problems and drive innovation. The talk will describe the most demanded features for building HPC and science clouds, and will illustrate using real-life case studies from leading research and industry organizations how OpenNebula effectively addresses these challenges of cloud usage, scheduling, security, networking and storage. The keynote will end with a view of private cloud's future in HPC and science, and grid as the foundation of cloud federation.
Introduction to Cloud Computing - CCGRID 2009James Broberg
Cloud computing has recently emerged as an exciting new trend in the ICT industry. Several IT vendors are promising to offer on-demand storage, application and computational hosting services, and provide coverage in several continents, offering Service-Level Agreements (SLA) backed performance and uptime promises for their services. While these ‘clouds’ are the natural evolution of traditional clusters and data centres, they are distinguished by following a ‘utility’ pricing model where customers are charged based on their utilisation of computational resources, storage and transfer of data. Whilst these emerging services have reduced the cost of computation, application hosting and content storage and delivery by several orders of magnitude, there is significant complexity involved in ensuring applications, services and data can scale when needed to ensure consistent and reliable operation under peak loads.
This tutorial endeavors to familiarise the audience with the new cloud computing paradigm, whilst comparing and contrasting it with existing approaches to scaling out computing resources such as cluster and grid computing. Case studies of numerous existing compute, storage and application cloud services will be given, familiarising the audience with the capabilities and limitations of current providers of cloud computing services. The hands-on interaction with these services during this tutorial will allow the audience to understand the mechanisms needed to harness cloud computing in their own respective endeavors. Finally, many open research problems that have arisen from the rapid uptake of cloud computing will be detailed, which will hopefully motivate the audience to address these in their own future research and development.
Cloud Computing Architecture with Open Nebula - HPC Cloud Use Cases - NASA A...Ignacio M. Llorente
OpenNebula is a fully open-source cloud management platform, with excellent performance and scalability to manage tens of thousands of virtual machines, and with the most advanced functionality for building virtualized enterprise data centers and private cloud infrastructures. OpenNebula is the result of many years of research and development in efficient and scalable management of virtual machines on large-scale distributed infrastructures. Its innovative features have been developed to address the requirements of business use cases from leading companies in the context of flagship international projects in cloud computing. OpenNebula is being used by many supercomputing and leading research centers to build HPC and science clouds for hosting virtualized computational environments, such as batch farms and computing clusters, or for providing users with new "HPC as a service" resource provisioning models. The talk describes how to design a cloud architecture with OpenNebula and its innovative features to enable the execution of flexible and elastic cluster and high performance computing services on demand while reducing the associated cost of building the datacenter infrastructure.
The presentation describes the different cloud federation scenarios, ranging from a federation built on commercial cloud providers that offer no real support for federation to one built on data centers of the same organization where the sites are completely dedicated to supporting all aspects of federation. The level of federation is defined based on the amount of information disclosed and how much control over the resources is provided across sites. The talk also presents the existing challenges for interoperability in federated and hybrid cloud computing scenarios, and ends with real-life examples of multi-cloud environments running OpenNebula.
Cloud Expo East 2013: Essential Open Source Software for Building the Open CloudMark Hinkle
Cloud computing is more than a buzz-phrase it’s a transformative IT paradigm shift. The emphasis in the cloud is on elasticity, scalability, agility and open. Not just open standards but open APIs and open source. The delivery of software is also going through a paradigm shift. Open source software was often a commoditization of a market leader; Unix to Linux or Oracle to MySQL what’s changing is that the iterative nature, user context and the motto of releasing early and often are driving real innovation in open source.
This session will cover those essential open source technologies for delivering cloud computing in the enterprise.
Speaker Bio:
Mark Hinkle is the Senior Director, Open Source Solutions at Citrix Systems Inc. He joined Citrix as a result of their July 2011 acquisition of Cloud.com where he was their Vice President of Community. He is currently responsible for Citrix open source efforts around the open source cloud computing platform, Apache CloudStack and the Xen Hypervisor. Previously he was the VP of Community at Zenoss Inc., a producer of the open source application, server, and network management software, where he grew the Zenoss Core project to over 100,000 users and 20,000 organizations on all seven continents. He also is a longtime open source expert and author having served as Editor-in-Chief for both LinuxWorld Magazine and Enterprise Open Source Magazine. His blog on open source, technology, and new media can be found at http://www.socializedsoftware.com.
OSCON 2013 - The Hitchiker’s Guide to Open Source Cloud ComputingMark Hinkle
And while the Hitchhiker’s Guide to the Galaxy (HHGTTG) is a wholly remarkable book it doesn’t cover the nuances of cloud computing. Whether you want to build a public, private or hybrid cloud there are free and open source tools that can help provide you a complete solution or help augment your existing Amazon or other hosted cloud solution. That’s why you need the Hitchhiker’s Guide to (Open Source) Cloud Computing (HHGTCC) or at least to attend this talk understand the current state of open source cloud computing. This talk will cover infrastructure-as-a-service, platform-as-a-service and developments in big data and how to more effectively deploy and manage open source flavors of these technologies. Specific the guide will cover:
Infrastructure-as-a-Service – The Systems Cloud – Get a comparison of the open source cloud platforms including OpenStack, Apache CloudStack, Eucalyptus and OpenNebula
Platform-as-a-Service – The Developers Cloud – Learn about the tools that abstract the complexity for developers and used to build portable auto-scaling applications ton CloudFoundry, OpenShift, Stackato and more.
Data-as-a-Service – The Analytics Cloud – Want to figure out the who, what, where, when and why of big data? You’ll get an overview of open source NoSQL databases and technologies like MapReduce to help parallelize data mining tasks and crunch massive data sets in the cloud.
Network-as-a-Service – The Network Cloud – The final pillar for truly fungible network infrastructure is network virtualization. We will give an overview of software-defined networking including OpenStack Quantum, Nicira, open Vswitch and others.
Finally this talk will provide an overview of the tools that can help you really take advantage of the cloud. Do you want to auto-scale to serve millions of web pages and scale back down as demand fluctuates. Are you interested in automating the total lifecycle of cloud computing environments You’ll learn how to combine these tools into tool chains to provide continuous deployment systems that will help you become agile and spend more time improving your IT rather than simply maintaining it.
[Finally, for those of you that are Douglas Adams fans please accept the deepest apologies for bad analogies to the HHGTTG.]
Federated Cloud Computing - The OpenNebula Experience v1.0sIgnacio M. Llorente
The talk mostly focuses on private cloud computing to support Science and High Performance Computing environments, the different architectures to federate cloud infrastructures, the existing challenges for cloud interoperability, and the OpenNebula's vision for the future of existing Grid infrastructures.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
2. 2/30Bringing Private Cloud Computing to HPC and Science !
Contents
Building Private Cloud Computing to HPC and Science
This presentation is about:
• The Private HPC Cloud Use Case
• Main Challenges for Private HPC Cloud
• Private HPC Cloud Case Studies
• Private Cloud Trends in Industry
• About Grid and Cloud
3. 3/30Bringing Private Cloud Computing to HPC and Science !
The Private HPC and Science Cloud Use Case
The Pre-cloud Era!
LRMS (LSF, PBS, SGE…)
Grid Middleware
AccessProvision
4. 4/30Bringing Private Cloud Computing to HPC and Science !
The Private HPC and Science Cloud Use Case
OpenNebula as an Infrastructure Tool – Enhanced Capabilities!
Virtual Worker Nodes
LRMS (LSF, PBS, SGE…)
Grid Middleware
AccessProvisionService
• Common interfaces
• Grid integration
• Custom environments
• Dynamic elasticity
• Consolidation of WNs
• Simplified management
• Physical – Virtual WNs
• Dynamic capacity partitioning
• Faster upgrades
Service/Provisioning Decoupling!
5. 5/30Bringing Private Cloud Computing to HPC and Science !
The Private HPC and Science Cloud Use Case
OpenNebula as an Provisioning Tool – Enhanced Capabilities!
Pilot Jobs, SSH…
IaaS Interface
AccessProvisionService
• Simple Provisioning Interface
• Raw/Appliance VMs
• Dynamic scalable computing
• Custom access to capacity
• Not only batch workloads
• Not only scientific workloads
• Improve utilization
• Reduced service management
• Cost efficiency
6. 6/30Bringing Private Cloud Computing to HPC and Science !
Main Challenges for Private HPC Cloud
Main Demands from Engineering, Research and Supercomputing !
Flexible Definition of
Multi-tier Applications
Resource
Management
Scale-out and
Provisioning
Application
Performance
7. 7/30Bringing Private Cloud Computing to HPC and Science !
Main Challenges for Private HPC Cloud
Using the Cloud – Execution of Multi-tiered Applications !
Management of interconnected multi-VM applications:
• Definition of application flows
• Catalog with pre-defined applications
• Sharing between users and groups
• Management of persistent scientific data
• Automatic elasticity
Front-end
Worker Nodes
{ "name": ”Computing_Cluster",
"deployment": "straight",
"roles": [
{
"name": "frontend",
"vm_template": 0
},
{
"name": "worker",
"parents": frontend,
"cardinality": 2,
"vm_template": 3,
"min_vms" : 1,
"max_vms" : 5,
"elasticity_policies" :
{
”expressions" : ”CPU> 90%”,
"type" : "CHANGE",
"adjust" : 2,
"period_number" : 3,
"period" : 10
}, …
8. 8/30Bringing Private Cloud Computing to HPC and Science !
Main Challenges for Private HPC Cloud
Using the Cloud – Performance Penalty as a Small Tax You Have to Pay!
Overhead in Virtualization
• Single processor performance penalty between 1% and 5%
• NASA has reported an overhead between 9% and 25% (HPCC and NPB)1
• Growing number of users demanding containers (OpenVZ and LXC)
Need for Low-Latency High-Bandwidth Interconnection
• Lower performance, 10 GigE typically, used in clouds has a significant
negative (x2-x10, especially latency) impact on HPC applications1
• FermiCloud has reported MPI performance (HPL benchmark) on VMs and
SR-IOV/Infiniband with only a 4% overhead2
• The Center for HPC at CSR has contributed the KVM SR-IOV Drivers for
Infiniband3
(1) An Application-Based Performance Evaluation of Cloud Computing, NASA Ames, 2013
(2) FermiCloud Update, Keith Chadwick!, Fermilab, HePIX Spring Workshop 2013
(3) http://wiki.chpc.ac.za/acelab:opennebula_sr-iov_vmm_driver , 2013
Overhead in Input/Output
• Growing number of Big Data apps
• Support for multiple datastores including automatic scheduling
9. 9/30Bringing Private Cloud Computing to HPC and Science !
Main Challenges for Private HPC Cloud
Operating the Cloud – Resource Management!
Optimal Placement of Virtual Machines
• Automatic placement of VM near input data
• Striping policy to maximize the resources available to VMs
Fair Share of Resources
• Resource quota management to allocate, track and limit resource utilization.
Management of Different Hardware Profiles
• Resource pools (physical clusters) with specific Hw and Sw profiles, or
security levels for different workload profiles (HPC and HTC)
Isolated Execution of Applications
• Full Isolation of performance-sensitive applications
10. 10/30Bringing Private Cloud Computing to HPC and Science !
Main Challenges for Private HPC Cloud
Operating the Cloud – Scale out and Provisioning!
Multi-tier Deployment
• Management of multiple cloud instances
that may be hosted in different sites
Provide VOs with Isolated Cloud Environ
• Automatic provision of Virtual Data Centers
Hybrid Cloud Computing
• Cloudbursting to address peak or fluctuating
demands for no critical and HTC workloads
11. 11/30Bringing Private Cloud Computing to HPC and Science !
Private HPC Cloud Case Studies
One of Our Main User Communities!
Supercomputing Centers
Research Centers
Distributed Computing Infrastructures
Industry
12. 12/30Bringing Private Cloud Computing to HPC and Science !
Private HPC Cloud Case Studies
FermiCloud!
Nodes KVM on 23 nodes (1 TB RAM - 368 cores) Koi Computer
Network Gigabit and Infiniband
Storage CLVM+GFS2 on shared 120TB NexSAN SataBeats
AuthN X509
Linux Scientific Linux
Interface Sunstone Self-service and EC2 API
App Profile Legacy, HTC and MPI HPC
http://www-fermicloud.fnal.gov/
Typical Workloads
• Scientific stakeholders get access to on-
demand VMs
• Developers & integrators of new Grid
applications
13. 13/30Bringing Private Cloud Computing to HPC and Science !
Private HPC Cloud Case Studies
CESGA Cloud!
Nodes KVM on 27 nodes (0.5 TB RAM – 216 cores) HP ProLiant
Network 2 x Gigabit (1G and 10G)
Storage ssh from remote EMC storage server
AuthN X509 and core password
Linux Scientific Linux
Interface Sunstone Self-service and OCCI
App Profile Individual VMs and virtualised computing clusters
Typical Workloads
• 103 users
• Genomic, rendering…
• Grid services on production at CESGA
• Node at FedCloud project
• UMD middleware testing
http://cloud.cesga.es/
14. 14/30Bringing Private Cloud Computing to HPC and Science !
Private HPC Cloud Case Studies
SARA Cloud!
Nodes KVM on 19 HPC nodes (256 GB RAM 608 cores) Dell PowerEdge and 10 “light”
nodes (64 GB RAM 80 cores) Supermicro
Network 4 x Gigabit (10G) with Arista switch
Storage NFS on 400 GB NAS for HPC and ssh for “light”
AuthN Core password
Linux CentOS
Interface Sunstone and OCCI
App Profile MPI clusters, windows clusters and independent VMs
ww.cloud.sara.nl
Typical Workloads
• Ad-hoc clusters with MPI and pilot jobs
• Windows clusters for Windows-bound
software
• Single VMs, sometimes acting as web
servers to disseminate results
15. 15/30Bringing Private Cloud Computing to HPC and Science !
Private HPC Cloud Case Studies
SZTAKI Cloud!
Nodes KVM on 7 nodes (1.8 TB RAM – 448 cores) DELL PowerEdge
Network 2 x Gigabit (1G and 10G)
Storage iSCSI on DELL storage server 72 TB shared
AuthN X509
Linux CentOS
Interface Sunstone Self-service, EC2 and OCCI
App Profile Individual VMs and virtualised computing cluster
http://cloud.sztaki.hu/
.
Typical Workloads
• Run standard and grid services (e.g.:web
servers, grid middlewares…)
• Development and testing of new codes
• Research on performance and
opportunistic computing
16. 16/30Bringing Private Cloud Computing to HPC and Science !
Private HPC Cloud Case Studies
KTh Cloud!
Nodes KVM on 768 cores (768 GB RAM) HP ProLiant
Network Infiniband and Gigabit
Storage NFS and LVM
AuthN X509 and core password
Linux Ubuntu
Interface Sunstone self-service, OCCI and EC2
App Profile Individual VMs and virtualised computing cluster
http://www.pdc.kth.se/
Typical Workloads
• Mainly BIO
• Hadoop, Spark, Galaxy, Cloud Bio Linux…
17. 17/30Bringing Private Cloud Computing to HPC and Science !
Private Cloud Trends in Industry
Experimenting with ARM for the Private Cloud!
Why?
• Decrease power consumption, reduce costs, simplify solutions…
• Mostly managing bare metal and early experiences with virtualization
Tiniest Cloud Ever! (by Citrix and Linaro at LCU 2013)
Ubuntu on Versatile Express
Cortex-A15 Dual core
Ubuntu on Arndale Board
Cortex-A15 Dual core
http://www.youtube.com/watch?v=xZP9YKv3P_E
18. 18/30Bringing Private Cloud Computing to HPC and Science !
Private Cloud Trends in Industry
Cloud for Mission-critical Applications!
Availability and redundancy to keep it running in case of failure
• Cloud services availability => HA Architectures
• Application availability => Failover Solutions
Service Continuity (by European Aeronautic Company)
OpenNebula 4.0
Automatic failover and recovery within 1 minute
KVM
19. 19/30Bringing Private Cloud Computing to HPC and Science !
Private Cloud Trends in Industry
Hybrid Cloud Deployments !
Transparent and automatic access to the public cloud
• Dev&testing to the public cloud
• Security and performance sensitive workloads on the private cloud
Cloudbursting Deployment (by Telecom Company)
Public
Cloud
1
Public
Cloud
2
Local data center
OpenNebula
Private
Cloud
Cloud API is not relevant
20. 20/30Bringing Private Cloud Computing to HPC and Science !
About Grid and Cloud
What is the Difference between a Grid Site and a Cloud Provider?!
Definitions of Grid Site
• “A resource provider is a site which provides services and resources (e.g.
data storage) to this VO“ (GridPP)
• “What makes a grid site a grid site?. A single grid resource (a grid site)
offers compute and/ or storage services to remote users via standardized
interfaces” (GridKa)
• “A typical (minimal) grid site provides computing and storage to supported
Virtual Organizations (VOs) and runs a few services to make those resources
visible on the grid” (StratusLab)
Definitions of Cloud Provider
• “A cloud provider is a service provider that offers storage and compute
resources on a private or public network“ (IBM)
• “(Cloud) Providers offer resources to the customer – either via dedicated
APIs (PaaS), virtual machines and / or direct access to the resources
(IaaS)” (EC Report on The Future of Cloud Computing)
21. 21/30Bringing Private Cloud Computing to HPC and Science !
Virtual CE, WN… Other (web, mail...) Raw machines
LRMS (LSF, PBS…)
Grid Middleware IaaS Interface
Access
• Batch Job Processing
• Custom Execution Environments
• Grid Service Integration
• Industry Applications
• Other WMS (pilots)
• Complete Services (cluster)
Grid Site External Providers
ProvisionService
About Grid and Cloud
The OpenNebula Vision for Grid Sites: Extending the Range of Applications!
22. 22/30Bringing Private Cloud Computing to HPC and Science !
About Grid and Cloud
What is the Difference between a Grid and Cloud Federation?!
Definitions of Grid
• Ian Foster’s definition lists these primary attributes: “Computing resources
are not administered centrally, open standards are used, and nontrivial
quality of service is achieved”
• Plaszczak/Wellner: “The technology that enables resource virtualization, on-
demand provisioning, and service (resource) sharing between organizations”
• IBM: “The ability, using a set of open standards and protocols, to gain access
to applications and data, processing power, storage capacity and a vast
array of other computing resources over the Internet”
• CERN: “A service for sharing computer power and data storage capacity
over the Internet”
Definition of Cloud Federation
• “Cloud federation is the practice of interconnecting the cloud computing
environments of two or more service providers for the purpose of load
balancing traffic and accommodating spikes in demand”, Wikipedia
23. 23/30Bringing Private Cloud Computing to HPC and Science !
Grid Services
Grid API Cloud API Grid API Cloud API
Appliance Repo
MarketPlace
Cloud/Grid Site Cloud/Grid Site
• Sharing existing VM images
• Registry of metadata
• Image are kept elsewhere
• Supports trust
• Federation facilities
• Security
• Grid specific services
• Storage VM images
• Distributed
• Multi-protocol
About Grid and Cloud
The OpenNebula Vision for Grid Infrastructures, October 2008!
24. 24/30Bringing Private Cloud Computing to HPC and Science !
CloudsGridsUsage
Job Processing
Big Batch System
File Sharing Services
Achievements
Federation of Resources
VO Concept
But…
User experience
Complexity
Usage
Raw infrastructure
Elasticity & Pay-per-use
Simple Web Interface
Achievements
Agile Infrastructures
IT is another Utility
But…
Interoperability
Federation
Customize Environments
Uniform Security
Resource Management
Scientific Applications
Resource Sharing
Flexibility & Simplicity
About Grid and Cloud
Grid and Cloud as Complementary Computing Models!
25. 25/30Bringing Private Cloud Computing to HPC and Science !
About Grid and Cloud
EGI Federated Cloud Doing a Pioneering Work in the Field!
26. 26/30Bringing Private Cloud Computing to HPC and Science !
About Grid and Cloud
Different Names for the Same Model? Same Challenges but Different Technologies?!
Grid Computing
Cloud Computing
27. 27/30Bringing Private Cloud Computing to HPC and Science !
Try it Out!
OpenNebula Sandboxes!
● OpenNebula pre-installed in a VM: VirtualBox, KVM, VMware, Amazon
29. 29/30Bringing Private Cloud Computing to HPC and Science !
Visit our Booth!
Interoperability Features in
OpenNebula
Wednesday, 18
12:10 - 12:25
Room: Escudo
Building a OCCI-compatible
Cloud with OpenNebula
Thursday, 19
13:30 - 16:00
Room: Toledo
30. 30/30Bringing Private Cloud Computing to HPC and Science !
Thanks to People and Organizations that Provided Info to Prepare this Presentation!
Questions?
OpenNebula.org @OpenNebula