The FermiCloud Project has been operating an Infrastructure-as-a-Service private Cloud using OpenNebula since the fall of 2010. FermiCloud has made significant contributions in X.509-based authentication and authorization, accounting, fabric deployment and high-availability cloud infrastructure. Our current program of work, carried out jointly with KISTI, focuses on interoperability and federation with the goal of running scientific cloud-based workflows across multiple clouds. I will identify some of the technical challenges that remain to be solved in widespread cloud deployment, as well as lessons that we have learned from grid computing and applied to the cloud environment.
Bio:
Steven C. Timm, PhD. is the group leader of the FermiGrid Services Group in the Grid and Cloud Computing Department at Fermi National Accelerator Laboratory, Batavia Illinois, USA, where he has been on the computing staff since 2000 with various responsibilities in large-scale distributed computing. He is the lead of the FermiCloud project and also responsible for leading the operations of the FermiGrid authentication and authorization servers and batch servers. Dr. Timm received his M.S. in Computer Science from Andrews University andhis Ph.D. in Physics from Carnegie Mellon University.
BBC Research & Development are in the process of deploying a department wide virtualization solution, catering for use cases including web development, machine learning, transcoding, media ingress and system testing. This talk discusses the implementation of a high performance Ceph storage backend and the challenges of virtualization in a broadcast research and development environment.
This talk describes the Fermilab Virtual Facility project, which incorporates bare-metal machines, our OpenNebula-based private cloud, and commercial clouds. After a number of years of research and development we are now doing stable production of data-intensive analysis and simulation for High Energy Experiments on the cloud.
I will pay special attention to the auxiliary services such as code caching, data caching, job submission, autoscaling, and load balancing that we are launching in the cloud. I will also review other significant developments by others in the field with which Fermilab is not directly involved.
Author Biography
Steven Timm has worked on cloud and virtualization issues for the Scientific Computing Division at Fermilab. The new Virtual Facility Project is a way to transparently extend Fermilab’s facility onto commercial and community clouds.
Enabling Scientific Workflows on FermiCloud using OpenNebulaNETWAYS
The FermiCloud Project has been operating an Infrastructure-as-a-Service private Cloud using OpenNebula since the fall of 2010. FermiCloud has made significant contributions in X.509-based authentication and authorization, accounting, fabric deployment and high-availability cloud infrastructure. Our current program of work, carried out jointly with KISTI, focuses on interoperability and federation with the goal of running scientific cloud-based workflows across multiple clouds. I will identify some of the technical challenges that remain to be solved in widespread cloud deployment, as well as lessons that we have learned from grid computing and applied to the cloud environment.
Monitoring Large-scale Cloud Infrastructures with OpenNebulaNETWAYS
Efficient monitoring is crucial when managing your Cloud infrastructure. The metrics collected by OpenNebula can be used to trigger automatic scaling, or quickly detect failures to automatically restart virtual machines. During this talk, I will show how OpenNebula can be used to efficiently monitor thousands of virtual machines at sub-1 minute interval. I will show how OpenNebula can be enhanced and optimized, and how different metrics collection tools such as Ganglia and Host-sFlow can be used with OpenNebula to monitor large-scale Cloud infrastructures.
The Contrail Virtual Execution Platform (VEP) allows Cloud administrators to manage data centers and monitor the usage of resources. Users can manage their distributed applications on IaaS Cloud providers under the control of Service Level Agreements (SLA). VEP applications are packaged in the standard OVF format and they are deployed inside Constrained Execution Environments (CEE) derived from the SLA, to support the specification of SLA contracts between users and providers.
These CEE environments allow to define constraints concerning virtual hardware performance, localization and affinity allowing the administrator to configure the monitoring system in order to feed external SLA enforcement services. VEP integrates elasticity management capabilities which can be controlled by external SLA enforcement services. A resource allocator service is integrated to dispatch the virtual components on the physical resources of the provider in accordance with the SLA terms.
The first version of VEP is currently implemented on OpenNebula. This talk presents the implementation of VEP on OpenNebula and discusses some implementation choices such as the resource allocator.
BBC Research & Development are in the process of deploying a department wide virtualization solution, catering for use cases including web development, machine learning, transcoding, media ingress and system testing. This talk discusses the implementation of a high performance Ceph storage backend and the challenges of virtualization in a broadcast research and development environment.
This talk describes the Fermilab Virtual Facility project, which incorporates bare-metal machines, our OpenNebula-based private cloud, and commercial clouds. After a number of years of research and development we are now doing stable production of data-intensive analysis and simulation for High Energy Experiments on the cloud.
I will pay special attention to the auxiliary services such as code caching, data caching, job submission, autoscaling, and load balancing that we are launching in the cloud. I will also review other significant developments by others in the field with which Fermilab is not directly involved.
Author Biography
Steven Timm has worked on cloud and virtualization issues for the Scientific Computing Division at Fermilab. The new Virtual Facility Project is a way to transparently extend Fermilab’s facility onto commercial and community clouds.
Enabling Scientific Workflows on FermiCloud using OpenNebulaNETWAYS
The FermiCloud Project has been operating an Infrastructure-as-a-Service private Cloud using OpenNebula since the fall of 2010. FermiCloud has made significant contributions in X.509-based authentication and authorization, accounting, fabric deployment and high-availability cloud infrastructure. Our current program of work, carried out jointly with KISTI, focuses on interoperability and federation with the goal of running scientific cloud-based workflows across multiple clouds. I will identify some of the technical challenges that remain to be solved in widespread cloud deployment, as well as lessons that we have learned from grid computing and applied to the cloud environment.
Monitoring Large-scale Cloud Infrastructures with OpenNebulaNETWAYS
Efficient monitoring is crucial when managing your Cloud infrastructure. The metrics collected by OpenNebula can be used to trigger automatic scaling, or quickly detect failures to automatically restart virtual machines. During this talk, I will show how OpenNebula can be used to efficiently monitor thousands of virtual machines at sub-1 minute interval. I will show how OpenNebula can be enhanced and optimized, and how different metrics collection tools such as Ganglia and Host-sFlow can be used with OpenNebula to monitor large-scale Cloud infrastructures.
The Contrail Virtual Execution Platform (VEP) allows Cloud administrators to manage data centers and monitor the usage of resources. Users can manage their distributed applications on IaaS Cloud providers under the control of Service Level Agreements (SLA). VEP applications are packaged in the standard OVF format and they are deployed inside Constrained Execution Environments (CEE) derived from the SLA, to support the specification of SLA contracts between users and providers.
These CEE environments allow to define constraints concerning virtual hardware performance, localization and affinity allowing the administrator to configure the monitoring system in order to feed external SLA enforcement services. VEP integrates elasticity management capabilities which can be controlled by external SLA enforcement services. A resource allocator service is integrated to dispatch the virtual components on the physical resources of the provider in accordance with the SLA terms.
The first version of VEP is currently implemented on OpenNebula. This talk presents the implementation of VEP on OpenNebula and discusses some implementation choices such as the resource allocator.
Supercomputing by API: Connecting Modern Web Apps to HPCOpenStack
Audience Level
Intermediate
Synopsis
The traditional user experience for High Performance Computing (HPC) centers around the command line, and the intricacies of the underlying hardware. At the same time, scientific software is moving towards the cloud, leveraging modern web-based frameworks, allowing rapid iteration, and a renewed focus on portability and reproducibility. This software still has need for the huge scale and specialist capabilities of HPC, but leveraging these resources is hampered by variation in implementation between facilities. Differences in software stack, scheduling systems and authentication all get in the way of developers who would rather focus on the research problem at hand. This presentation reviews efforts to overcome these barriers. We will cover container technologies, frameworks for programmatic HPC access, and RESTful APIs that can deliver this as a hosted solution.
Speaker Bio
Dr. David Perry is Compute Integration Specialist at The University of Melbourne, working to increase research productivity using cloud and HPC. David chairs Australia’s first community-owned wind farm, Hepburn Wind, and is co-founder/CTO of BoomPower, delivering simpler solar and battery purchasing decisions for consumers and NGOs.
Testing, CI Gating & Community Fast Feedback: The Challenge of Integration Pr...OPNFV
Jose Lausuch, Ericsson, Nikolas Hermanns, Ericsson
How can we make sure that new code in OPNFV does not break or stop CI?
How can we ensure quick feedback for each patch-set?
With the new way to snapshot a virtual deployment it is now possible to get virtual clouds up and running in about 2 min. In addition, through low amount of disk/cpu consumption and isolation of the networking it is possible to have a very high number of virtual deployments co-existing in the same bare-metal server.
OpenStack Networks the Web-Scale Way - Scott Laffer, Cumulus NetworksOpenStack
Audience Level
Beginner
Synopsis
Layer 2 versus Layer 3, MLAG, Spanning-Tree, switch mechanism drivers, overlays and routing-on-the-host — What scales and what does not? The underlying plumbing of an OpenStack network is something you’d rather not have to think about. This presentation examines the network architectures of web-scale and large enterprise OpenStack users and how those same efficiencies can be used in deployments of all sizes.
Speaker Bio:
Scott is a Member of Technical Staff at Cumulus Networks where he designs, supports and deploys web-scale technologies and architectures in enterprise networks globally. Prior to becoming a founding member of the Cumulus office in Australia, Scott started his career as a network administrator before joining Cisco Systems to support their data centre products.
OpenStack Australia Day Melbourne 2017
https://events.aptira.com/openstack-australia-day-melbourne-2017/
Accelerated dataplanes integration and deploymentOPNFV
Tim Rozet, Red Hat, Feng Pan, Red Hat
This session will explore the challenges and lessons learned with integrating accelerated dataplanes into OPNFV deployments. More specifically the talk will focus on FD.IO (VPP) and OVS DPDK integration into Apex, including different types of configuration options, platform requirements, performance tuning, and deployment challenges. This talk will also provide context to how OpenStack functions differently with these types of dataplanes, and how integration with the OpenDaylight controller works.
Watch this presentation and learn about Kubernetes Networking:
How to build applications without knowing subnets & IP addresses and build modern cloud-friendly applications in an agile fashion.
OpenNebulaConf 2016 - Budgeting: the Ugly Duckling of Cloud computing? by Mat...OpenNebula Project
After more than one year since the start of the operational phase, it is time for the Leibniz Supercomputing Centre to reconsider the usage model of its cloud infrastructure based on OpenNebula. Budgeting is the tool of choice to regulate the access to the resources and to translate the diverse access priorities into allocation policies. This talk will focus on the use case of a resource provider for research and education, giving an overview of the current needs together with a proposed solution.
Tips Tricks and Tactics with Cells and Scaling OpenStack - May, 2015Belmiro Moreira
Tips Tricks and Tactics with Cells and Scaling OpenStack
OpenStack Design Summit, Paris - May, 2015
Belmiro Moreira - CERN
Matt Van Winkle - Rackspace
Sam Morrison - NeCTAR, University of Melbourne
In this talk, Tim Bird will discuss the recent status of the Linux with regard to embedded systems. This will include a review of the last year's worth of mainline kernel releases, as well as topic areas specifically related to embedded, such as boot-up time, security, system size, etc. Tim will also present recent and planned work by the Core Embedded Linux Project of the Linux Foundation, and discuss the current status of Linux in various markets and fields. Tim will go over current areas of work, and discuss remaining challenges faced by Linux in embedded projects.
Cumulus Linux 2.5 makes it easy to adopt modern DataCenter Networking.
With new options for network architecture and validated designs, it is easier to migrate to modern data center networks.
Making it easy
– Design: Drop in switch running Cumulus Linux in existing network architecture, simplify migration with validated designs
– Procure: Making it simpler with expanded channel partners coverage (42 WW + Dell)
-- Deploy: Making it easy to support with First class Support organization –24x7 Coverage WW
While every new release of OpenStack offers improvements in functionality and the user experience, one thing’s for certain: troubleshooting is hard if you don’t know where to start.
Join us as we cover some common and not-so-common issues with Nova and Neutron that lead to some of our favorite error messages, including “No valid host was found”. Participants will learn basic troubleshooting procedures, including tips, tricks, and processes of elimination, to get their cloud back on track.
Supercomputing by API: Connecting Modern Web Apps to HPCOpenStack
Audience Level
Intermediate
Synopsis
The traditional user experience for High Performance Computing (HPC) centers around the command line, and the intricacies of the underlying hardware. At the same time, scientific software is moving towards the cloud, leveraging modern web-based frameworks, allowing rapid iteration, and a renewed focus on portability and reproducibility. This software still has need for the huge scale and specialist capabilities of HPC, but leveraging these resources is hampered by variation in implementation between facilities. Differences in software stack, scheduling systems and authentication all get in the way of developers who would rather focus on the research problem at hand. This presentation reviews efforts to overcome these barriers. We will cover container technologies, frameworks for programmatic HPC access, and RESTful APIs that can deliver this as a hosted solution.
Speaker Bio
Dr. David Perry is Compute Integration Specialist at The University of Melbourne, working to increase research productivity using cloud and HPC. David chairs Australia’s first community-owned wind farm, Hepburn Wind, and is co-founder/CTO of BoomPower, delivering simpler solar and battery purchasing decisions for consumers and NGOs.
Testing, CI Gating & Community Fast Feedback: The Challenge of Integration Pr...OPNFV
Jose Lausuch, Ericsson, Nikolas Hermanns, Ericsson
How can we make sure that new code in OPNFV does not break or stop CI?
How can we ensure quick feedback for each patch-set?
With the new way to snapshot a virtual deployment it is now possible to get virtual clouds up and running in about 2 min. In addition, through low amount of disk/cpu consumption and isolation of the networking it is possible to have a very high number of virtual deployments co-existing in the same bare-metal server.
OpenStack Networks the Web-Scale Way - Scott Laffer, Cumulus NetworksOpenStack
Audience Level
Beginner
Synopsis
Layer 2 versus Layer 3, MLAG, Spanning-Tree, switch mechanism drivers, overlays and routing-on-the-host — What scales and what does not? The underlying plumbing of an OpenStack network is something you’d rather not have to think about. This presentation examines the network architectures of web-scale and large enterprise OpenStack users and how those same efficiencies can be used in deployments of all sizes.
Speaker Bio:
Scott is a Member of Technical Staff at Cumulus Networks where he designs, supports and deploys web-scale technologies and architectures in enterprise networks globally. Prior to becoming a founding member of the Cumulus office in Australia, Scott started his career as a network administrator before joining Cisco Systems to support their data centre products.
OpenStack Australia Day Melbourne 2017
https://events.aptira.com/openstack-australia-day-melbourne-2017/
Accelerated dataplanes integration and deploymentOPNFV
Tim Rozet, Red Hat, Feng Pan, Red Hat
This session will explore the challenges and lessons learned with integrating accelerated dataplanes into OPNFV deployments. More specifically the talk will focus on FD.IO (VPP) and OVS DPDK integration into Apex, including different types of configuration options, platform requirements, performance tuning, and deployment challenges. This talk will also provide context to how OpenStack functions differently with these types of dataplanes, and how integration with the OpenDaylight controller works.
Watch this presentation and learn about Kubernetes Networking:
How to build applications without knowing subnets & IP addresses and build modern cloud-friendly applications in an agile fashion.
OpenNebulaConf 2016 - Budgeting: the Ugly Duckling of Cloud computing? by Mat...OpenNebula Project
After more than one year since the start of the operational phase, it is time for the Leibniz Supercomputing Centre to reconsider the usage model of its cloud infrastructure based on OpenNebula. Budgeting is the tool of choice to regulate the access to the resources and to translate the diverse access priorities into allocation policies. This talk will focus on the use case of a resource provider for research and education, giving an overview of the current needs together with a proposed solution.
Tips Tricks and Tactics with Cells and Scaling OpenStack - May, 2015Belmiro Moreira
Tips Tricks and Tactics with Cells and Scaling OpenStack
OpenStack Design Summit, Paris - May, 2015
Belmiro Moreira - CERN
Matt Van Winkle - Rackspace
Sam Morrison - NeCTAR, University of Melbourne
In this talk, Tim Bird will discuss the recent status of the Linux with regard to embedded systems. This will include a review of the last year's worth of mainline kernel releases, as well as topic areas specifically related to embedded, such as boot-up time, security, system size, etc. Tim will also present recent and planned work by the Core Embedded Linux Project of the Linux Foundation, and discuss the current status of Linux in various markets and fields. Tim will go over current areas of work, and discuss remaining challenges faced by Linux in embedded projects.
Cumulus Linux 2.5 makes it easy to adopt modern DataCenter Networking.
With new options for network architecture and validated designs, it is easier to migrate to modern data center networks.
Making it easy
– Design: Drop in switch running Cumulus Linux in existing network architecture, simplify migration with validated designs
– Procure: Making it simpler with expanded channel partners coverage (42 WW + Dell)
-- Deploy: Making it easy to support with First class Support organization –24x7 Coverage WW
While every new release of OpenStack offers improvements in functionality and the user experience, one thing’s for certain: troubleshooting is hard if you don’t know where to start.
Join us as we cover some common and not-so-common issues with Nova and Neutron that lead to some of our favorite error messages, including “No valid host was found”. Participants will learn basic troubleshooting procedures, including tips, tricks, and processes of elimination, to get their cloud back on track.
SDN 101: Software Defined Networking Course - Sameh Zaghloul/IBM - 2014SAMeh Zaghloul
Sameh Zaghloul
Technology Manager @ IBM
+2 0100 6066012
zaghloul@eg.ibm.com
SDN: Technology that enables data center team to use software to efficiently control network resources
SDN Overview
SDN Standards
NFV – Network Function Virtualization
SDN Scenarios and Use Cases
SDN Sample Research Projects
SDN Technology Survey
SDN Case Study
SDN Online Courses
SDN Lab SW Tools
- OpenStack Framework
- OpenDayLighyt – SDN Controller
- FloodLight – SDN Controller
- Open vSwitch – Virtual Switch
- MiniNet – Virtual Network: OpenFlow Switches, SDN Controllers, and Servers/Hosts
- OMNet++ Network Simulator
- Avior – Sample FloodLight Java Application
- netem - Network Emulation
- NOX/POX - C++/ Python OpenFlow API for building network control applications
- Pyretic = Python + Frenetic - Enables network programmers and operators to write modular network applications by providing powerful abstractions
- Resonance - Event-Driven Control for Software-Defined Networks (written in Pyretic)
SDN Project
Implementing vCPE with OpenStack and Software Defined NetworksPLUMgrid
Service providers and the broader vendor community have made progress in virtualizing key vCPE network functions. Concurrently, there is a strong push to bring these functions to the cloud. This session will discuss how Openstack is enabling this transformation and the role played by technologies like SDN and NFV. It will also discuss the latest advances in the networking stack of the Linux kernel which further enable these network functions to run in a fully distributed architecture. Finally, it will tie all these concepts together proposing a model for implementing virtual CPE services.
Automated Deployment and Management of Edge CloudsJay Bryant
This presentation discusses the challenges of cloud computing at the edge. From the exploding number of nodes, the need for integrated monitoring and zero touch discovery. We introduce Lenovo Open Cloud Automation, an automated framework built in collaboration with Red Hat to help address these challenges.
Openflow for Mobile Broadband service providers_Nov'11Radhakant Das
This is an analysis done on status of Open flow as on 2011 . It has many reasons to be embraced. SDNA architecture under cloud orchestration demands OPENFLOW to come live soon in commercial networks.
Red Hat multi-cluster management & what's new in OpenShiftKangaroot
More and more organisations are not only using container platforms but starting to run multiple clusters of containers. And with that comes new headaches of maintaining, securing, and updating those multiple clusters. In this session we'll look into how Red Hat has solved multi-cluster management, covering cluster lifecycle, app lifecycle, and governance/risk/compliance.
In this session, we’ll review how previous efforts, including Netfilter, Berkley Packet Filter (BPF), Open vSwitch (OVS), and TC, approached the problem of extensibility. We’ll show you an open source solution available within the Red Hat Enterprise Linux kernel, where extending and merging some of the existing concepts leads to an extensible framework that satisfies the networking needs of datacenter and cloud virtualization.
OpenNebulaConf2019 - Welcome and Project Update - Ignacio M. Llorente, Rubén ...OpenNebula Project
We've made our way into the world of open cloud — where each organization can find the right cloud for its unique needs. A single cloud management platform cannot be all things to all people. There will be a cloud space with several offerings focused on different environments and/or industries. The OpenNebula commitment to the open cloud is at the very base of its mission — to become the simplest cloud enabling platform — and its purpose — to bring simplicity to the private and hybrid enterprise cloud. OpenNebula exists to help companies build simple, cost-effective, reliable, open enterprise clouds on existing IT infrastructure. The OpenNebula Conference will be a great opportunity to communicate and share our vision and commitment, to look back at how the project has grown in the last 9 years, and to shed some insight into what to expect from the project in the near future.
OpenNebulaConf2019 - Building Virtual Environments for Security Analyses of C...OpenNebula Project
Computer networks are undergoing a phenomenal growth, driven by the rapidly increasing number of nodes constituting the networks. At the same time, the number of security threats on Internet and intranet networks is constantly increasing, and the testing and experimentation of cyber defense solutions require the availability of separate, test environments that best reflect the complexity of a real system. Such environments support the deployment and monitoring of complex mission-driven network scenarios, and cyber security training activities, thus enabling enterprises to study cyber defense strategies and allowing security researchers to evaluate their algorithms at scale.
The main objective is delivering to researchers and practitioners an overview of the technological means and the practical steps to setup a private cloud platform based on OpenNebula for the creation and management of virtual environments that support cyber-security activities of training and testing, as well as an overview of its possible applications in the cyber security domain.
In particular:
1. We describe our infrastructure based on OpenNebula
2. We overview our application, sitting on top of OpenNebula, as well as the technological tools involved in the management of its lifecycle (e.g., Ansible) .
3. We show how the platform can support various examples of security research activities
[References] Building an emulation environment for cyber security analyses of complex networked systems, Tanasache, Florin Dragos and Sorella, Mara and Bonomi, Silvia and Rapone, Raniero and Meacci, Davide, ICDCN '19, ACM, 2019
OpenNebulaConf2019 - CORD and Edge computing with OpenNebula - Alfonso Aureli...OpenNebula Project
I will be presenting the ongoing advances of the OnLife Networks project across Spain and Brasil, with a focus on use cases we have implemented in the Central Offices, which serve as the edge resources closest to the end-user. I will share an interesting synopsis of the the projects evolution, as well as provide several lessons learned.
OpenNebulaConf2019 - 6 years (+) OpenNebula - Lessons learned - Sebastian Man...OpenNebula Project
Insight into more than 6 years experience with OpenNebula from different perspectives: ISP & Datacenter Provider and Consultant / System Integrator
Lessons learned, "the dos and don'ts" and how we convince and enable customers with OpenNebula - and the NTS ecosystem.
OpenNebulaConf2019 - Performant and Resilient Storage the Open Source & Linux...OpenNebula Project
OpenNebula users have a range of storage options available to them, including proprietary appliances, proprietary software and Open Source software projects. This session will present a fully Open Source approach, that tightly integrates with Linux, and makes full use of the mature building blocks within the Linux kernel (LVM, Software RAID, DM-crypt, NVMe-oF Target, DRBD, etc...), and delivers one of the highest performance open source storage stacks currently available.
The core goal is to expose the improved performance of NVMe storage devices to VMs and containers. The solution covers both local NVMe drives and NVMe-oF. For interacting with NVMe-oF targets it supports the Swordfish-API and LVM & Linux’s software NVMe-oF target. The solution contains a storage addon for OpenNebula.
Our take on centralized and controlled VM image backups that deal with both CEPH and local QCOW2 datastores. As there are no default means of executing image backups in OpenNebula, I'd like to share our perspective on how we do it.
OpenNebulaConf2019 - How We Use GOCA to Manage our OpenNebula Cloud - Jean-Ph...OpenNebula Project
At Iguane Solutions, a lot of our "DevOps" tools are developed in Golang, and we have a good amount of experience in contributing to the Goca. I'll review just what contributions we make, as well as how we use Goca with different tools, on a daily basis, to manage and monitor our OpenNebula cloud.
I will delve into the concept of Infrastructure as Code - deployment of VM instances on cloud, as well as, also address the metrics collection of deployed VMs. Finally, I will present how we can abstract VM management with automation tools thanks to GOCA.
A deep insight into a project with codename "TARDIS" at HAUFE Lexware with the purpose to replace vCloud with OpenNebula. A technical deep dive into a focussed project done by real DevOps experts.
How and what we do with OpenNebula to enable our customers for a completely new way how it is consumed in a modern, service orientated IT. We will also talk about the question, why we have chosen OpenNebula and how deep is the level - and ability - of integration of the NTS CAPTAIN into existing 2nd and 3rd party tools like IPAM, CMDBs, backup, monitoring, approval processes and much more...
TeleData operates a purpose build IaaS enterprise ready cloud plattfom in the region of lake constance. OpenNebula is used in production since several years. TeleData will share an insight into the "Lessons learned" and a brief summary how to operate a public cloud, built on top of OpenNebula. Content is subject to change!
Performant and Resilient Storage: The Open Source & Linux WayOpenNebula Project
OpenNebula users have a range of storage options available to them, including proprietary appliances, proprietary software and Open Source software projects. This session will present a fully Open Source approach, that tightly integrates with Linux, and makes full use of the mature building blocks within the Linux kernel (LVM, Software RAID, DM-crypt, NVMe-oF Target, DRBD, etc...), and delivers one of the highest performance open source storage stacks currently available. The core goal is to expose the improved performance of NVMe storage devices to VMs and containers. The solution covers both local NVMe drives and NVMe-oF. For interacting with NVMe-oF targets it supports the Swordfish-API and LVM & Linux’s software NVMe-oF target. The solution contains a storage addon for OpenNebula.
NetApp’s Hybrid Cloud Infrastructure manages to leverage Kubernetes to a Hybrid Multi Cloud use case where OpenNebula integrates seamlessly. A technical deep dive in how NTS and NetApp integrated NTS Captain into NetApp’s DataFabric world on top of NetApp HC
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
GridMate - End to end testing is a critical piece to ensure quality and avoid...ThomasParaiso2
End to end testing is a critical piece to ensure quality and avoid regressions. In this session, we share our journey building an E2E testing pipeline for GridMate components (LWC and Aura) using Cypress, JSForce, FakerJS…
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Epistemic Interaction - tuning interfaces to provide information for AI support
OpenNebulaConf 2013 - Keynote: Enabling Scientific Workflows on FermiCloud using OpenNebula by by Steven C. Timm
1. Enabling Scientific Workflows
on FermiCloud using
OpenNebula
Steven Timm
Grid & Cloud Services Department
Fermilab
Work supported by the U.S. Department of Energy under contract No. DE-AC02-07CH11359
3. Fermilab and Scientific Computing
Fermi National Accelerator
Laboratory:
• Lead United States
particle physics
laboratory
• ~60 PB of data on tape
• High Throughput
Computing
characterized by:
• ―Pleasingly parallel‖ tasks
• High CPU instruction /
Bytes IO ratio
• But still lots of I/O. See
Pfister: ―In Search of
Clusters‖
25-Sep-2013S. Timm, OpenNebulaConf2
4. Grid and Cloud Services Dept.
Operations:
Grid Authorization
Grid Accounting
Computing Elements
Batch Submission
All require high availability
All require multiple
integration systems to test.
Also requires virtualization
And login as root
Solutions:
Development of
authorization, accounting,
and batch submission
software
Packaging and integration
Requires development
machines not used all the
time
Plus environments that are
easily reset
And login as root
25-Sep-2013S. Timm, OpenNebulaConf3
5. HTC Virtualization Drivers
Large multi-core servers have evolved from 2 to 64 cores per box,
• A single ―rogue‖ user/application can impact 63 other users/applications.
• Virtualization can provide mechanisms to securely isolate users/applications.
Typical ―bare metal‖ hardware has significantly more performance than usually needed
for a single-purpose server,
• Virtualization can provide mechanisms to harvest/utilize the remaining cycles.
Complicated software stacks are difficult to distribute on grid,
• Distribution of preconfigured virtual machines together with GlideinWMS and
HTCondor can aid in addressing this problem.
Large demand for transient development/testing/integration work,
• Virtual machines are ideal for this work.
Science is increasingly turning to complex, multiphase workflows.
• Virtualization coupled with cloud can provide the ability to flexibly reconfigure
hardware ―on demand‖ to meet the changing needs of science.
Legacy code:
• Data and code preservation for recently-completed experiments at Fermilab
Tevatron and elsewhere.
Burst Capacity:
• Systems are full all the time, need more cycles just before conferences.
25-Sep-2013S. Timm, OpenNebulaConf4
6. FermiCloud – Initial Project
Specifications
FermiCloud Project was established in 2009 with the goal of
developing and establishing Scientific Cloud capabilities for the
Fermilab Scientific Program,
• Building on the very successful FermiGrid program that supports the full
Fermilab user community and makes significant contributions as members of
the Open Science Grid Consortium.
• Reuse High Availabilty, AuthZ/AuthN, Virtualization from Grid
In a (very) broad brush, the mission of the FermiCloud project is:
• To deploy a production quality Infrastructure as a Service (IaaS) Cloud
Computing capability in support of the Fermilab Scientific Program.
• To support additional IaaS, PaaS and SaaS Cloud Computing capabilities based
on the FermiCloud infrastructure at Fermilab.
The FermiCloud project is a program of work that is split over
several overlapping phases.
• Each phase builds on the capabilities delivered as part of the previous phases.
25-Sep-2013S. Timm, OpenNebulaConf5
7. Overlapping Phases
25-Sep-2013S. Timm, OpenNebulaConf6
Phase 1:
“Build and Deploy the Infrastructure”
Phase 2:
“Deploy Management Services, Extend
the Infrastructure and Research
Capabilities”
Phase 3:
“Establish Production Services and
Evolve System Capabilities in
Response to User Needs & Requests”
Phase 4:
“Expand the service capabilities to serve
more of our user communities”
Time
Today
8. Current FermiCloud Capabilities
The current FermiCloud hardware capabilities include:
• Public network access via the high performance Fermilab network,
- This is a distributed, redundant network.
• Private 1 Gb/sec network,
- This network is bridged across FCC and GCC on private fiber,
• High performance Infiniband network,
- Currently split into two segments,
• Access to a high performance FibreChannel based SAN,
- This SAN spans both buildings.
• Access to the high performance BlueArc based filesystems,
- The BlueArc is located on FCC-2,
• Access to the Fermilab dCache and enStore services,
- These services are split across FCC and GCC,
• Access to 100 Gbit Ethernet test bed in LCC (Integration nodes),
- Intel 10 Gbit Ethernet converged network adapter X540-T1.
25-Sep-2013S. Timm, OpenNebulaConf7
9. Typical Use Cases
Public net virtual machine:
• On Fermilab Network open to Internet,
• Can access dCache and Bluearc Mass Storage,
• Common home directory between multiple VM’s.
Public/Private Cluster:
• One gateway VM on public/private net,
• Cluster of many VM’s on private net.
• Data acquisition simulation
Storage VM:
• VM with large non-persistent storage,
• Use for large MySQL or Postgres databases,
Lustre/Hadoop/Bestman/xRootd/dCache/OrangeFS/IRODS servers.
25-Sep-2013S. Timm, OpenNebulaConf8
10. FermiGrid-HA2 Experience
In 2009, based on operational experience and plans for
redevelopment of the FCC-1 computer room, the FermiGrid-HA2
project was established to split the set of FermiGrid services
across computer rooms in two separate buildings (FCC-2 and
GCC-B).
• This project was completed on 7-Jun-2011 (and tested by a building failure
less than two hours later).
• FermiGrid-HA2 worked exactly as designed.
Our operational experience with FermiGrid-HA and FermiGrid-
HA2 has shown the benefits of virtualization and service
redundancy.
• Benefits to the user community – increased service reliability and uptime.
• Benefits to the service maintainers – flexible scheduling of maintenance and
upgrade activities.
25-Sep-2013S. Timm, OpenNebulaConf9
11. Experience with FermiGrid =
Drivers for FermiCloud
25-Sep-2013S. Timm, OpenNebulaConf10
Access to pools of resources using common interfaces:
• Monitoring, quotas, allocations, accounting, etc.
Opportunistic access:
• Users can use the common interfaces to ―burst‖ to
additional resources to meet their needs
Efficient operations:
• Deploy common services centrally
High availability services:
• Flexible and resilient operations
12. Additional Drivers for FermiCloud
Existing development and integration (AKA the FAPL cluster)
facilities were:
• Technically obsolescent and unable to be used effectively to
test and deploy the current generations of Grid middleware.
• The hardware was over 8 years old and was falling apart.
• The needs of the developers and service administrators in
the Grid and Cloud Computing Department for reliable and
―at scale‖ development and integration facilities were
growing.
• Operational experience with FermiGrid had demonstrated
that virtualization could be used to deliver production class
services.
25-Sep-2013S. Timm, OpenNebulaConf11
13. OpenNebula
OpenNebula was picked as result of evaluation of Open
source cloud management software.
OpenNebula 2.0 pilot system in GCC available to users
since November 2010.
Began with 5 nodes, gradually expanded to 13 nodes.
4500 Virtual Machines run on pilot system in 3+ years.
OpenNebula 3.2 production-quality system installed in
FCC in June 2012 in advance of GCC total power
outage—now comprises 18 nodes.
Transition of virtual machines and users from ONe 2.0
pilot system to production system almost complete.
In the meantime OpenNebula has done five more
releases, will catch up shortly.
25-Sep-2013S. Timm, OpenNebulaConf12
14. FermiCloud – Fault Tolerance
As we have learned from FermiGrid, having a distributed fault
tolerant infrastructure is highly desirable for production operations.
We are actively working on deploying the FermiCloud hardware
resources in a fault tolerant infrastructure:
• The physical systems are split across two buildings,
• There is a fault tolerant network infrastructure in place that interconnects the
two buildings,
• We have deployed SAN hardware in both buildings,
• We have a dual head-node configuration with HB for failover
• We have a GFS2 + CLVM for our multi-user filesystem and distributed SAN.
• SAN replicated between buildings using CLVM mirroring.
GOAL:
• If a building is ―lost‖, then automatically relaunch ―24x7‖ VMs on surviving
infrastructure, then relaunch ―9x5‖ VMs if there is sufficient remaining capacity,
• Perform notification (via Service-Now) when exceptions are detected.
25-Sep-2013S. Timm, OpenNebulaConf13
15. FCC and GCC
25-Sep-2013S. Timm, OpenNebulaConf14
FC
C
GC
C
The FCC and GCC buildings
are separated by
approximately 1 mile (1.6 km).
FCC has UPS and Generator.
GCC has UPS.
16. Distributed Network Core
Provides Redundant Connectivity
25-Sep-2013S. Timm, OpenNebulaConf15
GCC-A
Nexus
7010
Robotic
Tape
Libraries
(4)
Robotic
Tape
Libraries
(3)
Fermi
Grid
Fermi
Cloud
Fermi
Grid
Fermi
Cloud
Disk Servers
Disk Servers
20 Gigabit/s L3 Routed Network
80 Gigabit/s L2 Switched Network
Note – Intermediate level switches and top of rack switches are
not shown in the this diagram.
Private Networks over dedicated fiber
Grid
Worker
Nodes
Nexus
7010
FCC-2
Nexus
7010
FCC-3
Nexus
7010
GCC-B
Grid
Worker
Nodes
Deployment completed in June 2012
17. Distributed Shared File System
Design:
Dual-port FibreChannel HBA in each node,
Two Brocade SAN switches per rack,
Brocades linked rack-to-rack with dark fiber,
60TB Nexsan Satabeast in FCC-3 and GCC-B,
Redhat Clustering + CLVM + GFS2 used for file system,
Each VM image is a file in the GFS2 file system
LVM mirroring RAID 1 across buildings.
Benefits:
Fast Launch—almost immediate as compared to 3-4 minutes with ssh/scp,
Live Migration—Can move virtual machines from one host to another for
scheduled maintenance, transparent to users,
Persistent data volumes—can move quickly with machines,
Can relaunch virtual machines in surviving building in case of building
failure/outage,
25-Sep-2013S. Timm, OpenNebulaConf16
18. FermiCloud – Network & SAN
―Today‖
Private Ethernet
over dedicated fiber
Fibre Channel
over dedicated fiber
25-Sep-201317 S. Timm, OpenNebulaConf
Nexus
7010
Nexus
7010
Nexus
7010
FCC-2 GCC-A
FCC-3 GCC-B
Nexus
7010
fcl315
To
fcl323
FCC-3
Brocade
Satabeast
Brocade
fcl001
To
fcl013
GCC-B
Brocade
Satabeast
Brocade
19. FermiCloud-HA
Head Node Configuration
25-Sep-2013S. Timm, OpenNebulaConf18
fcl001 (GCC-B) fcl301 (FCC-3)
ONED/SCHED
fcl-ganglia2
fermicloudnis2
fermicloudrepo2
fclweb2
fcl-cobbler
fermicloudlog
fermicloudadmin
fcl-lvs2
fcl-mysql2
ONED/SCHED
fcl-ganglia1
fermicloudnis1
fermicloudrepo1
fclweb1
fermicloudrsv
fcl-lvs1
fcl-mysql1
2 way rsync
Live Migration
Multi-master
MySQL
CLVM/rgmanager
20. Cooperative R+D Agreement
Partners:
• Grid and Cloud Computing Dept. @FNAL
• Global Science Experimental Data hub Center @KISTI
Project Title:
• Integration and Commissioning of a Prototype Federated Cloud for Scientific
Workflows
Status:
• Three major work items:
1. Virtual Infrastructure Automation and Provisioning,
2. Interoperability and Federation of Cloud Resources,
3. High-Throughput Fabric Virtualization.
25-Sep-2013S. Timm, OpenNebulaConf19
21. Virtual Machines as Jobs
OpenNebula (and all other open-source IaaS stacks) provide
an emulation of Amazon EC2.
HTCondor developers added code to their ―Amazon EC2‖
universe to support the X.509-authenticated protocol.
Currently testing in bulk, up to 75 VM’s OK thus far:
Goal to submit NOvA workflow to OpenNebula @ FermiCloud,
OpenStack @ Notre Dame, and Amazon EC2.
Smooth submission of many thousands of VM’s is key step to
making the full infrastructure of a site into a science cloud.
25-Sep-2013S. Timm, OpenNebulaConf20
26. True Idle VM Detection
In times of resource need, we want the ability to suspend or ―shelve‖ idle VMs
in order to free up resources for higher priority usage.
• This is especially important in the event of constrained resources (e.g. during
building or network failure).
Shelving of ―9x5‖ and ―opportunistic‖ VMs allows us to use FermiCloud
resources for Grid worker node VMs during nights and weekends
• This is part of the draft economic model.
Giovanni Franzini (an Italian co-op student) has written (extensible) code for
an ―Idle VM Probe‖ that can be used to detect idle virtual machines based on
CPU, disk I/O and network I/O.
Nick Palombo, consultant, has written the communication system and the
collector system to do rule-based actions based on the idle information.
25-Sep-2013S. Timm, OpenNebulaConf25
27. Idle VM Information Flow
25-Sep-2013S. Timm, OpenNebulaConf26
Raw VM
State DB
Idle VM
Collector
Idle VM
Logic
Idle VM
List
Idle VM
Trigger
Idle VM
Shutdown
Idle VM Management Process HOST
VM
VM
VM
Idle data
store
OpenNebula
HOST
VM
VM
VM
Idle data
store
IM
IM
XMLRPC
XMLRPC
28. Interoperability and Federation
Driver:
• Global scientific collaborations such as LHC experiments will have to interoperate across
facilities with heterogeneous cloud infrastructure.
European efforts:
• EGI Cloud Federation Task Force – several institutional clouds (OpenNebula, OpenStack,
StratusLab).
• HelixNebula—Federation of commercial cloud providers
Our goals:
• Show proof of principle—Federation including FermiCloud + KISTI ―G Cloud‖ + one or more
commercial cloud providers + other research institution community clouds if possible.
• Participate in existing federations if possible.
Core Competency:
• FermiCloud project can contribute to these cloud federations given our expertise in X.509
Authentication and Authorization, and our long experience in grid federation
25-Sep-2013S. Timm, OpenNebulaConf27
29. Virtual Image Formats
Different clouds have different virtual
machine image formats:
• File system ++, Partition table, LVM
volumes, Kernel?
We have identified the differences and
written a comprehensive step by step user
manual, soon to be public.
25-Sep-2013S. Timm, OpenNebulaConf28
30. Interoperability/Compatibility of API’s
Amazon EC2 API is not open source, it is a
moving target that changes frequently.
Open-source emulations have various feature
levels and accuracy of implementation:
• Compare and contrast OpenNebula,
OpenStack, and commercial clouds,
• Identify lowest common denominator(s) that
work on all.
25-Sep-2013S. Timm, OpenNebulaConf29
31. VM Image Distribution
Investigate existing image marketplaces
(HEPiX, U. of Victoria).
Investigate if we need an Amazon S3-like
storage/distribution method for OS images,
• OpenNebula doesn’t have one at present,
• A GridFTP ―door‖ to the OpenNebula VM
library is a possibility, this could be integrated
with an automatic security scan workflow
using the existing Fermilab NESSUS
infrastructure.
25-Sep-2013S. Timm, OpenNebulaConf30
32. High-Throughput Fabric Virtualization
Followed up earlier virtualized MPI work:
• Use it in real scientific workflows
• Now users can define a set of IB machines
in OpenNebula on their own
• DAQ system simulation
• Large multicast activity
• Also experiments done with virtualized
10GBe on 100GBit WAN testbed.
.
25-Sep-2013S. Timm, OpenNebulaConf31
33. Security
Main areas of cloud security development:
Secure Contextualization:
• Secrets such as X.509 service certificates and Kerberos keytabs are not stored in
virtual machines (See following talk for more details).
X.509 Authentication/Authorization:
• X.509 Authentication written by T. Hesselroth, code submitted to and accepted by
OpenNebula, publicly available since Jan-2012.
Security Policy:
• A security taskforce met and delivered a report to the Fermilab Computer Security
Board, recommending the creation of a new Cloud Computing Environment, now in
progress.
We also participated in the HEPiX Virtualisation Task Force,
• We respectfully disagree with the recommendations regarding VM endorsement.
25-Sep-2013S. Timm, OpenNebulaConf32
35. OpenNebula Authentication
OpenNebula came with ―pluggable‖ authentication, but few plugins
initially available.
OpenNebula 2.0 Web services by default used access key / secret
key mechanism similar to Amazon EC2. No https available.
Four ways to access OpenNebula:
• Command line tools,
• Sunstone Web GUI,
• ―ECONE‖ web service emulation of Amazon Restful (Query) API,
• OCCI web service.
FermiCloud project wrote X.509-based authentication plugins:
• Patches to OpenNebula to support this were developed at Fermilab and
submitted back to the OpenNebula project in Fall 2011 (generally available
in OpenNebula V3.2 onwards).
• X.509 plugins available for command line and for web services authentication.
25-Sep-2013S. Timm, OpenNebulaConf34
36. X.509 Authentication—how it works
• Command line:
• User creates a X.509-based token using ―oneuser login‖
command
• This makes a base64 hash of the user’s proxy and
certificate chain, combined with a username:expiration
date, signed with the user’s private key
• Web Services:
• Web services daemon contacts OpenNebula XML-RPC
core on the users’ behalf, using the host certificate to sign
the authentication token.
• Use Apache mod_proxy to pass the grid certificate DN to
web services.
• Limitations:
• With Web services, one DN can map to only one user.
25-Sep-2013S. Timm, OpenNebulaConf35
37. Grid AuthZ Interoperability Protocol
• Use XACML 2.0 to specify
• DN, CA, Hostname, CA, FQAN, FQAN signing entity, and more.
• Developed in 2007, has been used in Open Science Grid
and other grids
• Java and C bindings available for client
• Most commonly used C binding is LCMAPS
• Used to talk to GUMS, SAZ, others
• Allows one user to be part of different Virtual
Organizations and have different groups and roles.
• For Cloud authorization we will configure GUMS to map
back to individual user names, one per person
• Each personal account in OpenNebula created in
advance.
25-Sep-2013S. Timm, OpenNebulaConf36
38. ―Authorization‖ in OpenNebula
• Note: OpenNebula has pluggable
―Authorization‖ modules as well.
• These control Access ACL’s—namely which
user can launch a virtual machine, create a
network, store an image, etc.
• Not related to the grid-based notion of
authorization at all.
• Instead we make our ―Authorization‖
additions to the Authentication routines of
OpenNebula
25-Sep-2013S. Timm, OpenNebulaConf37
39. X.509 Authorization
• OpenNebula authorization plugins written in Ruby
• Use existing Grid routines to call to external GUMS and SAZ
authorization servers
• Use Ruby-C binding to call C-based routines for LCMAPS or
• Use Ruby-Java bridge to call Java-based routines from Privilege proj.
• GUMS returns uid/gid, SAZ returns yes/no.
• Works with OpenNebula command line and non-interactive web services
• Much effort spent in trying to send user credentials with extended
attributes into web browser
• Currently—ruby-java-bridge setup works for CLI
• For Sunstone we have shifted to have callout to VOMS done on server
side.
• We are always interested in talking to anyone who is doing X.509
authentication in any cloud.
25-Sep-2013S. Timm, OpenNebulaConf38
40. Reframing Cloud Discussion
Purpose of Infrastructure-as-a-service:
On demand only?
No—a whole new way to think about IT infrastructure both
internal and external.
Cloud API is just a part of rethinking IT infrastructure for data-
intensive science (and MIS).
Only as good as the hardware and software it’s built on.
Network fabric, storage, and applications all crucial.
Buy or build?
Both! Will always need some in-house capacity.
Performance hit?
Most can be traced to badly written applications or
misconfigured OS.
25-Sep-2013S. Timm, OpenNebulaConf39
41. FermiCloud Project Summary - 1
Science is directly and indirectly benefiting from FermiCloud:
• CDF, D0, Intensity Frontier, Cosmic Frontier, CMS, ATLAS, Open Science Grid,…
FermiCloud operates at the forefront of delivering cloud
computing capabilities to support scientific research:
• By starting small, developing a list of requirements, building on existing Grid
knowledge and infrastructure to address those requirements, FermiCloud has
managed to deliver a production class Infrastructure as a Service cloud computing
capability that supports science at Fermilab.
• FermiCloud has provided FermiGrid with an infrastructure that has allowed us to test
Grid middleware at production scale prior to deployment.
• The Open Science Grid software team used FermiCloud resources to support their
RPM ―refactoring‖ and is currently using it to support their ongoing middleware
development/integration.
25-Sep-201340 S. Timm, OpenNebulaConf
42. FermiCloud Project Summary
The FermiCloud collaboration with KISTI has leveraged the
resources and expertise of both institutions to achieve significant
benefits.
vCluster has demonstrated proof of principle Grid Bursting‖ using
FermiCloud and Amazon EC2 resources.
Using SRIOV drivers on FermiCloud virtual machines, MPI
performance has been demonstrated to be >96% of the native
―bare metal‖ performance.
The future is mostly cloudy.
25-Sep-201341 S. Timm, OpenNebulaConf
43. Acknowledgements
None of this work could have been
accomplished without:
• The excellent support from other departments of the
Fermilab Computing Sector – including Computing
Facilities, Site Networking, and Logistics.
• The excellent collaboration with the open source
communities – especially Scientific Linux and
OpenNebula,
• As well as the excellent collaboration and contributions
from KISTI.
• And talented summer students from Illinois Institute of
Technology
25-Sep-2013S. Timm, OpenNebulaConf42