In this video, Prof. John Morrison from University College Cork describes the CloudLightning project. CloudLightning’s vision is a European economy that thrives and leads the world in the provision and adoption of high performance cloud computing services. Funded by the European Commission’s Horizon 2020 Program, CloudLightning brings together eight project partners from five countries across Europe. Learn more: http://cloudlightning.eu Watch the video presentation: http://wp.me/p3RLHQ-fsb

1. SELF-ORGANISING, SELF-MANAGING
HETEROGENEOUS CLOUD
A Brief Overview
Prof J. P. Morrison

2. Overview Project Funding and Consortium
Specific Challenge
Typical IaaS Cloud Usage
Project Goals and Ambitions
Our Approach
The CloudLightning Architecture
Beneficiaries
Challenges

3. The principal goals are to ensure Europe produces
world-class science,
removes barriers to innovation and
enabling the public and private sectors to work together in delivering innovation.
The emphasis is on:
excellent science,
industrial leadership and
tackling societal challenges.
The CloudLightning project was funded under
Call H2020-ICT-2014-1 Advanced Cloud Infrastructures and Services
High performance heterogeneous cloud infrastructures
and runs from Feb 2014 - January 2017
Horizon 2020

4. Project
Consortium

5. Cloud computing is being transformed by new requirements such as
• heterogeneity of resources and devices
• software-defined data centres
• cloud networking, security, and
• the rising demands for better quality of user experience.
Cloud computing research will be oriented towards
• new computational and data management models (at both infrastructure and services levels)
that respond to the advent of faster and more efficient machines,
• rising heterogeneity of access modes and devices,
• demand for low energy solutions,
• widespread use of big data,
• federated clouds and
• secure multi-actor environments including public administrations.
The aim is to develop infrastructures, methods and tools for high performance, adaptive
cloud applications and services that go beyond the current capabilities
Specific Challenge
https://ec.europa.eu/research/participants/portal/desktop/en/opportunities/h2020/topics/290-ict-07-2014.html

6. Cloud computing is being transformed by new requirements such as
• heterogeneity of resources and devices
• software-defined data centres
• cloud networking, security, and
• the rising demands for better quality of user experience.
Cloud computing research will be oriented towards
• new computational and data management models (at both infrastructure and services levels)
that respond to the advent of faster and more efficient machines,
• rising heterogeneity of access modes and devices,
• demand for low energy solutions,
• widespread use of big data,
• federated clouds and
• secure multi-actor environments including public administrations.
The aim is to develop infrastructures, methods and tools for high performance, adaptive
cloud applications and services that go beyond the current capabilities
Specific Challenge
https://ec.europa.eu/research/participants/portal/desktop/en/opportunities/h2020/topics/290-ict-07-2014.html

7. Customer must do the hard work
 Research various offerings and build/compile solutions accordingly.
 Target the lowest common denominator to facilitate portability
 Solution often end up either being completely generic
 opportunity cost
 Or, they are focused on using some special features (inevitably tying them to particular
providers)
 portability lost
Providers support this usage pattern with over-provisioning
Typical IaaS Cloud Usage

8. Make Cloud Computing more accessible to the average customer.
Allow the provider to make their offering more efficient
• The current model is not sustainable. The cloud is now approaching 10% of the world’s
electricity consumption.
Exploit heterogeneous hardware type
Demonstrate our approach in a very challenging application domain – HPC
Project Goals and Ambitions

9. We see the adoption of a Service Interface as key.
• Provides a “clean” interface between the customer and provider
• This interface should not require the customer to specify resource requirements. Rather,
function requirements, workflows and SLAs
However, this implies moving the management complexity from the customer to the
provider, which in turn, gives rise to a large complex system.
Project Goals and Ambitions

10. The BP Creator forms the work-flow and stores the
Blueprint in the Blueprint Catalogue;
The Operator selects a Blueprint from the Blueprint
Catalogue and optionally constrains and
parameterizes it.
The Operator launches the Blueprint by:
(1) requesting an appropriate solution from the CL
and
(2) deploying the Blueprint on the resources
returned as part of that solution.
The End User then interacts with the deployed
Blueprint.
Our Approach

11. Managing complexity of this scale can be done using self-
organisation.
• Synergetic activities of elements when no single element
acts as a coordinator and the global patterns of behaviour
are distributed
• Prevalent in Nature
• Already being used to develop many control systems,
sensor networks, economic systems, ...
“Global order can arise from local interactions”. Alan Turing.
Conceptual Architecture

12. Architecture Components

13. Basic tenets:
• component autonomy
• awareness of the environment
• goal-driven behaviour of individual components
• self-configuration
Goals include:
• minimize energy consumption
• Improve service delivery
Goals are achieved by collaboration.
Self-configuration allows the system to create coalitions of resources, working in concert to
respond to the needs of a specific service request, rather than offering a menu of a limited number
of resource packages.
Self-Organisation

14. The CL system uses a single abstract concept of resource, known as a CL-Resource.
In response to a service request, the CL system identifies a specific CL-Resource
that will be used for the delivery of that service.
The physical realization of a CL-Resource depends on what aspect of the underlying physical
hardware is being exposed to the CL system.
CL-Resources can be
• bare metal,
• virtual machines,
• containers,
• networked commodity hardware (either offered as a bare metal cluster or as a cluster pre-configured to host
distributed applications),
• servers with attached accelerators such as GPUs, MICs and FPGAs.
CloudLightning Resources

15. CL-Resources aggregated together and given a specific identity, known as a Coalition.
Coalitions formed by a vRack Manager in response to specific service requirements.
Coalitions may be persisted for improved service delivery
The constituent CL-Resources of a Coalition may span multiple servers but are restricted to a single
vRack.
Resource Coalitions

16. vRack Manager Types and Groups

17. Plug and Play

18. Leveraging Existing OpenStack Components

19. Beneficiaries
The primary beneficiary is
the Infrastructure-as-a-
Service provider. They
benefit from activating the
HPC in the cloud market
and a reduction in cost
related to better
performance per cost and
performance per watt.
This increased energy
efficiency can result in
lower costs throughout the
cloud ecosystem and can
increase the accessibility
and performance in a wide
range of use cases
including Oil and Gas
discovery, Genomics and
Ray Tracing (e.g. 3D
Image Rendering)
• Oil and Gas
Improved physics
simulations and
higher resolution
RTM imaging.
Energy and cost
efficient scalable
solution for RTM and
OPM/DUNE
simulations.
Reduced risk and
costs of dry
exploratory wells.
Genomics
Improved
performance/cost
and
performance/Watt
Faster speed of
genome sequence
computation.
Reduced
development times.
Increased volume
and quality of related
research.
Ray Tracing (3D
Image Rendering)
Reduced CAPEX
and IT associated
costs.
Extra capacity for
overflow (“surge”)
workloads.
Faster workload
processing to meet
project timelines.

20. In Conclusion The Challenges Ahead
Separate the concerns of the IaaS consumer and the CSP
Create a Service Oriented Architecture for the emerging heterogeneous
cloud
Reduce energy consumption by improved IaaS management
Improve service delivery
Leverage heterogeneity to bring HPC to the cloud
Resource management in hyper-scale cloud deployments

21. THANK YOU
John Morrison j.morrison@cs.ucc.ie