Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

CloudLightning - Project Overview


Published on

This presentation introduces CloudLightning, a €4m Horizon 2020 research project that proposes a novel architecture for self-organising self-managing heterogeneous clouds. The proposed use cases include IAAS service provision for HPC to serve the oil and gas, genome processing and ray tracing (3D image rendering) markets.

Published in: Science
  • ..............ACCESS that WEBSITE Over for All Ebooks ................ ......................................................................................................................... DOWNLOAD FULL PDF EBOOK here { } ......................................................................................................................... Download Full EPUB Ebook here { } .........................................................................................................................
    Are you sure you want to  Yes  No
    Your message goes here
  • How To Cure Acne For Good, Achieve lasting acne freedom Simple proven science of clear skin ●●●
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

CloudLightning - Project Overview

  2. 2. OVERVIEW 1. Funding and Challenge 2. Consortium 3. Current IaaS Cloud Usage 4. Objectives 5. Approach 6. CloudLightning Architecture 7. HPC Use Cases 8. Challenges Ahead
  3. 3. HORIZON 2020 Horizon 2020 is the biggest EU Research and Innovation programme ever with nearly €80 billion of funding available over 7 years (2014 to 2020) – in addition to the private investment that this money will attract. It promises more breakthroughs, discoveries and world-firsts by taking great ideas from the lab to the market. The principal goals: • world-class science • removal of barriers to innovation • enable public and private sectors to work together The CloudLightning project was funded under Call H2020-ICT-2014-1 Advanced Cloud Infrastructures and Services High performance heterogeneous cloud infrastructures and runs from Feb 2014 - January 2017
  4. 4. SPECIFIC CHALLENGE The aim is to develop infrastructures, methods and tools forhigh performance, adaptive cloud applications and services that go beyond the current capabilities. Cloud computing is being transformed by new requirements such as • heterogeneity of resources and devices, • software-defined data centres, • cloud networking, security, and • the rising demands for better quality of user experience. Cloud computing researchwill be oriented towards • new computational and data management models (at both infrastructure and services levels) that respond to the advent of faster and more efficientmachines, • rising heterogeneity of access modes and devices, • demand for low energy solutions, • widespread use of big data, • federated clouds and • secure multi-actor environments including public administrations.
  5. 5. CONSORTIUM CloudLightning comprises of eight partners from academia and industry and is coordinated by University College Cork.
  6. 6. CURRENT IAAS CLOUD USAGE Consider the two actors: the customer, looking to build a solution on provider’s infrastructure, and the cloud service provider Customer: • Hard work • Research various offerings and build/compile solutions accordingly. • Sub-optimal • Create a generic solution to facilitate portability • Opt for provider specific offering and risk vendor lock-in Provider: • Relinquishes control • Over resource utilization and power management • The cloud is now approaching 10% of the world’s electricity consumption! • Offers Resources in limited, discrete sizes • Precipitates over- provisions and exacerbates waste
  7. 7. PROJECT OBJECTIVES Customer Level Objectives • Make cloud computing more accessible • Make cloud computing more efficient • Move towards “ease of everything” Provider level objectives • Re-establish control over their IaaS offerings • Facilitate better power management • Enable fast resource provisioning for quicker service initiation • Enable seamless exploitation of heterogeneous hardware • Exploit faster and cheaper service delivery offered by hardware accelerators • Employ different heterogeneous hardware types for different services or for different invocations of the same service Project level objectives • Demonstrate our approach in a very challenging HPC application domain • Construct small- scale test-bed • Construct Large- scale simulation
  8. 8. OUR APPROACH Separate the concerns of the customer and the provider 1. Create a Service Orient Architecture for the Heterogeneous Cloud 2. Customer focuses on service requirements, workflows and SLAs rather than resources 3. Provider focuses on efficient resource management and service delivery
  9. 9. SERVICE ORIENTED ARCHITECTURE This approach moves the management burden from the customer to the provider. The resulting complexity for the provider is very high. Creator forms the work-flow and stores the Blueprint in the Blueprint Catalogue; the Operator selects a Blueprint from the Blueprint Catalogue and optionally edits its constraints and parameters. The Operator launches the Blueprint by: (1) requesting an appropriate solution from the CL and (2) deploying the Blueprint on the resources returned as part of that solution. The End User then interacts with the deployed Blueprint.
  10. 10. SELF-ORGANISATION AND SELF-MANAGEMENT Moving from a distributed customer-based IaaS Management to centralised provider-based IaaS management introduces enormous complexity. This complexity can be addressed by self-organization and self-management. Basic tenets: • component autonomy • awareness of the environment • goal-driven behaviour of individual components • self-configuration Goals include: • minimize energy consumption • Improve service delivery Goals achieved by collaboration. Coalitions of resources, working in concert to respond to the needs of a specific service request rather than offering a menu of a limited number of resource packages.
  11. 11. CONCEPTUAL ARCHITECTURE A Cell is a collection of resources. There may be multiple Cells A Cell Manager is associated with each Cell The Cell resources are grouped into a number virtual racks, calledvRacks. A vRack Manager is associated with each vRack. It is self-managed, identifying and creating coalitions, to deliver on specific service requests. vRack Managers cooperate to form vRack Manager Groups. vRacks Managers in the same Group self-organize to meet specific objectives. They are aware of changes in the environment including new and disappearing resources and adapt, on a negotiated basis, with other vRacks Managers within the same vRack Manager Group to meet system objectives.
  13. 13. CLOUDLIGHTNING RESOURCES • In a heterogeneous cloud, there will be many different types of compute resources. • These resources may be available individually or they may be bundled into subsystems. • Individual resources and subsystems may have pre-installed software stacks • They may be physically located on interconnects with different characteristics • A CL-Resource is a generic term used to refer to any of the above • CL-Resources can thus be bare metal; virtual machines; containers; networked commodity or specialized hardware, servers with accelerators such as GPUs, MICs and FPGAs; pre- built HPC environments • In response to a service request, the CL system identifies specific CL-Resources to be used for the delivery of that service. • Workflows of services may be implemented on a mixture of CL- Resources – one resource type per service
  14. 14. RESOURCE COALITION A collection of CL-Resources used to execute a service is called a Coalition. A coalition may be composed of one or more CL- Resources of the same type. Multi CL-Resource coalitions support multi-process services. Coalitions are formed by a vRack Manager in response to specific service requirements. Coalitions may be persisted to eliminate delays in CL-Resource creation and so to improve service delivery Dynamic Coalition formation respects SLA requirements minimizes provider overheads maximizes resource utilization The constituent CL- Resources of a Coalition may span multiple servers within a single vRack.
  15. 15. VRACK MANAGER TYPES AND GROUPS vRack Managers are typed to • reflect differences in the CL-Resources under their control • constrain how vRack Manager Groups are formed and self- organized • leverage resource specific optimization opportunities resulting from grouping vRack Managers together
  16. 16. PLUG AND PLAY
  18. 18. BENEFICIARIES The primary beneficiary is the Infrastructure-as-a- Service provider. They benefit from activating the HPC in the cloud market and a reduction in cost related to better performance per cost and performance per watt. This increased energy efficiency can result in lowercosts throughout the cloud ecosystem and can increase the accessibility and performance in a wide range of use cases including Oil and Gas discovery, Genomics and Ray Tracing (e.g. 3D Image Rendering) • Improved physics simulations and higher resolution RTM imaging. • Energy and cost efficient scalable solution for RTM and OPM/DUNE simulations. • Reduced risk and costs of dry exploratory wells. • Improved performance/cost and performance/Watt. • Faster speed of genome sequence computation. • Reduced development times. • Increased volume and quality of related research. • Reduced CAPEX and IT associated costs. • Extra capacity for overflow (“surge”) workloads. • Faster workload processing to meet project timelines. Ray Tracing (3D Image Rendering) GenomicsOil and Gas
  19. 19. IN CONCLUSION The Challenges Ahead • Separate the concerns of the IaaS consumer and the CSP • Create a Service Oriented Architecture for the emerging heterogeneous cloud • Reduce energy consumption by improved IaaS management • Improve service delivery • Leverage heterogeneity • Bring HPC to the cloud • Resource management in hyper-scale cloud deployments