Using a Widely Distributed Federated Cloud System to Support Multiple Disparate User Communities

www.egi.euEGI-InSPIRE RI-261323
EGI-InSPIRE
Using a Widely Distributed Federated
Cloud System to Support Multiple
Disparate User Communities
David Wallom
University of Oxford
Chair, EGI Federated Cloud

Overview
• Why
• What
• Who

Growth of Providers
Grid
Limited academic resource providers
Cloud
Diversity and scale of new resource providers

Growth of Userbase
20k researchers 2M researchers!
Few related
use cases
Many diverse use cases

Hardware
Hardware
Hardware
Hardware
Hardware
Cloud Management
Cloud Management
Cloud Management
Cloud Management
Cloud Management
User Communities
User Communities
User Communities
Federated
interfaces
Federated
services
• Open Standards
• Technology independent
• Provider agnosticism
• Production integration and reuse
Principles of Federation

EGI Cloud Infrastructure
6
EGI Core Platform
Federated
AAI
Service
Registry
Monitoring Accounting
EGI Cloud Infrastructure Platform
Instance
Mgmt
Information
Discovery
Storage
Management
Cloud Management Stacks
(OpenStack, OpenNebula, Synnefo, …)
Help and
Support
Security Co-
ordination
Training and
Outreach
EGICollaborationTools
EGIApplication
DB
Image
Repository
EGICloudServiceMarketplace
Sustainable
Business
Models
User Community
GSIGLUE2
OCCI
CDMI
SAM UR
OVF
Cloudinit

Compatibility for technology
providers
Cloud Mgmt. Fram. Fed. AAI Monitoring Accounting Img. Mgmt. OCCI CDMI
OpenStack Yes Yes* Yes Yes Yes Yes
OpenNebula Yes Yes* Yes Yes Yes Yes
Synnefo Yes Yes Yes - Yes -
Cloudstack - - - - Yes -
Emotive Yes Yes* - N/A N/A Yes
SlipStream Yes N/A N/A Yes Yes Yes*
Perun Yes Yes N/A N/A N/A N/A
7
Example documentation for a CMF:-
https://wiki.egi.eu/wiki/Fedcloud-tf:ResourceProviders:OpenStack

Geographical dispersion
• 12 countries provide 19 certified
resources
– Czech Republic, Germany, Greece,
Hungary, Italy, Macedonia, Poland,
Slovakia, Spain, Sweden, Turkey,
United Kingdom
• 2 countries currently integrating
– France, Finland, Portugal, Croatia
• 5 countries interested
– Bulgaria, Israel*, The Netherlands,
Switzerland
• Worldwide interest
– South Africa* (SAGrid)
– South Korea* (KISTI)
– United States* (NIST, NSF Centres)
* Not shown on map

Utilisation
9
217352 VMs launched across 25 clouds

Federated Cloud Services
Federated IaaS and STaaS Cloud
EYR TC monthly meeting 10
Tier 1:
Reliable
Infrastructure Cloud
Tier 4:
Zero ICT
Infrastructures
Tier 3:
Platform as a Service
Tier 2:
General-purpose
platform services
PaaS
PaaS
DBaaS
Hadoop
aaS
VRE
Secure storage
KeyMgmt
Encryption
ACLmgmt
Virtual
eLaboratory
17 Feb 2014

AppDB – Virtual Appliance catalogue
11

EGI FedCloud Communities 9/2014
• Ecology – BioVeL: Biodiversity Virtual e-Laboratory
• Structural biology – WeNMR: a worldwide e-Infrastructure for NMR and structural biology
• Linguistics – CLARIN: ‘British National Corpus’ service (BNCWeb)
• Earth Observation – SSEP: European Space Agency’s Supersites Exploitation Platform for
volcano and earthquakes monitoring (Collaboration with Helix Nebula)
• Software Engineering – SCI-BUS: simulated environments for portal testing
• Software Engineering – DIRAC: deploying ready-to-use distributed computing systems
• Software Engineering – Catania Science Gateway Framework
• Musicology – Peachnote: dynamic analysis of musical scores
• Earth Observation – ENVRI: Common Operations of Environmental Research
infrastructures (collaboration with EISCAT3D)
• Geology – VERCE: Virtual Earthquake and seismology Research
• Ecology – LifeWatch: E-Science European Infrastructure for Biodiversity and Ecosystem
Research
• High Energy Physics – CERN ATLAS: ATLAS processing cluster via HelixNebula
More info: https://wiki.egi.eu/wiki/Fedcloud-tf:Users
12

New EGI FedCloud Communities
since launch
• Education – Cranfield University distributed systems course
• Cultural Heritage – DCH-RP management of preservation services in the cloud
• Hydrological Modelling – Running Hydrological models to support real time analysis
• Bioinformatics – ELIXIR execution of the Ensamble application in the Federated Cloud
environment
• Systems implementations – deployment of FTK developed tools and services and data
preservation
• Internet of Things – Smart Grid systems investigation
• Software Development – deployment of research PaaS
• RNA Sequencing – deployment of analysis engines in the cloud
• Physiological Modelling – Calibration, scenario mapping and development
More info: https://wiki.egi.eu/wiki/Fedcloud-tf:Users
13

WeNMR e-Infrastructure
(VirtualCing)
EGI Federated Cloud Infrastructure
…
NMR
Protein DB
ToPoS
Virtual
CINGVirtual
CINGVirtual
Cing
Virtual
CINGVirtual
CINGVirtual
Cing
Virtual
CINGVirtual
CINGVirtual
Cing
OCCI client
Virtual Common Interface for NMR structure generation (VirtualCing), developed in WeNMR,
has been deployed via the EGI Federated Cloud
14

ESA HN Flagship on the EGI
Federated Cloud
The European Space Agency deployed part of the ESA Helix Nebula
Flagship into the EGI Federated Cloud:
1. Use SlipStream Infrastructure Broker to deploy a custom
computing cluster on multiple Federated Cloud sites (CESNET,
GRNET)
2. Connect the computing cluster to the ESA G-POD system and run
SAR processing for volcano and earthquake monitoring
15
Automatic
Cluster
Deployment
EGI Federated Cloud
Computing Cluster
Cluster
Head
Working
Nodes
OCCI
Globus
Job submission
BLUE BOX

16
EISCAT 3D PoC
EISCAT_3D is a three-dimensional imaging radar to be located in the
northernmost parts of Europe, to study Earth atmosphere and its coupling to
space.
The EISCAT 3D Proof of Concept is a PaaS built on top of the EGI
Federated Cloud for Data Access and Dissemination, based on the
ENVRI tools, the EGI Federated Cloud Infrastructure and Storage services
and the current EISCAT data.

Peachnote
3/31/2015 17
Peachnote is a music score search engine and analysis platform.
Hundreds of thousands of music scores are
being digitized by libraries all over the world.
In contrast to books, they generally remain
inaccessible for content-based retrieval and
algorithmic analysis.
There is no analogue to Google Books for
music scores, and no large corpora exists that
can empower advanced analysis on music
scores.
Peachnote want to help change that providing
visitors and researchers access to a massive
amount of symbolic music data.
EGI Federated Cloud
OMR
Worker
PDF
Splitter
OMR
Feeder
OMR
Worker
…

OpenModeller on the
Biovel Portal
3/31/2015 18
Evalua
mod
BENELUX confe
Ghent, 02. April
The Ecological Niche Modeling (ENM) Workflow takes as input a
file containing species occurrence points to create a model with
the openModeller Web Service.
• The EUBrazilOpenBio ENM service is
exposed through an extended
openModeller Web Service interface
• Multi-staging and multi-parametric oM
experiments are implemented through
COMPSs that dynamically creates the
virtual resources to execute the
operations.
• An OCCI connector is used for the VMs
management while data management
supports CDMI endpoints.
ENM Service (OMWS2)
VENUS-C Cloud Middleware
COMPSs Workflow
Orchestrator
OCCI CDMI
EGI Federated Cloud
Service available at
https://portal.biovel.eu/

3/31/2015 20
Running GNU Octave and R on the
CHAIN-REDS Science Gateway
20
CHAIN-REDS Science Gateway
-CHAIN-REDS RI-306819

Strengthening the underpinning platform
Technology Evolution
• Broader support for open standards in Cloud
management frameworks
– Utilisation of rOCCI for interfaces to commercial cloud
frameworks
• New feature additions to foundational tools
depending on requests
– Accounting, monitoring, service discovery, Application
DB
• Innovative deployments of opensource PaaS
• Large improvements on documentation for all 3
stakeholder groups
22
ValueaddedservicesforUser
Communities

Conclusions
The EGI Federated Cloud, a federation of institutional private Clouds, offering Cloud
Services to researchers in Europe and worldwide
• A single cloud system able to
– Scale to user needs
– Integrate multiple different providers to give resilience
– Prevent vendor lock-in through open standards
– Enable resource provision targeted towards the research community
• Paving the way for a global federated cloud marketplace
– Established best practice
– Illustrating leadership
– Open standards, open technology
– Open membership, open processes
– Open competition
Current capability – ~6,000 cores, 300 TB storage
Promised through EGI-Inspire – 18,000 cores, 6000 TB storage
2020 Target – 1M cores, 1EB storage
23

Alison Packer, Álvaro López García, Alvaro Simon, Binh Minh
Nguyen, Björn Hagemeier, Boris Parak, Boro Jakimovski, Cal
Loomis, Christos Loverdos, Daniele Cesini, Daniele Lezzi, David
Blundell, Diego Scardaci, Elisabetta Ronchieri, Emir Imamagic,
Enol Fernandez, Feyza Eryol, Florian Feldhaus, Gergely Sipos,
Ivan Diaz, Jan Meizner, John Gordon, Kostas Koumantaros,
Malgorzata Krakowian, Marios Chatziangelou, Marco Verlato,
Marica Antonacci , Mattieu Puel, Matteo Turilli, Michel Jouvin,
Michel Drescher, Miroslav Ruda, Nuno L. Ferreira, Owen Synge,
Paul Miller, Peter Solagna, Piotr Kasprzak , Roberto Barbera,
Ruben Valles, Sándor Ács, Salvatore Pinto, Silvio Spardi,
Soonwook Hwang, Steven Newhouse, Stuart Pullinger, Thijs
Metsch, Tomasz Szepieniec, Viet Tran, Zeeshan Ali Shah and all
the other members of the EGI federated Cloud Task Force,
activity, workpackage

Thank you!
Questions?
25
Task Force resources
Mailing List: fedcloud-tf@mailman.egi.eu
Wiki site: http://go.egi.eu/tf-fedclouds
GitHub: https://github.com/EGI-FCTF
Indico site: https://www.egi.eu/indico/categoryDisplay.py?categId=112

Using a Widely Distributed Federated Cloud System to Support Multiple Disparate User Communities

More Related Content

What's hot

Similar to Using a Widely Distributed Federated Cloud System to Support Multiple Disparate User Communities

More from David Wallom

Recently uploaded

Using a Widely Distributed Federated Cloud System to Support Multiple Disparate User Communities

Editor's Notes