EGI Federated Cloud presentation Cloud Workshop talk, MYREN 7th National Conference 2014

1. www.egi.euEGI-InSPIRE RI-261323
EGI-InSPIRE
www.egi.euEGI-InSPIRE RI-261323
The EGI Federated Cloud, 7 months of
production
David Wallom
University of Oxford
Chair, EGI Federated Cloud

2. www.egi.euEGI-InSPIRE RI-261323
• Why
• Principles
• What
• Who
• For whom
• Future

3. www.egi.euEGI-InSPIRE RI-261323
Growth of Providers
• Grid
– Limited academic resource providers
• Cloud
– Diverse of resource providers

4. www.egi.euEGI-InSPIRE RI-261323
Growth of Userbase
20k researchers 2M researchers!
Few related
use cases
Many diverse use cases

5. www.egi.euEGI-InSPIRE RI-261323
Hardware
Hardware
Hardware
Hardware
Hardware
Cloud Management
Cloud Management
Cloud Management
Cloud Management
Cloud Management
User Communities
User Communities
User Communities
Federated
interfaces
Federated
services
• Standards and validation: Recommended and
common open standards for the interfaces and
images – OCCI, CDMI, OVF, GLUE2.
• Resource integration: Cloud Computing to be
integrated into the existing production
infrastructure.
• Heterogeneous implementation: no mandate
on the cloud technology.
• Provider agnosticism: the only condition to
federate resources is to expose the chosen
interfaces and services.
Principles of Federation

6. www.egi.euEGI-InSPIRE RI-261323
Value proposition
The EGI Federated Cloud, a federation of institutional private Clouds,
offering Cloud Services to researchers in Europe and worldwide
A single cloud system able to
• Scale to user needs
• Integrate multiple different providers to give resilience
• Prevent vendor lock-in
• Enable resource provision targeted towards the research
community
Standards based federation of IaaS cloud:
• Exposes a set of independent cloud services accessible to users
utilising a common standards profile
• Allows deployment of services across multiple providers and
capacity bursting

7. www.egi.euEGI-InSPIRE RI-261323
EGI Cloud Infrastructure
EGI Core Platform
Federated
AAI
Service
Registry
Monitoring Accounting
EGI Cloud Infrastructure Platform
Instance
Mgmt
7
Information
Discovery
GSIGLUE2
Cloudinit
Storage
Management
CDMI
Cloud Management Stack
(OpenStack, OpenNebula, Synnefo, …)
SAM UR
Help and
Support
Security Co-
ordination
Training
and
Outreach
EGICollaborationTools
EGIApplication
DB
ImageRepositoryEGICloudServiceMarketplace
OVF
Sustainable
Business
Models
OCCI

8. www.egi.euEGI-InSPIRE RI-261323
VM Instance Management through rOCCI
+ Contextualisation with Cloudinit
• Contextualisation, the process of installing, configuring and preparing software upon boot time
• All uploaded Fedcloud images should support Cloudinit
• All supported CMF OCCI implementations extended to support

9. www.egi.euEGI-InSPIRE RI-261323
Accounting
9
~384k VMs launched across 25 clouds
Usage Record Content
• Site
• User: local user/group, Digital Certificate
DN
• VMUUID (VM instance identifier)
• Start (and End) Time
• Status
• Wall and CPU Duration
• VM source image
• Cloud Type

10. www.egi.euEGI-InSPIRE RI-261323
Status
10

11. www.egi.euEGI-InSPIRE RI-261323
Federated AAI
Inherited from the EGI AAI framework
• Scalable with many users
• Uniform authentication across multiple service providers
• Users manage their community via Virtual Organisations

12. www.egi.euEGI-InSPIRE RI-261323
AppDB – Virtual Appliance catalogue
12

13. www.egi.euEGI-InSPIRE RI-261323
Compatibility for technology
providers
Cloud Mgmt. Fram. Fed. AAI Monitoring Accounting Img. Mgmt. OCCI CDMI
OpenStack Yes Yes* Yes Yes Yes Yes
OpenNebula Yes Yes* Yes Yes Yes Yes
Synnefo Yes Yes Yes - Yes -
StratusLab* Yes Yes Yes - Yes -
Cloudstack Yes
Emotive Yes Yes
13
Example documentation for a CMF:-
https://wiki.egi.eu/wiki/Fedcloud-tf:ResourceProviders:OpenStack

14. www.egi.euEGI-InSPIRE RI-261323
EGI Marketplace
• Rich and diverse set of individual offerings
• Application/VM Image repository
• User community choice of provider at all levels
• Academic and commercial resource & service providers
• User community from the public and private sector

15. www.egi.euEGI-InSPIRE RI-261323
Federated Cloud Services
Federated IaaS and STaaS Cloud
EYR TC monthly meeting 15
Tier 1:
Reliable
Infrastructure Cloud
Tier 4:
Zero ICT
Infrastructures
Tier 3:
Platform as a Service
Tier 2:
General-purpose
platform services
PaaS
PaaS
DBaaS
Hadoop
aaS
VRE
Secure storage
KeyMgmt
Encryption
ACLmgmt
Virtual
eLaboratory
17 Feb 2014

16. www.egi.euEGI-InSPIRE RI-261323
Partnership
• Resources
– 12 countries provide 19 certified resources
• Czech Republic, Germany, Greece, Hungary,
Italy, Macedonia, Poland, Slovakia, Spain,
Sweden, Turkey, United Kingdom
– 4 countries currently integrating
• France, Finland, Portugal, South Korea*
– 5 new countries interested+
• Bulgaria, Denmark, Israel*, The Netherlands,
Switzerland
– Worldwide interest
• South Africa* (SAGrid)
• United States* (NIST, NSF Centres)
• Australia* (NeCTAR)
• People
– >155 individuals participating and
contributing to the Federated Cloud group
– >150 conference calls (Sep 2011 – August
2014)
• Utilisation
– 98’860 VM instantiations in the last year * Not shown on map

17. www.egi.euEGI-InSPIRE RI-261323
User Model
• Total control over deployed
applications
• Elastic resource consumption
based on real needs
• Immediately processed
workloads – no more waiting
time
• An extended e-Infrastructure
across resource providers in
Europe
• Service performance scaled
with elastic resource
consumption
• Single sign-on at multiple,
independent providers

18. www.egi.euEGI-InSPIRE RI-261323
EGI FedCloud Communities 9/2014
• Ecology – BioVeL: Biodiversity Virtual e-Laboratory
• Structural biology – WeNMR: a worldwide e-Infrastructure for NMR and structural biology
• Linguistics – CLARIN: ‘British National Corpus’ service (BNCWeb)
• Earth Observation – SSEP: European Space Agency’s Supersites Exploitation Platform for
volcano and earthquakes monitoring (Collaboration with Helix Nebula)
• Software Engineering – SCI-BUS: simulated environments for portal testing
• Software Engineering – DIRAC: deploying ready-to-use distributed computing systems
• Software Engineering – Catania Science Gateway Framework
• Musicology – Peachnote: dynamic analysis of musical scores
• Earth Observation – ENVRI: Common Operations of Environmental Research
infrastructures (collaboration with EISCAT3D)
• Geology – VERCE: Virtual Earthquake and seismology Research
• Ecology – LifeWatch: E-Science European Infrastructure for Biodiversity and Ecosystem
Research
• High Energy Physics – CERN ATLAS: ATLAS processing cluster via HelixNebula
More info: https://wiki.egi.eu/wiki/Fedcloud-tf:Users
18

19. www.egi.euEGI-InSPIRE RI-261323
New EGI FedCloud Communities
since launch
• Education – Cranfield University distributed systems course
• Cultural Heritage – DCH-RP management of preservation services in the cloud
• Hydrological Modelling – Running Hydrological models to support real time analysis
• Bioinformatics – ELIXIR execution of the Ensamble application in the Federated Cloud
environment
• Systems implementations – deployment of FTK developed tools and services and data
preservation
• Internet of Things – Smart Grid systems investigation
• Software Development – deployment of research PaaS
• RNA Sequencing – deployment of analysis engines in the cloud
• Physiological Modelling – Calibration, scenario mapping and development
More info: https://wiki.egi.eu/wiki/Fedcloud-tf:Users
19

20. www.egi.euEGI-InSPIRE RI-261323
WeNMR e-Infrastructure
(VirtualCing)
EGI Federated Cloud Infrastructure
…
NMR
Protein DB
ToPoS
Virtual
CINGVirtual
CINGVirtual
Cing
Virtual
CINGVirtual
CINGVirtual
Cing
Virtual
CINGVirtual
CINGVirtual
Cing
OCCI client
Virtual Common Interface for NMR structure generation (VirtualCing), developed in WeNMR,
has been deployed via the EGI Federated Cloud
20

21. www.egi.euEGI-InSPIRE RI-261323
ESA HN Flagship on the EGI
Federated Cloud
The European Space Agency deployed part of the ESA Helix Nebula
Flagship into the EGI Federated Cloud:
1. Use SlipStream Infrastructure Broker to deploy a custom
computing cluster on multiple Federated Cloud sites (CESNET,
GRNET)
2. Connect the computing cluster to the ESA G-POD system and run
SAR processing for volcano and earthquake monitoring
21
Automatic
Cluster
Deployment
EGI Federated Cloud
Computing Cluster
Cluster
Head
Working
Nodes
OCCI
Globus
Job submission
BLUE BOX

22. www.egi.euEGI-InSPIRE RI-261323
22
EISCAT 3D PoC
EISCAT_3D is a three-dimensional imaging radar to be located in the
northernmost parts of Europe, to study Earth atmosphere and its coupling to
space.
The EISCAT 3D Proof of Concept is a PaaS built on top of the EGI
Federated Cloud for Data Access and Dissemination, based on the
ENVRI tools, the EGI Federated Cloud Infrastructure and Storage services
and the current EISCAT data.

23. www.egi.euEGI-InSPIRE RI-261323
Peachnote
3/31/2015 23
Peachnote is a music score search engine and analysis platform.
Hundreds of thousands of music scores are
being digitized by libraries all over the world.
In contrast to books, they generally remain
inaccessible for content-based retrieval and
algorithmic analysis.
There is no analogue to Google Books for
music scores, and no large corpora exists that
can empower advanced analysis on music
scores.
Peachnote want to help change that providing
visitors and researchers access to a massive
amount of symbolic music data.
EGI Federated Cloud
OMR
Worker
PDF
Splitter
OMR
Feeder
OMR
Worker
…

24. www.egi.euEGI-InSPIRE RI-261323
OpenModeller on the
Biovel Portal
3/31/2015 24
Evalua
mod
BENELUX confe
Ghent, 02. April
The Ecological Niche Modeling (ENM) Workflow takes as input a
file containing species occurrence points to create a model with
the openModeller Web Service.
• The EUBrazilOpenBio ENM service is
exposed through an extended
openModeller Web Service interface
• Multi-staging and multi-parametric oM
experiments are implemented through
COMPSs that dynamically creates the
virtual resources to execute the
operations.
• An OCCI connector is used for the VMs
management while data management
supports CDMI endpoints.
ENM Service (OMWS2)
VENUS-C Cloud Middleware
COMPSs Workflow
Orchestrator
OCCI CDMI
EGI Federated Cloud
Service available at
https://portal.biovel.eu/

25. www.egi.euEGI-InSPIRE RI-261323
BioSTIF - Biodiversity Spatio-temportal
interface on the BioVel Portal
3/31/2015 25
BioSTIF provides an easy GIS interface to vizualize and quality
control biodiversity data in time and space
Evaluat
mode
The interface contains;
• a map widget (displaying
geographical data)
• a timeline widget
(displaying data on a time
scale)
• a table widget (displaying
data in table format).
Service available at
https://portal.biovel.eu/

26. www.egi.euEGI-InSPIRE RI-261323
3/31/2015 26
Running GNU Octave and R on the
CHAIN-REDS Science Gateway
26
CHAIN-REDS Science Gateway
-CHAIN-REDS RI-306819

27. www.egi.euEGI-InSPIRE RI-261323
Usecase Integration Workflow
Status Description
Pre-
assessment
FCUS members identified a potential use case that can profit from the EGI Federated Cloud services
Assessed
The use case requirements are assessed with relevant information added to a dedicated wiki page, to manage the
full porting & integration process
Preparatory
The user & FCUS team setup the environment to execute tests on the FedCloud (configuration of client
environment, identify resource providers, upload VM images, ...)
Test &
Integration
Users are accessing the FedCloud through the fedcloud.egi.eu VO to execute tests while integrate their
applications to the infrastructure
Pre-production
Test & integration phases were successfully completed. Create a production Virtual Organization (VO) or join an
already existing VO
Production
Use case is being used in production. The use case completed all tests and is regularly making use of the
Federated Cloud using a production level VO
Closed Test & integration successfully completed. Use case either does not foresee moving into production.
Cancelled Test & integration did not successfully completed. User cancelled the use case.
10 6411 3
6

28. www.egi.euEGI-InSPIRE RI-261323
Strengthening the underpinning platform
Technology Evolution
• Broader support for open standards in Cloud
management frameworks
– Utilisation of rOCCI for interfaces to commercial cloud
frameworks
• New feature additions to foundational tools
depending on requests
– Accounting, monitoring, service discovery, Application
DB
• Innovate deployments of opensource PaaS
• Large improvements on documentation for all 3
stakeholder groups
28
ValueaddedservicesforUser
Communities

29. www.egi.euEGI-InSPIRE RI-261323
EGI’s Cloud Vision for 2020
To support the digital European Research Area through a pan-
European research infrastructure based on an open federation of
reliable services that provide uniform access to computing and
data resources provided by the public and private sector.
29
EGI Federated Cloud capability vision
10M cores Cloud compute
1 EB Cloud storage

30. www.egi.euEGI-InSPIRE RI-261323
Conclusions
• 40 individual usecases from 27 communities have
used the fedcloud activities since launch
– ‘Standard’ and innovative utilisation of cloud including
research and teaching
• Paving the way for a global federated cloud
marketplace
– Established best practice
– Illustrating leadership
– Open standards, open technology
– Open membership, open processes
– Open competition
30

31. Alison Packer, Álvaro López García, Alvaro Simon, Binh
Minh Nguyen, Björn Hagemeier, Boris Parak, Boro
Jakimovski, Cal Loomis, Daniele Cesini, Daniele Lezzi,
David Blundell, Diego Scardaci, Elisabetta Ronchieri,
Emir Imamagic, Enol Fernandez, Feyza Eryol, Florian
Feldhaus, Gergely Sipos, Jan Meizner, John Gordon,
Kostas Koumantaros, Malgorzata Krakowian, Marios
Chatziangelou, Marco Verlato, Marica Antonacci ,
Mattieu Puel, Matteo Turilli, Michel Jouvin, Michel
Drescher, Miroslav Ruda, Nuno L. Ferreira, Owen Synge,
Paul Miller, Peter Solagna, Piotr Kasprzak , Roberto
Barbera, Ruben Valles, Sándor Ács, Salvatore Pinto,
Silvio Spardi, Soonwook Hwang, Steven Newhouse,
Stuart Pullinger, Thijs Metsch, Tomasz Szepieniec, Viet
Tran, Zeeshan Ali Shah.

32. www.egi.euEGI-InSPIRE RI-261323
Thank you!
Questions?
32
Task Force resources
Mailing List: fedcloud-tf@mailman.egi.eu
Wiki site: http://go.egi.eu/tf-fedclouds
GitHub: https://github.com/EGI-FCTF
Indico site: https://www.egi.eu/indico/categoryDisplay.py?categId=112