Opening the Path to Technical Excellence

ONE Conference 2013 - 26th September 2013
J.Farres – European Space Agency
R&D for Earth Observation Ground Segment
OpenNebula Conference, Berlin
26/9/2013
Cloud Computing in Space:
Opening the path to technical excellence

Agenda
1.Background and Objectives
2.ESA Experiences
1. EO Re-processing on Amazon
2. Dissemination and Processing on Hetzner
3. SuperSites Exploitation platform with Helix Nebula
4. A sandbox service for Science
3.Summary of lessons learnt
4.Future prospects

Objectives (1)
3- Processing bursting
2- Dissemination peaks
1- ICT Costs savings
4- Collaboration platform
Cloud Computing
IaaS
SaaS
Hosting
(VPS, Rental)
CDN
PaaS
A model for enabling convenient, on-demand network
access to a shared pool of configurable computing
resources (e.g., networks, servers, storage, applications,
and services) that can be rapidly provisioned and
released with minimal management effort or service
provider interaction [NIST]

Background: ESA Mission Statement
ESA's purpose shall be to provide for, and to promote, for
exclusively peaceful purposes, cooperation among European
States in space research and technology and their space
applications, with a view to their being used for scientific
purposes and for operational space applications systems:
• by elaborating and implementing a long-term European
space policy, by recommending space objectives to the
Member States, and by concerting the policies of the
Member States with respect to other national and
international organisations and institutions;
• by elaborating and implementing activities and programmes
in the space field;
• by coordinating the European space programme and national
programmes, and by integrating the latter progressively and
as completely as possible into the European space
programme, in particular as regards the development of
applications satellites;
• by elaborating and implementing the industrial policy
appropriate to its programme and by recommending a
coherent industrial policy to the Member States.ONE Conference 2013 - 26th September 2013

Objectives (1)
3- Processing bursting
2- Dissemination peaks
1- ICT Costs savings
4- Collaboration platform
Cloud Computing
IaaS
SaaS
Hosting
(VPS, Rental)
CDN
PaaS
A model for enabling convenient, on-demand network
access to a shared pool of configurable computing
resources (e.g., networks, servers, storage, applications,
and services) that can be rapidly provisioned and
released with minimal management effort or service
provider interaction [NIST]
5- Lead effective use of modern
computing infrastructures
by European industry

Agenda
1.Background and Objectives
2.ESA Experiences
1. Mass processing on Amazon
2. Dissemination and Processing on Hetzner
3. SuperSites Exploitation platform with Helix Nebula
4. A sandbox service for Science
3.Summary of lessons learnt
4.Future prospects

Case 1: Mass processing on Amazon(1)
Purpose
•Fast re-processing of large EO products collections for CalVal
purposes.
Project / Service
•Timeframe: 2009 and 2011
•Provider: Amazon, EC2, S3
•Data: ERS SAR Wave, MIPAS (30,000 products)
•System: 200 Virtual Servers
configured as Working Nodes to an ESA grid.
•Usage: 11 CPU years of processing in 5 weeks

https://earth.esa.int/web/guest/missions/esa-operational-eo-missions/envisat/instruments/mipas

Pros Cons
Case 1: Mass processing on Amazon(2)
Pros
1. Excellent processing scalability
2. Efficient bulk-in/out data service
(via HD)
3. “The faster the cheaper” as it
reduces storing costs
4. Good application portability for
gridified applications
Cons
1. Complex and changing pricing,
e.g. periods with cheaper hosts
and free data upload.
2. In-bound / Out-bound costs
3. Ad-hoc scripts to command
provisioning
4. AMI format portability

Case 2: Dissemination and Processing on
Hetzner (1)
Purpose
•Couple large processing and dissemination capabilities for low cost.
Project / Service
•Timeframe: 2011
•Provider: Hetzner
•Data: 60TB
•System: 1 Head: Catalogue, Processor Register
n Nodes: Data Dissemination, Hadoop Processing Cluster
Packaged as a back-end for web portal services
•Usage: GeoHazards SuperSites (38,000 SAR images and 3,000 users)

InSAR Processing
~150 Satellite
images:
1.5TB
Time
Processing:
150h
CNR

Pros Cons
Case 2: Dissemination and Processing on
Hetzner (2)
Pros
1. Good archive scalability (chunks
of 8TB)
2. Synergy of processing -
dissemination: processing peaks
followed by dissemination peaks
3. Much cheaper than Amazon
services. No in-boud/out-boud
costs
4. Physical dedicated servers
enabled easier security
Cons
1. System scales storage,
dissemination and processing
capabilities simultaneously.
2. Lower service levels than Level
3 or Amazon
3. Virtualization layer had to be
deployed (KVM and ONE)

Case 3: Exploitation platform with Helix
Nebula (1)
Purpose
•Pilot a collaborative platform for EO exploitation using multisourced
cloud provisioning.
Project / Service
•Timeframe: 2012-2013
•Providers: ATOS, CloudSigma, Interoute, T-Systems, EGI
•Cloud Brokers: SlipStream, Entratius
•Data: GeoHazards data (ESA, CNES, DLR, …)
•Processors: ESA, CNR, Gamma, …
•System: 15 TB of raw data, 4 processing services,
dedicated VM for selected users, >200 users

Model
EO Data
Software
ICT
Exploitation
Platform
Science
Services
Services
ESA UNCLASSIFIED – For Internal Use – Version 2013/04/29

SSEP
SuperSites Exploitation Platform
ESA UNCLASSIED - For Offical Use
29/04/2013

/9/2013
18
ESA PoC on EGI Federated Cloud
The ESA Proof of Concepts on the EGI Federated Cloud focuses on
demonstrating the possibility to provide Processing Services to ESA scientists
using the EGI Federated Cloud resources
• Participants:
– User community
• ESA Research and Service Support section: Configuration and execution of the
tests
– Technology providers
• Sixsq: Provided Open Source SlipStream software and OCCI connector
– Resource providers
• CESNET (OpenNebula): Performance tests and multi-site tests
• GRNET (synnefo): Multi-site tests
• CESGA (OpenNebula): Hosting of the SlipStream server

Pros Cons
Case 3: Exploitation platform with Helix
Nebula (2)
Pros
1. High performance storage and
dissemination from cloud
2. Scalable processing co-located
with data (same VDC)
3. Multi-sourced via cloud
brokering services
4. Direct provision to virtual hosts
to users in the cloud, via ESA
5. Easy application deployment via
grid controller
Cons
1. Frequent platform upgrades
2. Need to distribute processing
resources near distributed data
3. No Cloud federation. Limitations
of brokers
4. COTS licensing
5. Grid on Cloud approach.
Workaround to high effort in
application “cloudification”

Case 4: A sandbox service for Science (1)
Purpose
• Provide researchers and service providers a development environment
for cloudifying and exploiting their algorithms/services.
Project / Service
•Timeframe: 2013-2014
•Providers: Private cloud + Helix Nebula
•Data: Multiple reference data sets from ESA archives
•Processors: Those developed by the users

10/9/2013
22
Many
Service
Providers

Pros Cons
Pros
1. Hybrid cloud model in support of
development (private) ->
deployment (public)
2. Deployment model via PaaS and
SaaS
3. Simplified “cloudification” via
Cloudera + supporting tools (for
SPMD paradigm)
Cons
1. Need for new CSP drivers for
Helix Nebula: T-Systems
(Zimory), Interoute (Jclouds)
2. Limited CSP support to PaaS
and SaaS services.
3. Slow adoption of cloud-reduce
paradigm among application
developers in Remote Sensing

Role of OpenNebula on the four Cases
Case ONE
1: Mass
processing on
Amazon
Use of GridWay to command IaaS provisioning from
Amazon
2: Dissemination
and Processing
on Hetzner
Use of ONE to host private IaaS on dedicated hosting
services
3: Exploitation
platform with
Helix Nebula
Use of ONE in resource providers to EGI Federated Cloud
4: A sandbox
service for
Science
Use of ONE for hybrid cloud provision ESA + Helix Nebula
Development of drivers for Zimory and Jclouds

Lessons Learnt: ICT provisioning
1. As soon as ICT needs can be predicted and planned, IaaS is
more expensive than other hosting solutions like (rental,
dedicated hosting).
2. Flexibility of Public IaaS is less appealing when internal
resources are pooled, virtualized and managed as an internal
cloud.
3. On the other hand, IaaS services allow to size down internal ICT
resources to the “fixed” need and ensure their maximum
utilization; e.g. using external provisioning for the “variable”
need.
Hybrid ICT provisioning

Lessons Learnt: Service Levels
• Terms & Conditions in Public Clouds express surprising low
commitment.
• Cloud opportunities can become risks when applied to critical
systems.
Develop multi-sourcing
Plan contingency scenarios for services hosted in Public
Clouds

Lessons Learnt: Application Areas
• Dissemination and on-demand processing
– because are very variable (depending on user demand)
• Secondary archive and re-processing
– because are limited in time
• Temporary resources for integration, testing and demonstration
– because are limited in time
• System sizing
– because needs are unknown
Important areas when remote sensing services can gain
from Cloud Computing

Lessons learnt: User expectations
• Open Data
– All data are discoverable, accessible online and free
– Data is arranged on long time series of coherent data from different providers.
• Open Computing
– Users will be able to perform processing directly on the cloud using virtual servers.
– Users can choose their preferred cloud provider
• Open Source Software
– All basic/platform software is open and freely available
– Application can be easily ported across clouds
• Open Collaboration
– Data and applications can be easily shared with other users
Be up to the users expectations

Lessons learnt and OpenNebula
OK, Case 4
OK, Case 3
Lesson learnt ONE
Hybrid cloud OK, Case4: A sandbox service for
Science
Cloud multi-sourcing OK, Case3: Exploitation platform
with Helix Nebula
Application Areas N/A
User Expectations OK, Open Source Software

Future Prospects (1)
ICT
• Set-up mid-term relation with 2-4 cloud providers;
similarly to current agreements with network providers.
• Cloudify present corporate computing resources
• Establish common ICT provisioning service based on:
– Hybrid and multi-sourced resources
– Based on in-house brokering layer

POLICY
• Mandate cloud hosting for specific activities: Integration
& Validation, Demonstrators, R&D …
• Promote the use of cloud computing solutions among
ESA service providers, in benefit for their competiveness
• Continue to launch specific flagship projects based on
public clouds
– SuperSites exploitation platform (continuation)
– Thematic Exploitation platforms

1. Adopt and promote Open data policy
– ESA Data Policy
– Data access agreements
2. Adopt and promote interoperability Standards
– Data discovery & ordering
– Processing discovery & ordering
– Data / Results access
– RAAA – Registration, Authentification, Authorisation, Accounting
3. Provide Open Source Software
– EO Toolboxes
– Data discovery/catalogue tools
– EO Data management and access
4. Promote Open computing infrastructures
– Cloud computing paradigm
– Standards for IaaS and brokering services
– Software Licensing Agreements

In summary: Manifest
+ Open Data
+ Open Software
+ Open Computing
=
Increased market development &
industrial competiveness

Opening the Path to Technical Excellence

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (11)

Similar to Opening the Path to Technical Excellence

Similar to Opening the Path to Technical Excellence (20)

Recently uploaded

Recently uploaded (20)

Opening the Path to Technical Excellence

Editor's Notes