Digital Infrastructures for Research 2018 https://www.digitalinfrastructures.eu/ Topic "Business Models, Sustainability and Policies" Session "Open Science: Skills and Credits" ISCTE, Lisbon, Tuesday 9th October 2018

1. Ellip: a collaborative workplace
for EO Open Science
Lisbon, October 9th
, 2018
Hervé Caumont
Platform Operations, Terradue

2. Copernicus Sentinels as the nextgen
Earth Observation satellites
Typical EO data volumes
500MB (S-2) < a single product < 8GB (S-1)

3. ~5000 ~400

4. Why Terradue ?
▪ Terradue is an ESA spin-off (2006)
▪ From day one providing support to application
builders in earth sciences, to use satellite EO
data as information source
▪ Early days Grid Computing. Nowadays, Cloud
PaaS, complemented with APIs for Cloud
bursting, aimed at the transfer in production of
released applications
▪ Business model: Platform-centered, a
collaborative workplace for value adders to
interact & co-createNow opening new Ellip solutions
to beta testers, who will help us
improve our Platform services

5. Why value co-creation ?
▪ There is still need for significant capacity
building within the earth sciences community
in exploiting EO data
▪ We believe the value co-creation model is the
way forward to unleash innovation
▪ Since 2014 we’re evolving Terradue Cloud
Platform, user-driven, via ESA & EU funding
▪ A steady and careful process, as well as a
learning process
Now opening new Ellip solutions
to beta testers, who will help us
improve our Platform services

6. Funding & User communities

7. Hosted Communities
↗
H2020 Copernicus App Lab
(2016-2018)
↗
H2020 INTAROS
(2016-2021)
ESA Exploitation Platforms
(2014 onwards)
70+
EC FP7 MELODIES
(2014-2016)
14
EC FP7 SenSyF
(2013-2015) 8
EC FP7 GEOWOW
(2011-2013) 5
Processing Campaigns
(2013-onwards) 35+
▪ Climate Change
▪ Carbon market
▪ Oceanography
▪ Climatology
▪ Biodiversity
▪ Volcanology
▪ Ship logistics
▪ Water quality
▪ Desertification
▪ Seismic analysis
▪ Urban accounting
▪ Land management
▪ Disaster Risk Reduction
H2020 Co-ReSyF
(2016-2018)15+
10+
H2020 NextGEOSS
(2016-2020)
↗
H2020 ECOPOTENTIAL
(2015-2019)5+
↗
H2020 EVER-EST
(2015-2018)4+
4+
3+

8. ▪ EO data is a unique source of global
measurements over decades
▪ When calibrated and combined with other sources, it
empowers validation & interpolation models
▪ Collaboration process: a feedback loop of
providers & consumers
▪ More providers of pre-processed data layers
▪ New demands for structured information
The growth model is pulled by the
communities
Earth Observation Platform
Producers
Consumers

9. Partnerships
We are always seeking for new Partnerships to
expand our Platform services to users :
▪ Data providers
▪ e-Infra / IaaS providers
▪ Technology providers
▪ Scientific Toolbox/Algorithm/Model providers
A particular focus from community requirements
is to boost the joint use of EO & in-situ data

10. EGI.eu & Terradue Partnership
M2M communications realised initially via OCCI and then via native APIs for
cloud computing, ensuring integration, portability, and interoperability.
Use of standard interfaces
We can rely on well-defined operational processes (e.g. FitSM), having the
EGI Foundation acting as single point of contact of the EGI Federated Cloud.
Reliable operations
We use Cloud resources within the European boundaries. In fact, it is
mandatory for certain processing services due to their organisations’ policies.
European cloud resources
The partnership with EGI Foundation enhances the sustainability of the
involved projects, via SLAs / OLAs helping to sustain operational lifetimes.
Project(s) sustainability
Oct. 2018
https://www.egi.eu/about/newsletters/egi-terradue-a-long-lasting-partnership/
Nov. 2015

11. Integration roadmap
2016/09 - 1st MoU
2015 2016 2017 2018
Feasibility
Setup
Dev
Acceptance
Pilots(cntd)
Monitor
(cntd)
Monitor
Pilots
Acceptance
(cntd)
2018/09 - 2nd MoU

12. Other partnership examples
Science providers:
● research and commercial
(GEP, HEP, NextGEOSS)
Data providers:
● EPOS RI on Geohazards TEP
for data uploads and
processing
● Open Data policies in the EO
data world
● (cf. RDA interview)
Technology providers:
● OpenNebula (since before ESA
TEPs early stage, circa 2012)
● OGC standards (Terradue as
standard editor)
Cloud IaaS providers:
● EGI.eu (NextGEOSS)
● EOSC.eu (EOSC-hub)
● DIAS (AppLab, GEP)
https://www.rd-alliance.org/interview-terradue

13. Opportunities to partner

14. ● Since Terradue’s inception 12 years ago, our
company DNA pushes us to innovate
● We believe technology serves new ways to
design products, that people can use to better
work collaboratively
● Platform Operations goals
○ Manage well defined usage scenarii as ‘products’
○ Pilot a unified foundation for software solutions
○ Innovate smoother, end-to-end business processes matching
customer needs
○ Guarantee the ability to share data among applications
○ ...
Why do Platform Operations exist ?

15. ● Platform Operations require a thoughtful mix of
stability & innovation
● We’re embarking on a new journey to learn how
to tune this mix, based on:
○ The market needs
○ Our own skills and capacities to deliver
software of high quality
○ The cost / benefit of the solutions that we
decide to market
○ The market recognition of what we do
How to run Platform Operations ?
Stability
Timely
Innovation

17. How to work with Ellip ?

18. Federated IdPs & Terradue SSO
Possibility to keep
your preferred
Identity Provider
OpenID Connect
preferred
Under work: User
Managed Access
(UMA) profile of
OAuth 2.0

20. • Business model for
sustainability of hosted
processing operations
• Public funding,
commercial revenues,
and costs-sharing
• Cloud providers,
• Processors developers,
• Applications
Integrators
• Scientific users
Subscription Plans
EXPLORER
Low cost
App. development
For
Data science users
SCALER
Low cost
Cluster processing
For
Producer users
PREMIUM
PARTNER
Higher cost
Innovation
& platform evolutions
For
Integrator users
4 Partner Programs
◆ Technology Alliance
◆ Virtual Archive
◆ Data Science
◆ Cloud Appliance

21. Ellip Dashboard

22. ● Ellip Notebooks
○ “Create interactive laboratory notebooks”
● Ellip Workflows
○ “Design scalable processing chains”
● Ellip Launchpads
○ “Select a data processor and deploy it”
● Ellip Infohubs
○ “Connect a WPS to a custom Geobrowser”
Ellip Solutions
Vestibulum
congue
Vestibulum
congue
Vestibulum
congue
Vestibulum
congue
Ellip
Infohubs
Ellip
N
otebooks
Ellip
Launchpads
Ellip
W
orkflow
s

23. Ellip Solutions in a nutshell
I want to build and maintain a processing service
Interactively
prototype
Integrate for
scalability
Run a scale
Monitor and
collaborate

24. Vestibulum
congue
Vestibulum
congue
Vestibulum
congueVestibulum
congue
Ellip
Exchange
Ellip
Infohubs
Ellip
N
otebooks
Ellip
Launchpads
Ellip
W
orkflow
s
The Ellip
Exchange as
the core enabler
of Ellip as a
‘collaborative
workplace’
. Application code
. Packaged application
. Tutorial
. Running WPS
at a selected
Cloud Provider
. New
information
layers
. Reproducible experiment as code
. Interactive data visualisations
. OWS Context template
. ad-hoc Community Portal

26. So, what does a platform mean nowadays?
The Ellip Solutions are built for earth sciences
practitioners:
▪ to share about their findings, streamline
their creation of new data products and make
these accessible, interoperable,
▪ and document their developments on the PaaS
as reproducible experiments
In line with the FAIR guiding principles to make
data findable, accessible, interoperable and
reusable

27. BuildIntegrate & Test Package Dockerize
Code
Commit
Scan
Repository
Production
Environments
Continuous Integration & Deployment
Code
Upload
Application
Release
Integration
Environment

28. Hybrid Cloud
Openstack API -
powered by jclouds
Poznań Supercomputing
and Networking Center
EC2 - powered by jclouds
Amazon Web Services
OCCI,
native cloud APIs
EGI Federated Cloud
Opennebula
registered partner
Terradue
Openstack API -
powered by libcloud
IPT.PL
AWS
EGI
IPT
PSNC
T2

29. www.terradue.com
Scheduled Data Processing
29
24 processing nodes provided by
EGI.eu (NextGEOSS partner)
Growing season
2016-2017
Biopar
products
Terradue Cloud Platform
Biopar Application Burst to EGI.eu

30. www.terradue.com
Scheduled Data Processing
30
It’s a data driven systematic processing. The
service will follow a ramp-up period starting
from September 2016 until end 2017:
■ EU Tectonic area
■ World tectonic area (40%)
■ World tectonic area (70%)
It currently processes 150+ Sentinel-1 SLC
pairs per day.
DLR InSAR Browse Medium
Resolution Service
Supported by
BELNET-BEGRID (Belgium)

31. www.terradue.com
User-Driven Data Processing
31
This service provides the coherence and
intensity changes for Sentinel-1 TOPSAR IW
data pairs performed through SNAP. SNAP is
a common architecture for all Sentinel
Toolboxes, which is ideal for Earth
Observation processing and analysis.
ESA SNAP Sentinel-1 COherence
and INtensity (COIN)
Supported by
GOEGRID-GWGD (Germany)

32. H2020 project BETTER

33. H2020 project BETTER

34. All of it: an Open Cloud strategy
■ Open APIs
Embrace Cloud bursting APIs that can be
easily plugged into the Platform’s codebase,
so to expand the Platform offering with
Providers offering complementary strategic
advantages for different user communities.
■ Developers community
Support and nurture Cloud communities that
collaborate on evolving open source
technologies, including at the level of the
Platform engineering team, when it comes to
deliver modular extensions.
■ Self-service provisioning and
management of resources
The Platform’s end-users are able to
self-provision their required ICT resources
and to work autonomously.
■ Users rights to move data as
needed
By supporting distributed instances of its EO
Data management layer, the Platform
delivers the required level of data locality to
ensure high performance processing with
optimized costs.

35. All of it: an Open Cloud strategy
▪ The Ellip subscribers remain the owners of
their created assets, and decide how to
share these on the platform, for other users
to find and reuse these assets.
▪ They are also empowered to decide on the
target production environment of their
packaged applications (no lock in with
Cloud provider)
Producers
Consumers

36. That’s it!
▪ Last message: we’re always looking for
new collaborators
▪ Join our team in Rome La Città Eterna
15 members, 5 EU nationalities
▪ We are welcoming experts in EO
applications and cloud technologies
▪ Exciting new challenges rolling out Ellip
for Earth Science

37. Looking forward
hearing from you!
https://www.terradue.com
Hervé Caumont, Platform Operations
herve.caumont@terradue.com