OGC November 2019 Technical Committee Earth Observation Exploitation Platforms (EOEP) session Toulouse, France

1. ®
Sponsor
OGC Interfaces in Thematic
Exploitation Platforms
113th OGC Technical Committee
Toulouse, France
Pedro Gonçalves, Terradue
18 November 2019
Copyright © 2019 Open Geospatial Consortium

2. OGC
®
Thematic Exploitation Platforms
• R&D activities started in 2016 to
create an ecosystem of Thematic
Exploitation Platforms
• Users access a work environment
containing the data and resources,
as opposed to downloading and
replicating the data “at home”
The fundamental principle is to
move the user to the data and tools
Copyright © 2019 Open Geospatial Consortium

3. OGC
®
Fields in each Thematic Domain
• Geohazards domain
– Earthquakes, Volcanoes,
Landslides …
• Hydrology domain
– Hydrological modelling, Flood
mapping, Water quality
monitoring …
• Data-intensive research
– Targeted data
– Targeted software
Copyright © 2019 Open Geospatial Consortium

4. OGC
®
Data and ICT Resources for Exploitation
• Fast access & large volume
– EO data
– In-situ data
• Distributed processing
– Hybrid Cloud infrastructure
– Commercial, Public and
Private Clouds
Copyright © 2019 Open Geospatial Consortium

5. OGC
®
Platform Tools and Capacity
• Collaborative tools
– Integrating resources (data
source, processor)
– Publishing Results
– Referencing Publications
– Sharing processing results
• Different roles
– Community scientists
– Data providers
– Resource providers
Copyright © 2019 Open Geospatial Consortium

6. OGC
®
Geohazard Exploitation Platform
• Platform based on virtualization
& federation of EO data
– Provide services & support to the
geohazards community
• On-demand & systematic processing
services
– Cloud Compute power, managing
multi-tenant resources
• Access to Copernicus Sentinels
– Hundred TBs of EO data archives (ERS & ENVISAT), and EO
missions (ALOS-2, Cosmo-Skymed and TerraSAR-X) under
CEOS WG Disaster and the GSNL agreements
Copyright © 2019 Open Geospatial Consortium

7. OGC
®
Geohazard Exploitation Platform
• Users from geoscience centres hosting
geohazard experts
– skills in EO (to process, analyze, validate data)
– generating DRM products for decision-makers
– decision-makers are not direct platform users
• Initially targeting three main scenarios:
– EO Data Exploitation
on-demand processing services
– New EO Service Development
Integrates new or improves processing chain
– New EO Product Development
Executess a global/systematic processing chain to generate a
new derived information layer
Copyright © 2019 Open Geospatial Consortium

8. OGC
®
Geohazards Services Portfolio
• 25+ on-demand services using Optical &
SAR data grouped in Thematic Apps organized
by Community Managers
• New basic services providing full resolution
and change detection imagery
for rapid online visualization
• 8 systematic services delivering
continuously updated information
• Large scale production of Sentinel-1
InSAR browse images at 100m and 50m
resolution over tectonic regions and
volcanoes
Copyright © 2019 Open Geospatial Consortium

9. OGC
®
Development Software Available
• Libraries and toolboxes:
– Orfeo Toolbox
– Gdal
– pandas/geopandas
– NumPy
• Environments
– Jupyter
– Xarray
– Dask
• Processing
– YARN, OOZIE
– Kubernetes

10. OGC
®
Some Statistics (Sep 2019)
• 12 205 processing CPU hours / month
consumed solely by users
• Products
– Generated: 8872 Published: 7796 Downloaded: 10749
• Overall 251 selected users (121 organisations)
– Carrying out projects on the platform via the early
adopters programme to use data processing services,
publish results and provide feedback to improve the platform
• 1545 registered users (96 countries)
– To search for EO data acquisitions, shared results
from processing jobs, and public information layers

11. OGC
®
API driven
• Data Discovery
– OGC OpenSearch
– Geo & Time Extensions
– EO Extensions
• Data Packaging
– OGC OWS Context
• Application Packaging
– OGC OWS Context + T13/14
• Data Processing Execution
– WPS 2.0 API

12. OGC
®
Data Discovery API
• Earth Observation Catalogues from a Portal or a Jupyter
• Accessing OGC OpenSearch API directly from Python
and explore catalogue contents as Panda/Geopanda data
structures

13. OGC
®
• Invoke on-demand data processing services from
authorized application Portals or a Jupyter Notebooks
with OGC WPS 2.0 API
• Analyse and exploit data results
On-demand Data Processing API

14. OGC
®
Systematic Data Processing API
• Invoke systematic data processing services from
authorized application Portals or a Jupyter Notebooks
• Analyse and exploit data results

15. OGC
®
Data Processing Application Integration
• Applications developed in any programming language supported
– C/C++, Java, Python, Matlab, IDL …
• Continuous Integration, automated build & packaging, deployment to
selected Production environment (Cloud providers)
Copyright © 2019 Open Geospatial Consortium

16. OGC
®
EO Application Development
Implement an EO application using your IDE
on your machine using the client APIs
Example shown:
PyCharm IDE with EO application to process
Sentinel-1 GRD data for sigma0 calculation
using SNAP

17. OGC
®
EO Application Version Control
Use a software repository like GitHub or
GitLab to host the EO application code and
files → versioning and well tracked
releases
Our Sentinel-1 sigma0 SNAP application is
is hosted by GitLab

18. OGC
®
EO Application build process
Continuous Integration/Deployment
environment to build, package and
dockerize the EO application release(s)
Our Sentinel-1 sigma0 SNAP application is
now built, packaged and dockerized

19. OGC
®
EO Application - Package
The build, package and dockerize process
produces an Application package with:
- A docker image
- Its associated OWS-Context
document
The docker image is accessible on a
docker-hub

20. OGC
®
EO Application Exploitation - Docker
To consume the EO Application docker
image:
- pull the docker image with the EO
Application from the docker-hub
- run the docker image in a container

21. OGC
®
EO Application Exploitation - Deploy
On TEPs, the Application Package is
deployed and the EO Application exposed
as a Web Processing Service (WPS 2.0
API)

22. OGC
®
EO Application exploitation on a TEP
The deployed Application Package as a
Web Processing Service (WPS 2.0 API) on
a TEP is exploited using Cloud resources
(e.g. DIAS)

23. OGC
®
EO Application Exploitation on a Notebook
Consume the EO application via WPS API
2.0, from a Jupyter Notebook or desktop
application client, to analyse and exploit
data results

24. OGC
®
Takeaway Messages
• Users benefit from fast access to EO data,
storage capacity and processing resources
offered by platform-based solutions
• A hosted processing platform is a
partnership that needs the support
of scientist/developer to offer operational
services providing value added information
to the community
• Platform e-collaboration and reproducible
knowledge promote innovation and
response capacity

25. OGC
®
Platform for Earth Sciences Communities
• to share their findings, streamline their
creation of new data products and
make these accessible, interoperable
• communities remain owners of their
created assets, and decide how to share
these on the platforms, for other users to
find and reuse without a platform lock-in
• to document their developments as
reproducible experiments
• with FAIR guiding principles to make
data findable, accessible, interoperable
and reusable

26. OGC
®
Thank You!
https://geohazards-tep.eu/

27. OGC
®
accessible by a portal

28. OGC
®