an API for standardised access to Big Earth Observation data in a landscape of a growing number of EO cloud providers 11:40 16/11/2019 With the opening of global archives of Earth Observation data streams from satellites we have arrived at a richness of operationally available observations over the whole globe, starting from the Landsat series of satellites and now the plethora of available data coming through Copernicus and its series of Sentinel satellites, that has never been available before. This created huge opportunities for research and businesses, being able to exploit the temporal domain of those observations in a powerful manner, but also poses challenges in terms of data management and processing capacities. As a consequence, a growing number of cloud services and customized solutions in various research centres have been developed, leading to processing workflows optimized for specific system architectures and back-end infrastructures. As such this is limiting portability and reproducibility of workflows across backends, both for science and business applications. (...)

1. Authors: Alexander Jacob, Jeroen Dries , Matthias Mohr , Luca Foresta , Markus Neteler, Edzer Pebesma, Prateek
Budhwar, Simone Tritini, Peter Zellner, Armin Costa, Matthias Schramm
Contact: alexander.jacob@eurac.edu, openEO@list.tuwien.ac.at
An API for Standardised Access to
Big Earth Observation Data in a
Landscape of a Growing Number of
EO Cloud Providers

2. Why do we need openEO?
1openEO – SFSCon – 2019-11-16
Eurac Research 2019, contains modified Copernicus Sentinel data [2016], processed by ESA.

3. Why do we need openEO?
1openEO – SFSCon – 2019-11-16
Eurac Research 2019, contains modified Copernicus Sentinel data [2016], processed by ESA.

4. Why do we need openEO?
1openEO – SFSCon – 2019-11-16
Eurac Research 2019, contains modified Copernicus Sentinel data [2016], processed by ESA.

5. Why do we need openEO?
1openEO – SFSCon – 2019-11-16
Eurac Research 2019, contains modified Copernicus Sentinel data [2016], processed by ESA.

6. Why do we need openEO?
1openEO – SFSCon – 2019-11-16
Eurac Research 2019, contains modified Copernicus Sentinel data [2016], processed by ESA.

7. 2
Why do we need openEO?
CEPH_FS CEPH_RBD
Hardware
File Systems
Cluster Orchestration &
Virtual Environments
Applications
Data Models &
Data Bases
1.4
PB
2 x 40 Gb/s
336 Cores
3 TB RAM
~50 Cores
~200 MB
RAM
e.g. pre -
processing
of S1 / S2
WCS
WCPS
SOS
openEO – SFSCon – 2019-11-16

8. 2
Why do we need openEO?
CEPH_FS CEPH_RBD
Hardware
File Systems
Cluster Orchestration &
Virtual Environments
Applications
Data Models &
Data Bases
1.4
PB
2 x 40 Gb/s
336 Cores
3 TB RAM
~50 Cores
~200 MB
RAM
e.g. pre -
processing
of S1 / S2
WCS
WCPS
SOS
openEO – SFSCon – 2019-11-16

9. 2
Why do we need openEO?
CEPH_FS CEPH_RBD
Hardware
File Systems
Cluster Orchestration &
Virtual Environments
Applications
Data Models &
Data Bases
1.4
PB
2 x 40 Gb/s
336 Cores
3 TB RAM
~50 Cores
~200 MB
RAM
e.g. pre -
processing
of S1 / S2
WCS
WCPS
SOS
openEO – SFSCon – 2019-11-16

10. 3
Why do we need openEO?
CEPH_FS CEPH_RBD
Hardware
File Systems
Cluster Orchestration &
Virtual Environments
Applications
Data Models &
Data Bases
1.4
PB
2 x 40 Gb/s
336 Cores
3 TB RAM
~50 Cores
~200 MB
RAM
e.g. pre -
processing
of S1 / S2
WCS
WCPS
SOS
openEO – SFSCon – 2019-11-16

11. 3
Why do we need openEO?
openEO – SFSCon – 2019-11-16

12. 3
Why do we need openEO?
openEO – SFSCon – 2019-11-16

13. 3
Why do we need openEO?
openEO – SFSCon – 2019-11-16

14. 3
Why do we need openEO?
openEO – SFSCon – 2019-11-16

15. 3
Why do we need openEO?
openEO – SFSCon – 2019-11-16

16. 3
Why do we need openEO?
openEO – SFSCon – 2019-11-16

17. 3
Why do we need openEO?
openEO – SFSCon – 2019-11-16

18. 3
Why do we need openEO?
openEO – SFSCon – 2019-11-16

19. 4
Why do we need openEO?
API
openEO – SFSCon – 2019-11-16

20. 5
Why do we need openEO?
API
openEO – SFSCon – 2019-11-16

21. 6
Getting started…
• First check for existing drivers @
https://github.com/Open-EO
https://open-eo.github.io/openeo-api/gettingstarted-backends/
RESTful implementation using
OpenAPI Specification Version 3.0.1
openEO – SFSCon – 2019-11-16

22. 6
Getting started…
• First check for existing drivers @
https://github.com/Open-EO
https://open-eo.github.io/openeo-api/gettingstarted-backends/
RESTful implementation using
OpenAPI Specification Version 3.0.1
openEO – SFSCon – 2019-11-16

23. 6
Getting started…
• First check for existing drivers @
https://github.com/Open-EO
• If an own implementation is needed:
• You can still rely on some base functionality in
the existing implementations
• Or start with openAPI code generator @
https://github.com/OpenAPITools/openapi-
generator
• Start with implementing the essential
endpoints
https://open-eo.github.io/openeo-api/gettingstarted-backends/
RESTful implementation using
OpenAPI Specification Version 3.0.1
openEO – SFSCon – 2019-11-16

24. 7
The Endpoints of openEO
root slash for capabilities and well-known
for versioning/.well-known/openeo
/
/output_formats
openEO – SFSCon – 2019-11-16
RESTful implementation using
OpenAPI Specification Version 3.0.1

25. 7
The Endpoints of openEO
openEO strives for compatibility
with STAC and OGC API as far as possible
for data discovery.
root slash for capabilities and well-known
for versioning
/collections
/.well-known/openeo
/
/output_formats
/collections/{collectionid}
openEO – SFSCon – 2019-11-16
RESTful implementation using
OpenAPI Specification Version 3.0.1

26. 7
The Endpoints of openEO
openEO strives for compatibility
with STAC and OGC API as far as possible
for data discovery.
The basis for all computation are
processes
root slash for capabilities and well-known
for versioning
/collections
/processes
/.well-known/openeo
/
/output_formats
/collections/{collectionid}
openEO – SFSCon – 2019-11-16
RESTful implementation using
OpenAPI Specification Version 3.0.1

27. 7
The Endpoints of openEO
openEO strives for compatibility
with STAC and OGC API as far as possible
for data discovery.
The basis for all computation are
processes
Processes can be chained into process
graphs
root slash for capabilities and well-known
for versioning
/collections
/processes
/.well-known/openeo
/
/process_graphs
/process_graphs/{graphID}
/output_formats
/collections/{collectionid}
openEO – SFSCon – 2019-11-16
RESTful implementation using
OpenAPI Specification Version 3.0.1

28. 7
The Endpoints of openEO
openEO strives for compatibility
with STAC and OGC API as far as possible
for data discovery.
The basis for all computation are
processes
Processes can be chained into process
graphs
Results can be processed and
downloaded synchronously
Process graphs can be submitted as batch-
jobs, queued for processing and results
can be download upon completion
root slash for capabilities and well-known
for versioning
/collections
/processes
/jobs/{jobid}
/results
/jobs/{jobid}/results
/.well-known/openeo
/
/jobs
/process_graphs
/process_graphs/{graphID}
/output_formats
/collections/{collectionid}
openEO – SFSCon – 2019-11-16
RESTful implementation using
OpenAPI Specification Version 3.0.1

29. 7
The Endpoints of openEO
openEO strives for compatibility
with STAC and OGC API as far as possible
for data discovery.
The basis for all computation are
processes
Processes can be chained into process
graphs
Results can be processed and
downloaded synchronously
Process graphs can be submitted as batch-
jobs, queued for processing and results
can be download upon completion
Handling of user authentication and
billing.
root slash for capabilities and well-known
for versioning
/collections
/processes
/jobs/{jobid}
/credentials/oidc
/results
/jobs/{jobid}/results
/.well-known/openeo
/
/credentials/basic
/jobs
/process_graphs
/process_graphs/{graphID}
/output_formats
/collections/{collectionid}
openEO – SFSCon – 2019-11-16
RESTful implementation using
OpenAPI Specification Version 3.0.1

30. 7
The Endpoints of openEO
openEO strives for compatibility
with STAC and OGC API as far as possible
for data discovery.
The basis for all computation are
processes
Processes can be chained into process
graphs
Results can be processed and
downloaded synchronously
Process graphs can be submitted as batch-
jobs, queued for processing and results
can be download upon completion
User can upload own files
Handling of user authentication and
billing.
root slash for capabilities and well-known
for versioning
/collections
/processes
/jobs/{jobid}
/files/{userid}
/credentials/oidc
/results
/jobs/{jobid}/results
/.well-known/openeo
/
/credentials/basic
/jobs
/process_graphs
/process_graphs/{graphID}
/output_formats
/collections/{collectionid}
openEO – SFSCon – 2019-11-16
RESTful implementation using
OpenAPI Specification Version 3.0.1

31. 7
The Endpoints of openEO
openEO strives for compatibility
with STAC and OGC API as far as possible
for data discovery.
The basis for all computation are
processes
Processes can be chained into process
graphs
Results can be processed and
downloaded synchronously
Process graphs can be submitted as batch-
jobs, queued for processing and results
can be download upon completion
User can upload own files
Handling of user authentication and
billing.
root slash for capabilities and well-known
for versioning
/collections
/processes
/jobs/{jobid}
/files/{userid}
/credentials/oidc
/results
/jobs/{jobid}/results
/service_types/{serviceid}
/.well-known/openeo
/
Consume results as secondary web
services (e.g. WMS, XYZ, WCS)
/credentials/basic
/jobs
/process_graphs
/process_graphs/{graphID}
/output_formats
/collections/{collectionid}
openEO – SFSCon – 2019-11-16
RESTful implementation using
OpenAPI Specification Version 3.0.1

32. 7
The Endpoints of openEO
openEO strives for compatibility
with STAC and OGC API as far as possible
for data discovery.
The basis for all computation are
processes
Processes can be chained into process
graphs
Results can be processed and
downloaded synchronously
Process graphs can be submitted as batch-
jobs, queued for processing and results
can be download upon completion
User can create and integrate user
defined functions into process graphs
User can upload own files
Handling of user authentication and
billing.
root slash for capabilities and well-known
for versioning
/collections
/processes
/jobs/{jobid}
/files/{userid}
/credentials/oidc
/results
/jobs/{jobid}/results
/service_types/{serviceid}
/.well-known/openeo
/
Consume results as secondary web
services (e.g. WMS, XYZ, WCS)
/credentials/basic
/jobs
/process_graphs
/process_graphs/{graphID}
/output_formats
/udf_runtimes
/collections/{collectionid}
openEO – SFSCon – 2019-11-16
RESTful implementation using
OpenAPI Specification Version 3.0.1

33. 8
The Architecture of openEO
everything is
JSON
openEO – SFSCon – 2019-11-16

34. 9
Useful resources http://docs.openeo.org
https://open-eo.github.io/openeo-api/apireference/
openEO – SFSCon – 2019-11-16

35. 9
Useful resources http://docs.openeo.org
https://open-eo.github.io/openeo-api/apireference/ http://processes.openeo.org
openEO – SFSCon – 2019-11-16

36. 10
The Implementations of openEO
Back-end Implementations Client Implementations
openEO – SFSCon – 2019-11-16

37. 11
Implementations: http://hub.openeo.org
openEO – SFSCon – 2019-11-16

38. 11
Backends: http://hub.openeo.org
openEO – SFSCon – 2019-11-16

39. 12
WCPS backend (EURAC): https://openeo.eurac.edu
CEPH_FS CEPH_RBD
Hardware
File Systems
Cluster Orchestration &
Virtual Environments
Applications
Data Models &
Data Bases
1.4
PB
2 x 40 Gb/s
336 Cores
3 TB RAM
~50 Cores
~200 MB
RAM
e.g. pre -
processing
of S1 / S2
WCS
WCPS
SOS
https://github.com/Open-EO/openeo-wcps-driver
openEO – SFSCon – 2019-11-16

40. 12
WCPS backend (EURAC) : https://openeo.eurac.edu
CEPH_FS CEPH_RBD
Hardware
File Systems
Cluster Orchestration &
Virtual Environments
Applications
Data Models &
Data Bases
1.4
PB
2 x 40 Gb/s
336 Cores
3 TB RAM
~50 Cores
~200 MB
RAM
e.g. pre -
processing
of S1 / S2
WCS
WCPS
SOS
https://github.com/Open-EO/openeo-wcps-driver
openEO – SFSCon – 2019-11-16

41. 13
WCPS backend (EURAC)
https://github.com/Open-EO/openeo-wcps-driver
Based on swagger-jersey2-jaxrs for rest
API implementation.
Sqlite for openEO related DB
=> Batch job management, storing of
process graphs
GDAL for image operations and
coordinate transformations
JJWT for openID connect implementation
=> Linked to Microsoft azure for
authentication
Packaged as web archive using maven
=> Deployable on any java capable web
container (e.g. tomcat or jetty)
openEO – SFSCon – 2019-11-16

42. 13
WCPS backend (EURAC)
https://github.com/Open-EO/openeo-wcps-driver
Based on swagger-jersey2-jaxrs for rest
API implementation.
Sqlite for openEO related DB
=> Batch job management, storing of
process graphs
GDAL for image operations and
coordinate transformations
JJWT for openID connect implementation
=> Linked to Microsoft azure for
authentication
Packaged as web archive using maven
=> Deployable on any java capable web
container (e.g. tomcat or jetty)
Configuration:
• Properties File
• WCPS endpoint
• openEO endpoint
• Authentication endpoint (for oidc)
• DB location
• TMP location
• Session timings (auth expiry, tmp duration, etc.)
openEO – SFSCon – 2019-11-16

43. 13
WCPS backend (EURAC)
https://github.com/Open-EO/openeo-wcps-driver
Based on swagger-jersey2-jaxrs for rest
API implementation.
Sqlite for openEO related DB
=> Batch job management, storing of
process graphs
GDAL for image operations and
coordinate transformations
JJWT for openID connect implementation
=> Linked to Microsoft azure for
authentication
Packaged as web archive using maven
=> Deployable on any java capable web
container (e.g. tomcat or jetty)
Configuration:
• Properties File
• WCPS endpoint
• openEO endpoint
• Authentication endpoint (for oidc)
• DB location
• TMP location
• Session timings (auth expiry, tmp duration, etc.)
• Setup of Host Environment
• Centos 7 or Ubuntu 18.04
• Install Tomcat 7 or later
• Configure for https
• Install sqlite (v3) & GDAL (v2.4)
• Deploy openEO.war
• Setup of proxy server for public access
openEO – SFSCon – 2019-11-16

44. 14
Client Examples (Python)
openEO – SFSCon – 2019-11-16

45. 15
Client Examples (Python)
openEO – SFSCon – 2019-11-16

46. 15
Client Examples (Python)
openEO – SFSCon – 2019-11-16

47. 16
Client Examples (JavaScript): https://editor.openeo.org
openEO – SFSCon – 2019-11-16

48. 17
Conclusions
• A RESTful API has been defined
• Interface human and machine readable (JSON)
• Process catalogue
• Extendable by user defined functions
• Standardized process graph for abstract description of EO computational workflows
• A number of reference implementations are currently in development
• Based on different programming languages
• Python
• Java
• R
• Java-script
• Based on different existing hard and software infrastructure
• Can be used as starting point for own implementation
• Together with extensive documentation
• And guides
• All of this is open source
openEO – SFSCon – 2019-11-16

49. Thank you for your attention!
H2020 openEO
EO-2-2017: EO Big Data Shift
Grant agreement No 776242
Contact:
http://openeo.org/
openEO@list.tuwien.ac.at
https://github.com/Open-EO
@open_EO
https://www.youtube.com/channel/UCMJQil8
j9sHBQkcSlSaEsvQ
https://www.researchgate.net/project/openEO
https://openeo-chat.eodc.eu/channel/public
https://zenodo.org/communities/openeo
alexander.jacob@eurac.edu
openEO – SFSCon – 2019-11-16