1.
THE CLOUD FOR EARTH
OBSERVATION DATA
PROCESSING

2.
CREODIAS © | Powered by CloudFerro
2
CREODIAS
HIGHLIGHTS
✓ The leading cloud platform for EO Data processing
✓ Operating under ESA/EC contract
✓ Offering cloud services and tools for EO data
storage, processing and dissemination
✓ Designed for the development and hosting of third
party EO applications and data processing chains.

3.
CREODIAS © | Powered by CloudFerro
3
CREODIAS © | Powered by CloudFerro
CREODIAS
DESIGN
ASSUMPTIONS
3

4.
CREODIAS © | Powered by CloudFerro
4
CREODIAS EXPLAINED
Cloud Services (IaaS)
Compute
EO Data processing (FaaS)
Block Storage
Object Storage
Virtual Networking
Backup
CREODIAS Portal Cloud DashboardEO Finder EO Browser
Third party
Applications
Data Acquisiton/Ingestion
Data Search
OCS/OpenSearch/SPARQL
Data Access
NFS/S3/HTTP
OGC W MS/WMTS/WCS
(via Sentinel Hub)
External EO data sources
(currently over 20)
Data Search
Data Access
User
generated
data
EO Data Repository &
Catalog
Public
Collections
Cache Private
Collections
EO Data & Metadata
4

5.
CREODIAS © | Powered by CloudFerro
5
Datasets Products Instrument Locally Held
Sentinel-1A & Sentinel-1B
GRD
SAR C-BAND
Full archive
OCN
RAW Last 6 months
SLC
- Europe: full archive
- Last 6 months / orderable
Sentinel-2A & Sentinel-2B
L1C
MSI
Full archive
L2A
- Orderable */**
- Cached ***
Sentinel-3A & Sentinel-3B
L1 SLSTR SLSTR
Full archive
L1 OLCI OLCI
L1 SRAL SRAL
L2 SLSTR (LST/WST) SLSTR
L2 OLCI OLCI
L2 SRAL SRAL
Sentinel-5P
L1B
TROPOMI Full archive
L2 ****
Landsat-5 L1G, L1T, L1GT TM Coverage of Europe, (1984-2011)
Landsat-7 L1G, L1T, L1GT ETM Coverage of Europe, (1999-2017)
Landsat-8 L1T, L1GT OLI, OLI TIRS Coverage of Europe, (2013- present)
Envisat L1 MERIS Global, (2002-2012)
Overview of original, uncompressed Earth Observation satellite data on CREODIAS. All data are indexed and available via CREODIAS Finder and API.
Can be ordered from ESA via CREODIAS Finder
If L2A does not exist in ESA archive, it can be processed by CloudFerro with Sen2Cor
Kept in rolling cache of 1 PB (mostly European coverage). Oldest data will be removed when cache is full
Sentinel-5P L2 products currently stored: Aerosol, Cloud, CO, HCHO, NO2, NP, O3, SO2
*
**
***
****
CREODIAS DATA OFFER

6.
CREODIAS © | Powered by CloudFerro
6
Datasets Products Data access
CAMS (Atmosphere) All collections Accessible over S3 or NFS protocol
CEMS (Emergency) All collections Accessible over S3 or NFS protocol
CLMS (Land) All collections Accessible over S3 or NFS protocol
CMEMS (Marine) All collections Accessible over S3 or NFS protocol
Datasets Products Instrument Locally Held Data access
DEM
Mapzen SRTM, others Aggregation of several DEM sources Accessible over S3 or NFS protocol
SRTM SRTM
Global (56S, 60N, 180W, 180E)
February 2000
Accessible over S3 or NFS protocol
Jason-3 Altimeter Altimeter Since mission began Accessible over S3 or NFS protocol
COPERNICUS SERVICES DATA ON CREODIAS
OTHER DATA COLLECTIONS ON CREODIAS
CREODIAS © | Powered by CloudFerro

7.
CREODIAS © | Powered by CloudFerro
7
EO DATA
ACQUISITION
✓ Over 20 external EO data sources
✓ HTTP(S), (S)FTP, S3 and physical media transfer
✓ Optimized acquisition algorithm (parallelization,
selection of fastest source)
✓ Fault tolerance and quality control
✓ Full data or metadata-only
✓ On demand acquisition (ordering)
✓ Subscription API for new products
✓ Customer-provided collections (restricted use)
Data Acquisiton/Ingestion
Data Search
OCS/OpenSearch/SPARQL
Data Access
NFS/S3/HTTP
OGC W MS/WMTS/WCS
(via Sentinel Hub)
External EO data sources
(currently over 20)
EO Data Repository &
Catalog
EO Data & Metadata
7
CREODIAS © | Powered by CloudFerro

8.
CREODIAS © | Powered by CloudFerro
8
REPOSITORY
✓ High capacity object storage for EO Data
(currently over 15PB)
✓ Data stored in its original, uncompressed form –
direct access to individual files
✓ Accessible over NFS and S3 protocols
✓ Direct file visibility in user VMs – no need to download data
✓ Dynamic generation of OGC WMS/WMTS/WCS tiles
(integrated Sentinel Hub)
✓ Storage of public and private data
✓ 1 PB cache area for data ordered or generated by users
✓ Possible dissemination to remote users
Data Acquisiton/Ingestion
Data Search
OCS/OpenSearch/SPARQL
Data Access
NFS/S3/HTTP
OGC W MS/WMTS/WCS
(via Sentinel Hub)
External EO data sources
(currently over 20)
EO Data Repository &
Catalog
Public
Collections
Cache Private
Collections
EO Data & Metadata

9.
CREODIAS © | Powered by CloudFerro
9
CATALOG
✓ Fast search based on collection, AOI, time, cloud-cover, etc.
✓ Access to external search engines and indices
✓ Standard OpenSearch interface
✓ Semantic search over multiple data sources using SPARQL
✓ Customer-defined collections (restricted use)
✓ INSPIRE-compliant
Data Acquisiton/Ingestion
Data Search
OCS/OpenSearch/SPARQL
Data Access
NFS/S3/HTTP
OGC W MS/WMTS/WCS
(via Sentinel Hub)
External EO data sources
(currently over 20)
EO Data Repository &
Catalog
EO Data & Metadata

10.
CREODIAS
CLOUD SERVICES (IaaS)Cloud Services (IaaS)
Compute
EO Data processing (FaaS)
Block Storage
Object Storage
Virtual Networking
Backup
CREODIAS © | Powered by CloudFerro
 PROCESSING :
✓ Virtual Machines
✓ Dedicated Server VM-s
✓ VM-s with GPU
✓ Containers
✓ FaaS
 STORAGE :
✓ Block & Object
✓ HDD/SSD/NVMe
✓ Fully redundant
✓ Snapshots
✓ Backups
 OTHER SERVICES :
✓ Virtual Networks & Routers
✓ Firewalls
✓ Floating IP
✓ DNS
✓ VPN
✓ Load Balancer

11.
CREODIAS © | Powered by CloudFerro
11
CREODIAS CLOUD SERVICES (IaaS)
FULLY
INTEGRATED WITH
EO REPOSITORY
AND CATALOGUE
BASED ON
OPENSTACK
ACCESSIBLE VIA
API AND GUI
SEPARATE
ADMINISTRATIVE
DOMAINS FOR
DIFFERENT
CUSTOMERS
11

12.
CREODIAS © | Powered by CloudFerro
12
USER TOOLS
CREODIAS Portal EO Browser EO Finder
Cloud Dashboard
Third party
Applications

13.
CREODIAS © | Powered by CloudFerro
13
EO FINDER
✓ Allows finding data products stored in the
repository by timeframe, area of interest,
cloud coverage and other selection criteria
✓ Data can be downloaded or selected for
further processing
✓ Orders can be placed for fetching remote
data into the repository cache
https://finder.creodias.eu/www/

14.
CREODIAS © | Powered by CloudFerro
14
CREODIAS © | Powered by CloudFerro
EO BROWSER
✓ Allows visualization, zooming and basic processing of
selected data collections
✓ The SPARQL interface provides extended search
capabilities (semantic search) linking metadata of products
stored in the repository with external data sources
✓ Tile browsers based on CREODIAS WMTS data can be
embedded in third party applications
14

15.
CREODIAS © | Powered by CloudFerro
15
03 Full flexibility - use any OS,
frameworks, tools or
processing chains
02 Industry standard APIs: OGC,
OpenStack, S3, SPARQL
01 All the data at your fingertips –
no more downloads
04 Smooth integration of
external data
05 Fast deployment and easy
scaling
CREODIAS
ENVIRONMENT -
OPEN AND
FLEXIBLE

16.
CREODIAS © | Powered by CloudFerro
16
EO APPLICATION DEVELOPMENT –
GENERAL WORKFLOW
Deploy To
Production
Verify
assumptions
Define use case Build Prototype Scale up!
CREODIAS © | Powered by CloudFerro
16

17.
CREODIAS © | Powered by CloudFerro
17
VERIFY ASSUMPTIONS
DIAS Browser
& Semantic Search
DIAS
DATA
Data HUB
& WMS
API for Products
Metadata
FAQ & Support
Serverless
FaaS Ordering API

18.
CREODIAS © | Powered by CloudFerro
18
BUILD THE PROTOTYPE
Data
Services
Metadata APIs
Virtual
Resources
Data access
(NFS/S3/API)

19.
CREODIAS © | Powered by CloudFerro
19
DEPLOY TO PRODUCTION
HA&
Load Balancer
CREODIAS © | Powered by CloudFerro

20.
CREODIAS © | Powered by CloudFerro
20
EXAMPLES OF LIVE PROJECTS
ON CREODIAS
20
CREODIAS © | Powered by CloudFerro

21.
CREODIAS © | Powered by CloudFerro
21
PROCESSING SAMPLE RUNNING ON
THE CREODIAS PLATFORM
LAND COVER MAP
SZGLC Extension – Europe
Classification of Sentinel-2 data from year 2017
LEGEND
5-2 tile
Clouds
Artificial surfaces and constructions
Cultivated areas
Vineyards
Broadleaf tree cover
Coniferous tree cover
Herbaceous vegetation
Moors and Healthland
Sclerophyllous vegetation
Marshes
Peatbogs
Natural material surfaces
Permanent snow covered surfaces
Water bodies

22.
CREODIAS © | Powered by CloudFerro
22
CloudFerro Sp. z o.o.
(CF) – basic information
✓ Polish technological company established in 2015 by a
group of experienced Polish IT managers
✓ Provider of computing cloud and storage space focused
on Big Data storage and processing
✓ Infrastructure of servers and systems based in Tier3
independent data centers in Warsaw, Poland
✓ Certified for compliance with ISO 9001 and ISO 27001
standards

23.
CREODIAS © | Powered by CloudFerro
23
CloudFerro Sp. z o.o.
(CF) – basic information
✓ Experienced 40 people team, including IT staff available on 24/7 basis:
▪ 30 full time engineers and programmers with specialization in
cloud computing, design, implementation and management of
advanced IT systems located in Warsaw and 1 full time engineer
delegated to ECMWF in Reading, Great Britain
▪ Team of specialists with knowledge of telecommunication
market and excellent business relations with telecommunication
carriers
✓ Services insured up to 3 million EUR
✓ Main clients and projects:
▪ European Space Agency (ESA) – CREODIAS.eu
▪ European Center for Medium Range Weather Forecast (ECMWF) –
Hybrid Cloud for Climate Data Store
▪ EUMETSAT + ECMWF + Mercator Ocean - WEkEO

24.
CREODIAS © | Powered by CloudFerro
24
LINKS TO
CREODIAS
TOOLS
Portal : https://portal.creodias.eu
EO Finder : https://finder.creodias.eu
EO Browser : https://browser.creodias.eu
EO Catalogue : https://discovery.creodias.eu
Jupyter Hub : https://jupyter.creodias.eu/
EO Cloud : https://cf2.cloudferro.com

25.
CREODIAS © | Powered by CloudFerro
25
FOR MORE, PLEASE VISIT :
www.creodias.eu
OR LINK UP WITH US ON:
https://twitter.com/CREODIAS
https://twitter.com/CloudFerro
https://www.linkedin.com/company/clfr/
https://www.facebook.com/cloudferro /
THANK YOU FOR
YOUR ATTENTION
Maciej Krzyżanowski
Email: mkrzyzanowski@cloudferro.com
Phone: +48 602 415 461