Fast-data management toolbox using Open Earth building blocks

1. FAST Modelling Tool Box
Deltares Software Days
Delft, November 26-28 2015
7FP – SPACE
no. 607131

2. 7FP – SPACE
no. 607131
DeltaresSoftwareDays,FASTWorkshops28October2015
Topics
1. OpenEarth
2. The Basic Web Tool?
3. Tool description
4. EO Data input:
1. Elevation
2. Vegetation or not
3. …

3. 7FP – SPACE
no. 607131
DeltaresSoftwareDays,FASTWorkshops28October2015
OpenEarth is a community
538 Members
• Data
• Models
• Tools
• Open Source
• Transparent
• Reuse
• Publicly available

4. 7FP – SPACE
no. 607131
DeltaresSoftwareDays,FASTWorkshops28October2015
Recognized: Dutch data prize in 2012
30 March 2015

5. 7FP – SPACE
no. 607131
DeltaresSoftwareDays,FASTWorkshops28October2015
Goal
Proper (re)use of data
Data/information to support:
• Decision making
• Efficient management/maintenance
• Research

6. 7FP – SPACE
no. 607131
DeltaresSoftwareDays,FASTWorkshops28October2015
Data stages of OpenEarth
tailored
data
> WCS
> WFS
> SOS
> SOAP
> netCDF-CF-
OPeNDAP
> ISO SQL–
PostGIS> SVN
> GIT
> http
> ftp
> KML
> WMS
> WFS
> CSWgraphics
of data
standard
data
raw
data
catalogue
of data
work done
on server
work done
on client
what
where
when
who
why
how
…
data URLs
smart phone & tablet usersscientists professionals

7. 7FP – SPACE
no. 607131
DeltaresSoftwareDays,FASTWorkshops28October2015
Open Earth components
standard
data
raw
data
catalogue
of data
ISO SQL:
postgresql
GIT
SubVersion
VECTOR DATA
OGC CSW: CATALOGUE: geonetworks, ckan, …
OGC NETCDF:
OPENDAP
thredds
OGC SFS: postgis
GRID DATA
LEGEND:
ROADMAP
DEVELOPMENT
OPERATIONAL
Lineage / version
control of raw data,
processing and ETL
tailored
data
OGC CF CONVENTIONS
OGC WXS: geoserver
OGC WXS: OTS WEB VIEWER (THIN CLIENT)
graphics
of data
OGC WXS: ncwms, …
OGC WPS: web processing service

8. 7FP – SPACE
no. 607131
DeltaresSoftwareDays,FASTWorkshops28October2015
Open Earth DataLab
(DDS)
(DAS)
(DMS)
Field
observations
Monitoring
Network
DAS – Data Acquisition Sub-
system
DMS - Data Management
Sub-system
DDS - The Data
Dissemination Sub-system
Cat
alog
ue
serv
ice
(DAS)
EO/RS
EO/RS
OGC services
(WMS,WFS,WCS,WPS)
Web page
Architecture

9. 7FP – SPACE
no. 607131
DeltaresSoftwareDays,FASTWorkshops28October2015
Geo Database (DMS)
• Vector data (points, lines, polygons)
• Multiple thematic data
• Extents of studyareas
• Footprints of downloaded images
• Sample transect
• Diva segments (in phase 2)
• …
• OGC services to:
• Visualise the data (in viewers)
• Extract information and present that in plots
• Make data available for third party
software (GIS, scripting)

10. 7FP – SPACE
no. 607131
DeltaresSoftwareDays,FASTWorkshops28October2015
GeoTIFF/NetCDF/THREDDS (DMS)
• Rasters
• Model outputs (time series of rasters)
• Thematic rasters (DEM, Landuse, EO data)
• Exchange via OGC services:
• WMS (pictures via ncWMS)
• WCS (raster exchange)
• OPeNDAP (direct access to the data)

11. 7FP – SPACE
no. 607131
DeltaresSoftwareDays,FASTWorkshops28October2015
FAST and standards
OGC services:
• WMS – images
• WFS – features
• WCS – rasters
• CSW – metadata
• WPS – processes

12. 7FP – SPACE
no. 607131
DeltaresSoftwareDays,FASTWorkshops28October2015
FAST and standards – WPS
?
Why
What
How
Via
WPS is a communication
standard which allows users to access tools or
models which can be applied on geographical
data.
- Open Standard
- Standardized Input/Output
- Multi Platform use
- OS independent
- multiple use
- Models
- GIS procedures (WPS – GRASS GIS modules)
- Non geospatial procedures
- Deltares (http://wps.openearth.nl)
- Project portals
- Python
- R (via Python)
- Fortron (via Python)
- Java

13. 7FP – SPACE
no. 607131
DeltaresSoftwareDays,FASTWorkshops28October2015
FAST Data Management
• All OpenEarth components listed before
• Portal
• OGC services
• Modelling as a service (online xbeach modelling)
+
• Automated download from scihub
• Datalab for processing:
• EO data using
S1Toobox
• Field observations

14. 7FP – SPACE
no. 607131
DeltaresSoftwareDays,FASTWorkshops28October2015
FAST product scheme

15. 7FP – SPACE
no. 607131
DeltaresSoftwareDays,FASTWorkshops28October2015
How? – data
Global data:
• SRTM+GEBCO (0.0042 degrees resolution (ca. 300 meter))
• Vegetation absence = 1 for elevations > 0.88 m MSL
• Wave height 4.9 + MSL

16. 7FP – SPACE
no. 607131
DeltaresSoftwareDays,FASTWorkshops28October2015
How? – Web Tool
Basics
• webviewer
• basic GUI
• based on OpenLayers
• HTML5 and js
• WFS, WMS and WCS
Specials
• automatic harvesting
geoserver contents
• WPS support

17. 7FP – SPACE
no. 607131
DeltaresSoftwareDays,FASTWorkshops28October2015
How? – WPS
yes no
WPS
Xbeach
Wave
attenuation
model
Global DEM
Bathymetry
Local detailed
DEM/Bathymetry
User draws line
Check for Local detailed information
WPS request
request=Execute&identifier=lin
esect&datainputs=[wktline=LINE
STRING(-6.29574 36.49846, -
6.22576 36.52938);crs=4326]

18. 7FP – SPACE
no. 607131
DeltaresSoftwareDays,FASTWorkshops28October2015
FAST product scheme current status

19. 7FP – SPACE
no. 607131
DeltaresSoftwareDays,FASTWorkshops28October2015
To be implemented
First things to do:
• Add vegetation from EO images
• Add global wave patterns
• Refine terrain models
• …?
Xbeach modelling
• performance
various
• Local data ingestion
• User consultation

20. 7FP – SPACE
no. 607131
DeltaresSoftwareDays,FASTWorkshops28October2015
FAST Demo