Serving earth observation data with GeoServer: addressing real world requirements - FOSS4G EU 2017

Serving EO Data
with GeoServer
Addressing real-world EO requirements
With GeoServer
Ing. Simone Giannecchini
Ing. Daniele Romagnoli
Ing. Andrea Aime
GeoSolutions

GeoSolutions
 Founded in Italy in late 2006
 Expertise
• Image Processing, GeoSpatial Data Fusion
• Java, Java Enterprise, C++, Python
• JPEG2000, JPIP, Advanced 2D visualization
 Supporting/Developing FOSS4G projects
 GeoServer, MapStore
 GeoNetwork, GeoNode, Ckan
 Clients
 Public Agencies
 Private Companies
 http://www.geo-solutions.it
FOSS4G EU 2017, Marne la Vallée
18th – 22nd July 2017

Reference scenario
18th – 22nd July 2017

ImageMosaic – the basics
18th – 22nd July 2017

Terminology
 Granule/Tile
 The individual raster element composing the
mosaic
 (Granule) Index
 The collection of metadata records describing
the location, spatial coverage and other
attributes of each single granule
 Dimensions/Domains
 The dimensions besides the spatial ones used
to distinguish individual granules
18th – 22nd July 2017

Complete freedom
 Granules can:
 Overlap as they please, can have different
resolutions
 Be in different file formats (faster if the format is
uniform)
 Have different color models, RGB, gray, paletted
(needs JAI-EXT enabled)
 Be in different coordinate reference systems
(needs GeoServer 2.12 to work best)
18th – 22nd July 2017

The mosaic index
 Always present
 A table of granules, their footprint, location,
and extra attributes
 Used to look up which granules to mosaick for
a given request
 Currently supported/tested sources
 PostGIS (JNDI)
 Oracle (JNDI) it’s been a nightmare because of
naming!
 Shapefile (no attr indexing, no timestamp)
 H2 (local embedded db)
 Can be customized to support custom
granule indexes (e.g. legacy catalog)
18th – 22nd July 2017

The Dimensions
 Maps to alphanumeric attributes in the index
 TIME and ELEVATION receive special
treatment for WMS and WCS
 Custom/Additional dimensions
 Everything besides TIME & ELEVATION
 They can be dynamically discovered
from GetCapabilities/DescribeCoverage
18th – 22nd July 2017

Filling the index
 Available options
 Have GeoServer do it for you from the UI
(one shot setup)
 Use the REST API to add granules and
make it populate the index
 Directly write the index (common for
moving window of data)
 Use the OpenSearch for EO extension
(later in the presentation)
18th – 22nd July 2017

Grabbing attributes from file names
 elevationregex.properties file
regex=(?<=_)(d{3})(?=_)
 elevationregex.properties file
regex=[0-9]{8}T[0-9]{9}Z(?!.*[0-9]{8}T[0-9]{9}Z.*)
NCOM_wattemp_020_20081031T0000000_12.tiff
 While building the index GeoServer can
fetch attributes from file name
 Or from within the file if needed
18th – 22nd July 2017

REST configuration of mosaic contents
 Granule Index CRUD Operations via REST
 Create mosaic
 Read its index schema
 Read its index contents
 Update/Delete/Insert granules
18th – 22nd July 2017
GS
Pixels Index
Client

Documentation links
 GeoServer reference docs:
 http://docs.geoserver.org/stable/en/user/data/raster/image
mosaic/index.html
 http://docs.geoserver.org/stable/en/user/data/raster/image
mosaic/tutorial.html
 http://docs.geoserver.org/api/#/1.0.0/structuredcoverages.y
aml
 GeoSolutions training:
 http://geoserver.geo-
solutions.it/edu/en/multidim/imagemosaic/index.html
solutions.it/edu/en/multidim/accessing_multidim/index.html
solutions.it/edu/en/multidim/rest/index.html
18th – 22nd July 2017

ImageMosaic – the fun stuff
18th – 22nd July 2017

Filtering and sorting
 The mosaic index can be filtered upon with CQL
 Can be sorted on using WFS like syntax
 More possibilities than just Time and Elevation
SENSOR = SAR AND SATELLITE=XYZ
18th – 22nd July 2017
SORTBY = TIME D

Masking Support
 Mask allow finding valid data
 Alternative to NODATA for lossy compressions
 Vector and raster masks supported
 Raster masks can be embedded (GDAL style)
 Vector ones can be WKB or shapefiles
 API to plug your own mask provider
 Available for image mosaic and GDAL based data
sources
18th – 22nd July 2017

Coverage Views
 NetCDF/Grib/Mosaic: one reader, many coverages
 Sometimes the coverages are strictly related
 E.g., U and V components of wind/current vectors
 Merge them back into a single coverage as bands
 Allows rendering engine to work with them both as one
18th – 22nd July 2017

Pluggability
 Lots of extension points and pluggable bits
 PropertyCollectors: extract index attributes from
granules
 CoverageNameCollector: when you have a hash of files
from multiple coverages
 GranuleHandler: pre-process granules before getting
them in the mosaic
 GranuleAcceptor: decide if the granule is actually part
of the mosaic
 SubMosaicProducer: custom way to mosaic granules
 GranuleCatalog: how you store the index
 See more examples at the end of presentation
18th – 22nd July 2017

NetCDF
18th – 22nd July 2017

NetCDF format support
 NetCDF support
 Support COARDS/CF* conventions loosely
 Expose NetCDF internal data as a set of 2D slices
 Fast 2D (time, elevation) slice extraction
 Supports NCML
18th – 22nd July 2017

One or more variables
 Polyphemus Sample Dataset
 1 File  Multiple Coverages!
18th – 22nd July 2017

NetCDF GridMapping to EPSG codes
18th – 22nd July 2017

NetCDF WCS output
 WCS can output NetCDF for any multidimensional source
 Several customizations to control output and preserve info
if the source is a NetCDF itself
 Source has to be multidimensional (current restriction)
18th – 22nd July 2017

Documentation links
 GeoServer documentation
 http://docs.geoserver.org/latest/en/user/extensions/n
etcdf/netcdf.html
 http://docs.geoserver.org/latest/en/user/extensio
ns/netcdf-out/index.html
solutions.it/edu/en/multidim/netcdf/index.html
solutions.it/edu/en/multidim/netcdf/index.html
18th – 22nd July 2017

ImageMosaic – NetCDF integration
18th – 22nd July 2017

NetCDF/Mosaic integration
 ImageMosaic NetCDF integration
 Allow the ImageMosaic to handle multiple NetCDF files
 Expose NetCDF internal structure (times, elevations)
 Make ImageMosaic handle slices of the NetCDF file as granules
 5-dim mosaic (space/time/runtime other dimensions)
Mosaic
NetCDF 1 NetCDF 2 NetCDF … N
…
18th – 22nd July 2017

Multiple coverages per mosaic
18th – 22nd July 2017

GRIB/GRIB2 Format Support
 Based on same UCAR NetCDF Java libs used
by NetCDF Format
 Same capabilities of NetCDF Format
 Same indexing logic
 Same ImageMosaic support
18th – 22nd July 2017

Documentation links
solutions.it/edu/en/multidim/mosaic_config/netc
df_mosaic.html
18th – 22nd July 2017

The services - visualize
18th – 22nd July 2017

WMS-T support
TIME, ELEVATION & Custom dimensions
http://localhost:8080/geoserver/geosolutions/wms?...
&time=2013-03-1T00:00:00.000Z
&elevation=35.0
&DIM_FILEDATE=2013-03-01T00:00:00.000Z
&DIM_UPDATED=2013-04-08T08:18:41.597Z
18th – 22nd July 2017

Enabling WMS-T support
18th – 22nd July 2017

WMS – custom filtering and sorting
18th – 22nd July 2017
 &CQL_FILTER=platform=SENTINEL2
 &sortBy=time D &sortBy=time D

WMS-EO
Extending LayerGroup
concept
Support same style on
both raster and vector
data
Support custom
dimensions
Alter map on the fly to
support band
combination
Wizard to configure EO
layer groups
 Product layer tree
 Mix of vector and raster info
 Raw data, flags (clouds,
snow), browse
18th – 22nd July 2017

WMS-EO
 Helper UI to build the tree
18th – 22nd July 2017

Rendering transformations
 SLD Based transformations
 On-the-fly contouring
 On-the-fly poligonalization
 Current arrows
 Wind Barbs
 Pluggable: add your own
18th – 22nd July 2017

Rendering transformations
windbarbs
currents
contouring
18th – 22nd July 2017

The services - download
18th – 22nd July 2017

WCS 2.0
 Basics
 Core service
 KVP binding
 XML binding
 Common extensions
 CRS
 Scaling
 Interpolation
 Range subsetting
 GeoTiff
 GML
 NetCDF
 Add the output format extensions
 GeoTIFF
 GML Grid
 NetCDF
18th – 22nd July 2017

WCS 2.0 processing chain
Crop
RangeSubset
Scale & Interpolate
Reproject & Interpolate
Encode
Core
Extension
Extension
Extension
Extension
18th – 22nd July 2017

WCS 2.0 DescribeCoverage
4D BBOX
TIME details
as extension
ELEVATION
details as
extension
CUSTOM
18th – 22nd July 2017

WCS – custom filtering and sorting
18th – 22nd July 2017
 &CQL_FILTER=platform=SENTINEL2
 &sortBy=time D &sortBy=time D

Download ND NetCDF
http://localhost:8080/geoserver/wcs?request=GetCoverage
&service=WCS&version=2.0.1&coverageId=geosolutions__NO2
&Format=application/x-netcdf
&subset=Long(5,20)
&subset=Lat(40,50)
&subset=elevation(300,1250)
&subset=time("2013-03-01T10:00:00.000Z","2013-03-
01T22:00:00.000Z")
• Can also handle custom
dimensions
• Described in DescribeCoverage
in a vendor metadata section
4D
BBOX
18th – 22nd July 2017

WCS EO
 Add support for WCS EO metadata in readers
 Associate each file with EO metadata
 Include such information in
DescribeCoverage/DescribeEODataset
 EODataset: multidimensional mosaic or NetCDF file
 Describe the inner structure (granules listing) in
DescribeEODataSet
18th – 22nd July 2017

OpenSearch for EO
18th – 22nd July 2017

Putting it all together (and finding it)
 Use case
 Lots of satellite imagery
 Different satellites
 Different sensor
 Deep time and space
distribution
 What to do about it?
 “OpenSearch for EO“ to
the rescue
18th – 22nd July 2017
OpenSearch
OGC OpenSearch Geo and
Time
OGC OpenSearch for EO

OpenSearch for EO
 Geo and Time: Extension to OpenSearch
adding search keywords for time and space
 Earth Observation: extension to the above
adding search keywords for and Earth
Observation properties
 E.g.
 Cloud cover
 Snow cover
 Off nadir
 …
 Simpler alternative to CSW with ebRIM
18th – 22nd July 2017

General information schema
18th – 22nd July 2017
Collection
Product
Granule
Search
Search
Mosaic/OGC

Two steps search
 First find collection of
similar products (by
satellite/sensor), e.g.
 Optical or SAR?
 Supported resolutions
 Availability of data in
particular timeframe
 Then search products in it,
by
 Time
 Space
 Cloud cover
 …
18th – 22nd July 2017

Simple search
 http://cloudsdi.
geo-solutions.it/
 geoserver/oseo/search?
parentId=SENTINEL2
&cloudCover=30]
 Output is RSS with
description,
thumb/quicklook and
links
18th – 22nd July 2017

Metadata and cross linking
 Links to metadata:
 ISO metadata for collection
 GML O&M for products
 Links to OGC services (cross linking)
 WMS/WMTS to see collection/product
 WFS to get “masks” (validity, sea, cloud, snow,
…)
 WCS to extract raster
 Direct download to get original package
18th – 22nd July 2017

Tight integration with mosaic possible
 The system creates one “feature type” per
collection
 Can be used as the index to a mosaic
 One mosaic per collection
 Filter using CQL and sort by attribute on the
WMS/WCS service
 Using all the attributes already searchable
by OpenSearch
18th – 22nd July 2017

REST API to manage OpenSearch
18th – 22nd July 2017
Swagger documentation

Real World Use cases
18th – 22nd July 2017

Use Case 1 – Light Integration
 GeoServer + MapStore + Legacy Ingestion
 Maritime Data
 SAR Time Series
 Oil Spill + other vectors
 Single Mosaic + CQL + Custom Access
Manager
 Integrated SSO with MapStore
 Precooked Download of Scenes
 Fine grain control over data access
 Download
 Visualization
 Clever usage of Mosaic Indexes via WFS
18th – 22nd July 2017

Use Case 1 – Light Integration
18th – 22nd July 2017

Use Case 2 – Deep Integration
 High Res Imagery
 SSO with CAS
 Legacy Ingestion
18th – 22nd July 2017

Use Case 2
 Database of satellite imagery, one satellite collect
(strip) materialized as many files (granules)
 Access filtering based on current user roles (custom
data store implementation)
 Custom stacking order (SORT BY) also user based if
not already provided in the request
Mosaic
POSTGIS
Custom store wrapper
Postgis store
18th – 22nd July 2017

Use Case 3 – Let’s go Deeper!
 XXX
18th – 22nd July 2017

That’s all folks!
Questions?
info@geo-solutions.it
18th – 22nd July 2017

Serving earth observation data with GeoServer: addressing real world requirements - FOSS4G EU 2017

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Serving earth observation data with GeoServer: addressing real world requirements - FOSS4G EU 2017

Similar to Serving earth observation data with GeoServer: addressing real world requirements - FOSS4G EU 2017 (20)

More from GeoSolutions

More from GeoSolutions (17)

Recently uploaded

Recently uploaded (20)

Serving earth observation data with GeoServer: addressing real world requirements - FOSS4G EU 2017