Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Serving earth observation data with GeoServer: addressing real world requirements - FOSS4G EU 2017

13,755 views

Published on

The presentation will cover GeoSolutions experience in setting up GeoServer based production systems providing access to earth observation products, with indications of technical challenges, solutions, and deployment suggestion. The presentations will cover topics such as setting up a single unified mosaic from all the available data sources, tailoring access to it to different users, determining the most appropriate stacking order, dealing with multiresolution, different coordinate systems, multiband data, SAR integration, searching for the most appropriate products using a mix of WFS, CSW and so on, serving imagery with high performance WMS and WMTS, performing small and large data extractions with WCS and WPS, closing up with deployment examples and suggestions.

Published in: Technology
  • Be the first to comment

Serving earth observation data with GeoServer: addressing real world requirements - FOSS4G EU 2017

  1. 1. Serving EO Data with GeoServer Addressing real-world EO requirements With GeoServer Ing. Simone Giannecchini Ing. Daniele Romagnoli Ing. Andrea Aime GeoSolutions
  2. 2. GeoSolutions  Founded in Italy in late 2006  Expertise • Image Processing, GeoSpatial Data Fusion • Java, Java Enterprise, C++, Python • JPEG2000, JPIP, Advanced 2D visualization  Supporting/Developing FOSS4G projects  GeoServer, MapStore  GeoNetwork, GeoNode, Ckan  Clients  Public Agencies  Private Companies  http://www.geo-solutions.it FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  3. 3. Reference scenario FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  4. 4. ImageMosaic – the basics FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  5. 5. Terminology  Granule/Tile  The individual raster element composing the mosaic  (Granule) Index  The collection of metadata records describing the location, spatial coverage and other attributes of each single granule  Dimensions/Domains  The dimensions besides the spatial ones used to distinguish individual granules FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  6. 6. Complete freedom  Granules can:  Overlap as they please, can have different resolutions  Be in different file formats (faster if the format is uniform)  Have different color models, RGB, gray, paletted (needs JAI-EXT enabled)  Be in different coordinate reference systems (needs GeoServer 2.12 to work best) FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  7. 7. The mosaic index  Always present  A table of granules, their footprint, location, and extra attributes  Used to look up which granules to mosaick for a given request  Currently supported/tested sources  PostGIS (JNDI)  Oracle (JNDI) it’s been a nightmare because of naming!  Shapefile (no attr indexing, no timestamp)  H2 (local embedded db)  Can be customized to support custom granule indexes (e.g. legacy catalog) FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  8. 8. The Dimensions  Maps to alphanumeric attributes in the index  TIME and ELEVATION receive special treatment for WMS and WCS  Custom/Additional dimensions  Everything besides TIME & ELEVATION  They can be dynamically discovered from GetCapabilities/DescribeCoverage FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  9. 9. Filling the index  Available options  Have GeoServer do it for you from the UI (one shot setup)  Use the REST API to add granules and make it populate the index  Directly write the index (common for moving window of data)  Use the OpenSearch for EO extension (later in the presentation) FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  10. 10. Grabbing attributes from file names  elevationregex.properties file regex=(?<=_)(d{3})(?=_)  elevationregex.properties file regex=[0-9]{8}T[0-9]{9}Z(?!.*[0-9]{8}T[0-9]{9}Z.*) NCOM_wattemp_020_20081031T0000000_12.tiff  While building the index GeoServer can fetch attributes from file name  Or from within the file if needed FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  11. 11. REST configuration of mosaic contents  Granule Index CRUD Operations via REST  Create mosaic  Read its index schema  Read its index contents  Update/Delete/Insert granules FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017 GS Pixels Index Client
  12. 12. Documentation links  GeoServer reference docs:  http://docs.geoserver.org/stable/en/user/data/raster/image mosaic/index.html  http://docs.geoserver.org/stable/en/user/data/raster/image mosaic/tutorial.html  http://docs.geoserver.org/api/#/1.0.0/structuredcoverages.y aml  GeoSolutions training:  http://geoserver.geo- solutions.it/edu/en/multidim/imagemosaic/index.html  http://geoserver.geo- solutions.it/edu/en/multidim/accessing_multidim/index.html  http://geoserver.geo- solutions.it/edu/en/multidim/rest/index.html FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  13. 13. ImageMosaic – the fun stuff FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  14. 14. Filtering and sorting  The mosaic index can be filtered upon with CQL  Can be sorted on using WFS like syntax  More possibilities than just Time and Elevation SENSOR = SAR AND SATELLITE=XYZ FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017 SORTBY = TIME D
  15. 15. Masking Support  Mask allow finding valid data  Alternative to NODATA for lossy compressions  Vector and raster masks supported  Raster masks can be embedded (GDAL style)  Vector ones can be WKB or shapefiles  API to plug your own mask provider  Available for image mosaic and GDAL based data sources FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  16. 16. Coverage Views  NetCDF/Grib/Mosaic: one reader, many coverages  Sometimes the coverages are strictly related  E.g., U and V components of wind/current vectors  Merge them back into a single coverage as bands  Allows rendering engine to work with them both as one FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  17. 17. Pluggability  Lots of extension points and pluggable bits  PropertyCollectors: extract index attributes from granules  CoverageNameCollector: when you have a hash of files from multiple coverages  GranuleHandler: pre-process granules before getting them in the mosaic  GranuleAcceptor: decide if the granule is actually part of the mosaic  SubMosaicProducer: custom way to mosaic granules  GranuleCatalog: how you store the index  See more examples at the end of presentation FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  18. 18. NetCDF FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  19. 19. NetCDF format support  NetCDF support  Support COARDS/CF* conventions loosely  Expose NetCDF internal data as a set of 2D slices  Fast 2D (time, elevation) slice extraction  Supports NCML FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  20. 20. One or more variables  Polyphemus Sample Dataset  1 File  Multiple Coverages! FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  21. 21. NetCDF GridMapping to EPSG codes FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  22. 22. NetCDF WCS output  WCS can output NetCDF for any multidimensional source  Several customizations to control output and preserve info if the source is a NetCDF itself  Source has to be multidimensional (current restriction) FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  23. 23. Documentation links  GeoServer documentation  http://docs.geoserver.org/latest/en/user/extensions/n etcdf/netcdf.html  http://docs.geoserver.org/latest/en/user/extensio ns/netcdf-out/index.html  GeoSolutions training:  http://geoserver.geo- solutions.it/edu/en/multidim/netcdf/index.html  http://geoserver.geo- solutions.it/edu/en/multidim/netcdf/index.html FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  24. 24. ImageMosaic – NetCDF integration FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  25. 25. NetCDF/Mosaic integration  ImageMosaic NetCDF integration  Allow the ImageMosaic to handle multiple NetCDF files  Expose NetCDF internal structure (times, elevations)  Make ImageMosaic handle slices of the NetCDF file as granules  5-dim mosaic (space/time/runtime other dimensions) Mosaic NetCDF 1 NetCDF 2 NetCDF … N … FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  26. 26. Multiple coverages per mosaic FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  27. 27. GRIB/GRIB2 Format Support  Based on same UCAR NetCDF Java libs used by NetCDF Format  Same capabilities of NetCDF Format  Same indexing logic  Same ImageMosaic support FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  28. 28. Documentation links  GeoSolutions training:  http://geoserver.geo- solutions.it/edu/en/multidim/mosaic_config/netc df_mosaic.html FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  29. 29. The services - visualize FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  30. 30. WMS-T support TIME, ELEVATION & Custom dimensions http://localhost:8080/geoserver/geosolutions/wms?... &time=2013-03-1T00:00:00.000Z &elevation=35.0 &DIM_FILEDATE=2013-03-01T00:00:00.000Z &DIM_UPDATED=2013-04-08T08:18:41.597Z FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  31. 31. Enabling WMS-T support FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  32. 32. WMS – custom filtering and sorting FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017  &CQL_FILTER=platform=SENTINEL2  &sortBy=time D &sortBy=time D
  33. 33. WMS-EO Extending LayerGroup concept Support same style on both raster and vector data Support custom dimensions Alter map on the fly to support band combination Wizard to configure EO layer groups  Product layer tree  Mix of vector and raster info  Raw data, flags (clouds, snow), browse FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  34. 34. WMS-EO  Helper UI to build the tree FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  35. 35. Rendering transformations  SLD Based transformations  On-the-fly contouring  On-the-fly poligonalization  Current arrows  Wind Barbs  Pluggable: add your own FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  36. 36. Rendering transformations windbarbs currents contouring FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  37. 37. The services - download FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  38. 38. WCS 2.0  Basics  Core service  KVP binding  XML binding  Common extensions  CRS  Scaling  Interpolation  Range subsetting  GeoTiff  GML  NetCDF  Add the output format extensions  GeoTIFF  GML Grid  NetCDF FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  39. 39. WCS 2.0 processing chain Crop RangeSubset Scale & Interpolate Reproject & Interpolate Encode Core Extension Extension Extension Extension FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  40. 40. WCS 2.0 DescribeCoverage 4D BBOX TIME details as extension ELEVATION details as extension CUSTOM FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  41. 41. WCS – custom filtering and sorting FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017  &CQL_FILTER=platform=SENTINEL2  &sortBy=time D &sortBy=time D
  42. 42. Download ND NetCDF http://localhost:8080/geoserver/wcs?request=GetCoverage &service=WCS&version=2.0.1&coverageId=geosolutions__NO2 &Format=application/x-netcdf &subset=Long(5,20) &subset=Lat(40,50) &subset=elevation(300,1250) &subset=time("2013-03-01T10:00:00.000Z","2013-03- 01T22:00:00.000Z") • Can also handle custom dimensions • Described in DescribeCoverage in a vendor metadata section 4D BBOX FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  43. 43. WCS EO  Add support for WCS EO metadata in readers  Associate each file with EO metadata  Include such information in DescribeCoverage/DescribeEODataset  EODataset: multidimensional mosaic or NetCDF file  Describe the inner structure (granules listing) in DescribeEODataSet FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  44. 44. OpenSearch for EO FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  45. 45. Putting it all together (and finding it)  Use case  Lots of satellite imagery  Different satellites  Different sensor  Deep time and space distribution  What to do about it?  “OpenSearch for EO“ to the rescue FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017 OpenSearch OGC OpenSearch Geo and Time OGC OpenSearch for EO
  46. 46. OpenSearch for EO  Geo and Time: Extension to OpenSearch adding search keywords for time and space  Earth Observation: extension to the above adding search keywords for and Earth Observation properties  E.g.  Cloud cover  Snow cover  Off nadir  …  Simpler alternative to CSW with ebRIM FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  47. 47. General information schema FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017 Collection Product Granule Search Search Mosaic/OGC
  48. 48. Two steps search  First find collection of similar products (by satellite/sensor), e.g.  Optical or SAR?  Supported resolutions  Availability of data in particular timeframe  Then search products in it, by  Time  Space  Cloud cover  … FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  49. 49. Simple search  http://cloudsdi. geo-solutions.it/  geoserver/oseo/search? parentId=SENTINEL2 &cloudCover=30]  Output is RSS with description, thumb/quicklook and links FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  50. 50. Metadata and cross linking  Links to metadata:  ISO metadata for collection  GML O&M for products  Links to OGC services (cross linking)  WMS/WMTS to see collection/product  WFS to get “masks” (validity, sea, cloud, snow, …)  WCS to extract raster  Direct download to get original package FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  51. 51. Tight integration with mosaic possible  The system creates one “feature type” per collection  Can be used as the index to a mosaic  One mosaic per collection  Filter using CQL and sort by attribute on the WMS/WCS service  Using all the attributes already searchable by OpenSearch FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  52. 52. REST API to manage OpenSearch FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017 Swagger documentation
  53. 53. Real World Use cases FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  54. 54. Use Case 1 – Light Integration  GeoServer + MapStore + Legacy Ingestion  Maritime Data  SAR Time Series  Oil Spill + other vectors  Single Mosaic + CQL + Custom Access Manager  Integrated SSO with MapStore  Precooked Download of Scenes  Fine grain control over data access  Download  Visualization  Clever usage of Mosaic Indexes via WFS FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  55. 55. Use Case 1 – Light Integration FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  56. 56. Use Case 1 – Light Integration FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  57. 57. Use Case 1 – Light Integration FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  58. 58. Use Case 2 – Deep Integration  High Res Imagery  SSO with CAS  Legacy Ingestion FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  59. 59. Use Case 2  Database of satellite imagery, one satellite collect (strip) materialized as many files (granules)  Access filtering based on current user roles (custom data store implementation)  Custom stacking order (SORT BY) also user based if not already provided in the request Mosaic POSTGIS Custom store wrapper Postgis store FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  60. 60. Use Case 3 – Let’s go Deeper!  XXX FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017
  61. 61. That’s all folks! Questions? info@geo-solutions.it FOSS4G EU 2017, Marne la Vallée 18th – 22nd July 2017

×