The purpose of this presentation is, on a side, to dissect the developments performed during last year as far as raster data support in GeoTools and GeoServer is concerned, while on the other side to introduce and discuss the future development directions.
Advancements and improvements for the management of raster mosaic and pyramids will be introduced and analyzed, as well as the latest developments for the exploitation of GDAL raster sources.
Extensive details will be provided on the latest updates for the management of multidimensional raster data used in the Remote Sensing and MetOc fields.
The presentation will also introduce and provide updates on the JAITools and ImageIO-Ext projects. JAITools provides a number of new raster data analysis operators, including powerful and fast raster algebra support. ImageIO-Ext bridges the gap across the Java world and native raster data access libraries providing high performance access to GDAL, Kakadu and other libraries.
The presentation will wrap up providing an overview of unresolved issues and challenges that still need to be addressed, suggesting tips and workarounds allowing to leverage the full potential of the systems.
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Raster data in GeoServer and GeoTools: Achievements, issues and future developments
1. Raster data in GeoServer and GeoTools:
Achievements, issues and future developments
Ing. Simone Giannecchini, GeoSolutions
Ing. Andrea Aime, GeoSolutions
FOSS4G 2014, Portland
8th-13th September 2014
2. Outline
Who we are
The building blocks
Recent Achievements
What’s cooking
FOSS4G 2014, Portland
8th-13th September 2014
3. GeoSolutions
Founded in Italy in late 2006
Expertise
• Image Processing, GeoSpatial Data Fusion
• Java, Java Enterprise, C++, Python
• JPEG2000, JPIP, Advanced 2D visualization
Supporting/Developing FOSS4G projects
MapStore, GeoServer
GeoBatch, GeoNetwork
Clients
Public Agencies
Private Companies
http://www.geo-solutions.it
FOSS4G 2014, Portland
8th-13th September 2014
5. GeoServer building blocks
WMS WFS WPS REST Google
GeoTools
GWC
ImageIO(-Ext) JTS
GDAL
WCS
JAI(-Ext)
Kakadu JAITools
Vector Style
Raster
FOSS4G 2014, Portland
8th-13th September 2014
6. JAI
Java Advanced Imaging Library
SUN then Oracle Image Processing library
Huge number of basic building operations
Tile-based, deferred execution model
Integrate Tile Caching (In Memory)
Easily Extensible
Natively Accelerated Code available for many
operations
No source code for native ops
Development more or less stopped!
FOSS4G 2014, Portland
8th-13th September 2014
7. JAI-Ext
Oracle JAI Drop-In Replacement
Open Source (Apache 2.0)
High Performance
Pure Java
Support for No-Data
Support for Processing ROI
Several fixes compared to JAI
Support for Band Masks
FOSS4G 2014, Portland
8th-13th September 2014
9. JAITools/Jiffle
High performance raster processing
Extends, complements, replaces Oracle Java Advanced
Imaging (JAI) Library
Open Source, BSD license
Low Level Operations, e.g. RangeLookup
Higher Level Operations
Vectorize, Contour
Raster algebra (Jiffle)
Works mostly in pure raster space (Image Processing)
Stable release is 1.3.0
FOSS4G 2014, Portland
8th-13th September 2014
10. ImageIO
SUN then Oracle Image I/O library
Partly ships with JDK partly as extension
Tile-based, deferred & immediate execution model
Easily Extensible
Natively Accelerated Code available for many
formats
No source code for native (de)coders
Development more or less stopped!
FOSS4G 2014, Portland
8th-13th September 2014
11. ImageIO-Ext
High Performance raster I/O
Extends, complements, replaces Oracle JAI ImageIO
Library
Open Source, LGPL and BSD licenses
Plugins:
BigTiff*, MatFile 5, TurboJPEG
JP2000 (Kakadu)
GDAL integration via ImageIO-ext
JPEG2000, MrSID, ECW
BigTIFF, ERDAS Image, HDF4
Complete list here
FOSS4G 2014, Portland
8th-13th September 2014
12. GeoTools
The Open Source Java GIS Toolkit
LGPL
Management of Geospatial data
CRUD Operations
Raster and vector
Rendering
GDAL Support
Based on best-of-breed Open Source projects
Java Advanced Imaging (JAI), ImageIO, ImageIO-Ext,
JAITools (soon JAI-Ext too)
Java Topology Suite (JTS)
FOSS4G 2014, Portland
8th-13th September 2014
13. GeoServer
GeoSpatial enterprise gateway
Java Enterprise
Management of raster and vector data
Based on GeoTools
Standards compliant
OGC WCS 1.0, 1.1.1 (RI), 2.0.1
OGC WFS 1.0, 1.1 (RI), 2.0
OGC WMS 1.1.1, 1.3.0
OGC WPS 1.0.0
OGC CSW 2.0.2
Google Earth/Maps support
FOSS4G 2014, Portland
8th-13th September 2014
15. GS – Faster PNG encoding
High Performance/Pure Java PNG output
format
More Control on bugs/enhancements
FOSS4G 2014, Portland
8th-13th September 2014
Based on the
open source
PNGJ library
Optimized
pixel
extraction to
PNG data
buffer code
16. GS – raster projection handling
Better polar projection support
FOSS4G 2014, Portland
8th-13th September 2014
18. GT – image mosaic masks
Support for vector masks as shp/wkb/wkt
FOSS4G 2014, Portland
8th-13th September 2014
19. GT – image mosaic masks
Also useful for clipping over a certain area
E.g., clip over administrative boundaries
FOSS4G 2014, Portland
8th-13th September 2014
20. GS – WCS 2.0
WCS 2.0
The basics
Core service
KVP binding
XML binding
GetCoverage extensions
CRS
Scaling
Interpolation
Range subsetting
GeoTiff
GML
NetCDF
Output format extensions
GeoTIFF
GML Grid
NetCDF
Ensure CITE tests compliancy
FOSS4G 2014, Portland
8th-13th September 2014
22. GS – WCS-EO
WCS Earth Observation profile
Build on top of a working WCS 2.0 with full extensions
WCS 2.0
CRS extension
Range subsetting extension
Interpolation extension
Scaling extension
GeoTiff extension
NetCDF extension
Add support for the WCS-EO extras
Listing coverage datasets in the capabilities documents (based on image
mosaic contents, which will have to be marked as “exposed” so that we
show their inner structure for EO)
Support describe coverage dataset
Support returning results for an entire dataset in GetCoverage
FOSS4G 2014, Portland
8th-13th September 2014
23. GS – Coverage Views
Virtual layers created merging bands from
different sources
FOSS4G 2014, Portland
8th-13th September 2014
25. GT – NetCDF and Grib
NetCDF/Grib official Support
Based on Unidata NetCDF JAVA library
Exposes current interfaces
(GridCoverage2DReader,…) too for GeoServer
compatibility before changes
1 coverage for each Dataset’s variable (multiple
coverages).
Internal data exposed as set of 2D slices
Custom dimensions support
Ancillary XML file:
filter on NetCDF variables/coverages
define low level index schema (for dimensions)
FOSS4G 2014, Portland
8th-13th September 2014
26. GT – NetCDF and GRIB
ImageMosaic NetCDF/GRIB integration
Allow the ImageMosaic to handle NetCDF/GRIB files
Expose NetCDF/GRIB internal structure (times, elevations)
Make ImageMosaic handle slices of the source files as granules
Mosaic
NetCDF 1 NetCDF 2 NetCDF … N
…
FOSS4G 2014, Portland
8th-13th September 2014
27. GT – NetCDF and GRIB
NetCDF Polyphemus Sample Dataset
FOSS4G 2014, Portland
8th-13th September 2014
28. GT – Multicoverage
Multidimentional interfaces
StructuredGridCoverage2DReader
exposes the underlying granule structure and dimensions
descriptors of a specific coverage
getGranules (return a GranuleSource)
getDimensionDescriptors
allows to create/remove coverages and add new sources
createCoverage
removeCoverage
Harvest
GetGranules
FOSS4G 2014, Portland
8th-13th September 2014
29. GS –multiple coverages per store
Multiple Coverages from Mosaic, NetCDF,
GRIB
FOSS4G 2014, Portland
8th-13th September 2014
30. GT – access mosaic contents
GranuleSource/GranuleStore
GranuleSource:
ReadOnly access to Granules (with Query support)
getGranules(Query) SimpleFeatureCollection
getSchema SimpleFeatureType
getCount(Query), getBounds(Query)
E.g.: get all granules having acquisitionYear > 2012 and Country=Italy
GranuleStore:
add/modify/delete capabilities using Transactions
addGranules (SimpleFeatureCollection)
removeGranules (Filter)
updateGranules
E.g.: remove all granules having acquisitionYear < 1990
FOSS4G 2014, Portland
8th-13th September 2014
31. GS – REST over mosaic contents
Granule Index CRUD Operations via REST
CREATE
curl -u admin:Geos -XPUT -H "Content-type:application/zip" --data-binary
@http://localhost:8080/geoserver/rest/workspaces/geosolutions/coveragestores/t
emperature/file.imagemosaic
READ index schema
curl -v -u admin:Geos -XGET
"http://localhost:8080/geoserver/rest/workspaces/geosolutions/coveragestores/polyphemus
/coverages/NO2/index.xml"
READ AND FILTER WFS like with CQL filtering and paging
curl -v -u admin:Geos -XGET
"http://localhost:8080/geoserver/rest/workspaces/geosolutions/coveragestores/p
olyphemus/coverages/NO2/index/granules.xml?limit=1&filter=time='2013-03-
03T00:00:00Z'"
FOSS4G 2014, Portland
8th-13th September 2014
32. GS – REST over mosaic contents
Granule Index CRUD Operations via REST
UPDATE
curl -v -u admin:Geos -XPOST -H "Content-type: text/plain" -d
"/polyphemus_20130303.nc"
"http://localhost:8080/geoserver/rest/workspaces/geosolutions/coveragestores/pol
yphemus/external.imagemosaic"
DELETE WFS like with CQL filtering and paging or by ID
curl -v -u admin:geoserver -XDELETE
"http://localhost:8080/geoserver/rest/workspaces/topp/coveragestores/polyphemus-v1/
coverages/NO2/index/granules.xml?filter=location='polyphemus_20130301.nc'"
curl -v -u admin:geoserver -XGET
"http://localhost:8080/geoserver/rest/workspaces/topp/coveragestores/polyphemus-v1/
coverages/NO2/index/granules/NO2.2689.xml"
FOSS4G 2014, Portland
8th-13th September 2014
33. GS – WCS NetCDF out
NetCDF Output Format
GetCoverage request parameter: &Format=NetCDF
Based on Unidata NetCDF JAVA library
Create a NetCDF output file on top of a GranuleStack:
A GridCoverage2D List:
a GridCoverage2D for each combination of requested
Dimension values
GridCoverages should come from a
StructuredGridCoverage2DReader
Supports WCS 2.0.1
Trimming: specify values range for a dimension
Slicing: specify single value for a dimension
Supports timeRanges/numberRanges on output
FOSS4G 2014, Portland
8th-13th September 2014
35. GS – NetCDF out in ToolsUI
FOSS4G 2013, Nottingham
21th September 2013
36. GS – WMS EO
Wizard to configure EO
layer groups
Extending LayerGroup
concept
Support same style on
both raster and vector
data
Support custom
dimensions
Alter map on the fly to
support band
combination
Earth observation profile
GSIP 84
FOSS4G 2014, Portland
8th-13th September 2014
37. GS – WMS-EO
WMS-EO Quirks
Root layer must respond with a specific layer rather than
the composition of the children
Kind of a default visualization for a certain Dataset
E.g. when I send a GetMap for the root layer I could get back the browse
image default layer
Same styles for both Raster as well as Vector data
E.g. yellow can be use for both flags as well as outlines
Peculiar behavior for Band layer ( raw data )
E.g. multiple bands at different wavelengths
Can request either 1 (grayscale image) or 3 (RGB image)
Different combinations are prohibited
Peculiar Behavior for GetFeatureInfo
FOSS4G 2014, Portland
8th-13th September 2014
39. What’s cooking
Plus what’s on our wish list
FOSS4G 2014, Portland
8th-13th September 2014
40. NODATA support
Natively handle NODATA with JAI-EXT
Support in all aspects
Scaling
Reprojection
Interpolation
FOSS4G 2014, Portland
8th-13th September 2014
41. GS – Raster algebra WPS
Raster Algebra as WPS
Gets an OGC Filter containing logical operations between
raster layers and produces a binary (0/1) layer
FOSS4G 2014, Portland
8th-13th September 2014
42. GS – Raster algebra in
Coverage views
JIFFLE
FOSS4G 2014, Portland
8th-13th September 2014
43. GS – Raster algebra
in Coverage Views
Define new bands using a Jiffle formula
Generate fully computed layers
Still using overviews and other native
abilities
FOSS4G 2014, Portland
8th-13th September 2014
44. The End
Questions?
andrea.aime@geo-solutions.it
simone.giannecchini@geo-solutions.it
FOSS4G 2014, Portland
8th-13th September 2014
45. Old news
FOSS4G 2014, Portland
8th-13th September 2014
46. ImageIO-Ext
LibJPEG-Turbo
derivative library of libjpeg based on SIMD
instructions (MMX, SSE2, NEON)
accelerate baseline JPEG compression and
decompression on x86, x86-64, and ARM systems
available on many platforms
Open Source, LGPL
ImageIO-Ext-TurboJPEG
A JPEG ImageIO plugin built on top of the JAVA
wrapper for LibJPEG-Turbo (TurboJPEG)
20%-30% SpeedUp* on JPEG encoding
10%-20% SpeedUp* on JPEG decoding
More info here
Integrates with TIFF reader and writer
FOSS4G 2014, Portland
8th-13th September 2014
47. ImageIO-Ext
NITF encoding Support (ImageIO-Ext-NITF)
Based on NITRO
Support for Tagged Record Extensions (TREs)
Support ImageSegments JPEG2000 compression through
Kakadu ImageIO-Ext plugin
(Numerically Lossless/VisuallyLossless/Lossy)
NPJE, EPJE BIIF (Basic Image Interchange Format) profiles
J2KLRA TRE support
Support for multiple TextSegments
Support for multiple ImageSegments
Support for ShapeFile Data Extension Segment (CSSHPA)
FOSS4G 2014, Portland
8th-13th September 2014
48. GeoTools
Band
Combine +
Symbology
On-the-fly
Contouring
Raster to
point vector
On-the-fly
Polygonalization
Improved internal JAI Concurrency
Introduced and Improved JAI Tile Recycling
Reduced depth of JAI Processing Chains
More aggressive factory caching
Added a few specific high perf JAI operations
e.g. crop
Created Specific high perf ROI class
Improving Tile Caching
Work ongoing
Transparent File vs Memory Caching when writing
raster
FOSS4G 2014, Portland
8th-13th September 2014
49. GeoTools
Optimized Rendering/Processing for raster data
Optimized Crop Chains Coalescing multiple Crops into
a single one
Optimize Affine Transformation Chains Coalescing
multiple Affine Transformations into a single one
NEAREST
FOSS4G 2014, Portland
8th-13th September 2014
53. GeoTools
ImageMosaic
Ordering
SORT_BY clause support on DBMS based mosaics
sorting=“time A, elevation D”
Stacking VS Merging
Merging: different granules merged together in
the final image as default
Stacking: different granules stacked on different
bands
E.g. MeteoSat data: 12 channels (1 granule for each Channel)
1 = VIS0.6, 2 = VIS0.8, …, 11 = IR13.4, 12 = HRV.
Want return a RGB image made of channels 1,3,7 Stack
them
FOSS4G 2014, Portland
8th-13th September 2014
54. GeoTools
New API (unsupported/coverage-experiment)
Driver: Get a connection to a coverage store.
CoverageAccess: a physical storage of coverage data. Get access
to a source of coverages.
CoverageSource: provides data access to a coverage.
Updated GridCoverage core API
Old GridCoverageReader:
inefficient and thread unsafe way to access multiple coverages
read GridCoverages from the input stream in a sequential order
New GridCoverage2DReader:
Coverages and properties may be accessed by coverage name:
get Envelope(name), get GridRange(name), get CRS(name),
read(name), get dynamicParameters (name), …
dynamicParameters support (for custom dimensions)
FOSS4G 2014, Portland
8th-13th September 2014
55. GeoServer
LibJPEG-Turbo Output format
Encoding WMS output JPEG images through
LibJPEG Turbo library
Based on the TurboJpeg ImageIO-Ext plugin built
on top of the JAVA interface for LibJPEG-Turbo
(TurboJPEG)
Provided as a GeoServer extension
Up to 80% SpeedUp measured on JPEG encoding
Can be disabled at startup:
-Ddisable.turbojpeg=true
More Info here
FOSS4G 2014, Portland
8th-13th September 2014
56. GeoServer
More Color Quantizers
OCTTREE
Speed loss is around 10%
MEDIANCUT
FOSS4G 2014, Portland
8th-13th September 2014
57. GeoServer
WMS Custom Dimensions
Work on GeoTools raster Additional Domains
NetCDF
ImageMosaic
Turn additional querying dimensions into
WCS/WMS Dimensions
ImageMosaic Domain Category
WMS 1.1.1 DIM_CATEGORY
WMS 1.3.0 DIM_CATEGORY
Perfect for large multidimensional mosaics!
E.g. meteo (forecast time, elevation, band, base
time, model name)
FOSS4G 2014, Portland
8th-13th September 2014
58. GeoServer
WMS TIME, ELEVATION & More
http://localhost:8080/geoserver/geosolutions/wms?...&time=2013-03-
01T00:00:00.000Z&elevation=35.0&DIM_FILEDATE=2013-03-
01T00:00:00.000Z&DIM_UPDATED=2013-04-08T08:18:41.597Z
FOSS4G 2014, Portland
8th-13th September 2014