Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

State of GeoServer 2.13


Published on

State of GeoServer provides an update on our community and reviews the new and noteworthy features for the Project. The community has a lot to cover in 2.12 and the recently released 2.13.

Each release provides exciting new features. This talk covers our work on supporting Java 9 and diverse improvements across GeoServer.

Attend this talk for a cheerful update on what is happening with this popular OSGeo project. Whether you are an expert user, a developer, or simply curious what GeoServer can do for you.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

State of GeoServer 2.13

  1. 1. State of GeoServer 2.13
  2. 2. Open Source Projects GeoTools GeoServer GeoWebCache OSGeo Foundation GeoServer PSC GeoWebCache Lead Kevin Smith Software Engineer smithkm Boundless Provides geospatial tools and services for managing data and building applications. Welcome
  3. 3. Jody Garnett Community Lead @jodygarnett Boundless Provides geospatial tools and services for managing data and building applications. Welcome Open Source Projects GeoTools GeoServer uDig OSGeo Foundation Board Member OSGeo Incubation Chair GeoTools Project Officer Eclipse Foundation LocationTech Steering Committee LocationTech Technology Project
  4. 4. Andrea Aime Technical Lead @geowolf GeoSolutions Innovative, robust and cost-effective solutions leveraging best-of-breed Open Source products. Welcome Open Source Projects GeoTools GeoServer ImageIO-Ext JAI-Ext JAI Tools OSGeo Foundation Charter Member GeoServer PSC GeoTools PSC
  5. 5. GeoServer 2018
  6. 6. GeoServer at a Glance Java Web Application to share and edit geospatial data. Publish data from any major spatial data source using open standards. Core Protocols WMS – maps WFS – vector WFS-T – editing WCS – coverage WPS* – process CSW* – search Tile Protocols WMTS – tiles TMS – tiles WMS-C – tiles
  7. 7. 28 → 37 members of team geoserver (to be updated 624 → 619 No clue this year 500+ → 525 pull requests New contributors being added (others stepping back) Smaller base of active contributors this year. Pull requests still going strong, very active codebase. Simone replaces Andrea as OSGeo Officer. 13 releases this year GeoServer Project Status How about team user-list? 2016: 2005 2017: 1400 SourceForge checking activity 2018: 491 SourceForge privacy policy change Also geoserver-italia, espanol,, geoserver-br (600) and mailing lists
  8. 8. Release Schedule stable maintenance July 2.13.2 June 2.12.4 May 2.13.1 April 2.12.3 March 2.13.0 February 2.11.5 January 2.12.2 December 2.11.4 November 2.12.1 October 2.11.3 2.12.0 September 2.12-RC1
  9. 9. User List Participation ● Answering users questions relies on a low number of people Action: ● We are sending reminder on effective use of the user list on subscription and at the bottom of each mail ● Community building activity - remind developers about the definition of “supported” (not only reviewing pull requests, also answering user questions, used to be obvious, but it’s not obvious to devs hired into open source)
  10. 10. ● Bug stomp introduced in 2016 ○ Relying on volunteer time not working out ○ allowed developers to reserve time ● Moved to fixed schedule in 2017 ○ Arranging day each month too random ● Last Friday of each month ○ Check blog post for tips on participation Standing “last friday of the month” bug stomp
  11. 11. Held “sprint” to update from Restlet to Spring MVC for our REST configuration API. Thanks to GeoSolutions for hosting, to Boundless and Astun for sending developers, and to all the sponsors who helped fund this important and badly needed, but boring work. Spring Results! REST API Migration 2.12Viareggio Code SprintCommunity
  12. 12. REST API Migration ● Migration complete ● REST API documented ○ Available online now! 2.12Viareggio Code SprintCommunity
  13. 13. Vector data sources
  14. 14. GeoPackage graduated from community module to “core”: ● Provides an sqlite database containing vector or raster data ○ Initial idea from spatialite project ○ Now an OGC Standard to replace shapefile and personal file geodatabase ● GeoPackage improvements: ○ Bug fix - raster mosaic geopackage had y-axis inverted. ○ Support for spatial indexes an RnD topic ● Additional GeoPackage community modules ○ GeoPackage WMS output available as a community module ○ GeoPackage WFS output format ○ GeoPackage WPS output format GeoPackage 2.12 GeoSolutions Boundless Community Boundless Suite
  15. 15. GeoPackage performance improvements GeoPackage reading and rendering: ● Up to two times faster on large datasets ● Up to 50% faster as you zoom in GeoPackage performance is now on par with PostGIS (Shapefile remains fastest / most optimized) 2.13 Andrea Aime GeoSolutions Customer
  16. 16. ● GeoServer importer supports GeoJSON files ○ Allows GeoJSON to be imported into PostGIS database and published in one step ○ Based on unsupported module in GeoTools ● Wish to package as a community module for GeoServer ○ Allow GeoServer to directly publish GeoJSON files ○ Text based, non indexed format, only suitable for small files GeoJSON store 2.12Ian TurtonCommunity
  17. 17. ● GeoServer importer supports CSV files ○ Allows CSV to be imported into PostGIS database and published in one step ○ Based on unsupported module in GeoTools (used as a tutorial for programmers) ● Package as a community module for GeoServer ○ Allow GeoServer to directly publish CSV files ○ Text based, non indexed format, only suitable for small files CSV store 2.12BoundlessCommunity
  18. 18. Support for more PostGIS data types HStore and JSON data types: ● HStore returned as a Map ● Render as a JSON formatted string field in WFS output No special query functionality provided, if you are interested please contact us!
  19. 19. Raster data and processing
  20. 20. ● Initial version in 2.11 ● Several fixes in 2.12, e.g. accuracy, performance, dateline crossing handling ● Being tested against global mosaics of Sentinel2 (each image in a different UTM zone), more improvements/fixes incoming Heterogeneous CRS image mosaics 2.11/2.12Boundless/GeoSolutionsRnD/DLR UTM60 image footprints UTM1 image footprints Dateline wrapped image mosaic
  21. 21. ● Reprojected images could have shown border effects Better handling of reprojection boundaries 2.11GeoSolutionsCommunity
  22. 22. ● Higher order interpolation did not play well with nodata, causing border effects: ○ Bilinear ○ Bicubic ● Requires jai-ext enabled Improved no-data handling in interpolation 2.11GeoSolutionsCommunity Before After
  23. 23. ● Respect scale/offset attributes ● Improved support for related dimensions (e.g. time/runtime in forecast), automatically finding the best related value when just one dimension is provided NetCDF reader improvements 2.11 Niels/Devon/Ben/Daniele Scitus/Boundless/Transient/GeoSolutions Customer Runtime
  24. 24. S3 GeoTIFF Community Module Support for GeoTiffs hosted on Amazon S3 ● Reader based off of the GeoTiff reader. ● Very basic caching of images from S3 based off of EhCache ● InputStream implementation from JAI for reading imagery from S3 2.12 Devon Boundless Boundless s3://landsat-pds/L8/001/002/LC80010022016230LGN00/LC8001 0022016230LGN00_B1.TIF?useAnon=true&awsRegion=US_WEST_2
  25. 25. Coverage views from heterogeneous bands 2.13 Image mosaics often pull data collected With different sensors and resolutions Into a single image. A new feature in 2.13 is the ability to define a coverage view using mix bands from seperate files into a single multiband coverage view. GeoSolutionsCustomer
  26. 26. GHRSST Community Module Group for High Resolution Sea Surface Temperature ● NetCDF output community module! 2.13GeoSolutionsCustomer
  27. 27. MongoDB (Community) Community module: ● Makes use of the supported GeoTools MongoDB DataStore ● Publish information stored in the MongoDB Document Database
  28. 28. Styling
  29. 29. YSLD extension module ● Style Layer Descriptor represented in YAML ○ Uses indentation rather than XML ○ Easier to read ○ More compact ○ More flexible syntax ○ Contains variables for reusable code ○ Compatible with SLD ● REST API can convert formats ○ Preview your existing styles as YSLD using REST API ○ View the SLD representation of YSLD 2.11Boundless symbolizers: - polygon: stroke-color: 'blue' stroke-width: 1 fill-color: '#7EB5D3' - text: label: ${name} fill-color: 'black' anchor: [0.5, 0.5] x-maxDisplacement: 40 x-autoWrap: 70 Community
  30. 30. ● JSON representation of styling ○ More limited than SLD/YSLD/CSS ● Works well with vector tiles ○ Cross platform styling (OpenLayers, MapBoxGL) ● Work in progress: ○ Currently a GeoTools community module ○ Should package and go to extension if we get time! MapBox Style Community Module 2.12David VickBoundless Suite MapBox Style in GeoServer and OpenLayers
  31. 31. Labelling improvements ● Kerning on by default (adjusts spacing between chars, e.g. Check space between T and r in Trimble) ● Underline and Strikethrough ● Char and word spacing control ● Work in progress: ○ Labelling border of polygon (instead of the centroid) ○ Filtering labels based on polygon screen size in pixels 2.11 2.12 Andrea Aime Nuno Oliveria Community/Geoinfo/ OpenGeoGroep
  32. 32. Determining a good location for a label can be tricky. Use of maxDisplacement indicates how far a label is allowed to roam ● Previously we seached in a circular area for a good location ● Now you can supply your preference using displacementMode Labelling improvements 2.13 Andrea Aime GeoSolutions
  33. 33. Improved QGIS Compatibility ● Additional custom “Well Known” marks ● Support for SLD 1.1 External Marks (e.g. TTF font based) ● SVG Parameters 1.0 spec <PointSymbolizer> <Graphic> <ExternalGraphic> <OnlineResource xlink:type="simple" xlink:href="firestation.svg?fill=#FF0000" /> <Format>image/svg</Format> </ExternalGraphic> <size>128</size> </Graphic> </PointSymbolizer> 2.11Andrea AimeCommunity
  34. 34. ● Label export, most shared configurations exported ● Some in the works (word/char spacing, strikethrough, polygon border labelling) ● Some changes needed in GeoServer SE 1.1 parser too QGIS 3.0 style export improvements 2.12 Andrea Aime GeoSolutions OpenGeoGroep
  35. 35. WMS
  36. 36. Vector Tiles Extension Extension for publishing vector tiles - Great solution for modern High Definition Screens - Use for GeoWebCache for “MapBox without MapBox” - Couple with MapBox style for consistent styling 2.11Gabriel RoldanBoundless Suite
  37. 37. Better centroid / label placement in “vector” outputs KML ● generate placemarks that are always inside polygon (for concave polygons) Vector Tiles ● Pregenerate label points in tiles ○ Reduce effort on client 2.12 Kevin Smith Just DeOlivera RnD Label Labelvs
  38. 38. WMS sortBy vendor param ● Sort by recency, cloud cover, and so on ● Supported by vector data and image mosaic 2.12 Andrea Aime GeoSolutions Community Sort by recencySort by cloud cover
  39. 39. WFS
  40. 40. App-schema improvements ● Improvements to connection pool usage (only one connection used per request now) ● Some improvements helping while using MongoDB as a source for app-schema records ● Hale mapping support being extended to support MongoDB 2.12 Nuno Oliveira GeoSolutions NPRA
  41. 41. NSG WFS Profile (Community) The WFS 2.0.2 specification defines basic pagination support. NSG Community module supports: ● NSG Profile of WFS 2.0.2 operation PageResults ● References a stored GetFeatureRequest by resultSetId 2.13Community
  42. 42. WCS
  43. 43. Vendor param for mosaics: cqlFilter and sortBy ● Sort by recency, cloud cover, and so on ● Filter on the same ● Maps to image mosaic read parameters ● Uses attributes in the mosaic index ● Consistent with WMS now, first see and then download 2.12 Andrea Aime GeoSolutions Community
  44. 44. NetCDF input/output improvements 2.12 Niels (Scitus) Ben (Transient) Community ● Can configure NCML files as sources ● Support variable attribute definition in NetCDF output ● Copy scalars from NetCDF/GRIB source ● Copy scalars from source granules to make vector in NetCDF output
  45. 45. Tiling
  46. 46. ● GeoServer supports “cascading” remote WMTS services ○ GeoServer acts as a client, rather than just rerouting each request ○ Same as WMS cascading WMTS cascading 2.12 Ian Turton/Emanuele Tajariol GeoSolutions Customer
  47. 47. Parameter filter cache drop improvements ● GeoWebCache can cache based on parameters like style, time, or elevation. ○ Could not truncate (remove cached tiles) across all parameter values, each must be known and truncated separately. ○ Tiles could be orphaned if the allowed values changed. ○ GeoServer can truncate tiles when the data within them changes, but only caught the default values for parameters other than style ● Added operations for ○ Cross parameter truncate ○ Orphan purge ● Enabled cross parameter truncate on data update GWC 1.12 GS 2.12 Kevin Smith Boundless MapStory
  48. 48. Manage and configure “blob store” used to hold cached tiles, and “gridsets” to define tile grids associated with a spatial reference system. GeoWebCache REST API - new endpoints! 2.13Boundless Torben Barsballe Boundless
  49. 49. WPS
  50. 50. CSV input/output for WPS ● Allows the input and output of CSV for use in WPS processes ● This will lead to GeoTools CSV Datastore being upgraded to community (and hopefully to supported) ● Come see more at “Building a Table Joining Like Service with Web Processing Services” Friday Afternoon 2.12 Ian Turton Astun RnD
  51. 51. WPS Download (Community) ● Control of GeoTIFF output structure (tiling compression) ● Download granules as part of heterogeneous mosaic ● Make use of jcodec library to download large maps and build animations (for async use)
  52. 52. Configuration and management
  53. 53. Security Consideration - Disabled writing by default ● WFS-T allows editing using Transaction requests ○ May not be what you expect? ● Default configuration includes security restrictions on WFS-T functionality ● Restricting editing of data to the administrator account. ● Makes service read-only (while still advertising wfs-t) 2.11Nuno OliveiraCommunity
  54. 54. ● Great performance improvement! ● Test with 10000 layers Faster startup times for large data dirs 2.11AndreaCommunity Version COLD (Startup time seconds) HOT (Startup time seconds) 2.10.1 428 39 2.10.1 + JDCBConfig 62 49 2. 11 68 29
  55. 55. ● Great performance improvement! ● Tested on topp:states to make catalog lookup times more evident (assume diff is a constant) Faster runtime times for large data dirs 2.11AndreaCommunity Version Throughput req/sec Avg resp. time ms 2.10.1 169 47 2.10.1 + JDCBConfig 68 117 2.11 233 34
  56. 56. ● When loading a shapefile the PRJ file only matched on AUTHORITY code ● Faster lookup implementation allows GeoServer to search the EPSG database and match far more often Better/Faster EPSG code lookups 2.11AndreaCommunity This reduces a common frustration when configuring new layers in GeoServer
  57. 57. Module “status” REST API (and GUI) ● Verifying what modules are installed was a challenge ● REST API access to the “status page” ● Easy to add additional entries ● Used to check environment for difficult to configure modules that use native code 2.11 Morgan Thompson Boundless Boundless Suite geoserver/rest/about/status
  58. 58. Isolated Workspaces Workspaces are used to partition what is published for ease of management: ● A workspace has a namespace (used to keep XML content from conflict) ● Workspace published as a web service Problem: ● INSPIRE standard requires the use of a specific namespace ● Cannot use same namespace, preventing reuse of INSPIRE namespace Isolated workspace: ● Allows one workspace to publish the common layers ● The remaining workspace (each one marked isolated) can reuse these common layer Allows the same application schema to be published from multiple workspaces, each mapped onto different datasets 2.13.0 Nuno Oliveira GeoSolutions INSPIRE Customers
  59. 59. User Interface - Autocompletion Entering in URLs for data files has been improved with autocomplete ● Entering URLs for data files, will scan the path suggest existing files ● Long lists of stores or layers also feature autocomplete 2.13.0Andrea AmieCommunity
  60. 60. User Interface - Raster Parameters Filling in raster layer, parameters is now easier and less error prone ● checkboxes ● dropdowns 2.13.0Andrea AmieCommunity
  61. 61. User Interface - Error messages and form validation Error messages are now reported: ● At the top of the page ● At the bottom of the page (near the submit button) 2.13.0Andrea AmieCommunity
  62. 62. JDBCConfig and JDBCStore (Community) Use a database, rather than data directory, for clustering: ● GeoServer “Catalog” configuration ● GeoServer support files (styles, icons, fonts, config files) Work done for 2.13: ● Reduce number of database queries to improve performance ● Consistently log configuration queries for analysis 2.13.0NielsCommunity
  63. 63. Security
  64. 64. Security Considerations
  65. 65. Take care to report the issue in a responsible fashion: ● Keep exploit details out of issue report ● Mark the issue as a vulnerability. ● Be prepared to work with Project Steering Committee (PSC) on a solution ● Keep in mind PSC members are volunteers and an extensive fix may require fundraising / resources If you are not in position to communicate in public please consider commercial support, contacting a PSC member, or reaching us via the Open Source Geospatial Foundation at Security Vulnerabilities - Responsible Disclosure
  66. 66. Responsible disclosure policy is working: ● Most users contacted us in a responsible fashion ● One user shared confidential customer report on the user list Where do reports come from? ● Majority reported from security audits ○ Usually performed under contract ○ Automated tools tend to focus on web administration ui, rather than web services ● GeoServer is subject to several security audits a year Security Vulnerabilities
  67. 67. Upgrade to get security fixes ● Security fixes being added to supported versions only ● Please upgrade your GeoServer installations!
  68. 68. Looking Ahead
  69. 69. Full-screen style editor 2.13.0Andrea & TorbenCommunity
  70. 70. Java 9 was delayed until 2017, but now that it is released Oracle is setting a very aggressive roadmap! Java Roadmap Planning Java 9 September 2017 Introduced “jigsaw” module system (for smaller/faster jvm) Java 10 / Java 18.3 March 2018 Java 11 / Java 18.9 LTS September 2018 Long-term-support release! (Java 8 will end-of-life January 2019)
  71. 71. jt-bandmerge-1.0.16 jt-bandselect-1.0.16 jt-binarize-1.0.16 jt-border-1.0.16 jt-buffer-1.0.16 jt-classifier-1.0.16 jt-colorconvert-1.0.16 jt-colorindexer-1.0.16 jt-concurrent-tile-cache-1.0.16 jt-contour-1.4.0 jt-crop-1.0.16 jt-errordiffusion-1.0.16 jt-format-1.0.16 jt-imagefunction-1.0.16 jt-iterators-1.0.16 jt-lookup-1.0.16 jt-mosaic-1.0.16 jt-nullop-1.0.16 jt-orderdither-1.0.16 jt-piecewise-1.0.16 jt-rangelookup-1.4.0 jt-rescale-1.0.16 jt-rlookup-1.0.16 jt-scale-1.0.16 jt-stats-1.0.16 jt-translate-1.0.16 jt-utilities-1.0.16 jt-utils-1.4.0 jt-vectorbin-1.0.16 jt-vectorbinarize-1.4.0 jt-vectorize-1.4.0 jt-warp-1.0.16 jt-zonal-1.0.16 jt-zonalstats-1.4.0 jts-core-1.14.0 jts-example-1.14.0 jts-io-1.14.0 log4j-1.2.17 mail-1.4 marlin-0.7.5-Unsafe commons-logging-1.1.1 commons-pool-1.5.4 core-0.26 ehcache-2.10.3 encoder-1.1 ezmorph-1.0.6 freemarker-2.3.18 GeographicLib-Java-1.44 guava-17.0 gwc-core-1.12.3 gwc-diskquota-core-1.12.3 gwc-diskquota-jdbc-1.12.3 gwc-georss-1.12.3 gwc-gmaps-1.12.3 gwc-kml-1.12.3 gwc-rest-1.12.3 gwc-tms-1.12.3 gwc-ve-1.12.3 gwc-wms-1.12.3 gwc-wmts-1.12.3 h2-1.1.119 hsqldb-2.3.0 itext-2.1.5 jackson-core-2.5.0 jai_codec-1.1.3 jai_core-1.1.3 jai_imageio-1.1 jasypt-1.8 JavaAPIforKml-2.2.0 Jdom-1.1.3 jettison-1.0.1 jgridshift-1.0 json-lib-2.2.3-jdk15 json-simple-1.1 jsr-275-1.0-beta-2 jsr305-2.0.3 jt-affine-1.0.16 jt-algebra-1.0.16 jt-attributeop-1.4.0 jt-bandcombine-1.0.16 Review activation 1.1 Aopalliance-1.0 Batik 1.7 batik-anim-1.7 batik-awt-util-1.7 batik-bridge-1.7 batik-css-1.7 batik-dom-1.7 batik-ext-1.7 batik-gvt-1.7 batik-js-1.7 batik-parser-1.7 batik-script-1.7 batik-svg-dom-1.7 batik-svggen-1.7 batik-transcoder-1.7 batik-util-1.7 batik-xml-1.7 bcprov-jdk15on-1.54 cglib-nodep-2.2 commons-beanutils-1.9.2-noclassprop commons-codec-1.9 commons-collections-3.2.2 commons-collections4-4.1 commons-dbcp-1.4 commons-fileupload-1.2.1 commons-httpclient-3.1 commons-io-2.1 commons-jxpath-1.3r commons-lang-2.6.ja Java Roadmap Planning Updated: ● JSR-275 → JSR-363 Update required: ● Spring 4.x → Spring 5.x ● Log4J 1.x → Log4j 3.x Review: ● Wicket 7.6.0 → ? ● GeoTools 19.0 → ? ● ImageIO 1.1.18 → ? miglayout-3.7-swing net.opengis.fes-18.3 net.opengis.ows-18.3 net.opengis.wcs-18.3 net.opengis.wfs-18.3 net.opengis.wmts-18.3 org.eclipse.emf.common-2.12.0 org.eclipse.emf.ecore-2.12.0 org.eclipse.emf.ecore.xmi-2.12.0 org.eclipse.xsd-2.12.0 org.json-2.0 org.w3.xlink-18.3 picocontainer-1.2 pngj-2.0.1 postgresql-42.1.1 slf4j-api-1.6.4 slf4j-log4j12-1.6.4 sqlite-jdbc-3.20.0 stax-1.2.0 stax-api-1.0.1 wicket-core-7.6.0 wicket-dnd-0.7.3 wicket-extensions-7.6.0 wicket-request-7.6.0 wicket-util-7.6.0 xml-apis-1.4.01 xml-apis-ext-1.3.04 xml-commons-resolver-1.2 xmlpull- xpp3_min-1.1.4c xpp3- xstream-1.4.10
  72. 72. The Java Advanced Imaging library enables GeoServer to do all kinds of great image processing! The JAI-EXT project extends this library with replacement no-data / footprint aware operations. We are working with LocationTech to ask Oracle to release this codebase to us as an open source project. We expect to hear back ... (see Replace-JAI on GeoTools wiki) Raster Processing Engine / Java Advanced Imaging Joint OSGeo / LocationTech game plan: ● Raster Processing Engine API definition of raster-processing-api ● Raster Processing Engine reference pure-java implementation ● Raster Processing Wrapper benchmark jai-wrapper implementation ● GeoTools Replace JAI migrate to raster processing engine api and coordinate transition affected projects (GeoServer, GeoNode, uDig, GeoScript, etc...)
  73. 73. Thanks!
  74. 74. Earth observation for EO
  75. 75. OpenSearch for Earth Observations ● Recent addition to OGC protocol for search of earth imagery ● Simpler alternative to CSW with ebRIM ● Two steps search ○ First find collection of similar products (by satellite/sensor) ○ Then search products in it (by other attributes, e.g., cloud cover, off nadir) ● Keeps database of searchable metadata ● Links to OGC services providing said data ● Allows downloading ISO/O&M metadata 2.12GeoSolutionsDLR/EVO-ODAS
  76. 76. OpenSearch for Earth Observations ● Result is RSS with human readable descriptions and ○ Links to OGC services providing said data ○ Links to download ISO/O&M metadata ○ Links to download the raw data ● Exposes one feature type per collection, allowing easy mosaic setup with same filtering and ordering 2.12GeoSolutionsDLR/EVO-ODAS
  77. 77. OpenSearch for Earth Observations parentId=SENTINEL2 &cloudCover=30] ● E.g., search all SENTINEL2- images with cloud cover less than 30% 2.12GeoSolutionsDLR/EVO-ODAS
  78. 78. OpenSearch for EO: admin REST API ● Create/Update/Delete the collections and its information ● Create/Update/Delete the product and its information ● Either one shot (throw a zip file with all the info) or granular (edit searchables first, then thumbnail, then metadata, links, …) Swagger documentation 2.12GeoSolutionsDLR/EVO-ODAS