Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

State of GeoServer 2.12

4,648 views

Published on

State of GeoServer provides an update on our community and reviews the new and noteworthy features for the Project. The community keeps an aggressive six month release cycle with GeoServer 2.11 and 2.12 being released this year. Each releases bring together exciting new features. This year a lot of work has been done on startup times, large catalogs, REST configuration upgrades, styling languages, styling interface, security,just to name a few. We will also take a look at community research into satellite imagery search and delivery, multi-resolution raster support and more. Attend this talk for a cheerful update on what is happening with this popular OSGeo project. Whether you are an expert user, a developer, or simply curious what these projects can do for you, this talk is for you.

Published in: Technology

State of GeoServer 2.12

  1. 1. State of GeoServer 2.12
  2. 2. Andrea Aime Technical Lead andrea.aime@geo-solutions.it @geowolf GeoSolutions Innovative, robust and cost-effective solutions leveraging best-of-breed Open Source products. Welcome Open Source Projects GeoTools GeoServer ImageIO-Ext JAI-Ext JAI Tools OSGeo Foundation Charter Member GeoServer PSC GeoTools PSC
  3. 3. Jody Garnett Community Lead jgarnett@boundlessgeo.com @jodygarnett Boundless Provides geospatial tools and services for managing data and building applications. Welcome Open Source Projects GeoTools GeoServer uDig OSGeo Foundation Board Member OSGeo Incubation Chair GeoTools Project Officer Eclipse Foundation LocationTech Steering Committee LocationTech Technology Project
  4. 4. Open Source Projects GeoTools GeoServer GeoWebCache OSGeo Foundation GeoServer PSC GeoWebCache Lead Kevin Smith Software Engineer ksmith@boundlessgeo.com smithkm Boundless Provides geospatial tools and services for managing data and building applications. Welcome
  5. 5. GeoServer 2017
  6. 6. GeoServer at a Glance Java Web Application to share and edit geospatial data. Publish data from any major spatial data source using open standards. Core Protocols WMS – maps WFS – vector WFS-T – editing WCS – coverage WPS* – process CSW* – search Tile Protocols WMTS – tiles TMS – tiles WMS-C – tiles
  7. 7. 28 → 37 members of team geoserver (to be updated 624 → 619 No clue this year (check with Kevin) 500+ → 525 pull requests New contributors being added (others stepping back) Smaller base of active contributors this year. Pull requests still going strong, very active codebase. Simone replaces Andrea as OSGeo Officer. 13 releases this year GeoServer Project Status How about team user-list? 2005 → 1400 (SF now checking on people not active on list) Also geoserver-italia, espanol and opengeo.cn mailing lists
  8. 8. Release Schedule stable maintenance July 2.12.5 June May 2.12.4 April March 2.12.3 February 2.11.5 January 2.12.2 December 2.11.4 November 2.12.1 October 2.11.3 September 2.12.0 August 2.10.5 2.11.2
  9. 9. User List Participation ● Answering users questions relies on a low number of people Action: ● We are sending reminder on effective use of the user list on subscription and at the bottom of each mail ● Community building activity - remind developers about the definition of “supported” (not only reviewing pull requests, also answering user questions, used to be obvious, but it’s not obvious to devs hired into open source)
  10. 10. ● Bug stomp introduced in 2016 ○ Relying on volunteer time not working out ○ allowed developers to reserve time ● Moved to fixed schedule in 2017 ○ Arranging day each month too random ● Last Friday of each month ○ Check blog post for tips on participation Standing “last friday of the month” bug stomp
  11. 11. Held “sprint” to update from Restlet to Spring MVC for our REST configuration API. Thanks to GeoSolutions for hosting, to Boundless and Astun for sending developers, and to all the sponsors who helped fund this important and badly needed, but boring work. REST API Migration 2.12Viareggio Code SprintCommunity
  12. 12. REST API Migration ● Migration complete ● REST API documented ○ Available online now! 2.12Viareggio Code SprintCommunity
  13. 13. Vector data sources
  14. 14. GeoPackage: ● Provides an sqlite database containing vector or raster data ○ Initial idea from spatialite project ○ Now an OGC Standard to replace shapefile and personal file geodatabase ● GeoServer module ○ GeoPackage is a supported plugin in GeoTools ○ We can now promote it to a GeoPackage extension ○ Needs some work, splitting into 4 GeoServer packages, store, wfs output format, wms output format, wps output format ○ Bug fix - raster mosaic geopackage had y-axis inverted. GeoPackage (community) 2.12 GeoSolutions Boundless Community Boundless Suite
  15. 15. ● GeoServer importer supports GeoJSON files ○ Allows GeoJSON to be imported into PostGIS database and published in one step ○ Based on unsupported module in GeoTools ● Wish to package as a community module for GeoServer ○ Allow GeoServer to directly publish GeoJSON files ○ Text based, non indexed format, only suitable for small files GeoJSON store 2.12Ian TurtonCommunity
  16. 16. ● GeoServer importer supports CSV files ○ Allows CSV to be imported into PostGIS database and published in one step ○ Based on unsupported module in GeoTools (used as a tutorial for programmers) ● Package as a community module for GeoServer ○ Allow GeoServer to directly publish CSV files ○ Text based, non indexed format, only suitable for small files CSV store 2.12BoundlessCommunity
  17. 17. CSV input/output for WPS ● Allows the input and output of CSV for use in WPS processes ● This will lead to GeoTools CSV Datastore being upgraded to community (and hopefully to supported) ● Come see more at “Building a Table Joining Like Service with Web Processing Services” Friday Afternoon 2.12 Ian Turton Astun RnD
  18. 18. Raster data and processing
  19. 19. ● Initial version in 2.11 ● Several fixes in 2.12, e.g. accuracy, performance, dateline crossing handling ● Being tested against global mosaics of Sentinel2 (each image in a different UTM zone), more improvements/fixes incoming Heterogeneous CRS image mosaics 2.11/2.12Boundless/GeoSolutionsRnD/DLR UTM60 image footprints UTM1 image footprints Dateline wrapped image mosaic
  20. 20. ● Reprojected images could have shown border effects Better handling of reprojection boundaries 2.10/2.11GeoSolutionsCommunity
  21. 21. ● Higher order interpolation did not play well with nodata, causing border effects: ○ Bilinear ○ Bicubic ● Requires jai-ext enabled Improved no-data handling in interpolation 2.10/2.11GeoSolutionsCommunity Before After
  22. 22. ● Respect scale/offset attributes ● Improved support for related dimensions (e.g. time/runtime in forecast), automatically finding the best related value when just one dimension is provided NetCDF reader improvements 2.11 Niels/Devon/Ben/Daniele Scitus/Boundless/Transient/GeoSolutions Customer Runtime
  23. 23. S3 GeoTIFF Community Module Support for GeoTiffs hosted on Amazon S3 ● Reader based off of the GeoTiff reader. ● Very basic caching of images from S3 based off of EhCache ● InputStream implementation from JAI for reading imagery from S3 2.12 Devon Boundless Boundless s3://landsat-pds/L8/001/002/LC80010022016230LGN00/LC8001 0022016230LGN00_B1.TIF?useAnon=true&awsRegion=US_WEST_2
  24. 24. Styling
  25. 25. Style Page 2.10BoundlessCommunity Refactored CSS Style Page into the main application. Works with SLD, CSS, YSLD, Mapbox Improved experience for everybody!
  26. 26. New features for CSS (2.10) ● Rendering transformations supported (feature parity with SLD!) ● Rule nesting, helps with selector duplication and makes styles more readable /* @title Levels */ * { transform: ras:Contour(levels: 1100 1200 1300 1400 1500 1600 1700); stroke: black; z-index: 0; } /* @title Values */ [@scale < 3000] { transform: ras:RasterAsPointCollection(); mark: symbol('square'); :mark { size: 2; fill: black; } label: [GRAY_INDEX]; label-anchor: 0 0.5; label-offset: 5 0; font-family: Arial; font-fill: black; z-index: 1; } Extract contours at the desired levels When zoomed-in enough, extract point values, display center with a mark, and value on top of it 2.10Andrea AimeCommunity
  27. 27. New features for CSS (2.12) ● Code completion in the editor ● LessCSS compatible color functions ○ Darken, lighten, saturate, …. ● More readable scale denominator rules ○ E.g. @sd < 100k ● Impressive CSS->SLD translation speedup ○ 50 times faster on large CSS sheets ● Easier env variable integration (in progress, not yet landed) ○ @varname or @varname(defaultValue) 2.10Andrea AimeCommunity
  28. 28. YSLD extension module ● Style Layer Descriptor represented in YAML ○ Uses indentation rather than XML ○ Easier to read ○ More compact ○ More flexible syntax ○ Contains variables for reusable code ○ Compatible with SLD ● REST API can convert formats ○ Preview your existing styles as YSLD using REST API ○ View the SLD representation of YSLD 2.11Boundless symbolizers: - polygon: stroke-color: 'blue' stroke-width: 1 fill-color: '#7EB5D3' - text: label: ${name} fill-color: 'black' anchor: [0.5, 0.5] x-maxDisplacement: 40 x-autoWrap: 70 Community
  29. 29. ● JSON representation of styling ○ More limited than SLD/YSLD/CSS ● Works well with vector tiles ○ Cross platform styling (OpenLayers, MapBoxGL) ● Work in progress: ○ Currently a GeoTools community module ○ Should package and go to extension if we get time! MapBox Style Community Module 2.12David VickBoundless Suite MapBox Style in GeoServer and OpenLayers
  30. 30. Labelling improvements ● Kerning on by default (adjusts spacing between chars, e.g. Check space between T and r in Trimble) ● Underline and Strikethrough ● Char and word spacing control ● Work in progress: ○ Labelling border of polygon (instead of the centroid) ○ Filtering labels based on polygon screen size in pixels 2.11 2.12 Andrea Aime Nuno Oliveria Community/Geoinfo/ OpenGeoGroep
  31. 31. Improved QGIS Compatibility ● Additional custom “Well Known” marks ● Support for SLD 1.1 External Marks (e.g. TTF font based) ● SVG Parameters 1.0 spec <PointSymbolizer> <Graphic> <ExternalGraphic> <OnlineResource xlink:type="simple" xlink:href="firestation.svg?fill=#FF0000" /> <Format>image/svg</Format> </ExternalGraphic> <size>128</size> </Graphic> </PointSymbolizer> 2.11Andrea AimeCommunity
  32. 32. ● Label export, most shared configurations exported ● Some in the works (word/char spacing, strikethrough, polygon border labelling) ● Some changes needed in GeoServer SE 1.1 parser too QGIS 3.0 style export improvements 2.12 Andrea Aime GeoSolutions OpenGeoGroep
  33. 33. WMS
  34. 34. Vector Tiles Extension Extension for publishing vector tiles - Great solution for modern High Definition Screens - Use for GeoWebCache for “MapBox without MapBox” - Couple with MapBox style for consistent styling 2.11Gabriel RoldanBoundless Suite
  35. 35. Better centroid / label placement in “vector” outputs KML ● generate placemarks that are always inside polygon (for concave polygons) Vector Tiles ● Pregenerate label points in tiles ○ Reduce effort on client 2.12 Kevin Smith Just DeOlivera RnD Label Labelvs
  36. 36. WMS sortBy vendor param ● Sort by recency, cloud cover, and so on ● Supported by vector data and image mosaic 2.12 Andrea Aime GeoSolutions Community Sort by recencySort by cloud cover
  37. 37. ncWMS like extensions to GetMap (community) ● A new styling language designed for dynamic colormaps ● Applies to all rasters based on their statistics ● Use can control min/max, log scale, animation ● Part of OGC TestBed 12 activities 2.10GeoSolutionsOGC
  38. 38. WFS
  39. 39. “null” prefix bug finally solved ● Occasionally GeoServer could return WFS GML with “null” as the feature prefix ● Solved for simple features ● Solved for complex features on global services 2.10 Nuno Oliveira GeoSolutions IGEA/NLS
  40. 40. App-schema improvements ● Improvements to connection pool usage (only one connection used per request now) ● Some improvements helping while using MongoDB as a source for app-schema records ● Hale mapping support being extended to support MongoDB 2.12 Nuno Oliveira GeoSolutions NPRA
  41. 41. GeoJSON output for complex features ● Complex features not limited to GML anymore 2.10 Nuno Oliveira GeoSolutions NPRA { "type":"FeatureCollection", "totalFeatures":"unknown", "features":[ { "type":"Feature", "id":"gu.93", "geometry":{ "type":"Polygon", "coordinates":[ ... ] }, "properties":{ "description":"Olivine basalt, tuff, microgabbro, minor sedimentary rocks", "occurrence":[ { "shape":{ "type":"Polygon", "coordinates":[ ... ] }, "name":"urn:x-test:GeologicUnit:16777549126931093", "observationMethod":{ "CGI_TermValue":{ "value":"urn:ogc:def:nil:OGC:missing" } }, "positionalAccuracy":{ "CGI_TermValue":{ "value":"urn:ogc:def:nil:OGC:missing" } }, { "shape":{ "type":"Polygon", "coordinates":[ ... ] }, "name":"urn:x-test:GeologicUnit:16777549126931093", "observationMethod":{ "CGI_TermValue":{ "value":"urn:ogc:def:nil:OGC:missing" } }, "positionalAccuracy":{ "CGI_TermValue":{ "value":"urn:ogc:def:nil:OGC:missing" } }, "specification":{ } } ], "name":"urn:x-test:GeologicUnit:16777549126931093" } // properties ends here }
  42. 42. WCS
  43. 43. Vendor param for mosaics: cqlFilter and sortBy ● Sort by recency, cloud cover, and so on ● Filter on the same ● Maps to image mosaic read parameters ● Uses attributes in the mosaic index ● Consistent with WMS now, first see and then download 2.12 Andrea Aime GeoSolutions Community
  44. 44. NetCDF input/output improvements 2.12 Niels Scitus Community ● Can configure NCML files as sources ● Support variable's attributes definition in NetCDF output ● Preserve scalars from input (NetCDF in/NetCDF out)
  45. 45. Tiling
  46. 46. ● GeoServer supports “cascading” remote WMTS services ○ GeoServer acts as a client, rather than just rerouting each request ○ Same as WMS cascading ● WMTS cascade is a work in progress ○ Should land in 2.12 WMTS cascading 2.12 Ian Turton/Emanuele Tajariol GeoSolutions Customer
  47. 47. Parameter filter cache drop improvements ● GeoWebCache can cache based on parameters like style, time, or elevation. ○ Could not truncate (remove cached tiles) across all parameter values, each must be known and truncated separately. ○ Tiles could be orphaned if the allowed values changed. ○ GeoServer can truncate tiles when the data within them changes, but only caught the default values for parameters other than style ● Added operations for ○ Cross parameter truncate ○ Orphan purge ● Enabled cross parameter truncate on data update GWC 1.12 GS 2.12 Kevin Smith Boundless MapStory
  48. 48. WMS/WMTS multidim domain discovery module (community) ● Data with N dimensions (time, elevation, custom ones) ● Dimensions can be related (forecasts, time and run time), the data can be scattered and found in pockets, hard to locate data in the ND cube ● Add extra calls to query the domains ● Developed as part of OGC Testbed 12 ● See the engineering report and give us feedback: 2.10 Andrea/Nuno GeoSolutions OGC http://docs.opengeospatial.org/per/16-042r1.html#WMTS_Multidimensional_GeoSolu tions
  49. 49. Configuration and management
  50. 50. Security Consideration - Disabled writing by default ● WFS-T allows editing using Transaction requests ○ May not be what you expect? ● Default configuration includes security restrictions on WFS-T functionality ● Restricting editing of data to the administrator account. ● Makes service read-only (while still advertising wfs-t) 2.11Nuno OliveiraCommunity
  51. 51. ● Great performance improvement! ● Test with 10000 layers Faster startup times for large data dirs 2.11AndreaCommunity Version COLD (Startup time seconds) HOT (Startup time seconds) 2.10.1 428 39 2.10.1 + JDCBConfig 62 49 GSIP 155 68 29
  52. 52. ● Great performance improvement! ● Tested on topp:states to make catalog lookup times more evident (assume diff is a constant) Faster runtime times for large data dirs 2.11AndreaCommunity Version Throughput req/sec Avg resp. time ms 2.10.1 169 47 2.10.1 + JDCBConfig 68 117 GSIP 155 233 34
  53. 53. ● When loading a shapefile the PRJ file only matched on AUTHORITY code ● Faster lookup implementation allows GeoServer to search the EPSG database and match far more often Better/Faster EPSG code lookups 2.11AndreaCommunity This reduces a common frustration when configuring new layers in GeoServer
  54. 54. Module “status” REST API (and GUI) ● Verifying what modules are installed was a challenge ● REST API access to the “status page” ● Easy to add additional entries ● Used to check environment for difficult to configure modules that use native code 2.11 Morgan Thompson Boundless Boundless Suite geoserver/rest/about/status
  55. 55. Parametric configuration ● Stick parameters in store configurations and some security bits ● Helps migrating a data dir between environments (e.g., integration/test/beta/production) ● Set the -DALLOW_ENV_PARAMETRIZATION=true system var ● Create a property file with variable values 2.10 Alessio Fabiani GeoSolutions Eumetsat World Bank store_url = file:///var/geoserver/store/teststore
  56. 56. Backup/restore (community) ● Save the configuration to a zip file (not the data!) ● Restore later, in the same or different environment ● UI to run backup and restore ● Asynch operations ○ During backup the configuration is forced read only ○ During restore it’s fully locked 2.10GeoSolutions Eumetsat World Bank
  57. 57. Web interface in Tools / Resource Browser ● Manage and edit resources ● Data Directory or JDBCStore ● Community module ● Packaged plugin coming soon ● REST API sporting equivalent functionality found in core Resource Browser (community) 2.10 Scitus Development Boundless/NGA
  58. 58. Security
  59. 59. OAuth (community) ● Allows GeoServer authentication to be provided by OAuth ● Allows users to login with: ○ GeoNode ○ Google ○ Github ○ … add yours here, it’s pluggable ● Developed for integration with GeoNode 2.10 Alessio Fabiani GeoSolutions Boundless
  60. 60. Security Considerations
  61. 61. Take care to report the issue in a responsible fashion: ● Keep exploit details out of issue report ● Mark the issue as a vulnerability. ● Be prepared to work with Project Steering Committee (PSC) on a solution ● Keep in mind PSC members are volunteers and an extensive fix may require fundraising / resources If you are not in position to communicate in public please consider commercial support, contacting a PSC member, or reaching us via the Open Source Geospatial Foundation at info@osgeo.org. Security Vulnerabilities - Responsible Disclosure
  62. 62. Responsible disclosure policy is working: ● Most users contacted us in a responsible fashion ● One user shared confidential customer report on the user list Where do reports come from? ● Majority reported from security audits ○ Usually performed under contract ○ Automated tools tend to focus on web administration ui, rather than web services ● GeoServer is subject to several security audits a year Security Vulnerabilities
  63. 63. Upgrade to get security fixes ● Security fixes being added to supported versions only ● Please upgrade your GeoServer installations!
  64. 64. Looking Ahead
  65. 65. Java 9 Compatibility Java 9 has been delayed until 2017, but we would like to ensure GeoServer will function. (Oracle stops free security updates to Java 8) We have a couple incompatibilities with the GeoTools “service provider interface” plugin system that we will look at resolving during the code sprint!
  66. 66. The Java Advanced Imaging library enables GeoServer to do all kinds of great image processing! The JAI-EXT project extends this library with replacement no-data / footprint aware operations. Sadly Oracle has not maintained the project, and has not released it as open source. (see Replace-JAI on GeoTools wiki) Raster Processing Engine Joint OSGeo / LocationTech game plan: ● Raster Processing Engine API definition of raster-processing-api ● Raster Processing Engine reference pure-java implementation ● Raster Processing Wrapper benchmark jai-wrapper implementation ● GeoTools Replace JAI migrate to raster processing engine api and coordinate transition affected projects (GeoServer, GeoNode, uDig, GeoScript, etc...)
  67. 67. Thanks!
  68. 68. Earth observation for EO
  69. 69. OpenSearch for Earth Observations ● Recent addition to OGC protocol for search of earth imagery ● Simpler alternative to CSW with ebRIM ● Two steps search ○ First find collection of similar products (by satellite/sensor) ○ Then search products in it (by other attributes, e.g., cloud cover, off nadir) ● Keeps database of searchable metadata ● Links to OGC services providing said data ● Allows downloading ISO/O&M metadata 2.12GeoSolutionsDLR/EVO-ODAS
  70. 70. OpenSearch for Earth Observations ● Result is RSS with human readable descriptions and ○ Links to OGC services providing said data ○ Links to download ISO/O&M metadata ○ Links to download the raw data ● Exposes one feature type per collection, allowing easy mosaic setup with same filtering and ordering 2.12GeoSolutionsDLR/EVO-ODAS
  71. 71. OpenSearch for Earth Observations http://cloudsdi.geo-solutions.it/geoserver/oseo/search? parentId=SENTINEL2 &cloudCover=30] ● E.g., search all SENTINEL2- images with cloud cover less than 30% 2.12GeoSolutionsDLR/EVO-ODAS
  72. 72. OpenSearch for EO: admin REST API ● Create/Update/Delete the collections and its information ● Create/Update/Delete the product and its information ● Either one shot (throw a zip file with all the info) or granular (edit searchables first, then thumbnail, then metadata, links, …) Swagger documentation 2.12GeoSolutionsDLR/EVO-ODAS

×