Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

State of GeoServer - FOSS4G 2016


Published on

State of GeoServer provides an update on our community and reviews the new and noteworthy features for the Project. The community keeps an aggressive six month release cycle with GeoServer 2.8 and 2.9 being released this year.

Each releases bring together exciting new features. This year a lot of work has been done on the user interface, clustering, security and compatibility with the latest Java platform. We will also take a look at community research into vector tiles, multi-resolution raster support and more.

Attend this talk for a cheerful update on what is happening with this popular OSGeo project. Whether you are an expert user, a developer, or simply curious what these projects can do for you, this talk is for you.

Published in: Technology
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ ◀ ◀ ◀ ◀
    Are you sure you want to  Yes  No
    Your message goes here

State of GeoServer - FOSS4G 2016

  1. 1. State of GeoServer 2016 FOSS4G Bonn
  2. 2. Andrea Aime Technical Lead @geowolf GeoSolutions Innovative, robust and cost-effective solutions leveraging best-of-breed Open Source products. Welcome Open Source Projects GeoTools GeoServer ImageIO-Ext JAI-Ext JAI Tools OSGeo Foundation Charter Member
  3. 3. Jody Garnett Community Lead @jodygarnett Boundless Provides geospatial tools and services for managing data and building applications. Welcome Open Source Projects GeoTools GeoServer uDig OSGeo Foundation Board Member OSGeo Incubation Chair GeoTools Project Officer Eclipse Foundation LocationTech Steering Committee LocationTech Technology Project
  4. 4. GeoServer 2016
  5. 5. GeoServer at a Glance Java Web Application to share and edit geospatial data. Publish data from any major spatial data source using open standards. Core Protocols WMS – maps WFS – vector WFS-T – editing WCS – coverage WPS – process Tile Protocols WMTS – tiles TileService – tiles WMS-C – tiles
  6. 6. 28 → 37 members of team geoserver 624 → 619 members on geoserver-devel 500+ → 525 pull requests New contributors being added (others stepping back) Smaller base of active contributors this year. Pull requests still going strong, very active codebase. Simone replaces Andrea as OSGeo Officer. 13 releases this year Health Checkup How about team user-list? 1983 → 2005 geoserver-users Also geoserver-italia, espanol and mailing lists
  7. 7. August 2.8.5 2.9.1 July June 2.8.4 May 2.9.0 April 2.9-beta2 March 2.7.6 2.8.3 February January 2.8.2 December 2.7.5 November 2.7.4 2.8.1 October 2.7.3 September 2.8.0 Releases stable maintenance } delay
  8. 8. Q: “Update the headers” was the most common feedback on incoming pull requests. Was it needed? A: Checked with OSGeo legal (thanks board): - headers are an historical artifact from before the Berne Convention - no longer needed, now just informative. Header Maintenance
  9. 9. /* (c) 2016 Open Source Geospatial Foundation - all rights reserved * This code is licensed under the GPL 2.0 license, available at the * root application directory. */ package org.geoserver.main; New Headers Policy The new copyright header management policy is as simple as: ● Add a copyright header, with the current year, to each new file ● Add a copyright header to each modified file without one (using the current year) ● Otherwise do not change the copyright header when an existing file is modified
  10. 10. Maintenance & Technical Debt
  11. 11. ● Relying on volunteer time to keep an eye on the codebase was not working out ○ team busy on customer work? ○ backlog of bugs piling up ● Trying an experiment ○ Once a month meeting ○ Distributed, on-line code sprint ● When: ○ First edition in July ○ Second at the end of FOSS4G Monthly bug fixing code sprints
  12. 12. Monthly bug fixing code sprints ● Actions ○ Vet existing tickets ○ close invalid/non reproducible ones ○ close feature requests that failed to gather resourcing in a long time... ○ Fix bugs ● Nice side effects: ○ Gets more developers familiar with the contents of the issue tracker ○ Opportunity to get proficient with new parts of the code base ○ Predictable, scheduled activity
  13. 13. Wicket 7 Held sprint in victoria to clear up this long standing technical debt! Thanks to OSGeo, Boundless, Vivid Solutions, How 2 Map, San Jose Water Company, Transient, GeoBeyond, Scitus, GeoSolutions, CCRi, Astun Technology, and Voyager Search for sponsoring this work. 2.9 2.10 Victoria Code Sprint
  14. 14. Java 8 GeoServer now requires Java 8. Why? What changed ... - Traditionally GeoServer takes forever to support the latest Java - Oracle now charges for Java 7 security updates 2.9 2.10Joint EffortCommunity
  15. 15. Required to upgrade the “Spring Framework” used to wire GeoServer together: - We needed a newer version of “Spring” for Java 8 compatibility - Upgraded to Spring 4 → 2 month delay to GeoServer 2.9 release Check that your application server supports: - Servlet 3.0 was released in 2009 - Supported by Tomcat 7 and Tomcat 8 Spring 4 2.9 2.10Joint-EffortCommunity
  16. 16. Vector data sources
  17. 17. GeoPackage module moving to supported land ● Currently a “community module”, not officially supported ● Increasing test coverage ● More compatibility testing with OGR and friends ● OGC compliance tests ● Moving towards supported status, hopefully in time for 2.10 2.10 Ian Turton GeoSolutions Swedish University of Agricultural Sciences
  18. 18. WFS cascading updates ● The original WFS client is being retired in 2.10 ● WFS-NG will be taking over ● Compatibility testing done with MapServer, TinyOWS, …. 2.10 Scitus Development Boundless TinyOWS ArcGIS WFS WFS/WMS/WMTS WFS
  19. 19. Raster data sources
  20. 20. Improved masking support ● Raster masking available for both image mosaic and GDAL data sources ● Both in vector (polygon footprint) and raster (binary mask) for 2.8 2.9 2.10GeoSolutionsLiguria digitale
  21. 21. Heterogeneous mosaics - color models ● Support for mosaics with sources in multiple color models (RGB, gray, indexed), since 2.8.x 2.8 2.9 2.10GeoSolutions
  22. 22. Heterogeneous mosaics - resolution / projections 2.10BoundlessSOCOM ● Support for mosaics in multiple coordinate reference systems (Coming in 2.10.x)
  23. 23. Excess granule removal ● Optimization for deep z-order mosaic stacks ● E.g., time series of scattered data ● Do not load imagery that is not contributing to the output 2.9 2.10GeoSolutionsBEV In development!
  24. 24. Multiple coverages for mosaic and pyramid (2.10) ● Already available for multidimensional data (NetCDF, Grib) ● Now available also for “flat” data sources ● Meaningful if the sources are normally managed and used together (e.g., multispectral satellite data and its derived products) 2.9 2.10ImplementorBEV
  25. 25. Coverage view optimizations (2.10) ● A coverage view allows to bind together bands from different coverages coming from the same source (e.g., NetCDF) ● Optimization to avoid reading all sources if the style is only using a subset of the bands A B C Combiner SLD Style selecting only A Map Hint about which bands are actually going to be used GeoSolutionsBEV 2.9 2.10 In development!
  26. 26. GeoServer support for rotated pole projections 2.9 2.10 Rotated pole projection support for GRIB2/NetCDF/ImageMosaic data stores and WCS NetCDF output. Publish the native GRIB2 file format of NOAA RAPv3 North American weather forecast model: Upgraded to NetCDF Java 4.6.6 (Ben contributed a fix in it) TransientBoundless/NOAA
  27. 27. Styling
  28. 28. Perpendicular Offset for Lines and Polygons <LineSymbolizer> <Stroke> <CssParameter name="stroke">#AAAAAA</CssParameter> <CssParameter name="stroke-width">3</CssParameter> </Stroke> <PerpendicularOffset>-2</PerpendicularOffset> </LineSymbolizer> <LineSymbolizer> <Stroke> <CssParameter name="stroke">#FF0000</CssParameter> <CssParameter name="stroke-width">3</CssParameter> <CssParameter name="stroke-dasharray">5 2</CssParameter> </Stroke> <PerpendicularOffset>3</PerpendicularOffset> </LineSymbolizer> 2.8 2.9 2.10GeoSolutionsDLR
  29. 29. ncWMS like extensions to GetMap (community) ● A new styling language designed for dynamic colormaps ● Applies to all rasters based on their statistics ● Use can control min/max, log scale, animation 2.10GeoSolutions
  30. 30. Style Page 2.10BoundlessCommunity Refactored CSS Style Page into the main application. Works with SLD, CSS, YSLD. Improved experience for everybody! In development!
  31. 31. New features for CSS (2.10) ● Rendering transformations supported (feature parity with SLD!) ● Rule nesting, helps with selector duplication and makes styles more readable /* @title Levels */ * { transform: ras:Contour(levels: 1100 1200 1300 1400 1500 1600 1700); stroke: black; z-index: 0; } /* @title Values */ [@scale < 3000] { transform: ras:RasterAsPointCollection(); mark: symbol('square'); :mark { size: 2; fill: black; } label: [GRAY_INDEX]; label-anchor: 0 0.5; label-offset: 5 0; font-family: Arial; font-fill: black; z-index: 1; } Extract contours at the desired levels When zoomed-in enough, extract point values, display center with a mark, and value on top of it 2.10Andrea Aime In development! Community
  32. 32. YSLD community module ● Style Layer Descriptor represented in YAML ○ Uses indentation rather than XML ○ Easier to read ○ More compact ○ More flexible syntax ○ Contains variables for reusable code ○ Compatible with SLD ● REST API can convert formats ○ Preview your existing styles as YSLD using REST API ○ View the SLD representation of YSLD 2.9 2.10Implementor symbolizers: - polygon: stroke-color: 'blue' stroke-width: 1 fill-color: '#7EB5D3' - text: label: ${name} fill-color: 'black' anchor: [0.5, 0.5] x-maxDisplacement: 40 x-autoWrap: 70 Community
  33. 33. Custom Legend Graphic Take control of the legend graphic (Advertised in WMS GetCapapabilities). Legend shown in compatible clients when listing your layer. 2.9 2.10BoundlessBoundless
  34. 34. WMS
  35. 35. image/vnd.jpeg-png ● Smart format, chooses between JPEG and PNG depending on image contents ● JPEG for max compression on fully opaque images ● PNG when transparency is needed ● Supported both in WMS and WMTS PNG PNG PNG PNG JPEG JPEG 2.9 2.10GeoSolutions
  36. 36. UTFGrid (format=application/json;type=utfgrid) ● MapBox style UTFGrid output for WMS and WMTS ● More flexibility ○ Non square tiles ○ Available in all projections ● Of course you can limit yourself to simple tiles 2.9 2.10GeoSolutionsCianad
  37. 37. Vector tiles Community module for publishing vector tiles - Great solution for modern High Definition Screens - Use for GeoWebCache for “MapBox without MapBox” - (see talk tomorrow) 2.8 2.9 2.10ImplementorCustomer
  38. 38. GetLegendGraphic layout control ● Vendor parameters to control layout of generated legend: layout, rows, cols, rowwidth, columnheight ● Horizontal or vertical, possibility to control number of columns ● Handy for printing maps horizontal horizontal, 2 rows horizontal, max row len vertical, 2 cols vertical, max col height 2.8 2.9 2.10GeoSolutionsUNEP
  39. 39. WFS
  40. 40. Breaking the limits of shapefile size (paging) ● A WFS shapefile extraction could go beyond the limits of shapefile max size ● If this happens, shapefiles are now paged out ● The dumping engine, including the ability to fan out different geometry types to separate shapefiles, is now available as a utility class in GeoTools 2.8 2.9 2.10GeoSolutions Swedish University of Agricultural Sciences
  41. 41. Easier setup for complex features with Hale ● Desktop tool to map a flat data source to a complex schema ● Visual ● Support for exporting the mapping as an app-schema mapping ● Still needs some work to improve workflow efficiency 2.8 2.9 2.10GeoSolutionsLAMMA
  42. 42. Speed up that complex feature query ● Filters on primary table fields already sent down to the DBMS as SQL ● Filters on joined fields were evaluated in memory ● Optimizations to send those down, leverage indexes and reduce network i/o 2.8 2.9 2.10GeoSolutionsCSI Piemonte In memory filtering DBMS filtering In development!
  43. 43. Tiling
  44. 44. MBTiles like storage ● Store tiles in MBTiles database ● Possibility to use many databases, following a pattern, e.g., ○ /path/to/{grid}/{dim}/{tileset}/{z}/{x}-{y}.sqlite (x and y are actually groups of values) ○ Helps with scalability, especially if not fully seeded ● Allows to store non Google Mercator projections (extension) ● Allows to store formats other than png or jpeg (extension) ● Stores in lookup table creation date for expiration support ● REST API to replace single files (automate migration from seeding machine to serving ones) ● Not compatible with disk quota! 2.9 2.10GeoSolutionsCapgemini
  45. 45. WMTS config page ● Allows WMTS to have its own service metadata configured from the UI ● Adds a point where the INSPIRE module can plug into for extra configuration 2.9 2.10GeoSolutionsBEV
  46. 46. WMTS workspace specific and INSPIRE compatibility ● WMTS now works also as a workspace specific service (finally tile caching for multitenant setups) ● Allows to setup INSPIRE harmonized layer names ● INSPIRE module extended to inject INSPIRE minimum metadata in WMTS GetCapabilities output Workspace 1 Workspace 2 INSPIRE compliant Generic tiles 2.9 2.10GeoSolutionsBEV
  47. 47. WMS/WMTS ND discovery extensions ● Data with N dimensions (time, elevation, custom ones) ● Dimensions can be related (forecasts, time and run time), the data can be scattered (remote sensing), hard to locate data in the ND cube ● Add extra calls to query the domains ● Check the current specification and give us feedback 2.9 2.10GeoSolutionsBEV In development!
  48. 48. WPS
  49. 49. Aggregation with group-by (2.9) ● The aggregation process has been extended with group-by capabilities ● Quick way to power custom charts on your client side { "GroupByAttributes": [ "groupingAttribute" ], "AggregationResults": [ [ "Class1", 18 ], [ "Class2", 1 ], [ "Class3", 3 ], [ "Class4", 3 ] ], "AggregationFunctions": [ "Count" ], "AggregationAttribute": "countingAttribute" } 2.8 2.9 2.10GeoSolutionsLiguria Digitale
  50. 50. Improved WPS resource control (2.10) ● Processes are run in thread pools, they can wait for their execution slot in queue ● Separate control for total time, and pure execution time
  51. 51. Download process improvements (community module) ● Community module for clip and ship support ● Asynch requests allow for large extractions ● Can have maximum extraction limits configured ● Added ability to also select bands, and improved output size estimation WPS 2.9 2.10GeoSolutionsBEV
  52. 52. Large raster extraction optimizations ● Optimizing many concurrent large masked raster extractions via Download process (e.g. 128 concurrent requests at the bigtiff threshold) ● Improved both scalability and reliability under sustained load on a “big iron” (40 cores, 128GB memory) 2.9 2.10GeoSolutions 2.8
  53. 53. Configuration and management
  54. 54. ● Allows GeoServer to store resources (icons, fonts, config files) in either file system or database “blobs”. ● Run without a “Data Directory” (Helpful in clustered/cloud environment) ● All instances of file access were converted to resource api ● Backwards compatible - file will be unpacked on disk if required ● Manage via REST API ● GUI available as community module Resource API 2.8 2.9 2.10 Scitus Development Boundless/NGA Web Services Configuration Data Directory ResourceCatalog Web Services Configuration JDBCStore ResourceCatalog
  55. 55. Web interface in Tools / Resource Browser ● Manage and edit resources ● Data Directory or JDBCStore ● Community module ● Packaged plugin coming soon Resource browser community module 2.10 Scitus Development Boundless/NGA
  56. 56. Module “status” REST API (and GUI) ● Automated testing of GeoServer is a challenge ● REST API access to the “status page” ● Easy to add additional entries ● Used to check environment for difficult to configure modules that use native code 2.9 2.10BoundlessBoundless QA Module name: Rendering Engine ● Module : jvm ● Component : java2d ● Version : 1.8.0_66 ● Enabled : true ● Available : true ● Message : Java 2D configured with DuctusRenderingEngine. ● Provider: OracleJDK geoserver/rest/about/status In development!
  57. 57. Parametric configuration ● Switching data directories between integration/beta/production environments can be a pain ● GeoServer 2.10 allows to stick parameters in the configuration, and read them from the environment ● Look ma, switching environments without grep/sed on the data dir :-) ● Store params in property file, system or environment variables 2.9 2.10GeoSolutions Eumetsat World Bank
  58. 58. Backup/restore community module ● Save the configuration to a zip file (not the data!) ● Restore later, in the same or different environment ● UI to run backup and restore ● Asynch operations ○ During backup the configuration is forced read only ○ During restore it’s fully locked 2.9 2.10GeoSolutions Eumetsat World Bank
  59. 59. Backup/restore community module (cont'd) ● Can do a “dry run” to check if there is any error prior to restore 2.9 2.10GeoSolutions Eumetsat World Bank
  60. 60. Backup/restore community module (cont'd) ● REST API to run, control, eventually cancel backups and restores 2.9 2.10GeoSolutions Eumetsat World Bank
  61. 61. Security
  62. 62. LDAP User Group Service ● We previously only had an “authentication provider”, meaning, it used the username/pw provided and tried to open a LDAP connection with them ● We now also have a “user group service”,it opens a LDAP connection using fixed credentials, and searches the user in the LDAP database, matching the password ● E.g., similar to storing users in a DBMS 2.10 Scitus development ?
  63. 63. Integrated GeoFence community module ● Allows to define complex rules without running a separate GeoFence server ● UI to edit rules ● REST API to automate rule editing 2.8 2.9 2.10 Scitus development Boundless
  64. 64. Integrated GeoFence community module (cnt’d) ● Separate page for “admin” rules, that is, rules controlling administration level access ● Used to setup workspace specific admins and the like 2.8 2.9 2.10GeosolutionsCamp2Camp
  65. 65. Research, Development and other news
  66. 66. GeoGig GUI Manage repositories and publish via WFS and WFS-T for web client access. 2.9 2.10BoundlessGeoGig
  67. 67. GeoGig Web API curl -X PUT -H "Content-Type: application/json" -d '{ "parentDirectory": "/opt/geogig/repos", "authorName": "geogig", "authorEmail": "" }' "http://localhost:8080/geoserver/geogig/repos/myrepo/init" REST API for managing repositories, when combined with GeoServer REST API a wide range of scripts are possible. 2.9 2.10BoundlessGeoGig
  68. 68. Looking Ahead
  69. 69. Java 9 Compatibility Java 9 has been delayed until 2017, but we would like to ensure GeoServer will function. Oracle will stop shipping free security updates to Java 8 on We have a couple incompatibilities with the GeoTools “service provider interface” plugin system that we will look at resolving during the code sprint!
  70. 70. The Java Advanced Imaging library enables GeoServer to do all kinds of great image processing! The JAI-EXT project extends this library with replacement no-data / footprint aware operations. Sadly Oracle has not maintained the project, and has not released it as open source. (see Replace-JAI on GeoTools wiki) Raster Processing Engine Joint OSGeo / LocationTech game plan: ● Raster Processing Engine API definition of raster-processing-api ● Raster Processing Engine reference pure-java implementation ● Raster Processing Wrapper benchmark jai-wrapper implementation ● GeoTools Replace JAI migrate to raster processing engine api and coordinate transition affected projects (GeoServer, GeoNode, uDig, GeoScript, etc...)
  71. 71. Thanks!