Processing data in GeoServer with WPS and SQL views

5,695 views

Published on

This presentation will provide the attendee with an introduction to data processing in GeoServer by means of WPS, rendering transformations and SQL views.
We will start by a brief introduction to GeoServer WPS capabilities, showing how to build processing request based on existing processes and how to build new processes leveraging scripting languages, and introducing unique GeoServer integration features, showing how processing can seamlessly integrate directly in the GeoServer data sources and complement existing services.
The presentation will move on showing how to integrate on the fly processing in WMS requests, achieving high performance data displays of heatmaps, point interpolation and contour line extraction without having to pre-process the data in advance, and allowing the caller to interactively choose processing parameters.
While the above shows how to make GeoSever perform the processing, the analytics abilities of spatial databases are not to be forgotten, the presentation will move on showing how certain classes of processing can be achieved directly in the database.
Eventually, the presentation will close with some guidance on how to choose the best processing approach depending on the application needs, data volumes and frequency of update, mentioning also the possibly to leverage GeoServer own processes from batch tools such as GeoBatch.
At the end the attendee will be able to easily issue WPS requests both for Vectors and Rasters to GeoServer trhough the WPS Demo Builder, enrich SLDs with awesome on-the-fly rendering transformations and play with virtal SQL views in order to create dynamic layers.

Published in: Technology

Processing data in GeoServer with WPS and SQL views

  1. 1. Processing Data In GeoServer With WPS And SQL Views Ing. Andrea Aime, GeoSolutions Ing. Alessio Fabiani, GeoSolutions FOSS4G-Europe 2014, Bremen 14th-17th July 2014
  2. 2. FOSS4G Europe 2014, Bremen 14th-17th July 2014 Some historical perspective  Year 2008:  Publish maps on the net  Nice styling, maybe time/elevation based selector, some little extra filtering  Maybe some editing, some PDF printing  Year 2013:  All of the above, but…  We can hardly make a new application without some data processing in it
  3. 3. WPS: some quick reminders
  4. 4. FOSS4G Europe 2014, Bremen 14th-17th July 2014 Web Processing Service  Wikipedia introduces OGC WPS as:  [A service] designed to standardize the way that GIS calculations are made available to the Internet.  WPS can describe any calculation including all of its inputs and outputs, and trigger its execution  The specific processes served up by a WPS implementation are defined by the owner of that implementation.  Although WPS was designed to work with spatially referenced data, it can be used with any kind of data.
  5. 5. FOSS4G Europe 2014, Bremen 14th-17th July 2014 An Example  Buffer a L shaped geometry with distance “2”  Get the result back as GML
  6. 6. FOSS4G Europe 2014, Bremen 14th-17th July 2014 GeoServer WPS integration WPS Remote WCS Remote WFS HTTP server WPS client All GeoServer Layers WMS clientWMS GeoServer UI
  7. 7. FOSS4G Europe 2014, Bremen 14th-17th July 2014 Rendering transformations  On-the-fly data transformations inside rendering chain  Calling WPS processes from SLD docs  Optimized for performance
  8. 8. FOSS4G Europe 2014, Bremen 14th-17th July 2014 • GeoServer ships with lots of built-in processes • Basic geometry manipulation • Aggregation options missing from WFS • Raster and vector clipping • And more! • Ability to chain processes to build more complex functionality • Open API to manage inputs and outputs • See all of it at http://demo.geo- solutions.it/share/foss4g2011/wps_aaime_foss4g20 11.pdf More On GeoServer WPS
  9. 9. When the WPS does not cut it…
  10. 10. FOSS4G Europe 2014, Bremen 14th-17th July 2014 • Never under-estimate the processing power of your BDMS: • Designed to efficiently juggle large quantities of data • Efficient spatial primitives (at least, in PostGIS) • Doesn’t get more local to your data than this! • Passing params down? • Parametric SQL views! Spatial DBMS!
  11. 11. FOSS4G Europe 2014, Bremen 14th-17th July 2014 Parametric SQL views WMS/WFS client GeoServer Spatial database &viewparams=low:2000000;high:5000000 Expanded Query
  12. 12. Enough theory Let’s get down to business!
  13. 13. FAO Tuna Atlas
  14. 14. FOSS4G Europe 2014, Bremen 14th-17th July 2014 Mapping Tuna Catches • Multiple Filtering • Aggregation • Joining quartely stats against the grid
  15. 15. FOSS4G Europe 2014, Bremen 14th-17th July 2014 Filtering, joining and aggregation Some example control regexps: • Y_INTERV: • Default: 1 • Regex: ^(d)+$ • OP • Default: sum • Regex: ^[avg|sum]$ SELECT (T.TS_VALUE / %Y_INTERV%) AS TS_VALUE, T.CD_TA_OCEANAREA, G.GEOMETRY FROM (SELECT CD_TA_OCEANAREA, OP%(TS_VALUE) AS TS_VALUE FROM FIGIS.TS_FI_TA WHERE FIC_ITEM IN (%FIC_ITEM%) AND CD_GEAR IN (%CD_GEAR%) AND YR_TA IN (%YR_TA%) AND QTR_TA IN (%QTR_TA%) GROUP BY CD_TA_OCEANAREA ) t LEFT OUTER JOIN FIGIS_GIS.GRID_G5 g ON T.CD_TA_OCEANAREA = g.CD_OAREA ORDER BY T.CD_TA_OCEANAREA
  16. 16. FOSS4G Europe 2014, Bremen 14th-17th July 2014 • Animator tool • GetMap + variying parameters + frame control => animated GIF! • http://docs.geoserver.org/stable/en/user/tutorials/animr eflector.html Animation
  17. 17. FOSS4G Europe 2014, Bremen 14th-17th July 2014 • …/geoserver/wms/animate?request=GetMap&… &format=image/gif;subtype=animated &aparam=viewparams:YR_TA &avalues=2000,2001,2002,2003,… Animator
  18. 18. What’s the wheater like today?
  19. 19. FOSS4G Europe 2014, Bremen 14th-17th July 2014 The Situation • MetOc data is extremely dynamic • In-Situ sensors acquire data in near real-time • Meteorological and Oceanographic model runs multiple times a day • Remote Sensing data is acquired at fast pace • Meteo Radar data is acquired at fast pace • Large amount of data is produced to model fast changing environmental variables • We must shorten to the small possible extent the time and resources for preprocessing!
  20. 20. FOSS4G Europe 2014, Bremen 14th-17th July 2014 Solutions: On-the-fly contouring
  21. 21. FOSS4G Europe 2014, Bremen 14th-17th July 2014 Solutions: On-the-fly Wind-Barbs
  22. 22. FOSS4G Europe 2014, Bremen 14th-17th July 2014 Solutions: On-the-fly Currents
  23. 23. FOSS4G Europe 2014, Bremen 14th-17th July 2014 Solutions: On-the-fly Storm Tracks
  24. 24. Download services
  25. 25. FOSS4G Europe 2014, Bremen 14th-17th July 2014 • New addition to the MapStore project • Requirements • Download large amounts of data • Generic data filtering • Clip on polygon/bbox/circle, both vector and raster • Reproject to target CRS • Mail notification when download is ready (or fails) • Work in a cluster • Solution: new WPS processes, asynch WPS calls and WPS customization for state sharing Advanced Clip and Ship
  26. 26. FOSS4G Europe 2014, Bremen 14th-17th July 2014 MapStore GUI Buffer process called synchronously when buffer size changes
  27. 27. FOSS4G Europe 2014, Bremen 14th-17th July 2014 MapStore GUI Tracking download status (asynch WPS)
  28. 28. FOSS4G Europe 2014, Bremen 14th-17th July 2014 Download service architecture WPS All GeoServer Layers MapStore WMS GetCapabilities List of layers Buffer DownloadEstimator Download GetStatus Fetch data Status database Store and retrieve status Mail server Custom process manager Start/ Complete/ Failed notifications
  29. 29. The CMRE-IDA Project
  30. 30. FOSS4G Europe 2014, Bremen 14th-17th July 2014 • The Integrated Decision Aid (IDA) has been developed at the Centre for Maritime Research and Experimentation (CMRE) in order to provide naval exercise planners with an effective tool to assess risk to marine mammals from acoustic transmissions. • Assessment of potential hazard is performed through sound propagation modeling available in the interface. Intro Sound Propagation Model
  31. 31. FOSS4G Europe 2014, Bremen 14th-17th July 2014 Sound Propagation Model • External Octave Matlab Process • Logs the asynchronous process status to the DB as a Feature Octave Matlab Script for SPM WFS Follows the process status through the WFSWFSLog Process allows to insert and update the status of SPM into the DB Dynamically builds an external Octave command line using the input parameters through a FreeMarker Template (FTL)
  32. 32. FOSS4G Europe 2014, Bremen 14th-17th July 2014 • The list of model runs is available on the bottom Data Grid. • It is possible to get information on the input parameters and execution from the WFS Sound Propagation Model
  33. 33. FOSS4G Europe 2014, Bremen 14th-17th July 2014 • Advanced Raster Algebra using Jiffle • Wraps Jiffle (http://code.google.com/p/jiffle/) in order to execute complex Raster Algebra scripts against the input layers Raster Algebra Processes Jiffle is a scripting language for creating and analysing raster images. Rather than having to write and test lots of JAI or Java AWT boiler-plate code to access and manipulate images, Jiffle lets you concentrate on the interesting bit: your algorithm. Jiffle is being developed as part of the JAITools project. Update 16 January 2013: Jiffle sources have moved to GitHub
  34. 34. FOSS4G Europe 2014, Bremen 14th-17th July 2014 • Advanced Raster Algebra using Jiffle • Gets an OGC Filter containing logical operations between raster layers and produces a binary light layer Raster Algebra Processes
  35. 35. FOSS4G Europe 2014, Bremen 14th-17th July 2014 • Raster Algebra detailed info • Input parameters and outcomes are logged into WFS • WPS Raster Stats Process provides statistics on the coverages Raster Algebra Processes Raster Statistics and Area in sq Km Raster Algebra Process user inputs
  36. 36. Script me plenty An example using GeoScript
  37. 37. FOSS4G Europe 2014, Bremen 14th-17th July 2014 Sample scripting application http://localhost:8080/geoserver/www/wps.html
  38. 38. FOSS4G Europe 2014, Bremen 14th-17th July 2014 Land property distrib. in the polygon
  39. 39. FOSS4G Europe 2014, Bremen 14th-17th July 2014 A look at the data dir 1. Data 2. WPS process scripted in Jython 3. The GUI
  40. 40. FOSS4G Europe 2014, Bremen 14th-17th July 2014 The Jython script http://geoscript.org/py/index.html
  41. 41. The Destination Project
  42. 42. FOSS4G Europe 2014, Bremen 14th-17th July 2014 • The Destination project computes the risk of accidents involving dangerous goods (chemicals, petrol, gases and so on) Intro Road segments and stats about car accidents Human and environmental «targets» Involved area, depending on type of good and amount of damage
  43. 43. FOSS4G Europe 2014, Bremen 14th-17th July 2014 • Road network of good part of northern Italy • Road divided into segments • 100m portions (500k of them) • 500m aggregation (120k of them) • 1 km square cells (few hundreds) • 51 buffer distances (depending on good, scenario, level of damage) • Several types of targets: schools, malls, hospitals, populated areas, superficial and underground acquifers, crops, woods, …. Large Data Volume
  44. 44. FOSS4G Europe 2014, Bremen 14th-17th July 2014 The road arc risk formula (s) • Adding togheter the risk caused by the different • Arc own propension to accidentds • Types of goods • Human and enviromental targets • The system allows to compute partial views of the formula, either by selection of targets/goods or by computing portions of it • Has a number of coefficients that can be hand- tuned by the caller
  45. 45. FOSS4G Europe 2014, Bremen 14th-17th July 2014 The results, visually • Rendering transformation • Read the arcs/polys from the DB, compute their risk based on the chosen formula, scenario, targets, and coefficients
  46. 46. FOSS4G Europe 2014, Bremen 14th-17th July 2014 How to compute it efficiently? • Using SQL Views? No, the possible aggregations variants are too many • Using a pure Java process? No, too much data to transfer from the DBMS • Fully on the fly? No, too much data involved •  Pre-compute all buffers and locate all involved targets before hand (pre-cooked per buffer risk) •  Use a process that builds a final aggregation query on the fly (dynamic sql views)
  47. 47. FOSS4G Europe 2014, Bremen 14th-17th July 2014 Efficient rendering tx Risk process Queries database Arcs/Buffer areas db Map renderer Build overall query, replace params Compute risk for a batch of arcs Raw arcs Arcs + risk • Compute risk on the fly in the viewing area • Batch requests to the BDMS to minimize round-trip overhead
  48. 48. FOSS4G Europe 2014, Bremen 14th-17th July 2014 Efficient cross layer filtering • Show only targets involved in the scenario under study, e.g., the ones crossing the buffer areas where there is significant risk • Limit query to the current bbox
  49. 49. FOSS4G Europe 2014, Bremen 14th-17th July 2014 Efficient cross layer filtering SELECT v_geo_popolazione_residente_pl.* FROM v_geo_popolazione_residente_pl WHERE v_geo_popolazione_residente_pl.fk_bersaglio_umano_pl in ( SELECT distinct bersaglio.fk_bersaglio_umano_pl FROM v_geo_popolazione_residente_pl bersaglio join siig_geo_ln_arco_1 on st_dwithin(bersaglio.geometria, siig_geo_ln_arco_1.geometria, %distanzaumano%) WHERE siig_geo_ln_arco_1.geometria && st_makeenvelope(%bounds%, 32632) ) • This is a job for a parametric sql view
  50. 50. FOSS4G Europe 2014, Bremen 14th-17th July 2014 The End Questions? andrea.aime@geo-solutions.it simone.giannecchini@geo-solutions.it

×