Thijs Brentjens - PostGIS as used by the Dutch government

1,828 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,828
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Thijs Brentjens - PostGIS as used by the Dutch government

  1. 1. National critical geoinfrastructure runs on open source database PostGIS TU Delft Geomatics Open Guest Lecture March 21st 2011 Thijs Brentjens
  2. 2. Who am I?● TU Delft MSc Geodetic Engineering (2004) ● Thesis: Web Feature Services● GIS Software engineer● Freelancer 2007● Member OpenGeoGroep● Open standards● Open sourceBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  3. 3. Contents National critical geo infrastructure runs on open source database PostGISBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  4. 4. Contents● Geo infrastructure● PostGIS in geo infrastructure: ● Usage on PostGIS ● Case: PDOK ● BAG: Postgis and authentic register● ExtrasBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  5. 5. Geo infrastructure● Spatial information is crucial in modern society, for example: ● ...Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  6. 6. Geo infrastructure● Spatial information is crucial in modern society, for example: ● Spatial planning ● Environmental issues ● Registrations / public authorities ● Agriculture ● Navigation● --> To make modern information society work (and fun)Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  7. 7. How could we efficiently andeffectively provide spatial data for this?Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  8. 8. Geo infrastructure● Need to re-use, share up-to-date data● Enabled by a geo information infrastructure http://www.flickr.com/photos/20013727@N02/Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  9. 9. Geo infrastructure● Some current driving programmes in NL: ● Authentic registers ● INSPIRE ● PDOK● From a bit earlier (and still running): ● WION, Spatial planning (RO Online), ...● Web developments, mobileBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  10. 10. Geo infrastructure● There is a need for geodata for critical applications● From mixed sources● Data is more and more available● An infrastructure to facilitate this is evolvingBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  11. 11. What is such a geo infrastructure in practice? (in terms of technical components)Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  12. 12. Geo infrastructure Basically:● Spatial datasets● Offered by standards based webservices● Described in (and searchable through) metadata● Consumed by client applications ● Web ● Desktop ● MobileBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  13. 13. Geo infrastructure● Simplified architecture:ClientsWebservices MetadataSpatial datasets MetadataBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  14. 14. Geo infrastructure● 100s services already publicly available● Services used in many applications● Cross-organization: startingBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  15. 15. PostGIS in geo infrastructureBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  16. 16. PostGIS● “PostGIS adds support for geographic objects to the PostgreSQL object- relational database.” http://postgis.refractions.net/● Spatial data types, spatial functions, spatial indexing for PostgresSQL● Able to store, query, manipulate and analyze geospatial data (and more)Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  17. 17. Applications using PostGISClientsWebservices MetadataSpatial datasets MetadataBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  18. 18. Applications using PostGIS● Data store for webservices: ● UMN Mapserver ● Geoserver ● Deegree ● ESRI ArcGIS Server ● Intergraph GeoMedia WebMap ● ...others...Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  19. 19. Applications using PostGIS● Desktop: ● Qgis ● Udig ● OpenJUMP ● Intergraph GeoMedia ● ... others..Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  20. 20. Applications using PostGIS● PostGIS often used for: ● Storage ● Retrieval (querying) ● Some data processing● Also: ● On-the-fly reprojection ● Specialist GIS / geometric analysis ● Geometric validation ● ...Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  21. 21. PDOK and PostGISBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  22. 22. Projects: PDOK● Who, why and what: project backgrounds● How Postgis is used● Next challengesBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  23. 23. Projects: PDOK● “Publieke Dienstverlening op de Kaart” (in English “Public Service on the Map”)● Organisations joint forces ● the Dutch Cadastre ● the Ministry of Economic Affairs, Agriculture and Innovation ● the Ministry of Infrastructure and the Environment ● TNO, an independent research organisationBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  24. 24. Projects: PDOK● Serve a core set of geographic data to other governments● Because of: ● the need to reduce government expenditure ● to make national geographic data services widely accessible to each other and (possibly) society in order to improve public services ● to address requirements set forth in INSPIRE● Very important contribution national geo infrastructureBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  25. 25. Projects: PDOK● Central and decentral servicesBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  26. 26. Projects: PDOK● Example of datasets: ● Authentic register for topographic data (BasisRegistratie Topografie (BRT))Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  27. 27. Projects: PDOK● Top10NLBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  28. 28. Projects: PDOK● Administrative borders● Natura2000● Geology● Addresses: ACN, later Authentic register (BAG)● Transport networks● Currently ~10 new datasets being added● Many more from PDOK partners planned, this year and beyondBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  29. 29. Projects: PDOK● Offer data-delivery services● For many users● From several data providers● High level requirements: ● High performance ● High availability (7x24h) – Emergency servicesBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  30. 30. Projects: PDOK● Guiding principles central services: ● components based ● proven solutions / best practices ● use Open Source software components ● account for future growth ● open standards ● existing software components ● Keep It Simple principle: only design for things that are needed in the phase of the implementation.Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  31. 31. Projects: PDOK● Central service componentsBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  32. 32. Projects: PDOK● Why Postgis for central services? Amongst others: ● PostGIS is able to do what is needed ● Proven: many examples, also big infrastructures IGN for example ● Scalable. Technically and license-wise. ● Open source software (OSS) ● Other software supports PostGIS well● Other dbmsses would have been possible too, but PostGIS the best choiceBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  33. 33. PostGIS usage in PDOKBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  34. 34. PostGIS usage in PDOK● Storage & loading data● Querying & optimizations● Users (autorization)● Scalability & deploymentBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  35. 35. PostGIS usage in PDOK● Storage currently (March 2011): ● Vector data ● around 25 GB vector data in PDOK central services – Example: Top10NL = 6Gb ● Growing● Schemas: per dataset, public only for generic objects (tables geometry_columns, functions)● Chosen for one geom per tableBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  36. 36. PostGIS usage in PDOK● Import: use unique table for import, suffix “_imp”● Load data: View ● check on valid geometry Valid ● result in “valid”-column● Use views to publish to web Table_imp ● Valid geometries only ● Column aliasses Data SourceBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  37. 37. PostGIS usage in PDOK● Querying: what kind of queries?Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  38. 38. PostGIS usage in PDOK● Queries mostly: “create a map”● Many spatial intersects (bbox / polygon).● Sometimes more: ● administrative constraints (classification), scale dependency (e.g. roads classfication)● Reproject coordinates on-the-flyBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  39. 39. PostGIS usage in PDOK SELECT "gid",encode(ST_AsBinary(ST_Force_2D("the_geom ")),base64) as "the_geom" FROM "grenzen"."cbs_wijken" WHERE "the_geom" && ST_GeomFromText( POLYGON ((-113908 197172, -113908 632427, 396528 632427, 396528 197172, -113908 197172)), 28992)Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  40. 40. PostGIS usage in PDOK● Tuning for querying: ● Indexing: – geometry is essential! – primary key (default indexed by PostGIS) – foreign keys (PDOK: not many now) – indices for known searches ● For now sufficient, because webservices are using more resources than databaseBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  41. 41. PostGIS usage in PDOK● PostGIS spatial indexing used: ● GiST-index, Generalized Search Tree ● to speed up searches on irregular data structures● Indexes break up data into: ● "things to one side" ● "things which overlap" ● "things which are inside"Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  42. 42. PostGIS usage in PDOK● Two advantages over R-Tree indexes: ● "null safe" ● Only store the "important" part in an index. --> spatial objects: the bounding box. Why? GIS objects larger than 8K cause R-Tree indexes to fail in the process of being built. create index geoname_geom_idx on geoname using gist (the_geom); http://postgis.refractions.net/docs/ch04.html#id2638705Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  43. 43. PostGIS usage in PDOK● Geometry Relationship Functions, include implicit bounding box overlap operators ● exceptions of ST_Disjoint and ST_Relate● very fast searches / processing http://postgis.refractions.net/docs/ch04.html#id2638955Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  44. 44. PostGIS usage in PDOK● User roles ● Default: only postgres (“admin”) ● Added: owner of all tables ● Read access only ● Also for more specific usage, depending on requirementsBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  45. 45. PostGIS usage in PDOK● Deployment & scalability● Simplified production view (currently) Webservices Webservices Server 1 Server n Webservices Farm All data Server dbBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  46. 46. PostGIS usage in PDOK● Deployment using development street● Many servers involved Webservices Webservices Webservices Webservices OK OK OK Development Test Acceptance ProductionDataimport Data SourcesBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  47. 47. PostGIS usage in PDOK● Upcoming challenges database in PDOK: ● More datasets --> consequences? ● More updates --> consequences? ● Temporal data & store history?● Solutions will depend on requirementsBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  48. 48. PostGIS usage in PDOK● Some options: ● more database servers & replication? ● split up databases?● --> more database instances● More tuning for performance?Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  49. 49. BAG: Example Authentic RegisterBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  50. 50. BAG: Example Authentic Register● What is the BAG?● Data model● BAG in PostGIS ● Model & database ● Temporal aspectsBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  51. 51. What is the BAG?● Authentic Register● All official buildings and addresses of NL● Municipalities provide & maintain data● Centralized services for access / delivery (XML)● Currently being implemented● Mandatory to use by this summer for all governmentsBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  52. 52. Data model Example:● 1 building● Many residences● With each an address● In a street● In a cityBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  53. 53. Data Model● Data model for delivery / clients● Places● “Streets”● Addresses● Addressable objects: ● Residences ● Site (for e.g. caravans) ● Berths (for ships / boats)● BuildingsBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  54. 54. BAG in PostGIS● ~ 20 million objects with geom● ~ 10 million address references Using BAG Extract tool:● Each type a table● Relations stored in database● --> ~ 1-to-1 translation of model● Views for retrievalBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  55. 55. BAG in PostGIS● Example, “Verblijfsobject” / “residence”:● 1 main table, several tables for references● Identification● Geometry (polygon)● “Simple” administrative dataBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  56. 56. BAG in PostGIS● Temporal: start-date (begindatum), end- date (einddatum) ● to store history & reconstruct● Relations / references to others: ● Main address (1) --> in table for references ● Secondary addresses (0-N) --> in table for references ● Building (1-N) --> table BuildingsBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  57. 57. BAG in PostGIS● Views for data access● Verblijfsobjectactueel / “Current residences”: ● “reconstructs” main address (using references) ● Filter on time: now● Other views could be defined, as neededBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  58. 58. BAG in PostGIS ●Filter on time: nowSELECT <verblijfsobject.attributes>FROM verblijfsobjectWHERE verblijfsobject.begindatum <= now::text::dateAND verblijfsobject.einddatum >= now::text::dateBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  59. 59. BAG in PostGISDemo:http://www.brentjensgeoict.nl/appinaday_RD.htmlBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  60. 60. ExtrasBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  61. 61. Extras● PostGIS (Extra) advanced features & functions● Easy export: AsKML(), AsGML()● Linear Referencing● Routing: PGRouting● GIS analyses using database● Geography type (calculations over a sphere, e.g. distance)Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  62. 62. Extras● Calculate height profiles, using: ● Linear Referencing ● Height contours ● Geography type (calculations over a sphere, e.g. Distance)Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  63. 63. Extras● Route:Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  64. 64. Extras● Contours:Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  65. 65. Extras● Calculate distances: ● Intersect of route and contours ● Distance calculation over line ● Using geography type (coordinates over sphere, lat/lon)● Linear referencing: walk aling a line and use distance from start (or something else) as referenceBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  66. 66. ExtrasSELECT road_heights.gid, ... ,ST_line_locate_point(road_geom,road_heights_geom)*ST_Length(Geography(road_geom)) asdistancefromstart, road_heights.height, road_heights_geomFROM ( SELECT pb_etappe4.gid, ..., contours.height as height, (ST_Dump(ST_Intersection(contours.the_geom, pb_etappe4.the_geom))).geom As road_heights_geom, pb_etappe4.the_geom as road_geom FROM contours INNER JOIN pb_etappe4 ON ST_Intersects(contours.way, pb_etappe4.the_geom) ) As road_heightsorder by distancefromstart;Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  67. 67. Extras● Result:Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  68. 68. Extras● Distance:Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  69. 69. Extras● Height:Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  70. 70. Extras● Distance and height graph:Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  71. 71. Extras: performance● Performance tips ● CLUSTERing on geometry indices, to physically reorder the data rows in the index order. ● Speedup look ups of data on disk-- first, set the geom to not being nullalter table planet_osm_line alter the_geom set not null;-- second, cluster on the index-name (of the geom-column)cluster planet_osm_line_index on planet_osm_line;Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  72. 72. Extras: performance● Use and tune tablespaces for particular usage (dynamic datasets vs. static data)● “Tablespaces define locations in the file system where the files representing database objects can be stored.”● Different file systems for storagethanks to Agustin Matilla Sanz (Geodan, PDOK)Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  73. 73. Extras● PDOK: Why open source? 1. Requirements mean many servers and processing power needed. Proprietary software is often licensed by number of cores/cpu’s which are used by the software simultaneously. A highly scalable system may therefore be confronted with high license costs.Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  74. 74. Extras PDOK: Why open source? (continued) 2.Open source software components will not provide all functionality which is required by PDOK, but this will also be the case with proprietary software. In case of open source there are more means to exert influence on the directions of future developments.Brentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  75. 75. Applications using PostGIS● Dutch examples (not all infrastructure):● Ruimtelijkeplannen.nl ● national database & web service ● for all spatial planning● PDOK ● Many datasets ● Some authentic registers with geometryBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  76. 76. Applications using PostGIS● Waterschapshuis Central Geo facility● Kadaster INSPIRE● RVOB: real estate of government● Police (vtsPN)● WION decentral systems: ● provinces Drenthe, Gelderland, LimburgBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  77. 77. Applications using PostGIS● Geo applications in municipalities: ● Several, e.g. 5 in Land van Cuijk, Maasdonk, Vlaardingen, Maarssen, Barneveld, Arnhem, ...● Groningen Seaports● ... and many moreBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  78. 78. Extras● Infrastructure is the basic physical and organizational structures needed for the operation of a society or enterprise,[1] or the services and facilities necessary for an economy to function.[2] The term typically refers to the technical structures that support a society, such as roads, water supply, sewers, power grids, telecommunications, and so forthBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011
  79. 79. References● http://postgis.refractions.net/docs/ch04.html#id2638705● http://bag.vrom.nl●● http://www.postgresql.org/docs/8.4/static/manage-ag-tablespacBrentjens Geo-ICT - TU Delft Geomatics Open Guest Lecture - March 21 st 2011

×