OSM data in MariaDB / MySQL - All the world in a few large tables

1,981 views

Published on

FOSDEM 2014 talk on importing OpenStreetMap data into MySQL and latest MySQL/MariaDB feature improvements

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,981
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
17
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

OSM data in MariaDB / MySQL - All the world in a few large tables

  1. 1. OSM data in MariaDB/MySQL All the world in a few large tables Well, almost ... Hartmut Holzgraefe SkySQL AB hartmut@skysql.com February 1, 2014 Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 1 / 30
  2. 2. Overview 1 MySQL, MariaDB and GIS 2 OpenStreetMap 3 osm2pgsql 4 Examples 5 The End ... Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 2 / 30
  3. 3. MySQL, MariaDB and GIS 1 MySQL, MariaDB and GIS History Current Status Roadmap 2 OpenStreetMap 3 osm2pgsql 4 Examples 5 The End ... Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 3 / 30
  4. 4. History First appeard in MySQL 4.1 (2004) ... with MBR relations only Lab Release adds true spatial relations (200?) MariaDB 5.3 GA with true spatial relations (2011) MySQL 5.6 GA with true spatial relations (2013) ... to be continued ... Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 4 / 30
  5. 5. MBR is not enough MBR CONTAINS() Hartmut Holzgraefe (SkySQL) True ST CONTAINS() OSM data in MariaDB/MySQL February 1, 2014 5 / 30
  6. 6. Current Status Spatial relations work ... but the world is still flat (no projections) OK for many use cases ... but be aware of gotchas like DISTANCE() GEOMETRY types in all storage engines ... but only MyISAM has SPATIAL indexes Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 6 / 30
  7. 7. MariaDB 10.1 Roadmap System Tables (spatial ref sys, geometry columns, ...) Precision math calculations and storage Coordinate transformations / projections 3rd coordinate (e.g. for altitude) all spatial functions required by OGC spatial aware optimizer Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 7 / 30
  8. 8. ... and beyond SPATIAL indexes in other storage engines 3D calculations client side support for GIS transformations Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 8 / 30
  9. 9. Openstreetmap 1 MySQL, MariaDB and GIS 2 OpenStreetMap Intro Core Data Model Data Access Data Import 3 osm2pgsql 4 Examples 5 The End ... Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 9 / 30
  10. 10. OpenStreetMap History founded in 2004 by Steve Coast data under open license (CC-BY-SA first, now ODBL) 1.5 million contributors 2 billion map nodes 200 million ways 2 million relations almost 4 billion GPX points Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 10 / 30
  11. 11. Pretty Tiles Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 11 / 30
  12. 12. ... and raw data Raw map data can be used for other things, too: for routing for coverage checks for flight simulators for science Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 12 / 30
  13. 13. Core Data Model Just three simple things Nodes (Points) Ways Relations Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 13 / 30
  14. 14. Nodes Nodes describe a single point at a specific location using: A numeric ID Object version, Timestamp of last change, User Node coordinates Node attributes as key/value pairs Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 14 / 30
  15. 15. Ways Ways form an open or closed line by connecting nodes, using: A numeric ID Object version, Timestamp of last change, User An ordered list of node IDs Way attributes as key/value pairs Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 15 / 30
  16. 16. Relations Relations bundle objects to describe more complex relations, using: A numeric ID Object version, Timestamp of last change, User Ordered lists of member nodes, ways and sub-relations Optional member roles Attributes as key/value pairs Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 16 / 30
  17. 17. Data Access The main database is not exposed directly Only one central instance, accessible via “The API” API meant for editor applications only Full data export once a week (“the planet”) Plus daily, hourly, minutely diffs Two file formats for planets: .osm XML based, usually bz2 compressed (32GB packed, 400GB unpacked) .pbf compact binary format based on Google ProtoBuf (23GB) Regional extracts available by 3rd parties, e.g. GeoFabrik.de Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 17 / 30
  18. 18. Data Import The raw data is not really suitable for most purposes, esp. rendering Several import/preprocessing tools provide more convenient schemas, e.g. by ... only extracting certain attributes ... making a difference beween ways and areas ... resolving relations into simpler objects Besides osm2pgsql that I’m about to talk about in a minute there are also impOsm, ... Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 18 / 30
  19. 19. osm2pgsql 1 MySQL, MariaDB and GIS 2 OpenStreetMap 3 osm2pgsql Block Diagram Data Model Again Adding MySQL Support Performance 4 Examples 5 The End ... Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 19 / 30
  20. 20. osm2pgsql osm2pgsql is a tool written in C with a small C++ part now reads OSM data preprocesses it stores results in relational tables originally in PostGIS only Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 20 / 30
  21. 21. Block Diagram Figure : osm2pgsql block diagram Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 21 / 30
  22. 22. Data Model Again prefix point for single node POIs prefix line for linear 2D objects like roads, rivers, power lines ... prefix roads a subset of the above, optimizer for rendering prefix polygon objects covering an area: buildings, landuse, administrative borders ... Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 22 / 30
  23. 23. Adding MySQL Support turned out to be more tricky than thought some core parts directly called PostgreSQL functions a lot of general functionality was hidden in PostgreSQL specific modules MySQL output module works, could be faster though MySQL middle layer is “code complete” ... ... but crashes while processing relations :( so for now imports are limited by RAM size Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 23 / 30
  24. 24. Import performance imports currently take about 4-5 times as long ... as we have no direct equivalent to COPY osm2pgsql at less than 50% CPU only ... so switching to async API would be a 2x win already ... with multi-insert even more so index building is faster ... ... but may not be once we get I/O bound tables on disk are of similar size Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 24 / 30
  25. 25. Query performance select from join on where and count(*) nrw_point n nrw_polygon p st_contains(p.way,n.way) p.name = ’Bielefeld’ n.amenity=’post_box’; Data Set Germany Northrhine-Westfalia same with better indexes Hartmut Holzgraefe (SkySQL) MySQL 5.5 (MBR) MariaDB 5.5 PostGIS 15.8s 2.7s 0.2s 16.5s 3.1s 0.2s ? 6.1s .04s OSM data in MariaDB/MySQL February 1, 2014 25 / 30
  26. 26. Examples 1 MySQL, MariaDB and GIS 2 OpenStreetMap 3 osm2pgsql 4 Examples 5 The End ... Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 26 / 30
  27. 27. The End ... 1 MySQL, MariaDB and GIS 2 OpenStreetMap 3 osm2pgsql 4 Examples 5 The End ... Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 27 / 30
  28. 28. References Contact hartmut@skysql.com MariaDB GIS https://mariadb.com/kb/en/gis-functionality/ MySQL GIS http://dev.mysql.com/doc/refman/5.6/en/spatialextensions.html OpenStreetMap http://openstreetmap.org/ MapCompare http://mc.bbbike.org/mc/ RiverMap http://www.kompf.de/gps/rivermap.html osm2pgsql https://wiki.openstreetmap.org/wiki/Osm2pgsql My Code https://github.com/hholzgra/osm2pgsql/tree/devel-mysql Table Files http://php-groupies.de/gis-data/ (soon) Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 28 / 30
  29. 29. Questions! Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 29 / 30
  30. 30. The End? Or just the beginning? Hartmut Holzgraefe (SkySQL) OSM data in MariaDB/MySQL February 1, 2014 30 / 30

×