The document discusses storing OpenStreetMap data in MariaDB/MySQL. It provides an overview of the spatial capabilities of MariaDB/MySQL, describes the OpenStreetMap data model and how data is accessed. It also explains how the osm2pgsql tool can be used to import OSM data into MySQL tables and discusses performance. The document concludes by noting some examples and references for further information.
This presentation was given by Pavan Naik in Open Source India (OSI) 2014 even held in Nimans Convention Centre, Bangalore. It talks about GIS features in MySQL 5.7.
Presentation covers following topics :
1. Introduction to GIS
2. Common Terms and Concepts
3. What's new in MySQL 5.7
4. A Real World Example
5. What's next for MySQL GIS
Advanced Cartographic Map Rendering In GeoServerGeoSolutions
Various software can style maps and generate a proper SLD document for OGC-compliant WMS. However the styling allowed by the graphical tools is often limited and not adequate for good cartographic output. This presentation will provide hints, examples and enough information to master SLD in order to create beautiful maps with GeoServer.
State of GeoServer provides an update on our community and reviews the new and noteworthy features for the Project. The community keeps an aggressive six month release cycle with GeoServer 2.8 and 2.9 being released this year.
Each releases bring together exciting new features. This year a lot of work has been done on the user interface, clustering, security and compatibility with the latest Java platform. We will also take a look at community research into vector tiles, multi-resolution raster support and more.
Attend this talk for a cheerful update on what is happening with this popular OSGeo project. Whether you are an expert user, a developer, or simply curious what these projects can do for you, this talk is for you.
Vector Tiles with GeoServer and OpenLayersJody Garnett
The latest release of GeoServer adds support for creating Vector Tiles in GeoJSON, TopoJSON, and MapBox Vector Tiles format through its WMS service for all the vector data formats it supports. These tiles can be cached using GeoWebCache (built into GeoServer), and served with the various tiling protocols (TMS, WMTS, and WMS-C). Thanks to very recent OpenLayers 3 development, these Vector Tiles can be easily and efficiently styled on a map.
This technical talk will look at how GeoServer makes Vector Tiles accessible through standard OGC services and how they differ from normal WMS and WFS usage. It will also look at how OpenLayers 3 - as a simple-to-use vector tiles client - interacts with GeoServer to retrieve tiles and effectively manage and style them. OpenLayer 3’s extensive style infrastructure will be investigated.
This presentation was given by Pavan Naik in Open Source India (OSI) 2014 even held in Nimans Convention Centre, Bangalore. It talks about GIS features in MySQL 5.7.
Presentation covers following topics :
1. Introduction to GIS
2. Common Terms and Concepts
3. What's new in MySQL 5.7
4. A Real World Example
5. What's next for MySQL GIS
Advanced Cartographic Map Rendering In GeoServerGeoSolutions
Various software can style maps and generate a proper SLD document for OGC-compliant WMS. However the styling allowed by the graphical tools is often limited and not adequate for good cartographic output. This presentation will provide hints, examples and enough information to master SLD in order to create beautiful maps with GeoServer.
State of GeoServer provides an update on our community and reviews the new and noteworthy features for the Project. The community keeps an aggressive six month release cycle with GeoServer 2.8 and 2.9 being released this year.
Each releases bring together exciting new features. This year a lot of work has been done on the user interface, clustering, security and compatibility with the latest Java platform. We will also take a look at community research into vector tiles, multi-resolution raster support and more.
Attend this talk for a cheerful update on what is happening with this popular OSGeo project. Whether you are an expert user, a developer, or simply curious what these projects can do for you, this talk is for you.
Vector Tiles with GeoServer and OpenLayersJody Garnett
The latest release of GeoServer adds support for creating Vector Tiles in GeoJSON, TopoJSON, and MapBox Vector Tiles format through its WMS service for all the vector data formats it supports. These tiles can be cached using GeoWebCache (built into GeoServer), and served with the various tiling protocols (TMS, WMTS, and WMS-C). Thanks to very recent OpenLayers 3 development, these Vector Tiles can be easily and efficiently styled on a map.
This technical talk will look at how GeoServer makes Vector Tiles accessible through standard OGC services and how they differ from normal WMS and WFS usage. It will also look at how OpenLayers 3 - as a simple-to-use vector tiles client - interacts with GeoServer to retrieve tiles and effectively manage and style them. OpenLayer 3’s extensive style infrastructure will be investigated.
Using GeoServer for spatio-temporal data management with examples for MetOc a...GeoSolutions
This presentation will provide detailed information on how to ingest and configure spatio-temporal data in GeoServer, to be served using OGC services, with examples from WMS and WCS services.
Slides from my Introduction to PostGIS workshop at the FOSS4G conference in 2009. The material is available at http://revenant.ca/www/postgis/workshop/
State of GeoServer provides an update on our community and reviews the new and noteworthy features for 2018. GeoServer is a web service for publishing your geospatial data. using industry standards for vector, raster and mapping.
We have an active community and a lot to cover for 2.12 and 2.13 release, as well what is cooking in September’s 2.14 release.
Each release provides exciting new features, this talk covers diverse improvements across GeoServer:
* OGC compliance work for WFS 2.0 and WMTS 1.0, WFS 3.0 support
* improvements for cloud deployments
* cascade WMTS services
* progress in NetCDF support
* getting ready for the Java 18.9 roadmap
* And much more…
Attend this talk for a cheerful update on what is happening with this popular OSGeo project. Whether you are an expert user, a developer, or simply curious what GeoServer can do for you.
Java Image Processing for Geospatial CommunityJody Garnett
The Java Advanced Imaging is a powerful Java image processing engine underlines our popular OSGeo open source projects - including GeoTools, GeoServer, GeoNetwork, and GeoNode, and more! Tragically there has been one problem with this, the JAI library is not open source!
The library originated at Sun Microsystem as a core component of the Java Runtime Environment, but was not included as part of OpenJDK collaboration.
This talk explores:
* Capabilities that make JAI attractive for GeoSpatial work
* How JAI has been used in our community
* The exciting JAI-EXT project by GeoSolutions
One of the reasons our community has been so addicted to this library is its power. It explored concepts like parallel processing, and distributed parallel processing in 1999, well ahead of the curve. It is an excellent example of engineering and software design.
Importantly we will cover the search for an open source alternative, and are the exciting progress in producing an open source alternative.
Come see how our this foundational library is being propelled into an open source future by our community.
Hashing has witnessed an increase in popularity over the
past few years due to the promise of compact encoding and fast query
time. In order to be effective hashing methods must maximally preserve
the similarity between the data points in the underlying binary representation.
The current best performing hashing techniques have utilised
supervision. In this paper we propose a two-step iterative scheme, Graph
Regularised Hashing (GRH), for incrementally adjusting the positioning
of the hashing hypersurfaces to better conform to the supervisory signal:
in the first step the binary bits are regularised using a data similarity
graph so that similar data points receive similar bits. In the second
step the regularised hashcodes form targets for a set of binary classifiers
which shift the position of each hypersurface so as to separate opposite
bits with maximum margin. GRH exhibits superior retrieval accuracy to
competing hashing methods.
Overview of MassGIS Web Mapping Servicesaleda_freeman
An overview of what MassGIS web mapping services are, how to connect and use them and a showcase of a selection of applications made by MassGIS and other agencies and organizations that use the services.
Creating Stunning Maps in GeoServer: mastering SLD and CSS stylesGeoSolutions
Various software can style maps and generate a proper SLD document for OGC compliant WMS like GeoServer to use. However, in most occasions, the styling allowed by the graphical tools is pretty limited and not good enough to achieve good looking, readable and efficient cartographic output. For those that like to write their own styles CSS also represents a nice alternatives thanks to its compactness and expressiveness.
Several topics will be covered, providing examples in both SLD and CSS for each, including: mastering multi-scale styling, using GeoServer extensions to build common hatch patterns, line styling beyond the basics, such as cased lines, controlling symbols along a line and the way they repeat, leveraging TTF symbol fonts and SVGs to generate good looking point thematic maps, using the full power of GeoServer label lay-outing tools to build pleasant, informative maps on both point, polygon and line layers, including adding road plates around labels, leverage the labeling subsystem conflict resolution engine to avoid overlaps in stand alone point symbology, blending charts into a map, dynamically transform data during rendering to get more explicative maps without the need to pre-process a large amount of views.
The presentation aims to provide the attendees with enough information to master SLD/CSS documents and most of GeoServer extensions to generate appealing, informative, readable maps that can be quickly rendered on screen.
Serving earth observation data with GeoServer: addressing real world requirem...GeoSolutions
The presentation will cover GeoSolutions experience in setting up GeoServer based production systems providing access to earth observation products, with indications of technical challenges, solutions, and deployment suggestion. The presentations will cover topics such as setting up a single unified mosaic from all the available data sources, tailoring access to it to different users, determining the most appropriate stacking order, dealing with multiresolution, different coordinate systems, multiband data, SAR integration, searching for the most appropriate products using a mix of WFS, CSW and so on, serving imagery with high performance WMS and WMTS, performing small and large data extractions with WCS and WPS, closing up with deployment examples and suggestions.
This presentation will dive into a development team’s use case for choosing MongoDB as their spatially enabled NoSQL solution. The talk will also cover how the integration of GeoServer can expand the accessibility of your data. GeoServer is the open source implementation of Open Geospatial Consortium (OGC) standards and a core component of the Geospatial Web.
Using GeoServer for spatio-temporal data management with examples for MetOc a...GeoSolutions
This presentation will provide detailed information on how to ingest and configure spatio-temporal data in GeoServer, to be served using OGC services, with examples from WMS and WCS services.
Slides from my Introduction to PostGIS workshop at the FOSS4G conference in 2009. The material is available at http://revenant.ca/www/postgis/workshop/
State of GeoServer provides an update on our community and reviews the new and noteworthy features for 2018. GeoServer is a web service for publishing your geospatial data. using industry standards for vector, raster and mapping.
We have an active community and a lot to cover for 2.12 and 2.13 release, as well what is cooking in September’s 2.14 release.
Each release provides exciting new features, this talk covers diverse improvements across GeoServer:
* OGC compliance work for WFS 2.0 and WMTS 1.0, WFS 3.0 support
* improvements for cloud deployments
* cascade WMTS services
* progress in NetCDF support
* getting ready for the Java 18.9 roadmap
* And much more…
Attend this talk for a cheerful update on what is happening with this popular OSGeo project. Whether you are an expert user, a developer, or simply curious what GeoServer can do for you.
Java Image Processing for Geospatial CommunityJody Garnett
The Java Advanced Imaging is a powerful Java image processing engine underlines our popular OSGeo open source projects - including GeoTools, GeoServer, GeoNetwork, and GeoNode, and more! Tragically there has been one problem with this, the JAI library is not open source!
The library originated at Sun Microsystem as a core component of the Java Runtime Environment, but was not included as part of OpenJDK collaboration.
This talk explores:
* Capabilities that make JAI attractive for GeoSpatial work
* How JAI has been used in our community
* The exciting JAI-EXT project by GeoSolutions
One of the reasons our community has been so addicted to this library is its power. It explored concepts like parallel processing, and distributed parallel processing in 1999, well ahead of the curve. It is an excellent example of engineering and software design.
Importantly we will cover the search for an open source alternative, and are the exciting progress in producing an open source alternative.
Come see how our this foundational library is being propelled into an open source future by our community.
Hashing has witnessed an increase in popularity over the
past few years due to the promise of compact encoding and fast query
time. In order to be effective hashing methods must maximally preserve
the similarity between the data points in the underlying binary representation.
The current best performing hashing techniques have utilised
supervision. In this paper we propose a two-step iterative scheme, Graph
Regularised Hashing (GRH), for incrementally adjusting the positioning
of the hashing hypersurfaces to better conform to the supervisory signal:
in the first step the binary bits are regularised using a data similarity
graph so that similar data points receive similar bits. In the second
step the regularised hashcodes form targets for a set of binary classifiers
which shift the position of each hypersurface so as to separate opposite
bits with maximum margin. GRH exhibits superior retrieval accuracy to
competing hashing methods.
Overview of MassGIS Web Mapping Servicesaleda_freeman
An overview of what MassGIS web mapping services are, how to connect and use them and a showcase of a selection of applications made by MassGIS and other agencies and organizations that use the services.
Creating Stunning Maps in GeoServer: mastering SLD and CSS stylesGeoSolutions
Various software can style maps and generate a proper SLD document for OGC compliant WMS like GeoServer to use. However, in most occasions, the styling allowed by the graphical tools is pretty limited and not good enough to achieve good looking, readable and efficient cartographic output. For those that like to write their own styles CSS also represents a nice alternatives thanks to its compactness and expressiveness.
Several topics will be covered, providing examples in both SLD and CSS for each, including: mastering multi-scale styling, using GeoServer extensions to build common hatch patterns, line styling beyond the basics, such as cased lines, controlling symbols along a line and the way they repeat, leveraging TTF symbol fonts and SVGs to generate good looking point thematic maps, using the full power of GeoServer label lay-outing tools to build pleasant, informative maps on both point, polygon and line layers, including adding road plates around labels, leverage the labeling subsystem conflict resolution engine to avoid overlaps in stand alone point symbology, blending charts into a map, dynamically transform data during rendering to get more explicative maps without the need to pre-process a large amount of views.
The presentation aims to provide the attendees with enough information to master SLD/CSS documents and most of GeoServer extensions to generate appealing, informative, readable maps that can be quickly rendered on screen.
Serving earth observation data with GeoServer: addressing real world requirem...GeoSolutions
The presentation will cover GeoSolutions experience in setting up GeoServer based production systems providing access to earth observation products, with indications of technical challenges, solutions, and deployment suggestion. The presentations will cover topics such as setting up a single unified mosaic from all the available data sources, tailoring access to it to different users, determining the most appropriate stacking order, dealing with multiresolution, different coordinate systems, multiband data, SAR integration, searching for the most appropriate products using a mix of WFS, CSW and so on, serving imagery with high performance WMS and WMTS, performing small and large data extractions with WCS and WPS, closing up with deployment examples and suggestions.
This presentation will dive into a development team’s use case for choosing MongoDB as their spatially enabled NoSQL solution. The talk will also cover how the integration of GeoServer can expand the accessibility of your data. GeoServer is the open source implementation of Open Geospatial Consortium (OGC) standards and a core component of the Geospatial Web.
Slides for a talk at PyCon AU 2013. Integrating PyDAP + WMS + OpenLayers + IPython Notebook.
Video: http://www.youtube.com/watch?v=YJqBGi48RAM
The IPython Notebook is a powerful web app for exploring ideas and data sets with Python. It has excellent integration with Matplotlib, giving the user highly customisable static plots with ease. But for larger data sets, a static plot may not be ideal - the ability to pan, zoom, choose dynamic layers and sample the data at particular points would be nice. This talk will demonstrate just how easy it is to integrate a Web Map Service/client such as Pydap/Leaflet.js into the IPython Notebook.
This talk discusses Spark (http://spark.apache.org), the Big Data computation system that is emerging as a replacement for MapReduce in Hadoop systems, while it also runs outside of Hadoop. I discuss why the issues why MapReduce needs to be replaced and how Spark addresses them with better performance and a more powerful API.
Unified Big Data Processing with Apache Spark (QCON 2014)Databricks
While early big data systems, such as MapReduce, focused on batch processing, the demands on these systems have quickly grown. Users quickly needed to run (1) more interactive ad-hoc queries, (2) sophisticated multi-pass algorithms (e.g. machine learning), and (3) real-time stream processing. The result has been an explosion of specialized systems to tackle these new workloads. Unfortunately, this means more systems to learn, manage, and stitch together into pipelines. Spark is unique in taking a step back and trying to provide a *unified* post-MapReduce programming model that tackles all these workloads. By generalizing MapReduce to support fast data sharing and low-latency jobs, we achieve best-in-class performance in a variety of workloads, while providing a simple programming model that lets users easily and efficiently combine them.
Today, Spark is the most active open source project in big data, with high activity in both the core engine and a growing array of standard libraries built on top (e.g. machine learning, stream processing, SQL). I'm going to talk about the latest developments in Spark and show examples of how it can combine processing algorithms to build rich data pipelines in just a few lines of code.
Talk by Databricks CTO and Apache Spark creator Matei Zaharia at QCON San Francisco 2014.
(Berkeley CS186 guest lecture) Big Data Analytics Systems: What Goes Around C...Reynold Xin
(Berkeley CS186 guest lecture)
Big Data Analytics Systems: What Goes Around Comes Around
Introduction to MapReduce, GFS, HDFS, Spark, and differences between "Big Data" and database systems.
OpenStreetMap is an open/collaborative map considered the Wikipedia of maps. There are tons of information available online and even books about it… anyway, not the history of OSM we will discuss here, but rather we will take a look on the impressive dataset behind and how such a big map can be analysed with modern technologies as Apache Spark.
This page provides access to information about how to integrate Apache Hadoop with Lustre. We have made several enhancements to improve the use of Hadoop with Lustre and have conducted performance tests to compare the performance of Lustre vs. HDFS when used with Hadoop.
http://wiki.lustre.org/index.php/Integrating_Hadoop_with_Lustre
This is a slide deck that I have been using to present on GeoTrellis for various meetings and workshops. The information is speaks to GeoTrellis pre-1.0 release in Q4 of 2016.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
Le nuove frontiere dell'AI nell'RPA con UiPath Autopilot™UiPathCommunity
In questo evento online gratuito, organizzato dalla Community Italiana di UiPath, potrai esplorare le nuove funzionalità di Autopilot, il tool che integra l'Intelligenza Artificiale nei processi di sviluppo e utilizzo delle Automazioni.
📕 Vedremo insieme alcuni esempi dell'utilizzo di Autopilot in diversi tool della Suite UiPath:
Autopilot per Studio Web
Autopilot per Studio
Autopilot per Apps
Clipboard AI
GenAI applicata alla Document Understanding
👨🏫👨💻 Speakers:
Stefano Negro, UiPath MVPx3, RPA Tech Lead @ BSP Consultant
Flavio Martinelli, UiPath MVP 2023, Technical Account Manager @UiPath
Andrei Tasca, RPA Solutions Team Lead @NTT Data
By Design, not by Accident - Agile Venture Bolzano 2024
OSM data in MariaDB / MySQL - All the world in a few large tables
1. OSM data in MariaDB/MySQL
All the world in a few large tables
Well, almost ...
Hartmut Holzgraefe
SkySQL AB hartmut@skysql.com
February 1, 2014
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
1 / 30
2. Overview
1
MySQL, MariaDB and GIS
2
OpenStreetMap
3
osm2pgsql
4
Examples
5
The End ...
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
2 / 30
3. MySQL, MariaDB and GIS
1
MySQL, MariaDB and GIS
History
Current Status
Roadmap
2
OpenStreetMap
3
osm2pgsql
4
Examples
5
The End ...
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
3 / 30
4. History
First appeard in MySQL 4.1 (2004)
... with MBR relations only
Lab Release adds true spatial relations (200?)
MariaDB 5.3 GA with true spatial relations (2011)
MySQL 5.6 GA with true spatial relations (2013)
... to be continued ...
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
4 / 30
5. MBR is not enough
MBR CONTAINS()
Hartmut Holzgraefe (SkySQL)
True ST CONTAINS()
OSM data in MariaDB/MySQL
February 1, 2014
5 / 30
6. Current Status
Spatial relations work
... but the world is still flat (no projections)
OK for many use cases
... but be aware of gotchas like DISTANCE()
GEOMETRY types in all storage engines
... but only MyISAM has SPATIAL indexes
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
6 / 30
7. MariaDB 10.1 Roadmap
System Tables (spatial ref sys, geometry columns, ...)
Precision math calculations and storage
Coordinate transformations / projections
3rd coordinate (e.g. for altitude)
all spatial functions required by OGC
spatial aware optimizer
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
7 / 30
8. ... and beyond
SPATIAL indexes in other storage engines
3D calculations
client side support for GIS transformations
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
8 / 30
9. Openstreetmap
1
MySQL, MariaDB and GIS
2
OpenStreetMap
Intro
Core Data Model
Data Access
Data Import
3
osm2pgsql
4
Examples
5
The End ...
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
9 / 30
10. OpenStreetMap History
founded in 2004 by Steve
Coast
data under open license
(CC-BY-SA first, now
ODBL)
1.5 million contributors
2 billion map nodes
200 million ways
2 million relations
almost 4 billion GPX points
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
10 / 30
12. ... and raw data
Raw map data can be used for
other things, too:
for routing
for coverage checks
for flight simulators
for science
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
12 / 30
13. Core Data Model
Just three simple things
Nodes (Points)
Ways
Relations
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
13 / 30
14. Nodes
Nodes describe a single point at a specific location using:
A numeric ID
Object version, Timestamp of last change, User
Node coordinates
Node attributes as key/value pairs
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
14 / 30
15. Ways
Ways form an open or closed line by connecting nodes, using:
A numeric ID
Object version, Timestamp of last change, User
An ordered list of node IDs
Way attributes as key/value pairs
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
15 / 30
16. Relations
Relations bundle objects to describe more complex relations, using:
A numeric ID
Object version, Timestamp of last change, User
Ordered lists of member nodes, ways and sub-relations
Optional member roles
Attributes as key/value pairs
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
16 / 30
17. Data Access
The main database is not exposed directly
Only one central instance, accessible via “The API”
API meant for editor applications only
Full data export once a week (“the planet”)
Plus daily, hourly, minutely diffs
Two file formats for planets:
.osm XML based, usually bz2 compressed (32GB packed, 400GB
unpacked)
.pbf compact binary format based on Google ProtoBuf (23GB)
Regional extracts available by 3rd parties, e.g. GeoFabrik.de
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
17 / 30
18. Data Import
The raw data is not really suitable for most purposes, esp. rendering
Several import/preprocessing tools provide more convenient schemas,
e.g. by
... only extracting certain attributes
... making a difference beween ways and areas
... resolving relations into simpler objects
Besides osm2pgsql that I’m about to talk about in a minute there are also
impOsm, ...
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
18 / 30
19. osm2pgsql
1
MySQL, MariaDB and GIS
2
OpenStreetMap
3
osm2pgsql
Block Diagram
Data Model Again
Adding MySQL Support
Performance
4
Examples
5
The End ...
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
19 / 30
20. osm2pgsql
osm2pgsql is a tool
written in C
with a small C++ part now
reads OSM data
preprocesses it
stores results in relational tables
originally in PostGIS only
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
20 / 30
21. Block Diagram
Figure : osm2pgsql block diagram
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
21 / 30
22. Data Model Again
prefix point for single node POIs
prefix line for linear 2D objects like roads, rivers, power lines ...
prefix roads a subset of the above, optimizer for rendering
prefix polygon objects covering an area: buildings, landuse,
administrative borders ...
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
22 / 30
23. Adding MySQL Support
turned out to be more tricky than thought
some core parts directly called PostgreSQL functions
a lot of general functionality was hidden in PostgreSQL specific
modules
MySQL output module works, could be faster though
MySQL middle layer is “code complete” ...
... but crashes while processing relations :(
so for now imports are limited by RAM size
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
23 / 30
24. Import performance
imports currently take about 4-5 times as long
... as we have no direct equivalent to COPY
osm2pgsql at less than 50% CPU only
... so switching to async API would be a 2x win already
... with multi-insert even more so
index building is faster ...
... but may not be once we get I/O bound
tables on disk are of similar size
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
24 / 30
25. Query performance
select
from
join
on
where
and
count(*)
nrw_point n
nrw_polygon p
st_contains(p.way,n.way)
p.name = ’Bielefeld’
n.amenity=’post_box’;
Data Set
Germany
Northrhine-Westfalia
same with better indexes
Hartmut Holzgraefe (SkySQL)
MySQL 5.5 (MBR)
MariaDB 5.5
PostGIS
15.8s
2.7s
0.2s
16.5s
3.1s
0.2s
?
6.1s
.04s
OSM data in MariaDB/MySQL
February 1, 2014
25 / 30
26. Examples
1
MySQL, MariaDB and GIS
2
OpenStreetMap
3
osm2pgsql
4
Examples
5
The End ...
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
26 / 30
27. The End ...
1
MySQL, MariaDB and GIS
2
OpenStreetMap
3
osm2pgsql
4
Examples
5
The End ...
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
27 / 30
28. References
Contact hartmut@skysql.com
MariaDB GIS https://mariadb.com/kb/en/gis-functionality/
MySQL GIS http://dev.mysql.com/doc/refman/5.6/en/spatialextensions.html
OpenStreetMap http://openstreetmap.org/
MapCompare http://mc.bbbike.org/mc/
RiverMap http://www.kompf.de/gps/rivermap.html
osm2pgsql https://wiki.openstreetmap.org/wiki/Osm2pgsql
My Code https://github.com/hholzgra/osm2pgsql/tree/devel-mysql
Table Files http://php-groupies.de/gis-data/ (soon)
Hartmut Holzgraefe (SkySQL)
OSM data in MariaDB/MySQL
February 1, 2014
28 / 30