OpenStreetMap is an open/collaborative map considered the Wikipedia of maps. There are tons of information available online and even books about it… anyway, not the history of OSM we will discuss here, but rather we will take a look on the impressive dataset behind and how such a big map can be analysed with modern technologies as Apache Spark.

1. OpenStreetMap in the age of Spark
@adrianulbona

2. OpenSteetMap
- The Wikipedia of Maps
- https://www.openstreetmap.org
- nodes: geo-localized points on the map
- ways: roads, building contours, borders (multiple nodes), …
- relations: highways (multiple ways), schools (building contours), ...

3. OpenSteetMap - the data
The data is available at http://planet.openstreetmap.org and it comes in two
formats:
- XML ~ around 53 GB
- PBF ~ around 34 GB
Take a look at http://osmstats.neis-one.org, if interested on how the map evolves
on a daily basis.

4. OpenSteetMap - story 1
- download the PBF/XML files
- wait days to import the data in PostgreSQL
- write some SQL
- grab a coffee
- grab a second coffee
- see some query results
- …
- manage scary scripts that keep your OSM DB updated

5. OpenSteetMap - story 2
- download the PBF/XML files
- extract various pieces of information in obscure CSVs
- write MR jobs full of string parsing bugs
- run the jobs
- grab a coffee
- fix your job
- grab a second coffee
- …
- MR is not cool anymore

6. OpenSteetMap - story 3
- one day some weird guy comes and asks:
what is the total road network length from OSM?
- you have ways as collections node ids
- you have nodes with ids and coordinates (latitude, longitude)
- all this mixed-up in a huge protobuf file
- options?

7. what the actual problem is?

8. we need parallel data access

9. we need the data structured

10. Spark will handle the rest

11. Parquet will give you more …
performance

12. Apache Parquet
Apache Parquet is a columnar storage format available to any project in the
Hadoop ecosystem, regardless of the choice of data processing framework, data
model or programming language.
google paper: Dremel - Interactive Analysis of Web-Scale Datasets
twitter blog: Dremel Made Simple with Parquet

13. Apache Parquet
row-based storage
column-based storage
data

14. Apache Parquet
- protobuf-like
- primitives, arrays, structs
- bonus: nested structs

15. Parquet + Spark

16. osm-parquetizer
- github.com/adrianulbona/osm-parquetizer
- input: one OSM PBF file
- output: one parquet file for each entity type (nodes, ways, relations)
- minutes for countries like Romania
- between 2 and 4 hours for the entire planet
- parquet files size ~ 3 x original PBF (~ 100 GB the planet)

17. http://bit.ly/2n9TRF3