High Performance OSM Data Manipulation With Osmium - State of the Map 2013


Published on

*** Presented by Jochen Topf at State of the Map 2013
*** For the video of this presentation please see http://lanyrd.com/2013/sotm/scpkqw/
*** Full schedule available at http://wiki.openstreetmap.org/wiki/State_Of_The_Map_2013

Osmium is a highly flexible and performant C++ library for working with OpenStreetMap data. It helps with reading, writing, and filtering OSM data, OSM history data and OSM change files in XML or PBF format. Osmium assembles way and (multi)polygon geometries and converts them into many GIS formats such as shapefiles. It also contains code to store OSM data efficiently and much more. All of this in an easy to use C++ header-only library. Some functions are also available from Javascript to make it's use even easier. The talks shows what we have learned over the years about high- performance OSM data manipulation, the current state of Osmium and what the future might bring.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

High Performance OSM Data Manipulation With Osmium - State of the Map 2013

  1. 1. High Performance OSM Data Manipulation With Osmium Jochen Topf
  2. 2. CC-BY http://www.flickr.com/photos/x1brett/4562610437/
  3. 3. Typical Problems Slow. Needs a lot of memory/disk space. Doesn't work with entire planet.
  4. 4. OSM Data There isn't all that much data (current planet PBF: 23 GB) But we need to store it efficiently!
  5. 5. OSM Data Often we can work on the data piece by piece Streaming
  6. 6. C++
  7. 7. Osmium A fast and flexible C++ library for working with OSM data
  8. 8. Modular CC-BY http://www.flickr.com/photos/jronaldlee/4479381576/
  9. 9. Has to work with data of entire planet! ...or a small extract!
  10. 10. Features Basic OSM objects: Nodes, ways, relations, tags, ... And operations on them. Tag filtering
  11. 11. Input/Output Read from: file, stdin or URL. Write to: file or stdout. XML or PBF. Compressed or uncompressed. OSM data (.osm) or changes (.osc). With or without history.
  12. 12. Geometry Add node locations to ways Assemble Multipolygons Convert geometries to WKT, WKB, OGR, GEOS Line length (haversine)
  13. 13. Handler OSM file Reader Handler Writer Temp. Storage For converter and filter OSM file
  14. 14. Example: main #include <osmium/io/any_input.hpp> int main(int argc, char* argv[]) { osmium::io::Reader reader(argv[1]); NamesHandler handler; } reader.open(); reader.push(handler);
  15. 15. Example: handler #include <iostream> #include <osmium/handler.hpp> struct NamesHandler : public osmium::handler::Handler<NamesHandler> { void node(const osmium::Node& node) { auto n = node.tags().get_value_by_key("name"); if (n) std::cout << n << std::endl; } };
  16. 16. taginfo.openstreetmap.org Statistics for 61 million different tags on 2.2 billion objects. Runs for about two hours every day. Needs less than 8 GB RAM.
  17. 17. Linux Mac OS X Windows
  18. 18. Osmium History Development started October 2010 Recently started „New Osmium“
  19. 19. Th The New Osmium Object Storage/Transport Indexes Multithreading (no multipolygon support yet) e N ew O sm iu m
  20. 20. Th C++11 e N ew Modern C++ Official ISO standard Works with GCC 4.7.3, clang 3.2 Easier to write, more efficient, cleaner code O sm iu m
  21. 21. Th e Multithreading N ew O sm Better design to take advantage of multithreading Dynamic memory allocation is even worse than with single thread iu m
  22. 22. Th e osmcode.org Osmium and Osmium-based software github.com/osmcode N ew O sm iu m
  23. 23. Javascript Old Osmium: osmjs New Osmium: Working on NodeJS module
  24. 24. Status Old Osmium: Tried and tested, In production for >2 years New Osmium: New and untested, Not production ready yet
  25. 25. Thanks!
  26. 26. Hackday tomorrow! Thanks! wiki.osm.org/wiki/Osmium github.com/joto/osmium osmcode.org github.com/osmcode/libosmium Jochen Topf jochen@topf.org jochentopf.com