OpenStreetMap : Open Licensed GeoData

Harry Wood OpenStreetMap : Open Licensed Geo Data British Computer Society : Monday 27 th April 2009

Topics
OpenStreetMap purpose and premise
Data structures: Nodes, Ways ,Tags etc
Editor demo
OpenStreetMap servers and architecture
Rendering and map displays
The license
CloudMade products and services
Imports and other mapping techniques
Getting Involved
NOTE: Ran out of time for these topics on the day. The slides for these have also been removed from this deck. Could present them on another occasion!

Free as in Freedom
Open license:
Creative Commons Attribution Share-alike
“Open Content” like “Open Source”
Contributors retain ownership of copyright
People and commercial companies can use the maps for free under this license.
Details of license requirements?... coming up

Getting an Open Licensed Map
Can't copy copyrighted maps
Not allowed to import copyrighted data
Not allowed to copy from copyrighted maps
Not allowed to trace over copyrighted maps
Not allowed to “derive”
Can copy some maps, but only...
Public domain. Unrestricted (incl. relicensing)
Get permission to release with an open license (big ask)
Can create maps completely from scratch
crazy idea?

GPS traces
How it started. Gadgets!
Cheap consumer GPS units or location-aware mobiles
Record a line of dots

Record many lines of dots.... Looks like a street map. Kind of

Recording data Names of streets Types of streets (trunk, residential, motorway) One-way restrictions Footpaths, tracks, pedestrian, rivers, railways Parks, woodland, industrial areas, cemeteries POI (pubs, cash machines, post offices, post boxes, bus stops, toilets, supermarkets, restaurants, kebab shops, monuments, hotels, picnic sites, barriers, light houses, piers, sports centres, petrol stations, playgrounds, cinemas, car parks, universities, bicycle parking, tourist information, etc etc etc

Mapping Techniques
Photo Mapping (geo-located photos)
Audio mapping
Taking notes
Ditch the GPS
Taking notes
Local knowledge!
Yahoo Aerial Imagery

Mapping: A lot of effort
Gather data
GPS traces and other information
Input data
using OSM “editor” software
Requires a lot of effort
Requires a lot of people!

Community Contribution
Built by a large online community
Many hands make light work
Openly editable (and easy)
Poor quality contributions?
Gradual refinement
Assume good faith
Monitoring and correction
....Remarkably it works! Sounds familiar?

Wikipedia
Large community coming together to build something great!
Wikipedia Principles
Openly editable
Open content license
Gradual refinement
Assume good faith
“ Soft Security” Monitoring and correction

looks like wikipedia OpenStreetMap = The wikipedia of maps

Community It's big. 100,000 registered users

Community Increasing editing activity

Community Very Active Less Active Lopsided Long tail of less active users

Open Licensed Data A copyrighted map ...is a justification for OpenStreetMap (It can't be used freely therefore OSM is better) ...cannot be a source for openstreetmap Existing maps are very rarely free

Ordnance Survey
Wonderful data in the UK
OS license use of maps (and charge ££££££)
Never allow re-distributing with a different license
Very strict about copying and their definition of “derived” work

Guardian 'Free Our Data' Campaign
Lobbying government
Tax paid for data collection
Tax still pays indirectly
Economic benefits of free
Slow progress
OS might release
low quality data first
less-than-free license
Or might be privatised!
Whine about it or take action?

OSM and Ordnance Survey £many £0 Low quality High quality OS OSM

We can't can't use google maps
License their data from teleatlas
...who license data from Navteq / Teleatlas
...Ordnance Survey!
No access to underlying data
Google terms & conditions
Don't allow deriving data from their maps
Don't allow copying & re-distributing with a different license
Wonderful hi-res aerial imagery
T&Cs do not allow deriving maps (tracing)
Bought in (licensed) from multiple suppliers
Google Maps

Why not use Google Maps? Wonderful “free” (beer) mash-up API but...
Errors and omissions
Car centric. Footpaths and other details
Cycle routes and Pistes
Colours / branding - Google maps fatigue
SVG export. Custom cartography
Underlying data access!
Details of OSM map access coming up
Help OpenStreetMap!

Oxford University Website OSM has better maps of Oxford Encourages the OSM community Other uses coming up...

Nodes, Ways, Relations Node Has latitude and longitude Can stand alone, or form part of a way Way Joins together several nodes Direction sometimes matters Can form a 'closed way' (area) Relation For complex things such as routes

Tags Applied to the Nodes, Ways, & Relations Key value pairs amenity=pub name=Hare & Hounds highway=residential name=Court Street

http://www.openstreetmap.org/?lat=53.548078&lon=-2.004771&zoom=18 Data Browser demo

Data Browser

Permalink

JOSM demo

OpenStreetMap Servers Hosted in UCL Loads of bandwidth ~10 servers: Where does the data go? http://wiki.openstreetmap.org/wiki/Servers

OpenStreetMap Foundation Custodian of servers and sysadmin access Oversees funding and vehicle for fund raising Protection from copyright and liability suits

Database Server Motherboard Supermicro X7DWN+ motherboard with Intel 5400 (Seaburg) Chipset CPU 2x Intel Xeon Processor E5420 Quad Core 2.5Ghz Memory 32GB DDR2 667 ECC Disk 2x 73GB (3.5) SAS 15K 10x 450GB (3.5) SAS 15K donate.openstreetmap.org Raised £10,000 in 2 days

API
REST web service
HTTP GET & PUT
Get elements at URLs http://www.openstreetmap.org/api/0.6/node/297556642
No bloated request payloads
Ruby on Rails

Ruby on Rails
It's easy. Web + REST
Fashionable. Developers like it
Developers are our most limited resource.
It's what SteveC used
Problems?
Can't stream data from db
Memory hungry and leaks somewhere
Maybe use something else for core API

Nodes http://www.openstreetmap.org/api/0.6/node/297556642 <node id="297556642" lat="53.548223" lon="-2.0056012" version="2" changeset="648346" user="Guy" uid="10983" visible="true" timestamp="2008-09-16T20:42:44Z"> <tag k="name" v="Hare & Hounds"/> <tag k="created_by" v="Potlatch 0.10b"/> <tag k="amenity" v="pub"/> </node>

Ways http://www.openstreetmap.org/api/0.6/way/27120827 <way id="27120827" visible="true" timestamp="2008-09-19T13:19:53Z" version="2" changeset="664390" user="Guy" uid="10983"> <nd ref="298116100"/> <nd ref="297555192"/> <nd ref="297555193"/> <nd ref="297555194"/> <tag k="name" v="Court Street"/> <tag k="created_by" v="Potlatch 0.10b"/> <tag k="highway" v="residential"/> </way>

Other API calls GET a map All elements within a bounding box http://api.openstreetmap.org/api/0.6/map?bbox=11.54,48.14,11.543,48.145 PUT elements Now requires “changeset open” request Various other operations History and changeset access Get GPS points/tracks

Some database details Switched from MySQL to Postgres last weekend! Rails migrations in theory In practice. C++ scripts running all weekend Why the switch? Lots of other planned restructuring including new DB hardware Good time to do it

MySQL Generally fast and scalable enough
Quadtile indexing extension
Several annoying flaws:
schema changes cause table copies
different features on different db engines
(transactions on InnoDB, spatial on MyISAM)
silently accepts invalid utf8
constraints can't be deferred
some non-standard SQL syntax

Postgres
Addresses a lot of MySQL flaws:
Faster schema changes
Better support for transactions, utf8, etc
Personal preference of our sysops

Full Revision History Store a full history edits to elements Essential wiki-like feature Ideally provide simple roll-back Access old versions of an element Difficult to reconstruct old version of a map

Changesets
Brand new feature
Every edit belongs in a change set
Every numbered version of every object belongs in one particular changeset
Changesets have comments
Great for monitoring

Changeset Displays http://www.openstreetmap.org/browse/changesets

Changeset revert?
Reverting is still a difficult problem
Changesets are not atomic
Changeset 1 User:Sam Node 12345 v1
Changeset 2 User:Sally Node 12345 v2
Changeset 3 User:Sid Node 12345 v3
Changeset 4 User:Sally Node 12345 v3
Many interlinked elements

Conflicts
Two users editing the same element
Rarely happens actually
Version mismatch now reported
“Optimistic locking”
Editors (should) do CVS style conflict resolution
Download reveals conflict
Upload not allowed until resolved

(Watch nice video) OSM 2008: A Year of Edits http://vimeo.com/2598878

planet.osm
Snapshot of the OpenStreetMap database
Entire planet. Every node, way, relation, tag
Only 'current' data. Not history
XML formatted .osm file
5.2 GB with bzip2 compression
Uncompressed... 150 GB
Takes several hours to dump. Every Wednesday
Important part of Openness. Ensures longevity.

Osmosis
Java toolkit for OpenStreetMap
Various data transformations
Minutely, Hourly, Daily diffs .osc.gz files
Created by Osmosis. Consumed by osmosis
Streamable changes

Open Tagging
Mentioned tags briefly
amenity=pub highway=residential
Free-form open tagging. Any tags you like!
Agree on standards
Main openstreetmap.org map rendering uses one set of tags
Other map renderers, other tools, can use other tagging schemes

'Map Features' wiki page
BIG list of tags
Which tags go on this page?
Wiki proposal process
Wiki discussion and voting
Wiki dabates (& blazing rows!)
Different ways of tagging the same thing.
Things which should not be tagged
Wiki documentation

Smoothness Debate
Vehement Objections
Too subjective
Verifiability
Poor english
Disruption
Disregarding vote
Wiki fiddlers
vs Mappers
Wiki edit wars
New process?
Lock down?

The wrong way to think about tags
Come up with lots of ideas for new tags
Submit proposals, organise votes, generally fiddle with the OSM wiki a lot
Pester people to use tags in map renderings
...oh and maybe do a bit of actual mapping

The right way to think about tags
Do mapping!
Found something without a documented tag?
Search thoroughly (in mailing list too)
Use a less specific tag and qualify with type=
Use a note= tag
Just invent a tag
Do more mapping!
Discuss politely. Improve existing docs.
maybe... possibly.... do a proposal
Focus on mapping. Don't worry about rendering

TagWatch
Tag usage stats
Split by country
Tags used in conjunction

Rendering
Topic follows on although...
tagging is not just about rendering
Go from geodata (nodes, ways, relations & tags) to rasterized map images
Rendering

Which tags to render?
Thousands of different tags in the DB
Can't show them all
Choose features to show at different zoom levels
Cartography!
What do you want to emphasise ?

Rendering Toolchain

Mapnik
Open Source rendering software
Fast!
C++
Requires PostGIS database

Mapnik Stylesheet
XML format
'styles', 'filters' and 'rules'
>7000 lines long
Pre-processing steps
Cascadenik
and also...

osm2pgsql
Step before using Mapnik (& stylesheet)
load OSM data into a Postgres database
Lossy conversion. Only take tags of interest
nodes and ways -> linestrings and polygons
Slippy Map Display

Open Source JavaScript library
Dynamic slippy map on your website
WMS layers
Tile based map layers
Transparent overlay layers
Markers, Boxes, Polygons, Click events
In the end we want a map display...

Tiles
Small map images
Cacheable
Fast loading
Sized to optimize speed
Too big. Unneeded map area
Too small. Too many requests
256x256pixels

Tile Naming
Slice the world into tiles at each zoom level
Tiles are always 256x256 pixels
Represent different sized area of the world at different zoom levels

Tile Naming
Zoom level 0 has only one tile (whole world):

Tile Naming
Zoom level 1 has 2x2 tiles

Tile Naming
...
Zoom level n has 2 n x2 n tiles
...
Zoom level 18 has
262144 x 262144 tiles

Tile Naming
Every tile has a URL
http://tile.openstreetmap.org/12/2047/1362.png y Zoom Level (0-18) x
Tile naming scheme followed by OpenLayers
Same used by google maps
Looks like filesystem URL

Tiles =High Performance Computing 262144 x 262144 = 68,719,476,736 tiles inode problem! 5 kB each = 320 terabytes But then there's zoom 17.... another 80 terabytes etc...

Tiles =High Performance Computing OpenStreetMap updates?
Apply diffs
Re-render tile images!
CPU problem!

Caching and mod_tile
mod_tile
Apache module. Very fast
Render-on-demand if necessary
Clever caching
Serves old cached images and labels as dirty
Dirty tiles get re-rendered by render daemon

Bandwidth
Serving terrabytes of tile data. High bandwidth
UCL

OpenCycleMap.org
Using OpenStreetMap
Presenting special interest map
Same data. Different cartographic choices
Toolchain running on another server
Updates fed in
Passionate sub-set of the OSM community

Route relations, Cycle Parking, Bike Shops, Drink

Relief maps!

SRTM
NASA - Shuttle Radar Tomography Mission
Public Domain
Problems
Spot heights – not contours
Course grid
Voids and other anomalies

CycleMap tool chain
Downloads weekly planet dump
SRTM. More steps in the chain!
Bandwidth problems. Now hosted by CloudMade

OpenPisteMap

Hiking Map

Whitewater Map

Bus map

Kosmos
.NET (windows only)
Desktop app
Can generate tiles
wiki based
style config

osmarender
First good OSM renderer
Used to be the only way to get SVG
Complex perl XSLT
Generates SVG (XML vector graphics format)
Feed in .osm file and style config
Can't be used to generate tiles.... or can it?

'osmarender' layer

[email_address]
Distributed tile rendering
Instructions dished out from [email_address] server
Many clients download via API and upload images
'osmarender' layer
Used to provide the fastest updates
XSLT transforms & inkscape SVG rendering
Eats massive amounts of CPU
Mapnik more sensible. ...no need to distribute

Other renderers?
Plenty of scope to develop but..
high performance problem
Complex graphics problem
e.g. phprender
Needs a bit of work!

OSM License Requirements
Free to bring maps into “collective” works
Must give “attribution”
Free to create “derivative” works
must share-alike
Awkward complications:
What exactly counts as “derivative work”?
How do you give credit to the “authors”?
We want people to be free to use our maps!

ODbL + ODC-Factual
Open Data Commons
Open Database License
Factual Information License
Benefits:
copyright, database right, and contract
Expressly written for data
More strict about underlying data (forcing sharing), but less strict about end products

Commercial use is allowed!
OSM destroys business models ...or does it?
Destroys monopolies on geo data
Allowed to charge for distribution
Can't disallow further distribution
Monetary value tends towards zero
Allowed to charge for services
Distribute different formats / renderings
Solve difficult problems (+time dependant problems)
Hosting
Consulting services
Just use maps. Core business not in geo-data

Flickr

Nestoria

'Trails' iPhone app

Get Involved! http://wiki.openstreetmap.org/ http://lists.openstreetmap.org/ irc://irc.oftc.net #osm http://trac.openstreetmap.org

Harry Wood worked as an enterprise integration consultant for 8 years, but led a secret double-life as addicted contributor to wikipedia and other collaborative open content projects. He got involved in OpenStreetMap three years ago, as a mapper, wiki gardener, and developer. Since January this year (2009) he has worked for CloudMade, as a full time OpenStreetMap developer CloudMade is a company providing products and services around OpenStreetMap. More information at http://cloudmade.com These slides are (of course) freely re-usable under the Creative Commons Attribution-ShareAlike 2.0 License

OpenStreetMap : Open Licensed GeoData

by Harry Wood on Apr 29, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 7

Statistics

OpenStreetMap : Open Licensed GeoData — Webinar Transcript

http://www.slideshare.net	4
http://www.harrywood.co.uk	2
http://benjamin.chartier.free.fr	1