Lost in the Clouds
An example of a schemaless database
application in the cloud
SlipstreamUSA :: March 25, 2009

The Project
OpenStreetMap

A free map of the planet

Free geospatial data

Licensed under CC-BY-SA

Created from

Crowd-sourced contributions

90,000 registered users

1,000+ contributors per day

Used for:

Wikipedia

Flikr

OpenCycleMap.org

OpenPisteMap.org

YourNavigation.org

Mobile devices

© Constantin Litvak

Coverage
CC-BY-SA 2.0 © OpenStreetMap

The OpenStreetMap Database Server
In the cloud

MySQL

Ways: 24,956,171

Nodes: 305,956,583

Tags: 3.6 billion

highway=motorway

name=Broadway

foo=bar

CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap

Interfaces
API

For updates

(1,000 users per day)
Queries of up to

0.25² degrees
Weekly Planet Dump

100Gbyte XML file

Changes

Feed of changes

Daily / Hourly / Minutely

One file per minute

(~30Kb compressed)
CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap

Problem Statement
Problem

Want to be able to query
the data in bigger chunks
and to do it more
selectively
Solution

Cloud based schemaless
database with simple to
use query interface and
lots of indexes
CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap

XAPI Service
GT.M or Cache

250Gbyte database

Nodes: 30Gb

Node tags: 115Gb

Node indexes: 70Gb

Ways: 21Gb

Way tags: 10Gb

Way indexes: 18Gb

Apache + serverLink

Licensed as AGPL

CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap

XAPI Service
In the cloud

Queries of up to 100² degrees

Query by tags and tag values

Scaleable

REST style interface

http://xapi.openstreetmap.org/api/0.6/*
[tourism=hotel]
[bbox=-75.7,40.0,-75.4,40.2]
Import planet dump

Import minute diff

CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap

XAPI Service
Geospatial Index – quadstrings

adaabcdcabaadab

XAPI Service
<way id='27016525'>
<nd ref='296138118'/>
<nd ref='296138119'/>
^way(27016525)=\"adaabcdcabaadab\" <nd ref='296138120'/>
^way(27016525,1)=296138118 <nd ref='296138121'/>
^way(27016525,2)=296138119 <nd ref='296138118'/>
<tag k='addr:housenumber' v='2'/>
^way(27016525,3)=296138120
<tag k='building' v='yes'/>
^way(27016525,4)=296138121
</way>
^way(27016525,5)=296138118
^waytag(27016525,\"addr:housenumber\")=2
^waytag(27016525,\"building\")=\"yes\"
^wayx(\"building\",\"*\",\"adaabcdcabaadab\",27016525)=\"\"
^wayx(\"building\",\"*\",\"adaabcdcabaadab\",27028298)=\"\"
^wayx(\"building\",\"*\",\"adaabcdcabaadab\",27028299)=\"\"
^wayx(\"building\",\"*\",\"adaabcdcabaadab\",27028326)=\"\"
^wayx(\"building\",\"*\",\"adaabcdcabaadab\",27028327)=\"\"
^wayx(\"building\",\"*\",\"adaabcdcabaadab\",27035972)=\"\"
^wayx(\"building\",\"*\",\"adaabcdcabaadab\",27035973)=\"\"
^wayx(\"building\",\"*\",\"adaabcdcabaadab\",27035974)=\"\"
^wayx(\"building\",\"*\",\"adaabcdcabaadab\",27035975)=\"\"
^wayx(\"building\",\"*\",\"adaabcdcabaadab\",27035984)=\"“

Deployment
Currently 5 instances:

London

Paris

San Diego

Zurich

Amsterdam

(Wikimedia-de)
More servers needed:

Internet connectivity

300Gb disk storage

CC-BY-SA 2.0 Image © Nikolaj Data © OpenStreetMap

Wikipedia: Query to Map
{{osm-query |way=name=Arizona Canal}}
CC-BY-SA 2.0 © OpenStreetMap

Internals
Debian

Apache web-server

REST

URI represents a

resource
Xpath flavoring

serverLink

WebLink emulator

GT.M or Cache

POM

AGPL


XAPI Future
Load Sharing / balancing

More Xpath-like queries

JSON output

Generic indexing service

More servers


Thank you
Thank you
Questions?
CC-BY-SA 2.0 Image © ITO! Data © OpenStreetMap