More Related Content
Similar to Lost In The Clouds
Similar to Lost In The Clouds (20)
More from george.james (20)
Lost In The Clouds
- 1. Lost in the Clouds
An example of a schemaless database
application in the cloud
SlipstreamUSA :: March 25, 2009
- 2. The Project
OpenStreetMap
A free map of the planet
Free geospatial data
Licensed under CC-BY-SA
Created from
Crowd-sourced contributions
90,000 registered users
1,000+ contributors per day
Used for:
Wikipedia
Flikr
OpenCycleMap.org
OpenPisteMap.org
YourNavigation.org
Mobile devices
© Constantin Litvak
- 3. Coverage
CC-BY-SA 2.0 © OpenStreetMap
- 4. The OpenStreetMap Database Server
In the cloud
MySQL
Ways: 24,956,171
Nodes: 305,956,583
Tags: 3.6 billion
highway=motorway
name=Broadway
foo=bar
CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
- 5. Interfaces
API
For updates
(1,000 users per day)
Queries of up to
0.25² degrees
Weekly Planet Dump
100Gbyte XML file
Changes
Feed of changes
Daily / Hourly / Minutely
One file per minute
(~30Kb compressed)
CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
- 6. Problem Statement
Problem
Want to be able to query
the data in bigger chunks
and to do it more
selectively
Solution
Cloud based schemaless
database with simple to
use query interface and
lots of indexes
CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
- 7. XAPI Service
GT.M or Cache
250Gbyte database
Nodes: 30Gb
Node tags: 115Gb
Node indexes: 70Gb
Ways: 21Gb
Way tags: 10Gb
Way indexes: 18Gb
Apache + serverLink
Licensed as AGPL
CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
- 8. XAPI Service
In the cloud
Queries of up to 100² degrees
Query by tags and tag values
Scaleable
REST style interface
http://xapi.openstreetmap.org/api/0.6/*
[tourism=hotel]
[bbox=-75.7,40.0,-75.4,40.2]
Import planet dump
Import minute diff
CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
- 9. XAPI Service
Geospatial Index – quadstrings
adaabcdcabaadab
- 10. XAPI Service
<way id='27016525'>
<nd ref='296138118'/>
<nd ref='296138119'/>
^way(27016525)=quot;adaabcdcabaadabquot; <nd ref='296138120'/>
^way(27016525,1)=296138118 <nd ref='296138121'/>
^way(27016525,2)=296138119 <nd ref='296138118'/>
<tag k='addr:housenumber' v='2'/>
^way(27016525,3)=296138120
<tag k='building' v='yes'/>
^way(27016525,4)=296138121
</way>
^way(27016525,5)=296138118
^waytag(27016525,quot;addr:housenumberquot;)=2
^waytag(27016525,quot;buildingquot;)=quot;yesquot;
^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27016525)=quot;quot;
^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028298)=quot;quot;
^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028299)=quot;quot;
^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028326)=quot;quot;
^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028327)=quot;quot;
^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035972)=quot;quot;
^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035973)=quot;quot;
^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035974)=quot;quot;
^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035975)=quot;quot;
^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035984)=quot;“
- 11. Deployment
Currently 5 instances:
London
Paris
San Diego
Zurich
Amsterdam
(Wikimedia-de)
More servers needed:
Internet connectivity
300Gb disk storage
CC-BY-SA 2.0 Image © Nikolaj Data © OpenStreetMap
- 13. Internals
Debian
Apache web-server
REST
URI represents a
resource
Xpath flavoring
serverLink
WebLink emulator
GT.M or Cache
POM
AGPL
- 14. XAPI Future
Load Sharing / balancing
More Xpath-like queries
JSON output
Generic indexing service
More servers
- 15. Thank you
Thank you
Questions?
CC-BY-SA 2.0 Image © ITO! Data © OpenStreetMap