Your SlideShare is downloading. ×
Lost In The Clouds
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Lost In The Clouds

988
views

Published on

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
988
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Lost in the Clouds An example of a schemaless database application in the cloud SlipstreamUSA :: March 25, 2009
  • 2. The Project OpenStreetMap  A free map of the planet  Free geospatial data  Licensed under CC-BY-SA  Created from  Crowd-sourced contributions  90,000 registered users  1,000+ contributors per day  Used for:  Wikipedia  Flikr  OpenCycleMap.org  OpenPisteMap.org  YourNavigation.org  Mobile devices  © Constantin Litvak
  • 3. Coverage CC-BY-SA 2.0 © OpenStreetMap
  • 4. The OpenStreetMap Database Server In the cloud  MySQL  Ways: 24,956,171  Nodes: 305,956,583  Tags: 3.6 billion  highway=motorway  name=Broadway  foo=bar  CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
  • 5. Interfaces API  For updates  (1,000 users per day) Queries of up to  0.25² degrees Weekly Planet Dump  100Gbyte XML file  Changes  Feed of changes  Daily / Hourly / Minutely  One file per minute  (~30Kb compressed) CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
  • 6. Problem Statement Problem  Want to be able to query the data in bigger chunks and to do it more selectively Solution  Cloud based schemaless database with simple to use query interface and lots of indexes CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
  • 7. XAPI Service GT.M or Cache  250Gbyte database  Nodes: 30Gb  Node tags: 115Gb  Node indexes: 70Gb  Ways: 21Gb  Way tags: 10Gb  Way indexes: 18Gb  Apache + serverLink  Licensed as AGPL  CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
  • 8. XAPI Service In the cloud  Queries of up to 100² degrees  Query by tags and tag values  Scaleable  REST style interface  http://xapi.openstreetmap.org/api/0.6/* [tourism=hotel] [bbox=-75.7,40.0,-75.4,40.2] Import planet dump  Import minute diff  CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
  • 9. XAPI Service Geospatial Index – quadstrings  adaabcdcabaadab
  • 10. XAPI Service <way id='27016525'> <nd ref='296138118'/> <nd ref='296138119'/> ^way(27016525)=quot;adaabcdcabaadabquot; <nd ref='296138120'/> ^way(27016525,1)=296138118 <nd ref='296138121'/> ^way(27016525,2)=296138119 <nd ref='296138118'/> <tag k='addr:housenumber' v='2'/> ^way(27016525,3)=296138120 <tag k='building' v='yes'/> ^way(27016525,4)=296138121 </way> ^way(27016525,5)=296138118 ^waytag(27016525,quot;addr:housenumberquot;)=2 ^waytag(27016525,quot;buildingquot;)=quot;yesquot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27016525)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028298)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028299)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028326)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028327)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035972)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035973)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035974)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035975)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035984)=quot;“
  • 11. Deployment Currently 5 instances:  London  Paris  San Diego  Zurich  Amsterdam  (Wikimedia-de) More servers needed:  Internet connectivity  300Gb disk storage  CC-BY-SA 2.0 Image © Nikolaj Data © OpenStreetMap
  • 12. Wikipedia: Query to Map {{osm-query |way=name=Arizona Canal}} CC-BY-SA 2.0 © OpenStreetMap
  • 13. Internals Debian  Apache web-server  REST  URI represents a  resource Xpath flavoring  serverLink  WebLink emulator  GT.M or Cache  POM  AGPL 
  • 14. XAPI Future Load Sharing / balancing  More Xpath-like queries  JSON output  Generic indexing service  More servers 
  • 15. Thank you Thank you Questions? CC-BY-SA 2.0 Image © ITO! Data © OpenStreetMap