• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Lost In The Clouds
 

Lost In The Clouds

on

  • 2,023 views

 

Statistics

Views

Total Views
2,023
Views on SlideShare
1,996
Embed Views
27

Actions

Likes
0
Downloads
5
Comments
0

3 Embeds 27

http://www.outoftheslipstream.com 25
http://www.outoftheslipstream.org 1
http://outoftheslipstream.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Lost In The Clouds Lost In The Clouds Presentation Transcript

    • Lost in the Clouds An example of a schemaless database application in the cloud SlipstreamUSA :: March 25, 2009
    • The Project OpenStreetMap  A free map of the planet  Free geospatial data  Licensed under CC-BY-SA  Created from  Crowd-sourced contributions  90,000 registered users  1,000+ contributors per day  Used for:  Wikipedia  Flikr  OpenCycleMap.org  OpenPisteMap.org  YourNavigation.org  Mobile devices  © Constantin Litvak
    • Coverage CC-BY-SA 2.0 © OpenStreetMap
    • The OpenStreetMap Database Server In the cloud  MySQL  Ways: 24,956,171  Nodes: 305,956,583  Tags: 3.6 billion  highway=motorway  name=Broadway  foo=bar  CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
    • Interfaces API  For updates  (1,000 users per day) Queries of up to  0.25² degrees Weekly Planet Dump  100Gbyte XML file  Changes  Feed of changes  Daily / Hourly / Minutely  One file per minute  (~30Kb compressed) CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
    • Problem Statement Problem  Want to be able to query the data in bigger chunks and to do it more selectively Solution  Cloud based schemaless database with simple to use query interface and lots of indexes CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
    • XAPI Service GT.M or Cache  250Gbyte database  Nodes: 30Gb  Node tags: 115Gb  Node indexes: 70Gb  Ways: 21Gb  Way tags: 10Gb  Way indexes: 18Gb  Apache + serverLink  Licensed as AGPL  CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
    • XAPI Service In the cloud  Queries of up to 100² degrees  Query by tags and tag values  Scaleable  REST style interface  http://xapi.openstreetmap.org/api/0.6/* [tourism=hotel] [bbox=-75.7,40.0,-75.4,40.2] Import planet dump  Import minute diff  CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
    • XAPI Service Geospatial Index – quadstrings  adaabcdcabaadab
    • XAPI Service <way id='27016525'> <nd ref='296138118'/> <nd ref='296138119'/> ^way(27016525)=quot;adaabcdcabaadabquot; <nd ref='296138120'/> ^way(27016525,1)=296138118 <nd ref='296138121'/> ^way(27016525,2)=296138119 <nd ref='296138118'/> <tag k='addr:housenumber' v='2'/> ^way(27016525,3)=296138120 <tag k='building' v='yes'/> ^way(27016525,4)=296138121 </way> ^way(27016525,5)=296138118 ^waytag(27016525,quot;addr:housenumberquot;)=2 ^waytag(27016525,quot;buildingquot;)=quot;yesquot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27016525)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028298)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028299)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028326)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028327)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035972)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035973)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035974)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035975)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035984)=quot;“
    • Deployment Currently 5 instances:  London  Paris  San Diego  Zurich  Amsterdam  (Wikimedia-de) More servers needed:  Internet connectivity  300Gb disk storage  CC-BY-SA 2.0 Image © Nikolaj Data © OpenStreetMap
    • Wikipedia: Query to Map {{osm-query |way=name=Arizona Canal}} CC-BY-SA 2.0 © OpenStreetMap
    • Internals Debian  Apache web-server  REST  URI represents a  resource Xpath flavoring  serverLink  WebLink emulator  GT.M or Cache  POM  AGPL 
    • XAPI Future Load Sharing / balancing  More Xpath-like queries  JSON output  Generic indexing service  More servers 
    • Thank you Thank you Questions? CC-BY-SA 2.0 Image © ITO! Data © OpenStreetMap