Lost in the Clouds
An example of a schemaless database
      application in the cloud




      SlipstreamUSA :: March 25,...
The Project

    OpenStreetMap

    A free map of the planet

        Free geospatial data
    
        Licensed under ...
Coverage




           CC-BY-SA 2.0 © OpenStreetMap
The OpenStreetMap Database Server




    In the cloud

    MySQL

    Ways: 24,956,171

    Nodes: 305,956,583

    T...
Interfaces


    API

        For updates
    
        (1,000 users per day)
        Queries of up to
    
         0.2...
Problem Statement


    Problem

    Want to be able to query
    the data in bigger chunks
    and to do it more
    sel...
XAPI Service



    GT.M or Cache

    250Gbyte database

        Nodes: 30Gb
    
        Node tags: 115Gb
    
     ...
XAPI Service
    In the cloud

    Queries of up to 100² degrees

    Query by tags and tag values

    Scaleable

   ...
XAPI Service

        Geospatial Index – quadstrings
    




             adaabcdcabaadab
XAPI Service
                                              <way id='27016525'>
                                           ...
Deployment



    Currently 5 instances:

        London
    
        Paris
    
        San Diego
    
        Zurich...
Wikipedia: Query to Map


         {{osm-query |way=name=Arizona Canal}}




                                             ...
Internals


    Debian

    Apache web-server

    REST

        URI represents a
    
        resource
        Xpath ...
XAPI Future

        Load Sharing / balancing
    
        More Xpath-like queries
    
        JSON output
    
      ...
Thank you




            Thank you
              Questions?




                           CC-BY-SA 2.0 Image © ITO! Data...
Upcoming SlideShare
Loading in...5
×

Lost In The Clouds

1,034

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,034
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Lost In The Clouds

  1. 1. Lost in the Clouds An example of a schemaless database application in the cloud SlipstreamUSA :: March 25, 2009
  2. 2. The Project OpenStreetMap  A free map of the planet  Free geospatial data  Licensed under CC-BY-SA  Created from  Crowd-sourced contributions  90,000 registered users  1,000+ contributors per day  Used for:  Wikipedia  Flikr  OpenCycleMap.org  OpenPisteMap.org  YourNavigation.org  Mobile devices  © Constantin Litvak
  3. 3. Coverage CC-BY-SA 2.0 © OpenStreetMap
  4. 4. The OpenStreetMap Database Server In the cloud  MySQL  Ways: 24,956,171  Nodes: 305,956,583  Tags: 3.6 billion  highway=motorway  name=Broadway  foo=bar  CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
  5. 5. Interfaces API  For updates  (1,000 users per day) Queries of up to  0.25² degrees Weekly Planet Dump  100Gbyte XML file  Changes  Feed of changes  Daily / Hourly / Minutely  One file per minute  (~30Kb compressed) CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
  6. 6. Problem Statement Problem  Want to be able to query the data in bigger chunks and to do it more selectively Solution  Cloud based schemaless database with simple to use query interface and lots of indexes CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
  7. 7. XAPI Service GT.M or Cache  250Gbyte database  Nodes: 30Gb  Node tags: 115Gb  Node indexes: 70Gb  Ways: 21Gb  Way tags: 10Gb  Way indexes: 18Gb  Apache + serverLink  Licensed as AGPL  CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
  8. 8. XAPI Service In the cloud  Queries of up to 100² degrees  Query by tags and tag values  Scaleable  REST style interface  http://xapi.openstreetmap.org/api/0.6/* [tourism=hotel] [bbox=-75.7,40.0,-75.4,40.2] Import planet dump  Import minute diff  CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
  9. 9. XAPI Service Geospatial Index – quadstrings  adaabcdcabaadab
  10. 10. XAPI Service <way id='27016525'> <nd ref='296138118'/> <nd ref='296138119'/> ^way(27016525)=quot;adaabcdcabaadabquot; <nd ref='296138120'/> ^way(27016525,1)=296138118 <nd ref='296138121'/> ^way(27016525,2)=296138119 <nd ref='296138118'/> <tag k='addr:housenumber' v='2'/> ^way(27016525,3)=296138120 <tag k='building' v='yes'/> ^way(27016525,4)=296138121 </way> ^way(27016525,5)=296138118 ^waytag(27016525,quot;addr:housenumberquot;)=2 ^waytag(27016525,quot;buildingquot;)=quot;yesquot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27016525)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028298)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028299)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028326)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028327)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035972)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035973)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035974)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035975)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035984)=quot;“
  11. 11. Deployment Currently 5 instances:  London  Paris  San Diego  Zurich  Amsterdam  (Wikimedia-de) More servers needed:  Internet connectivity  300Gb disk storage  CC-BY-SA 2.0 Image © Nikolaj Data © OpenStreetMap
  12. 12. Wikipedia: Query to Map {{osm-query |way=name=Arizona Canal}} CC-BY-SA 2.0 © OpenStreetMap
  13. 13. Internals Debian  Apache web-server  REST  URI represents a  resource Xpath flavoring  serverLink  WebLink emulator  GT.M or Cache  POM  AGPL 
  14. 14. XAPI Future Load Sharing / balancing  More Xpath-like queries  JSON output  Generic indexing service  More servers 
  15. 15. Thank you Thank you Questions? CC-BY-SA 2.0 Image © ITO! Data © OpenStreetMap
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×