• Save
Lost In The Clouds
Upcoming SlideShare
Loading in...5
×
 

Lost In The Clouds

on

  • 835 views

 

Statistics

Views

Total Views
835
Views on SlideShare
833
Embed Views
2

Actions

Likes
0
Downloads
0
Comments
0

2 Embeds 2

http://outoftheslipstream.org 1
http://www.outoftheslipstream.com 1

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Lost In The Clouds Lost In The Clouds Presentation Transcript

    • Lost in the Clouds An example of a schemaless database application in the cloud SlipstreamUSA :: March 25, 2009
    • The Project
        • OpenStreetMap
        • A free map of the planet
          • Free geospatial data
          • Licensed under CC-BY-SA
        • Created from
          • Crowd-sourced contributions
          • 90,000 registered users
          • 1,000+ contributors per day
        • Used for:
          • Wikipedia
          • Flikr
          • OpenCycleMap.org
          • OpenPisteMap.org
          • YourNavigation.org
          • Mobile devices
      © Constantin Litvak
    • Coverage CC-BY-SA 2.0 © OpenStreetMap
    • The OpenStreetMap Database Server
        • In the cloud
        • MySQL
        • Ways: 24,956,171
        • Nodes: 305,956,583
        • Tags: 3.6 billion
          • highway=motorway
          • name=Broadway
          • foo=bar
      CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
    • Interfaces
        • API
          • For updates (1,000 users per day)
          • Queries of up to 0.25 ² degrees
        • Weekly Planet Dump
          • 100Gbyte XML file
        • Changes
          • Feed of changes
          • Daily / Hourly / Minutely
          • One file per minute (~30Kb compressed)
      CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
    • Problem Statement
        • Problem
        • Want to be able to query the data in bigger chunks and to do it more selectively
        • Solution
        • Cloud based schemaless database with simple to use query interface and lots of indexes
      CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
    • XAPI Service
        • GT.M or Cache
        • 250Gbyte database
          • Nodes: 30Gb
          • Node tags: 115Gb
          • Node indexes: 70Gb
          • Ways: 21Gb
          • Way tags: 10Gb
          • Way indexes: 18Gb
        • Apache + serverLink
        • Licensed as AGPL
      CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
    • XAPI Service
        • In the cloud
        • Queries of up to 100 ² degrees
        • Query by tags and tag values
        • Scaleable
        • REST style interface
      http://xapi.openstreetmap.org/api/0.6/* [tourism=hotel] [bbox=-75.7,40.0,-75.4,40.2]
        • Import planet dump
        • Import minute diff
      CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
    • XAPI Service
        • Geospatial Index – quadstrings
      adaabcdcabaadab
    • XAPI Service ^way(27016525)=&quot;adaabcdcabaadab&quot; ^way(27016525,1)=296138118 ^way(27016525,2)=296138119 ^way(27016525,3)=296138120 ^way(27016525,4)=296138121 ^way(27016525,5)=296138118 ^waytag(27016525,&quot;addr:housenumber&quot;)=2 ^waytag(27016525,&quot;building&quot;)=&quot;yes&quot; ^wayx(&quot;building&quot;,&quot;*&quot;,&quot;adaabcdcabaadab&quot;,27016525)=&quot;&quot; ^wayx(&quot;building&quot;,&quot;*&quot;,&quot;adaabcdcabaadab&quot;,27028298)=&quot;&quot; ^wayx(&quot;building&quot;,&quot;*&quot;,&quot;adaabcdcabaadab&quot;,27028299)=&quot;&quot; ^wayx(&quot;building&quot;,&quot;*&quot;,&quot;adaabcdcabaadab&quot;,27028326)=&quot;&quot; ^wayx(&quot;building&quot;,&quot;*&quot;,&quot;adaabcdcabaadab&quot;,27028327)=&quot;&quot; ^wayx(&quot;building&quot;,&quot;*&quot;,&quot;adaabcdcabaadab&quot;,27035972)=&quot;&quot; ^wayx(&quot;building&quot;,&quot;*&quot;,&quot;adaabcdcabaadab&quot;,27035973)=&quot;&quot; ^wayx(&quot;building&quot;,&quot;*&quot;,&quot;adaabcdcabaadab&quot;,27035974)=&quot;&quot; ^wayx(&quot;building&quot;,&quot;*&quot;,&quot;adaabcdcabaadab&quot;,27035975)=&quot;&quot; ^wayx(&quot;building&quot;,&quot;*&quot;,&quot;adaabcdcabaadab&quot;,27035984)=&quot;“ <way id='27016525'> <nd ref='296138118'/> <nd ref='296138119'/> <nd ref='296138120'/> <nd ref='296138121'/> <nd ref='296138118'/> <tag k='addr:housenumber' v='2'/> <tag k='building' v='yes'/> </way>
    • Deployment
        • Currently 5 instances:
          • London
          • Paris
          • San Diego
          • Zurich
          • Amsterdam (Wikimedia-de)
        • More servers needed:
          • Internet connectivity
          • 300Gb disk storage
      CC-BY-SA 2.0 Image © Nikolaj Data © OpenStreetMap
    • Wikipedia: Query to Map {{osm-query |way=name=Arizona Canal}} CC-BY-SA 2.0 © OpenStreetMap
    • Internals
        • Debian
        • Apache web-server
        • REST
          • URI represents a resource
          • Xpath flavoring
        • serverLink
          • WebLink emulator
        • GT.M or Cache
          • POM
        • AGPL
    • XAPI Future
        • Load Sharing / balancing
        • More Xpath-like queries
        • JSON output
        • Generic indexing service
        • More servers
    • Thank you Thank you Questions? CC-BY-SA 2.0 Image © ITO! Data © OpenStreetMap