Lost in the Clouds An example of a schemaless database  application in the cloud SlipstreamUSA :: March 25, 2009
The Project <ul><ul><li>OpenStreetMap
A free map of the planet </li><ul><li>Free geospatial data
Licensed under CC-BY-SA </li></ul><li>Created from </li><ul><li>Crowd-sourced contributions
90,000 registered users
1,000+ contributors per day </li></ul><li>Used for: </li><ul><li>Wikipedia
Flikr
OpenCycleMap.org
OpenPisteMap.org
YourNavigation.org
Mobile devices </li></ul></ul></ul>©  Constantin Litvak
Coverage CC-BY-SA 2.0  ©  OpenStreetMap
The OpenStreetMap Database Server <ul><ul><li>In the cloud
MySQL
Ways:  24,956,171
Upcoming SlideShare
Loading in...5
×

Lost In The Clouds

381

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
381
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Lost In The Clouds"

  1. 1. Lost in the Clouds An example of a schemaless database application in the cloud SlipstreamUSA :: March 25, 2009
  2. 2. The Project <ul><ul><li>OpenStreetMap
  3. 3. A free map of the planet </li><ul><li>Free geospatial data
  4. 4. Licensed under CC-BY-SA </li></ul><li>Created from </li><ul><li>Crowd-sourced contributions
  5. 5. 90,000 registered users
  6. 6. 1,000+ contributors per day </li></ul><li>Used for: </li><ul><li>Wikipedia
  7. 7. Flikr
  8. 8. OpenCycleMap.org
  9. 9. OpenPisteMap.org
  10. 10. YourNavigation.org
  11. 11. Mobile devices </li></ul></ul></ul>© Constantin Litvak
  12. 12. Coverage CC-BY-SA 2.0 © OpenStreetMap
  13. 13. The OpenStreetMap Database Server <ul><ul><li>In the cloud
  14. 14. MySQL
  15. 15. Ways: 24,956,171
  16. 16. Nodes: 305,956,583
  17. 17. Tags: 3.6 billion </li><ul><li>highway=motorway
  18. 18. name=Broadway
  19. 19. foo=bar </li></ul></ul></ul>CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
  20. 20. Interfaces <ul><ul><li>API </li><ul><li>For updates (1,000 users per day)
  21. 21. Queries of up to 0.25 ² degrees </li></ul><li>Weekly Planet Dump </li><ul><li>100Gbyte XML file </li></ul><li>Changes </li><ul><li>Feed of changes
  22. 22. Daily / Hourly / Minutely
  23. 23. One file per minute (~30Kb compressed) </li></ul></ul></ul>CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
  24. 24. Problem Statement <ul><ul><li>Problem
  25. 25. Want to be able to query the data in bigger chunks and to do it more selectively
  26. 26. Solution
  27. 27. Cloud based schemaless database with simple to use query interface and lots of indexes </li></ul></ul>CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
  28. 28. XAPI Service <ul><ul><li>GT.M or Cache
  29. 29. 250Gbyte database </li><ul><li>Nodes: 30Gb
  30. 30. Node tags: 115Gb
  31. 31. Node indexes: 70Gb
  32. 32. Ways: 21Gb
  33. 33. Way tags: 10Gb
  34. 34. Way indexes: 18Gb </li></ul><li>Apache + serverLink
  35. 35. Licensed as AGPL </li></ul></ul>CC-BY-SA 2.0 Image © Maning Sambale Data © OpenStreetMap
  36. 36. XAPI Service <ul><ul><li>In the cloud
  37. 37. Queries of up to 100 ² degrees
  38. 38. Query by tags and tag values
  39. 39. Scaleable
  40. 40. REST style interface </li></ul></ul>http://xapi.openstreetmap.org/api/0.6/* [tourism=hotel] [bbox=-75.7,40.0,-75.4,40.2] <ul><ul><li>Import planet dump
  41. 41. Import minute diff </li></ul></ul>CC-BY-SA 2.0 Image © Ojw Data © OpenStreetMap
  42. 42. XAPI Service <ul><ul><li>Geospatial Index – quadstrings </li></ul></ul>adaabcdcabaadab
  43. 43. XAPI Service ^way(27016525)=&quot;adaabcdcabaadab&quot; ^way(27016525,1)=296138118 ^way(27016525,2)=296138119 ^way(27016525,3)=296138120 ^way(27016525,4)=296138121 ^way(27016525,5)=296138118 ^waytag(27016525,&quot;addr:housenumber&quot;)=2 ^waytag(27016525,&quot;building&quot;)=&quot;yes&quot; ^wayx(&quot;building&quot;,&quot;*&quot;,&quot;adaabcdcabaadab&quot;,27016525)=&quot;&quot; ^wayx(&quot;building&quot;,&quot;*&quot;,&quot;adaabcdcabaadab&quot;,27028298)=&quot;&quot; ^wayx(&quot;building&quot;,&quot;*&quot;,&quot;adaabcdcabaadab&quot;,27028299)=&quot;&quot; ^wayx(&quot;building&quot;,&quot;*&quot;,&quot;adaabcdcabaadab&quot;,27028326)=&quot;&quot; ^wayx(&quot;building&quot;,&quot;*&quot;,&quot;adaabcdcabaadab&quot;,27028327)=&quot;&quot; ^wayx(&quot;building&quot;,&quot;*&quot;,&quot;adaabcdcabaadab&quot;,27035972)=&quot;&quot; ^wayx(&quot;building&quot;,&quot;*&quot;,&quot;adaabcdcabaadab&quot;,27035973)=&quot;&quot; ^wayx(&quot;building&quot;,&quot;*&quot;,&quot;adaabcdcabaadab&quot;,27035974)=&quot;&quot; ^wayx(&quot;building&quot;,&quot;*&quot;,&quot;adaabcdcabaadab&quot;,27035975)=&quot;&quot; ^wayx(&quot;building&quot;,&quot;*&quot;,&quot;adaabcdcabaadab&quot;,27035984)=&quot;“ <way id='27016525'> <nd ref='296138118'/> <nd ref='296138119'/> <nd ref='296138120'/> <nd ref='296138121'/> <nd ref='296138118'/> <tag k='addr:housenumber' v='2'/> <tag k='building' v='yes'/> </way>
  44. 44. Deployment <ul><ul><li>Currently 5 instances: </li><ul><li>London
  45. 45. Paris
  46. 46. San Diego
  47. 47. Zurich
  48. 48. Amsterdam (Wikimedia-de) </li></ul><li>More servers needed: </li><ul><li>Internet connectivity
  49. 49. 300Gb disk storage </li></ul></ul></ul>CC-BY-SA 2.0 Image © Nikolaj Data © OpenStreetMap
  50. 50. Wikipedia: Query to Map {{osm-query |way=name=Arizona Canal}} CC-BY-SA 2.0 © OpenStreetMap
  51. 51. Internals <ul><ul><li>Debian
  52. 52. Apache web-server
  53. 53. REST </li><ul><li>URI represents a resource
  54. 54. Xpath flavoring </li></ul><li>serverLink </li><ul><li>WebLink emulator </li></ul><li>GT.M or Cache </li><ul><li>POM </li></ul><li>AGPL </li></ul></ul>
  55. 55. XAPI Future <ul><ul><li>Load Sharing / balancing
  56. 56. More Xpath-like queries
  57. 57. JSON output
  58. 58. Generic indexing service
  59. 59. More servers </li></ul></ul>
  60. 60. Thank you Thank you Questions? CC-BY-SA 2.0 Image © ITO! Data © OpenStreetMap

×