Your SlideShare is downloading. ×
0
OpenStreetMap Core Architecture and DevOps
OpenStreetMap Core Architecture and DevOps
OpenStreetMap Core Architecture and DevOps
OpenStreetMap Core Architecture and DevOps
OpenStreetMap Core Architecture and DevOps
OpenStreetMap Core Architecture and DevOps
OpenStreetMap Core Architecture and DevOps
OpenStreetMap Core Architecture and DevOps
OpenStreetMap Core Architecture and DevOps
OpenStreetMap Core Architecture and DevOps
OpenStreetMap Core Architecture and DevOps
OpenStreetMap Core Architecture and DevOps
OpenStreetMap Core Architecture and DevOps
OpenStreetMap Core Architecture and DevOps
OpenStreetMap Core Architecture and DevOps
OpenStreetMap Core Architecture and DevOps
OpenStreetMap Core Architecture and DevOps
OpenStreetMap Core Architecture and DevOps
OpenStreetMap Core Architecture and DevOps
OpenStreetMap Core Architecture and DevOps
OpenStreetMap Core Architecture and DevOps
OpenStreetMap Core Architecture and DevOps
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

OpenStreetMap Core Architecture and DevOps

1,350

Published on

Presentation at State of the Map US 2013. …

Presentation at State of the Map US 2013.
Video: http://vimeopro.com/openstreetmapus/state-of-the-map-us-2013/video/68087543

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,350
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
21
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. OSM Core Architecture and DevOps Grant Slater @firefishy1 sotm.us - 8 June 2013
  • 2. About Me ● South African living in London ● Member of OSM sysadmin team since 2007 ● Hardware Monkey ● Operation Working Group (planning, scaling) ● Day Job: Web Developer / Sysadmin
  • 3. OSM: Technical challenge ● >1,200,000 signed up users ● 3000 mappers a day ● 3.5 million unique visitors per month (osm.org) ● 3.0 TB Postgresql Database ● High IOPs: Concurrent Read + Write ● tile.openstreetmap.org (rendered map) – Live Map updates (Minute rendering) – Average of 2700 tiles per second. (5400 tiles per second peak) – Average: 175Mbits/s (340Mbit/s peak)
  • 4. Small Volunteer Team ● OSM Sysadmins – Tom Hughes – Grant Slater – Matt Amos ● Local sysadmin – Jon Burgess (Tile) – Sarah Hoffmann (Nominatim) ● Operation Work Group (Planning): – Andy Allan – Emilie Laffray – OSM sysadmins
  • 5. Core Services Primary – OSM.org Website – api.osm.org Editing API – planet.osm.org Raw osm data exports: weekly, daily, minutely, streaming
  • 6. Core Services Secondary Core – tile.osm.org – nominatim.osm.org (osm map search) – wiki.osm.org
  • 7. Services Tertiary ● Run by us – help.osm.org (Q&A "stackoverflow") – blog.osm.org – wiki.osmfoundation.org – otrs.osm.org (support tickets) – piwik.osm.org (site visitor analytics) – munin.osm.org (monitoring) – lists.osm.org – svn.osm.org – git.osm.org – trac.osm.org – irc.osm.org – dev.osm.org (aka toolserver) – switch2osm.org – imagery ● Run by others – taginfo.osm.org – forum.osm.org – ci.osm.org (Continuous Integration)
  • 8. Servers! FTW ● Here be Dragons bunyip draco errol eustace faffy fume gorynych grisu horntail idris jakelong katla konqi lurien nepomuk norbert orm ouroboros poldi ramoth ridgeback ridley sarel shenron smaug spike-01 spike-02 spike-03 tabaluga thorn-01 thorn-02 thorn-03 trogdor urmel yevaud zark Total: 36 ● Standardizing hardware (in progress) – HP Proliant – Supermicro Superserver
  • 9. OSM.org (www) Database Server Web/Rails Server Web/Rails Server Web/Rails Server File Server ● Ruby on Rails http://github.com/openstreetmap/openstreetmap-website
  • 10. OSM.org (www + api) Database Server Web/Rails Server Web/Rails Server Web/Rails Server Rails Server + cgimap Rails Server +cgimap Rails Server +cgimap File Server ● cgimap (api map call) http://github.com/zerebubuth/openstreetmap-cgimap
  • 11. OSM.org Database ● PostgreSQL 9.1 ● Streaming Replication to 2 read servers ● 3.0TB of data Primary DB Server Slave DB Server Slave DB Server
  • 12. Data Export ● Raw OSM XML Data http://planet.osm.org/ – Streaming – Minutely – Daily – Weekly – 370GB uncompressed
  • 13. Tile Rendering ● 1 (+1) live rendering servers ● PostGIS (osm2pgsql) ● mod_tile (rendering) http://github.com/openstreetmap/mod_tile ● Map Stylesheet Re-implement of the standard OpenStreetMap mapnik style in CartoCSS Andy Allan is presenting on Carto @ 12:15pmAndy Allan is presenting on Carto @ 12:15pm http://github.com/gravitystorm/openstreetmap-carto Preview: http://orm.osm.org/
  • 14. Tile CDN Source: http://dns.openstreetmap.org/tile.openstreetmap.org.html + http://git.osm.org/dns.git/blob/HEAD:/src/tile.openstreetmap
  • 15. Nominatim (Search) http://wiki.osm.org/Nominatim
  • 16. DevOps "Infrastructure as Code" ● Small team ● Many services, often repeated
  • 17. Chef + Git = Win
  • 18. Chef + Git ● OpenStreetMap Chef Cookbook Public Repo http://git.osm.org/chef.git ● All updates reported in #osm-dev (OFTC) http://irc.osm.org/ ● DNS in git (with commit updating) http://git.osm.org/dns.git
  • 19. Monitoring ● Proactive – http://munin.osm.org with alerts to sysadmin team ● Long term planning. Growth. ● Reactive – Pingdom (sms + email alerts)
  • 20. Future plans ● API in JSON ● Routing (osm.org + debugging) ● Additional Read + Write Partitioning of Database ● Additional Tile caching in Americas + Asia ● Scaling with growth curve – > 3% month on month – > 40% year on year
  • 21. Hardware Funding Donation Drive http://donate.osm.org 74% funding reached
  • 22. Questions?

×