Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

CartoDB Inside Out

1,693 views

Published on

Technical overview of the CartoDB platform, describing the most important pieces of an architecture that serves millions of maps every day.

Published in: Internet

CartoDB Inside Out

  1. 1. CartoDB Inside Out Jorge Sanz · jorge@cartodb.com · @xurxosanz SASIG · OSGeo.pt · Lisbon · 2015 A technical overview of a web mapping platform
  2. 2. Agenda ● Why ● CartoDB nowadays ● Architecture of the platform ○ Components ○ Setup ● How to serve dynamic data at scale No maps in this talk sorry
  3. 3. https://congosto.cartodb.com/viz/e5da12e2-9fe7-11e4-bc43-0e853d047bba/embed_map
  4. 4. http://elpais.com/elpais/2015/08/20/media/1440070260_752772.html
  5. 5. Why ● @jatorre and @saleiva start Vizzuality in April 2008 ● Company focused on visualization projects ● Every geospatial project had the same stack: ○ Database + Map Server + APIs + client code ● CartoDB: internal product to deploy faster this geo-stack
  6. 6. Mission With CartoDB everybody can analyse, visualise and share insights on location data. We are democratising location intelligence
  7. 7. CartoDB nowadays ● A VC founded company of ~70 people (NY + Madrid) ● A leading mapping service with +130.000 users ● An Open Source product that can be deployed on the cloud or on just one server (on-premises) ● Full stack solution from storage to client side libraries ● It’s a mapping editor supported by a platform (APIs)
  8. 8. CartoDB architecture
  9. 9. http://www.di.unito.it/~schifane/smellymaps/explorer.html
  10. 10. https://team.cartodb.com/u/piensaenpixel/viz/8a6530e0-5239-11e5-9cc4-0e0c41326911/public_map
  11. 11. The CartoDB platform ● Spatial database for users ● NoSQL database for configurations and cache ● Rendering, imports, database and other services ● Cache and Content Delivery Network ● JavaScript Library ● Map Editor Platform On-premises or in the Cloud Data services Your data Editor
  12. 12. The CartoDB platform CartoDB.js Odyssey.jsTorque.js Leaflet /Gmaps PostgreSQL PostGIS SQL API Redis Maps API TorqueMapnik Varnish Caching Fastly CDN CartoDBonAmazon elasticRecipes CartoDBon Amazonelastic Recipes PostgreSQL PostGIS CartoDB Server API Rest API Imports API CartoDB Editor Browser
  13. 13. Cloud setup ● 1 - load balancer (nginx) ● 1 - caching (varnish) ● N - UI servers (CartoDB UI) ● N - database servers (user data) ● 1 - Metadata server (CartoDB metadata) ● 1 - redis server (+slave) ● N - Maps API + API SQL servers
  14. 14. Postgres + Postgis ● Geodata, users and editor metadata storage ● Tiny Well Known Binary ● CartoDB - Postgres extension Redis ● key-value highly performant storage ● configuration for APIs ● cache
  15. 15. Backend services ● Ruby on Rails ● Import API (ogr2ogr) ● Background management tasks Invalidation service ● Node.js + HA Proxy + Consul ● Invalidates caches by request ● Multi-agent, replicated
  16. 16. SQL API ● node.js ● Exposes an entry point for running queries ● Acts as an export API Maps API · Windshaft ● node.js + mapnik ● renders tiles from CartoCSS + SQL ● SQL API client
  17. 17. nginx ● web server ● load balancer Varnish + Fastly/GCP ● web cache ● Content Delivery Network
  18. 18. CartoDB Map Editor the easiest web mapping tool ever
  19. 19. Map Editor ● Ruby on Rails + JavaScript ● Consumes the Imports, Maps, SQL API and its own API ● Highly focused on UX-UI ● Manage datasets (schema, privacy, etc) ● Manage maps (basemap, infowindows, other visual elements, privacy, etc) ● Define layer data source with filters and SQL editor ● Define symbology through wizards and CartoCSS editor ● Publish maps as embeds, links or with CartoDB.js
  20. 20. CartoDB.js ● Based on jQuery + Backbone + Underscore ● Leverages Leaflet or Google Maps API ● Easy interaction with the SQL and Maps API ● Manages transparently utf-grids for interactivity ● Allows easy customizations on infowindows, layer SQL and CartoCSS definitions, etc
  21. 21. Serving data at scale
  22. 22. https://scordivano.cartodb.com/viz/f55a3268-0c29-11e3-827e-5fd91394b353/embed_map
  23. 23. http://mappingkat.github.io/indian-removal/
  24. 24. Story of a tile: first request
  25. 25. Story of a tile: next request
  26. 26. Story of a tile: insert & next request
  27. 27. Cache invalidation Write queries fire a trigger that updates cdb_tablemetadata and starts the invalidation process for the tiles that use that table
  28. 28. Summary ● CartoDB is an Open Source, full stack mapping product ● Runs on a single server or distributed ● Delivers to thousands of users the superpowers of Postgis and Mapnik, not just pretty maps but also new insights ● Provides tools for users and developers ● It’s designed to scale, but we are always improving it
  29. 29. We are hiring!! http://cartodb.com/jobs
  30. 30. CartoDB Inside Out Jorge Sanz · jorge@cartodb.com · @xurxosanz SASIG · OSGeo.pt · Lisbon · 2015 http://bit.ly/cartodb-sasig2015 Muito obrigado! · Questions?
  31. 31. Links ● https://cartodb.com ● https://github.com/CartoDB/cartodb ● https://github.com/CartoDB/cartodb.js ● https://github.com/CartoDB/cartodb-postgresql ● https://github.com/CartoDB/Windshaft-cartodb ● https://github.com/CartoDB/CartoDB-SQL-API ● http://docs.cartodb.com ● http://cartodb.readthedocs.org

×