Mongodb, our Swiss Army Knife Database


Published on

Experience feedback on 10 monthes of happy mongodb usage at fotopedia.

You may also checkout:

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Mongodb, our Swiss Army Knife Database

  1. 1. MongoDB at fotopedia Timeline storage Our Swiss Army Knife Database
  2. 2. MongoDB at fotopedia • Context • Wikipedia data storage • Metacache
  3. 3. Fotopedia • Fotonauts, an American/French company • Photo — Encyclopedia • Heavily interconnected system : flickr, facebook, wikipedia, picassa, twitter… • MongoDB in production since last october • main store lives in MySQL… for now
  4. 4. First contact • Wikipedia imported data
  5. 5. Wikipedia queries • wikilinks from one article • links to one article • geo coordinates • redirect • why not use wikipedia API ?
  6. 6. Download ~ 5.7GB gzip XML Geo Redirect Backlink Related ~12GB tabular data
  7. 7. Problem Load ~12GB into a K/V store
  8. 8. CouchDB 0.9 attempt • CouchDB had no dedicated import tool • need to go through HTTP / Rest API
  9. 9. “DATA LOADING” LOADING! (obviously hijacked from
  10. 10. Problem, rephrased Load ~12GB into any K/V store in hours, not days
  11. 11. Hadoop HBase ? • as we were already using Hadoop Map/ Reduce for preparation • bulk load was just emerging at that time, requiring to code against HBase private APIs, generate the data in an ad-hoc binary format, ...
  12. 12. photo by on Flickr
  13. 13. Problem, rerephrased Load ~12GB into any K/V store in hours, not days without wasting a week on development and another week on setup and several months on tuning please ?
  14. 14. MongoDB attempt • Transforming the tabular data into a JSON form : about half an hour or code, 45 minutes of hadoop parallel processing • setup mongo server : 15 minutes • mongoimport : 3 minutes to start it, 90 minutes to run • plug RoR app on mongo : minutes • prototype was done in a day
  15. 15. Batch Synchronous Download ~ 5.7GB gzip Geo Redirect Backlink Ruby on Rails Related ~12GB, 12M docs
  16. 16. Hot swap ? • Indexing was locking everything. • Just run two instances of MongoDB. • One instance is servicing the web app • One instance is asleep or loading data • One third instance knows the status of the two instances.
  17. 17. We loved: • JSON import format • efficiency of mongoimport • simple and flexible installation • just one cumbersome dependency • easy to start (we use runit) • easy to have several instances on one box
  18. 18. Second contact • itʼs just all about graphes, anyway. • wikilinks • people following people • related community albums • and soon, interlanguage links
  19. 19. all about graphes... • ... and itʼs also all about cache. • The application needs to “feel” faster, letʼs cache more. • The application needs to “feel” right, so letʼs cache less. • or — big sigh — invalidate.
  20. 20. Page fragment caching photo by Mykl Roventine on Flickr Nginx SSI photo by Aires Dos Santos Varnish HTTP cache RoR application photo by Leslie Chatfield on Flickr
  21. 21. Haiku ? There are only two hard things in Computer Science: cache invalidation and naming things. Phil Karlton
  22. 22. Naming things • REST have been a strong design principle in fotopedia since the early days, and the efforts are paying.
  23. 23. /en/2nd_arrondissement_of_Paris /en/Paris/fragment/left_col /users/john/fragment/contrib /en/Paris/fragment/related
  24. 24. Invalidating • Rest allows us to invalidate by URL prefix. • When the Paris album changes, we have to invalidate /en/Paris.*
  25. 25. Varnish invalidation • Varnish built-in regexp based invalidation is not designed for intensive, fine grained invalidation. • We need to invalidate URLs individually.
  26. 26. /en/Paris.* /en/Paris /en/Paris/fragment/left_col /en/Paris/photos.json?skip=0&number=20 /en/Paris/photos.json?skip=13&number=27
  27. 27. Metacache workflow /en/Paris.* invalidation worker Nginx SSI /en/Paris /en/Paris/fragment/left_col /en/Paris/photos.json?skip=0&number=20 /en/Paris/photos.json?skip=13&number=27 Varnish HTTP cache varnish log /en/Paris/fragment/left_col RoR application metacache feeder
  28. 28. Waw. • This time we are actually using MongoDB as a BTree. Impressive. • The metacache has been running fine for several months, and we want to go further.
  29. 29. Invalidate less • We need to be more specific as to what we invalidate. • Today, if somebody votes on a photo in the Paris album, we invalidate all /en/Paris prefix, and most of it is unchanged. • We will move towards a more clever metacache.
  30. 30. Metacache reloaded • Pub/Sub metacache • Have the backend send a specific header to be caught by the metacache-feeder, conaining “subscribe” message. • This header will be a JSON document, to be pushed to the metacache. • The purge commands will be mongo search queries.
  31. 31. /en/Paris /en/Paris/fragment/left_col /en/Paris/photos.json?skip=0&number=20 /en/Paris/photos.json?skip=13&number=27 {url:/en/Paris, observe:[summary,links]} {url:/en/Paris/fragment/left_col, observe: [cover]} {url:/en/Paris/photos.json?skip=0&number=20, observe:[photos]} {url:/en/Paris/photos.json?skip=0&number=20, observe:[photos]}
  32. 32. when somebody votes { url:/en/Paris.*, observe:photos } when the summary changes { url:/en/Paris.*, observe:summary } when the a new link is created { url:/en/Paris.*, observe:links } {url:/en/Paris, observe:[summary,links]} {url:/en/Paris/fragment/left_col, observe: [cover]} {url:/en/Paris/photos.json?skip=0&number=20, observe:[photos]} {url:/en/Paris/photos.json?skip=0&number=20, observe:[photos]}
  33. 33. Other uses cases • Timeline activities storage: just one more BTree usage. • Moderation workflow data: tiny dataset, but more complex queries, map/reduce. • Suspended experimentation around log collection and analysis
  34. 34. Current situation • Mysql: main data store • CouchDB: old timelines (+ chef) • MongoDB: metacache, wikipedia, moderation, new timelines • Redis: raw data cache for counters, recent activity (+ resque)
  35. 35. What about the main store ? • albums are good fit for documents • votes and score may be more tricky • recent introduction of resque
  36. 36. In short • Simple, fast. • Hackable: in a language most can read. • Clear roadmap. • Very helpful and efficient team. • Designed with application developer needs in mind.