Your SlideShare is downloading. ×
Mongodb, our Swiss Army Knife Database
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Mongodb, our Swiss Army Knife Database


Published on

Experience feedback on 10 monthes of happy mongodb usage at fotopedia. …

Experience feedback on 10 monthes of happy mongodb usage at fotopedia.

You may also checkout:

Published in: Technology

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. MongoDB at fotopedia Timeline storage Our Swiss Army Knife Database
  • 2. MongoDB at fotopedia • Context • Wikipedia data storage • Metacache
  • 3. Fotopedia • Fotonauts, an American/French company • Photo — Encyclopedia • Heavily interconnected system : flickr, facebook, wikipedia, picassa, twitter… • MongoDB in production since last october • main store lives in MySQL… for now
  • 4. First contact • Wikipedia imported data
  • 5. Wikipedia queries • wikilinks from one article • links to one article • geo coordinates • redirect • why not use wikipedia API ?
  • 6. Download ~ 5.7GB gzip XML Geo Redirect Backlink Related ~12GB tabular data
  • 7. Problem Load ~12GB into a K/V store
  • 8. CouchDB 0.9 attempt • CouchDB had no dedicated import tool • need to go through HTTP / Rest API
  • 9. “DATA LOADING” LOADING! (obviously hijacked from
  • 10. Problem, rephrased Load ~12GB into any K/V store in hours, not days
  • 11. Hadoop HBase ? • as we were already using Hadoop Map/ Reduce for preparation • bulk load was just emerging at that time, requiring to code against HBase private APIs, generate the data in an ad-hoc binary format, ...
  • 12. photo by on Flickr
  • 13. Problem, rerephrased Load ~12GB into any K/V store in hours, not days without wasting a week on development and another week on setup and several months on tuning please ?
  • 14. MongoDB attempt • Transforming the tabular data into a JSON form : about half an hour or code, 45 minutes of hadoop parallel processing • setup mongo server : 15 minutes • mongoimport : 3 minutes to start it, 90 minutes to run • plug RoR app on mongo : minutes • prototype was done in a day
  • 15. Batch Synchronous Download ~ 5.7GB gzip Geo Redirect Backlink Ruby on Rails Related ~12GB, 12M docs
  • 16. Hot swap ? • Indexing was locking everything. • Just run two instances of MongoDB. • One instance is servicing the web app • One instance is asleep or loading data • One third instance knows the status of the two instances.
  • 17. We loved: • JSON import format • efficiency of mongoimport • simple and flexible installation • just one cumbersome dependency • easy to start (we use runit) • easy to have several instances on one box
  • 18. Second contact • itʼs just all about graphes, anyway. • wikilinks • people following people • related community albums • and soon, interlanguage links
  • 19. all about graphes... • ... and itʼs also all about cache. • The application needs to “feel” faster, letʼs cache more. • The application needs to “feel” right, so letʼs cache less. • or — big sigh — invalidate.
  • 20. Page fragment caching photo by Mykl Roventine on Flickr Nginx SSI photo by Aires Dos Santos Varnish HTTP cache RoR application photo by Leslie Chatfield on Flickr
  • 21. Haiku ? There are only two hard things in Computer Science: cache invalidation and naming things. Phil Karlton
  • 22. Naming things • REST have been a strong design principle in fotopedia since the early days, and the efforts are paying.
  • 23. /en/2nd_arrondissement_of_Paris /en/Paris/fragment/left_col /users/john/fragment/contrib /en/Paris/fragment/related
  • 24. Invalidating • Rest allows us to invalidate by URL prefix. • When the Paris album changes, we have to invalidate /en/Paris.*
  • 25. Varnish invalidation • Varnish built-in regexp based invalidation is not designed for intensive, fine grained invalidation. • We need to invalidate URLs individually.
  • 26. /en/Paris.* /en/Paris /en/Paris/fragment/left_col /en/Paris/photos.json?skip=0&number=20 /en/Paris/photos.json?skip=13&number=27
  • 27. Metacache workflow /en/Paris.* invalidation worker Nginx SSI /en/Paris /en/Paris/fragment/left_col /en/Paris/photos.json?skip=0&number=20 /en/Paris/photos.json?skip=13&number=27 Varnish HTTP cache varnish log /en/Paris/fragment/left_col RoR application metacache feeder
  • 28. Waw. • This time we are actually using MongoDB as a BTree. Impressive. • The metacache has been running fine for several months, and we want to go further.
  • 29. Invalidate less • We need to be more specific as to what we invalidate. • Today, if somebody votes on a photo in the Paris album, we invalidate all /en/Paris prefix, and most of it is unchanged. • We will move towards a more clever metacache.
  • 30. Metacache reloaded • Pub/Sub metacache • Have the backend send a specific header to be caught by the metacache-feeder, conaining “subscribe” message. • This header will be a JSON document, to be pushed to the metacache. • The purge commands will be mongo search queries.
  • 31. /en/Paris /en/Paris/fragment/left_col /en/Paris/photos.json?skip=0&number=20 /en/Paris/photos.json?skip=13&number=27 {url:/en/Paris, observe:[summary,links]} {url:/en/Paris/fragment/left_col, observe: [cover]} {url:/en/Paris/photos.json?skip=0&number=20, observe:[photos]} {url:/en/Paris/photos.json?skip=0&number=20, observe:[photos]}
  • 32. when somebody votes { url:/en/Paris.*, observe:photos } when the summary changes { url:/en/Paris.*, observe:summary } when the a new link is created { url:/en/Paris.*, observe:links } {url:/en/Paris, observe:[summary,links]} {url:/en/Paris/fragment/left_col, observe: [cover]} {url:/en/Paris/photos.json?skip=0&number=20, observe:[photos]} {url:/en/Paris/photos.json?skip=0&number=20, observe:[photos]}
  • 33. Other uses cases • Timeline activities storage: just one more BTree usage. • Moderation workflow data: tiny dataset, but more complex queries, map/reduce. • Suspended experimentation around log collection and analysis
  • 34. Current situation • Mysql: main data store • CouchDB: old timelines (+ chef) • MongoDB: metacache, wikipedia, moderation, new timelines • Redis: raw data cache for counters, recent activity (+ resque)
  • 35. What about the main store ? • albums are good fit for documents • votes and score may be more tricky • recent introduction of resque
  • 36. In short • Simple, fast. • Hackable: in a language most can read. • Clear roadmap. • Very helpful and efficient team. • Designed with application developer needs in mind.