Experience feedback on 10 monthes of happy mongodb usage at fotopedia. You may also checkout: http://www.slideshare.net/octplane/mongodb-vs-mysql-a-devops-point-of-view

1. MongoDB at fotopedia
Timeline storage
Our Swiss Army Knife Database

2. MongoDB at fotopedia
• Context
• Wikipedia data storage
• Metacache

3. Fotopedia
• Fotonauts, an American/French company
• Photo — Encyclopedia
• Heavily interconnected system : flickr,
facebook, wikipedia, picassa, twitter…
• MongoDB in production since last october
• main store lives in MySQL… for now

4. First contact
• Wikipedia imported data

9. Wikipedia queries
• wikilinks from one article
• links to one article
• geo coordinates
• redirect
• why not use wikipedia API ?

10. Download
~ 5.7GB gzip
XML
Geo
Redirect
Backlink
Related
~12GB tabular data

11. Problem
Load ~12GB into a K/V store

12. CouchDB 0.9 attempt
• CouchDB had no dedicated import tool
• need to go through HTTP / Rest API

13. “DATA LOADING”
LOADING!
(obviously hijacked from xkcd.com)

14. Problem, rephrased
Load ~12GB into any K/V store
in hours, not days

15. Hadoop HBase ?
• as we were already using Hadoop Map/
Reduce for preparation
• bulk load was just emerging at that time,
requiring to code against HBase private APIs,
generate the data in an ad-hoc binary
format, ...

16. photo by neural.it on Flickr

17. Problem, rerephrased
Load ~12GB into any K/V store
in hours, not days
without wasting a week on development
and another week on setup
and several months on tuning
please ?

18. MongoDB attempt
• Transforming the tabular data into a JSON
form : about half an hour or code, 45 minutes
of hadoop parallel processing
• setup mongo server : 15 minutes
• mongoimport : 3 minutes to start it, 90 minutes
to run
• plug RoR app on mongo : minutes
• prototype was done in a day

19. Batch Synchronous
Download
~ 5.7GB gzip
Geo
Redirect
Backlink Ruby on Rails
Related
~12GB, 12M docs

20. Hot swap ?
• Indexing was locking everything.
• Just run two instances of MongoDB.
• One instance is servicing the web app
• One instance is asleep or loading data
• One third instance knows the status of the two
instances.

21. We loved:
• JSON import format
• efficiency of mongoimport
• simple and flexible installation
• just one cumbersome dependency
• easy to start (we use runit)
• easy to have several instances on one box

22. Second contact
• itʼs just all about graphes, anyway.
• wikilinks
• people following people
• related community albums
• and soon, interlanguage links

24. all about graphes...
• ... and itʼs also all about cache.
• The application needs to “feel” faster, letʼs
cache more.
• The application needs to “feel” right, so letʼs
cache less.
• or — big sigh — invalidate.

26. Page fragment caching
photo by Mykl Roventine on Flickr
Nginx SSI
photo by Aires Dos Santos
Varnish HTTP cache
RoR application photo by Leslie Chatfield on Flickr

27. Haiku ?
There are only two hard things
in Computer Science:
cache invalidation and naming things.
Phil Karlton

28. Naming things
• REST have been a strong design principle in
fotopedia since the early days, and the efforts
are paying.

29. /en/2nd_arrondissement_of_Paris
/en/Paris/fragment/left_col
/users/john/fragment/contrib
/en/Paris/fragment/related

30. Invalidating
• Rest allows us to invalidate by URL prefix.
• When the Paris album changes, we have to
invalidate /en/Paris.*

31. Varnish invalidation
• Varnish built-in regexp based invalidation is
not designed for intensive, fine grained
invalidation.
• We need to invalidate URLs individually.

32. /en/Paris.*
/en/Paris
/en/Paris/fragment/left_col
/en/Paris/photos.json?skip=0&number=20
/en/Paris/photos.json?skip=13&number=27

33. Metacache workflow
/en/Paris.*
invalidation worker
Nginx SSI
/en/Paris
/en/Paris/fragment/left_col
/en/Paris/photos.json?skip=0&number=20
/en/Paris/photos.json?skip=13&number=27
Varnish HTTP cache
varnish log
/en/Paris/fragment/left_col
RoR application
metacache feeder

34. Waw.
• This time we are actually using MongoDB as a
BTree. Impressive.
• The metacache has been running fine for
several months, and we want to go further.

35. Invalidate less
• We need to be more specific as to what we
invalidate.
• Today, if somebody votes on a photo in the
Paris album, we invalidate all /en/Paris prefix,
and most of it is unchanged.
• We will move towards a more clever
metacache.

36. Metacache reloaded
• Pub/Sub metacache
• Have the backend send a specific header to
be caught by the metacache-feeder, conaining
“subscribe” message.
• This header will be a JSON document, to be
pushed to the metacache.
• The purge commands will be mongo search
queries.

37. /en/Paris
/en/Paris/fragment/left_col
/en/Paris/photos.json?skip=0&number=20
/en/Paris/photos.json?skip=13&number=27
{url:/en/Paris, observe:[summary,links]}
{url:/en/Paris/fragment/left_col, observe: [cover]}
{url:/en/Paris/photos.json?skip=0&number=20,
observe:[photos]}
{url:/en/Paris/photos.json?skip=0&number=20,
observe:[photos]}

38. when somebody votes
{ url:/en/Paris.*, observe:photos }
when the summary changes
{ url:/en/Paris.*, observe:summary }
when the a new link is created
{ url:/en/Paris.*, observe:links }
{url:/en/Paris, observe:[summary,links]}
{url:/en/Paris/fragment/left_col, observe: [cover]}
{url:/en/Paris/photos.json?skip=0&number=20,
observe:[photos]}
{url:/en/Paris/photos.json?skip=0&number=20,
observe:[photos]}

39. Other uses cases
• Timeline activities storage: just one more BTree
usage.
• Moderation workflow data: tiny dataset, but more
complex queries, map/reduce.
• Suspended experimentation around log
collection and analysis

40. Current situation
• Mysql: main data store
• CouchDB: old timelines (+ chef)
• MongoDB: metacache, wikipedia, moderation,
new timelines
• Redis: raw data cache for counters, recent
activity (+ resque)

41. What about the main store ?
• albums are good fit for documents
• votes and score may be more tricky
• recent introduction of resque

42. In short
• Simple, fast.
• Hackable: in a language most can read.
• Clear roadmap.
• Very helpful and efficient team.
• Designed with application developer needs in
mind.