Silentale mongo slides

546 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
546
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Silentale mongo slides

  1. 1. keep track and backup your contacts & conversations Erwan Arzur erwan@silentale.com +33 6 09 33 76 42 http://silentale.com vendredi 18 juin 2010
  2. 2. Silentale in a few words • Backup and indexes messages from all accounts • Creates the user’s social address book • Freemium service - free and paid plans Silentale June 18, 2010 2 vendredi 18 juin 2010
  3. 3. Silentale One convenient place to backup all your contacts and messages across multiple services Silentale June 18, 2010 3 vendredi 18 juin 2010
  4. 4. Find what you need, faster • Retrieve contact details, messages or attachments, across all your accounts • Complete, consolidated message history • Unified address book • Wherever you need it (mobile, browser, desktop clients) Silentale June 18, 2010 4 vendredi 18 juin 2010
  5. 5. How does it work? Contacts & Messages from everywhere Silentale June 18, 2010 5 vendredi 18 juin 2010
  6. 6. How does it work? Contacts & Messages Applications from everywhere API 3rd party apps Silentale June 18, 2010 5 vendredi 18 juin 2010
  7. 7. Distributed platform • Crawlers retrieve messages & contacts • Store raw messages in S3 buckets • Parsers extract useful information • update databases (MySQL & MongoDB) • message meta-data • Indexers • full-text index • MongoDB • Message Queue service Silentale June 18, 2010 6 vendredi 18 juin 2010
  8. 8. Mongo Setup • 1 collection / user • sharding • current sharding implementation doesn’t work for us • very crude custom sharding algorithm • replica pairs • seem like a waste of resource ... until it’s maintenance time • m1.x-large instances • 15 GB RAM • 8x70 GB EBS volumes in raid0 (xfs) Silentale June 18, 2010 7 vendredi 18 juin 2010
  9. 9. Our history with Mongo • Our first beta in sep ’09 was using 0.9 • why did we choose mongo ? • more than a simple key-value store • easy port from MySQL (MongoMapper) • store arrays and search in them (rare feature among mongodb competition) • very fast answers from 10gen and community • a few bugs hurt us • ruby driver • replication problems • quickly fixed and has to be expected when deploying an early release ! • Now using 1.4.3 in production Silentale June 18, 2010 8 vendredi 18 juin 2010
  10. 10. Scales • ~100M documents • ~1.5 TB of data • >1.5 TB of indices • 15GB of RAM / shard • I/O bound • first search into a user’s collection can take several minutes (!!) • acceptable performance afterwards, but don’t try to search another user’s collection ! Silentale June 18, 2010 9 vendredi 18 juin 2010
  11. 11. Scales • ~100M documents • ~1.5 TB of data • >1.5 TB of indices • 15GB of RAM / shard Too Slow ! • I/O bound • first search into a user’s collection can take several minutes (!!) • acceptable performance afterwards, but don’t try to search another user’s collection ! Silentale June 18, 2010 9 vendredi 18 juin 2010
  12. 12. Solutions / wish-list • separate intents in different databases • meta-data needs quasi-instant retrieval • full-text doesn’t • simple searches works well thanks to indexing of array values • http://jira.mongodb.org/browse/SERVER-380 • moving to bare-metal • specialized hardware. • cache management Silentale June 18, 2010 10 vendredi 18 juin 2010
  13. 13. Thank you ! Questions ? Erwan Arzur / erwan@silentale.com / @zuzur Silentale June 18, 2010 11 vendredi 18 juin 2010

×