Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Implementing MongoDB at Shutterfly (Kenny Gorman)

10,015 views

Published on

Published in: Technology
  • also, what manages the sync between oracle mongodb for the write/read through cache set-up thats mentioned?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • How many documents do you store in MongoDB? All of your 20TB from the RDBMS?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Implementing MongoDB at Shutterfly (Kenny Gorman)

  1. 1. Implementing MongoDB MongoSF April 2010 Kenny Gorman, Data Architect
  2. 2. Shutterfly Inc. •  Founded in December 1999 •  Public company (NASDAQ: SFLY) •  Millions of customers have billions of pictures on Shutterfly •  Photo site, books, sharing, prints, gifts •  Only photo sharing site that doesn’t down- sample, compress, or force delete photos •  > 6B photos, adding 400TB/mo April 30, 2010 Business Confidential 2
  3. 3. Existing Metadata Storage Architecture •  Metadata is persisted in RDBMS •  Images/media stored outside DB •  Java/Spring, C#,.Net •  Oracle™ RDBMS •  Sun™ servers and storage •  Vertically partitioned by function •  Hot Standbys used for availability •  > 20tb of RDBMS storage •  > 10000 ex/sec •  Extreme uptime requirements April 30, 2010 Business Confidential 3
  4. 4. Problems •  Time to Market •  Cost •  Performance •  Scalability April 30, 2010 Business Confidential 4
  5. 5. New Metadata Storage Architecture •  Performance ! Reduce complexity ! Partition data •  Scalability ! Move to clustered system •  Time to Market ! Simple API •  Cost ! OSS software ! Simple hardware April 30, 2010 Business Confidential 5
  6. 6. New Data Architecture Fundamentals •  Partition data •  Relax consistency (where applicable) •  Data locality •  Highly available configuration •  Keep design simple/fast •  Keep hardware simple/cheap •  Keep software simple/cheap April 30, 2010 Business Confidential 6
  7. 7. MongoDB •  Open Source •  Best of RDBMS, yet not quite k,v store •  Features we need •  Commercial support •  Active community •  Performance April 30, 2010 Business Confidential 7
  8. 8. MongoDB Development •  Data modeling •  Java, .Net •  Simple, fast development •  JSON just makes sense •  Data access layer •  GridFS April 30, 2010 Business Confidential 8
  9. 9. MongoDB in production •  Simple use case, simple project •  Primary and 2 replica DB’s, 1 ‘lagged’ •  Manual failover •  Monitoring: http interface •  Tools: mongostat, custom rrd graphs •  Linux on Intel™ •  MongoDB 1.4.2 (stable) April 30, 2010 Business Confidential 9
  10. 10. Going Live Plan •  Walk before you run •  Shutterfly project/product selection •  Write through architecture •  •  Good metrics •  Subset of MongoDB features April 30, 2010 Business Confidential 10
  11. 11. So how did we do? •  Time to Market •  Application developed in 1 sprint •  Cost •  500% improvement •  Performance •  900% improvement •  18ms to 2ms avg latency for inserts •  Scalability •  Shard on demand April 30, 2010 Business Confidential 11
  12. 12. The future •  More MongoDB •  Replication as durability (getLasterror(w=2)) •  Replica sets •  Excitement from developers •  Lots of attribute and media metadata types •  Object mapper •  New projects and old systems •  Evaluate as they come up April 30, 2010 Business Confidential 12
  13. 13. Lessons Learned •  Keep it simple •  Data Modeling •  Walk before you run •  Use Jira for MongoDB issues •  There is life after Larry April 30, 2010 Business Confidential 13
  14. 14. Q&A Questions? Contact: kg@kennygorman.com http://www.kennygorman.com http://github.com/kgorman http://www.shutterfly.com kgorman@shutterfly.com April 30, 2010 Business Confidential 14

×