Implementing MongoDB at Shutterfly (Kenny Gorman)

9,804 views

Published on

Published in: Technology
2 Comments
15 Likes
Statistics
Notes
  • also, what manages the sync between oracle mongodb for the write/read through cache set-up thats mentioned?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • How many documents do you store in MongoDB? All of your 20TB from the RDBMS?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
9,804
On SlideShare
0
From Embeds
0
Number of Embeds
1,148
Actions
Shares
0
Downloads
179
Comments
2
Likes
15
Embeds 0
No embeds

No notes for slide

Implementing MongoDB at Shutterfly (Kenny Gorman)

  1. 1. Implementing MongoDB MongoSF April 2010 Kenny Gorman, Data Architect
  2. 2. Shutterfly Inc. •  Founded in December 1999 •  Public company (NASDAQ: SFLY) •  Millions of customers have billions of pictures on Shutterfly •  Photo site, books, sharing, prints, gifts •  Only photo sharing site that doesn’t down- sample, compress, or force delete photos •  > 6B photos, adding 400TB/mo April 30, 2010 Business Confidential 2
  3. 3. Existing Metadata Storage Architecture •  Metadata is persisted in RDBMS •  Images/media stored outside DB •  Java/Spring, C#,.Net •  Oracle™ RDBMS •  Sun™ servers and storage •  Vertically partitioned by function •  Hot Standbys used for availability •  > 20tb of RDBMS storage •  > 10000 ex/sec •  Extreme uptime requirements April 30, 2010 Business Confidential 3
  4. 4. Problems •  Time to Market •  Cost •  Performance •  Scalability April 30, 2010 Business Confidential 4
  5. 5. New Metadata Storage Architecture •  Performance ! Reduce complexity ! Partition data •  Scalability ! Move to clustered system •  Time to Market ! Simple API •  Cost ! OSS software ! Simple hardware April 30, 2010 Business Confidential 5
  6. 6. New Data Architecture Fundamentals •  Partition data •  Relax consistency (where applicable) •  Data locality •  Highly available configuration •  Keep design simple/fast •  Keep hardware simple/cheap •  Keep software simple/cheap April 30, 2010 Business Confidential 6
  7. 7. MongoDB •  Open Source •  Best of RDBMS, yet not quite k,v store •  Features we need •  Commercial support •  Active community •  Performance April 30, 2010 Business Confidential 7
  8. 8. MongoDB Development •  Data modeling •  Java, .Net •  Simple, fast development •  JSON just makes sense •  Data access layer •  GridFS April 30, 2010 Business Confidential 8
  9. 9. MongoDB in production •  Simple use case, simple project •  Primary and 2 replica DB’s, 1 ‘lagged’ •  Manual failover •  Monitoring: http interface •  Tools: mongostat, custom rrd graphs •  Linux on Intel™ •  MongoDB 1.4.2 (stable) April 30, 2010 Business Confidential 9
  10. 10. Going Live Plan •  Walk before you run •  Shutterfly project/product selection •  Write through architecture •  •  Good metrics •  Subset of MongoDB features April 30, 2010 Business Confidential 10
  11. 11. So how did we do? •  Time to Market •  Application developed in 1 sprint •  Cost •  500% improvement •  Performance •  900% improvement •  18ms to 2ms avg latency for inserts •  Scalability •  Shard on demand April 30, 2010 Business Confidential 11
  12. 12. The future •  More MongoDB •  Replication as durability (getLasterror(w=2)) •  Replica sets •  Excitement from developers •  Lots of attribute and media metadata types •  Object mapper •  New projects and old systems •  Evaluate as they come up April 30, 2010 Business Confidential 12
  13. 13. Lessons Learned •  Keep it simple •  Data Modeling •  Walk before you run •  Use Jira for MongoDB issues •  There is life after Larry April 30, 2010 Business Confidential 13
  14. 14. Q&A Questions? Contact: kg@kennygorman.com http://www.kennygorman.com http://github.com/kgorman http://www.shutterfly.com kgorman@shutterfly.com April 30, 2010 Business Confidential 14

×