@February 2013 Sean Sullivan
About me• software engineer• ~2 years at Gilt• work in Gilt’s Portland office• back office applications
Agenda• Gilt Groupe• Gilt’s technology stack• MongoDB at Gilt• Q&A
Gilt Groupe http://www.gilt.com
flash sales
Everyday at 12 noon
what does Gilt sell?
Apparel
Kids merchandise
Home furnishings
Food
Local deals
Travel
Gilt technology stack
Gilt architecture
2007             2013monolithic    service-orientedapplication     architecture
2007
2013           service A           service B           service C                          legacy                         w...
Data storage @ Gilt
@
Why MongoDB?• Ease of use• Horizontal scaling• High availability• Automatic failover
Why MongoDB?• Stability• Support• Drivers
• MongoDB 2.0• sharded and non-sharded data• Solid State Drives• MMS for monitoring
Use case #1: user profiles
Challenges• user data in Postgres• legacy Rails app expects to find user data in  Postgres• we wanted Gilt’s customer facin...
Solution• keep user data in both MongoDB and  Postgres• replicate from MongoDB to Postgres
Replicating user datauser service                              legacy                             web app               re...
Replication service• listens for RabbitMQ messages   ‣ UserCreated message   ‣ UserUpdated message• retrieve data using RE...
Use case #2:   feature configurationGoal manage the release of new features on gilt.comHow feature configuration persisted i...
Rolling out a new feature 1. deploy new application code to production 2. enable feature for Gilt Tech employees 3. ... th...
Feature configuration           Feature            config           servicegilt.com
Use case #3:favorite brands
Favorite brandsAJAX      user       preference         service
Application development
mongo-java-driver   CasbahMorphia
MongoDB Java driver• main class: com.mongodb.Mongo• Mongo object maintains a pool of connections• Latest version: 2.10.1
MongoDB Java driverconnection pool tuningMongoOptions opts = new MongoOptions();// example valuesopts.connectionsPerHost =...
com.mongodb.WriteConcern • NORMAL • SAFE • REPLICAS_SAFE • MAJORITY • FSYNC_SAFE • JOURNAL_SAFE
Morphia library
Morphia• object-document mapper for Java• built on top of mongo-java-driver• map fields using Java annotations
// Morphia exampleimport com.google.code.morphia.annotations.*;@Entity(value="features", noClassnameStored = true)public c...
http://code.google.com/p/morphia/
Casbah library
Casbah• Scala toolkit for MongoDB• built on top of the mongo-java-driver• current version: 2.5.0
Casbah• Scala idioms• Scala collections• fluid query syntax
https://twitter.com/max4f/status/230503836958199808
Gilt Scala code
trait MongoWriteConcern {    def withSession[T](f: => T)(implicit mongoDb: MongoDB) {        mongoDb.requestStart()       ...
Best Practices
Best practicesConnection tuningexplicitly configure Mongo Java Driverconnection pool size
Best practices   Use caution when creating new indexes “creating a new index on a production mongo server canbasically cau...
Minimizing impact of        building a new index     • remove one secondary from replica set     • create/rebuild index on...
Best practices    use short names for fields to avoid wasted space{                                                {    cit...
Best practicesuse explain() during development    db.collection.find(query).explain()
Best practicesuse caution when choosing a shard key“It is generally not a good idea to use the defaultObjectId as the shar...
FutureMore Gilt data in MongoDB   • shipping addresses   • discounts   • SKU’s and brandsUpgrade to MongoDB 2.2  • tag awa...
Gilt Groupe is hiring! http://techjobs.gilt.com
Questions?
ssullivan@gilt.com@tinyrobots
The end
Bonus slides
Gilt tech talks• Apache Camel and MongoDB @ Gilt   http://bit.ly/OYO37K• Deploying new features @ Gilt   http://slidesha.r...
https://twitter.com/stripe/status/298858032421535744
MongoDB at Gilt Groupe
MongoDB at Gilt Groupe
MongoDB at Gilt Groupe
MongoDB at Gilt Groupe
MongoDB at Gilt Groupe
MongoDB at Gilt Groupe
MongoDB at Gilt Groupe
MongoDB at Gilt Groupe
MongoDB at Gilt Groupe
MongoDB at Gilt Groupe
MongoDB at Gilt Groupe
MongoDB at Gilt Groupe
MongoDB at Gilt Groupe
MongoDB at Gilt Groupe
MongoDB at Gilt Groupe
MongoDB at Gilt Groupe
Upcoming SlideShare
Loading in …5
×

MongoDB at Gilt Groupe

1,286 views

Published on

MongoDB at Gilt Groupe

  1. 1. @February 2013 Sean Sullivan
  2. 2. About me• software engineer• ~2 years at Gilt• work in Gilt’s Portland office• back office applications
  3. 3. Agenda• Gilt Groupe• Gilt’s technology stack• MongoDB at Gilt• Q&A
  4. 4. Gilt Groupe http://www.gilt.com
  5. 5. flash sales
  6. 6. Everyday at 12 noon
  7. 7. what does Gilt sell?
  8. 8. Apparel
  9. 9. Kids merchandise
  10. 10. Home furnishings
  11. 11. Food
  12. 12. Local deals
  13. 13. Travel
  14. 14. Gilt technology stack
  15. 15. Gilt architecture
  16. 16. 2007 2013monolithic service-orientedapplication architecture
  17. 17. 2007
  18. 18. 2013 service A service B service C legacy web app service Dgilt.com service E
  19. 19. Data storage @ Gilt
  20. 20. @
  21. 21. Why MongoDB?• Ease of use• Horizontal scaling• High availability• Automatic failover
  22. 22. Why MongoDB?• Stability• Support• Drivers
  23. 23. • MongoDB 2.0• sharded and non-sharded data• Solid State Drives• MMS for monitoring
  24. 24. Use case #1: user profiles
  25. 25. Challenges• user data in Postgres• legacy Rails app expects to find user data in Postgres• we wanted Gilt’s customer facing applications to retrieve user data from MongoDB
  26. 26. Solution• keep user data in both MongoDB and Postgres• replicate from MongoDB to Postgres
  27. 27. Replicating user datauser service legacy web app replication service
  28. 28. Replication service• listens for RabbitMQ messages ‣ UserCreated message ‣ UserUpdated message• retrieve data using REST API• write data to Postgres using JDBC
  29. 29. Use case #2: feature configurationGoal manage the release of new features on gilt.comHow feature configuration persisted in MongoDB
  30. 30. Rolling out a new feature 1. deploy new application code to production 2. enable feature for Gilt Tech employees 3. ... then enable for all Gilt employees 4. ... then enable for a subset of users 5. gradually ramp up to 100% of users
  31. 31. Feature configuration Feature config servicegilt.com
  32. 32. Use case #3:favorite brands
  33. 33. Favorite brandsAJAX user preference service
  34. 34. Application development
  35. 35. mongo-java-driver CasbahMorphia
  36. 36. MongoDB Java driver• main class: com.mongodb.Mongo• Mongo object maintains a pool of connections• Latest version: 2.10.1
  37. 37. MongoDB Java driverconnection pool tuningMongoOptions opts = new MongoOptions();// example valuesopts.connectionsPerHost = 50;opts.threadsAllowedToBlockForConnectionMultiplier = 5;
  38. 38. com.mongodb.WriteConcern • NORMAL • SAFE • REPLICAS_SAFE • MAJORITY • FSYNC_SAFE • JOURNAL_SAFE
  39. 39. Morphia library
  40. 40. Morphia• object-document mapper for Java• built on top of mongo-java-driver• map fields using Java annotations
  41. 41. // Morphia exampleimport com.google.code.morphia.annotations.*;@Entity(value="features", noClassnameStored = true)public class Feature { @Id ObjectId featureId; @Property("feature_key") @Indexed(unique=true) String featureKey; @Property("release_to_percentage") int releaseToPercentage;}
  42. 42. http://code.google.com/p/morphia/
  43. 43. Casbah library
  44. 44. Casbah• Scala toolkit for MongoDB• built on top of the mongo-java-driver• current version: 2.5.0
  45. 45. Casbah• Scala idioms• Scala collections• fluid query syntax
  46. 46. https://twitter.com/max4f/status/230503836958199808
  47. 47. Gilt Scala code
  48. 48. trait MongoWriteConcern { def withSession[T](f: => T)(implicit mongoDb: MongoDB) { mongoDb.requestStart() mongoDb.requestEnsureConnection() try { f } finally { mongoDb.requestDone() } }}
  49. 49. Best Practices
  50. 50. Best practicesConnection tuningexplicitly configure Mongo Java Driverconnection pool size
  51. 51. Best practices Use caution when creating new indexes “creating a new index on a production mongo server canbasically cause it to stop working while it builds the index” (source: Gilt production incident report)
  52. 52. Minimizing impact of building a new index • remove one secondary from replica set • create/rebuild index on this instance • rejoin replica set • repeat on all remaining secondaries • run rs.stepDown() on primary memberhttp://docs.mongodb.org/manual/administration/indexes/#index-building-replica-sets
  53. 53. Best practices use short names for fields to avoid wasted space{ { city: “Portland”, ci: “Portland”, state: “Oregon”, vs st: “Oregon”, country: “US” co: “US”} } http://christophermaier.name/blog/2011/05/22/MongoDB-key-names
  54. 54. Best practicesuse explain() during development db.collection.find(query).explain()
  55. 55. Best practicesuse caution when choosing a shard key“It is generally not a good idea to use the defaultObjectId as the shard key”source: http://stackoverflow.com/questions/9164356/sharding-by-objectid-is-it-the-right-way
  56. 56. FutureMore Gilt data in MongoDB • shipping addresses • discounts • SKU’s and brandsUpgrade to MongoDB 2.2 • tag aware sharding
  57. 57. Gilt Groupe is hiring! http://techjobs.gilt.com
  58. 58. Questions?
  59. 59. ssullivan@gilt.com@tinyrobots
  60. 60. The end
  61. 61. Bonus slides
  62. 62. Gilt tech talks• Apache Camel and MongoDB @ Gilt http://bit.ly/OYO37K• Deploying new features @ Gilt http://slidesha.re/OoCYfd• Voldemort @ Gilt http://bit.ly/b9Qhib
  63. 63. https://twitter.com/stripe/status/298858032421535744

×