Getting Started with MongoDB: 4 Application Designs

1,524 views
1,354 views

Published on

Published in: Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,524
On SlideShare
0
From Embeds
0
Number of Embeds
163
Actions
Shares
0
Downloads
22
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Getting Started with MongoDB: 4 Application Designs

  1. 1. 2 App DesignsUsing MongoDB
  2. 2. Kyle Bankerkyle@10gen.com
  3. 3. @hwaet
  4. 4. What well cover:Brief introFeed readerWebsite analyitcsQuestions
  5. 5. Intro
  6. 6. MongoDBRich data modelReplication and automatic failoverSharding
  7. 7. Use casesSocial networking and geolocationE-commerceCMSAnalyticsProduction deployments: http://is.gd/U0VkG6
  8. 8. Demo
  9. 9. Feed reader
  10. 10. Four collectionsUsersFeedsEntriesBuckets
  11. 11. Users{ _id: ObjectId("4e316f236ce9cca7ef17d59d"), username: kbanker, feeds: [ { _id: ObjectId("4e316f236ce9cca7ef17d54c"), name: "GigaOM" }, { _id: ObjectId("4e316f236ce9cca7ef17d54d"), name: "Salon.com" } { _id: ObjectId("4e316f236ce9cca7ef17d54e"), name: "Foo News" } ], latest: Date(2011, 7, 25)}
  12. 12. Indexdb.users.ensureIndex( { username: 1 }, { unique: true } )
  13. 13. Feeds{ _id: ObjectId("4e316b8a6ce9cca7ef17d54b"), url: http://news.foo.com/feed.xml/, name: Foo News, subscriber_count: 2, latest: Date(2011, 7, 25)} Indexdb.feeds.ensureIndex( { url: 1 }, { unique: true
  14. 14. Adding a feed subscription// Upsertdb.feeds.update( { url: http://news.foo.com/feed.xml/, name: Foo News}, { $inc: {subscriber_count: 1} }, true )
  15. 15. Adding a feed subscription// Add this feed to user feed listdb.users.update( {_id: ObjectId("4e316f236ce9cca7ef17d59d") }, { $addToSet: { feeds: { _id: ObjectId("4e316b8a6ce9cca7ef17d54b"), name: Foo News } })
  16. 16. Removing a feed subscriptiondb.users.update( { _id: ObjectId("4e316f236ce9cca7ef17d59d") }, { $pull: { feeds: { _id: ObjectId("4e316b8a6ce9cca7ef17d54b"), name: Foo News } } )
  17. 17. Removing a feed subscriptiondb.feeds.update( {url: http://news.foo.com/feed.xml/}, { $inc: {subscriber_count: -1} } )
  18. 18. Entries (populate in background){ _id: ObjectId("4e316b8a6ce9cca7ef17d54b"), feed_id: ObjectId("4e316b8a6ce9cca7ef17d54b"), title: Important person to resign, body: A person deemed very important has decided..., reads: 5, date: Date(2011, 7, 27)}
  19. 19. What we need nowPopulate personal feeds (buckets)Avoid lots of expensive queriesRecord whats been read
  20. 20. Without bucketing// Naive query runs every timedb.entries.find( feed_id: {$in: user_feed_ids}).sort({date: 1}).limit(25)
  21. 21. With bucketing// A bit smarter: only runs onceentries = db.entries.find( { date: {$gt: user_latest }, feed_id: { $in: user_feed_ids } ).sort({date: 1})
  22. 22. Indexdb.entries.ensureIndex( { date: 1, feed_id: 1} )
  23. 23. bucket = { _id: ObjectId("4e3185c26ce9cca7ef17d552"), user_id: ObjectId("4e316f236ce9cca7ef17d59d"), date: Date( 2011, 7, 27 ) n: 100, entries: [ { _id: ObjectId("4e316b8a6ce9cca7ef17d5ac"), feed_id: ObjectId("4e316b8a6ce9cca7ef17d54b"), title: Important person to resign, body: A person deemed very important has decided..., date: Date(2011, 7, 27), read: false }, { _id: ObjectId("4e316b8a6ce9cca7ef17d5c8"), feed_id: ObjectId("4e316b8a6ce9cca7ef17d54b"), title: Panda bear waves hello, body: A panda bear at the local zoo..., date: Date(2011, 7, 27), read: false } ]}
  24. 24. db.buckets.insert( buckets )db.users.update( { _id: ObjectId("4e316f236ce9cca7ef17d59d") } { $set: { latest: latest_entry_date } })
  25. 25. Viewing a personal feed// Newestdb.buckets.find( { user_id: ObjectId("4e316f236ce9cca7ef17d59d") }).sort({date: -1}).limit(1)
  26. 26. Viewing a personal feed// Next newest (thats how we paginate)db.buckets.find( { user_id: ObjectId("4e316f236ce9cca7ef17d59d"), date: { $lt: previous_reader_date } }).sort({date: -1}).limit(1)
  27. 27. Indexdb.buckets.ensureIndex( { user_id: 1, date: -1 } )
  28. 28. Marking a feed itemdb.buckets.update( { user_id: ObjectId("4e316f236ce9cca7ef17d59d"), entries.id: ObjectId("4e316b8a6ce9cca7ef17d5c8")}, { $set: { entries.$.read : true })
  29. 29. Marking a feed itemdb.entries.update( { _id: ObjectId("4e316b8a6ce9cca7ef17d5c8") }, { $inc: { read: 1 } })
  30. 30. Sharding note:Buckets collection is eminently shardableShard key: { user_id: 1, date: 1 }
  31. 31. Websiteanalyitcs
  32. 32. Challenges:Real-time reporting.Efficient use of space.Easily wipe unneeded data.
  33. 33. TechniquesPre-aggregate the data.Pre-construct document structure.Store emphemeral data in a separatedatabase.
  34. 34. Two collections:Each collection gets its own database.Collections names are time-scoped.Clean, fast removal of old data.
  35. 35. // Collections holding totals for each day, stored// in a database per monthdays_2011_5days_2011_6days_2011_7...// Totals for each month...months_2011_1_4months_2011_5_8months_2011_9_12...
  36. 36. Hours and minutes{ _id: { uri: BinData("0beec7b5ea3f0fdbc95d0dd47f35"), day: 2011-5-1 }, total: 2820, hrs: { 0: 500, 1: 700, 2: 450, 3: 343, // ... 4-23 go here } // Minutes are rolling. This gives real-time // numbers for the last hour. So when you increment // minute n, you need to $set minute n-1 to 0. mins: { 1: 12, 2: 10, 3: 5, 4: 34 // ... 5-60 go here }}
  37. 37. Updating daydocument...
  38. 38. // Update days collection at 5:37 p.m. on 2011-5-1// Might want to queue increments so that $inc is greater// than 1 for each write...id = { uri: BinData("0beec7b5ea3f0fdbc95d0dd47f35"), day: 2011-5-1 };update = { $inc: { total: 1, hrs.17: 1, mins.37: 1 }, $set: { mins.36: 0} };// Update collection containing days 1-5db.days_2011_5.update( id, update );
  39. 39. Months and days{ _id: { uri: BinData("0beec7b5ea3f0fdbc95d0dd47f35"), month: 2011-5 }, total: 34173, months: { 1: 4000, 2: 4300, 3: 4200, 4: 5000, 5: 5100, 6: 5700, 7: 5873 // ... 8-12 go here }}
  40. 40. Updating month document...
  41. 41. // Update months collection at 5:37 p.m. on 2011-5-1query = { uri: BinData("0beec7b5ea3f0fdbc95d0dd47f35"), month: 2011-5 };update = { $inc: { total: 1, months.5: 1 } };// Update collection containing days 1-5db.month_2011_1_4.update( query, update );
  42. 42. ReportingMust provide data at multiple resolutions(second, minute, etc.).We have the raw materials for that.Application assembles the dataintelligently.
  43. 43. Summary
  44. 44. Feed ReaderRich documentsIncremental modifiersBucketing strategy
  45. 45. Website AnalyticsPre-aggregate dataTime-scoped databases and collections
  46. 46. http://www.10gen.com/presentations
  47. 47. http://manning.com/banker
  48. 48. Thank you

×