• Like

Retail Reference Architecture Part 2: Real-Time, Geo Distributed Inventory

  • 423 views
Uploaded on

During this session we will cover the best practices for implementing a real-time inventory with MongoDB. This includes properly model quantities and stores to avoid large numbers of documents being …

During this session we will cover the best practices for implementing a real-time inventory with MongoDB. This includes properly model quantities and stores to avoid large numbers of documents being indexed, how to efficiently use geo-indexing to find the closest store with a specific item available and how to run aggregation to gather interesting inventory stats. We will also cover operational considerations, like how to make inventory queries and updates from anywhere be low-latency and resilient to network partitions via tag-aware sharding.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
423
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
27
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Senior Consulting Engineer, MongoDB Norman Graham #MongoDBWorld Retail Reference Architecture: Real-Time, Geo-Distributed Inventory
  • 2. Inventory Inventory MongoDB External Inventory Internal Inventory Regional Inventory Purchase Orders Fulfillment Promotions
  • 3. Inventory – Traditional Architecture Relational DB System of Records Nightly Batches Analytics, Aggregations, Reports Caching Layer Field Inventory Internal & External Apps Point-in-time Loads
  • 4. Inventory – Opportunities Missed • Can’t reliably detect availability • Can't redirect purchasers to in-store pickup • Can’t do intra-day replenishment • Degraded customer experience • Higher internal expense
  • 5. Inventory – Principles • Single view of the inventory • Used by most services and channels • Read-dominated workload • Local, real-time writes • Bulk writes for refresh • Geographically distributed • Horizontally scalable
  • 6. Requirement Challenge MongoDB Single view of inventory Ensure availability of inventory information on all channels and services Developer-friendly, document-oriented storage High volume, low latency reads Anytime, anywhere access to inventory data without overloading the system of record Fast, indexed reads Local reads Horizontal scaling Bulk updates, intra-day deltas Provide window-in-time consistency for highly available services Bulk writes Fast, in-place updates Horizontal scaling Rapid application development cycles Deliver new services rapidly to capture new opportunities Flexible schema Rich query language Agile-friendly iterations Inventory – Requirements
  • 7. Inventory – Target Architecture Relational DB System of Records Analytics, Aggregations, Reports Field Inventory Internal & External Apps Inventory Assortments Shipments Audits Products Stores Point-in-time Loads Nightly Refresh Real-time Updates
  • 8. Horizontal Scaling Inventory – Technical Decisions Store Inventory Schema Indexing
  • 9. Inventory – Collections Stores Inventory Products Audits AssortmentsShipments
  • 10. > db.stores.findOne() { "_id" : ObjectId("53549fd3e4b0aaf5d6d07f35"), "className" : "catalog.Store", "storeId" : "store0", "name" : "Bessemer store", "address" : { "addr1" : "1st Main St", "city" : "Bessemer", "state" : "AL", "zip" : "12345", "country" : "US" }, "location" : [ -86.95444, 33.40178 ], ... } Stores – Sample Document
  • 11. Stores – Sample Queries • Get a store by storeId db.stores.find({ "storeId" : "store0" }) • Get a store by zip code db.stores.find({ "address.zip" : "12345" })
  • 12. What’s near me?
  • 13. Stores – Sample Geo Queries • Get nearby stores sorted by distance db.runCommand({ geoNear : "stores", near : { type : "Point", coordinates : [-82.8006, 40.0908] }, maxDistance : 10000.0, spherical : true })
  • 14. Stores – Sample Geo Queries • Get the five nearest stores within 10 km db.stores.find({ location : { $near : { $geometry : { type : "Point", coordinates : [-82.80, 40.09] }, $maxDistance : 10000.0 } } }).limit(5)
  • 15. Stores – Indices • { "storeId" : 1 }, { "unique" : true } • { "name" : 1 } • { "address.zip" : 1 } • { "location" : "2dsphere" }
  • 16. > db.inventory.findOne() { "_id": "5354869f300487d20b2b011d", "storeId": "store0", "location": [-86.95444, 33.40178], "productId": "p0", "vars": [ { "sku": "sku1", "q": 14 }, { "sku": "sku3", "q": 7 }, { "sku": "sku7", "q": 32 }, { "sku": "sku14", "q": 65 }, ... ] } Inventory – Sample Document
  • 17. Inventory – Sample Queries • Get all items in a store db.inventory.find({ storeId : "store100" }) • Get quantity for an item at a store db.inventory.find({ "storeId" : "store100", "productId" : "p200" })
  • 18. Inventory – Sample Queries • Get quantity for a sku at a store db.inventory.find( { "storeId" : "store100", "productId" : "p200", "vars.sku" : "sku11736" }, { "vars.$" : 1 } )
  • 19. Inventory – Sample Update • Increment / decrement inventory for an item at a store db.inventory.update( { "storeId" : "store100", "productId" : "p200", "vars.sku" : "sku11736" }, { "$inc" : { "vars.$.q" : 20 } } )
  • 20. Inventory – Sample Aggregations • Aggregate total quantity for a product db.inventory.aggregate( [ { $match : { productId : "p200" } }, { $unwind : "$vars" }, { $group : { _id : "result", count : { $sum : "$vars.q" } } } ] ) { "_id" : "result", "count" : 101752 }
  • 21. Inventory – Sample Aggregations • Aggregate total quantity for a store db.inventory.aggregate( [ { $match : { storeId : "store100" } }, { $unwind : "$vars" }, { $match : { "vars.q" : { $gt : 0 } } }, { $group : { _id : "result", count : { $sum : 1 } } } ] ) { "_id" : "result", "count" : 29347 }
  • 22. Inventory – Sample Aggregations • Aggregate total quantity for a store db.inventory.aggregate( [ { $match : { storeId : "store100" } }, { $unwind : "$vars" }, { $group : { _id : "result", count : { $sum : "$vars.q" } } } ] ) { "_id" : "result", "count" : 29347 }
  • 23. Inventory – Sample Geo-Query • Get inventory for an item near a point db.runCommand( { geoNear : "inventory", near : { type : "Point", coordinates : [-82.8006, 40.0908] }, maxDistance : 10000.0, spherical : true, limit : 10, query : { "productId" : "p200", "vars.sku" : "sku11736" } } )
  • 24. Inventory – Sample Geo-Query • Get closest store with available sku db.runCommand( { geoNear : "inventory", near : { type : "Point", coordinates : [-82.800672, 40.090844] }, maxDistance : 10000.0, spherical : true, limit : 1, query : { productId : "p200", vars : { $elemMatch : { sku : "sku11736", q : { $gt : 0 } } } } } )
  • 25. Inventory – Sample Geo- Aggregation • Get count of inventory for an item near a point db.inventory.aggregate([ { $geoNear: { near : { type : "Point", coordinates : [-82.800672, 40.090844] }, distanceField: "distance", maxDistance: 10000.0, spherical : true, query: { productId: "p200", vars : { $elemMatch : { sku : "sku11736", q : {$gt : 0} } } }, includeLocs: "dist.location", num: 5 } }, { $unwind: "$vars" }, { $match: { "vars.sku" : "sku11736" } }, { $group: { _id: "result", count: {$sum: "$vars.q"} } }])
  • 26. Inventory – Sample Indices • { storeId : 1 } • { productId : 1, storeId : 1 } • { productId : 1, location : "2dsphere" } • Why not "vars.sku"? – { productId : 1, storeId : 1, "vars.sku" : 1 }
  • 27. Horizontal Scaling Inventory – Technical Decisions Store Inventory Schema Indexing
  • 28. Shard East Shard Central Shard West East DC Inventory – Sharding Topology West DC Central DC Legacy Inventory Primary Primary Primary
  • 29. Inventory – Shard Key • Choose shard key – { storeId : 1 } ? – { productId : 1, storeId : 1 }? – { storeId : 1, productId : 1 } ? • Set up sharding – sh.enableSharding("inventoryDB") – sh.shardCollection( "inventoryDB.inventory", { storeId : 1, productId : 1 })
  • 30. Inventory – Shard Tags • Set up shard tags – sh.addShardTag("shard0000", "west") – sh.addShardTag("shard0001", "central") – sh.addShardTag("shard0002", "east") • Set up tag ranges – sh.addTagRange("inventoryDB.inventory", { storeId : 0 }, { storeId : 100}, "west" ) – sh.addTagRange("inventoryDB.inventory", { storeId : 100 }, { storeId : 200 }, "central" ) – sh.addTagRange("inventoryDB.inventory", { storeId : 200 }, { storeId : 300 }, "east" )
  • 31. Senior Consulting Engineer, MongoDB Norman Graham #MongoDBWorld Thank You