Senior Consulting Engineer, MongoDB
Norman Graham
#MongoDBWorld
Retail Reference Architecture:
Real-Time, Geo-Distributed
...
Inventory
Inventory
MongoDB
External Inventory
Internal Inventory
Regional Inventory
Purchase Orders
Fulfillment
Promotions
Inventory – Traditional Architecture
Relational DB
System of Records
Nightly
Batches
Analytics,
Aggregations,
Reports
Cach...
Inventory – Opportunities Missed
• Can’t reliably detect availability
• Can't redirect purchasers to in-store pickup
• Can...
Inventory – Principles
• Single view of the inventory
• Used by most services and channels
• Read-dominated workload
• Loc...
Requirement Challenge MongoDB
Single view of
inventory
Ensure availability of
inventory information on
all channels and
se...
Inventory – Target Architecture
Relational DB
System of Records
Analytics,
Aggregations,
Reports
Field Inventory
Internal ...
Horizontal Scaling
Inventory – Technical Decisions
Store
Inventory
Schema
Indexing
Inventory – Collections
Stores Inventory
Products
Audits AssortmentsShipments
> db.stores.findOne()
{
"_id" : ObjectId("53549fd3e4b0aaf5d6d07f35"),
"className" : "catalog.Store",
"storeId" : "store0",...
Stores – Sample Queries
• Get a store by storeId
db.stores.find({ "storeId" : "store0" })
• Get a store by zip code
db.sto...
What’s near me?
Stores – Sample Geo Queries
• Get nearby stores sorted by distance
db.runCommand({
geoNear : "stores",
near : {
type : "Po...
Stores – Sample Geo Queries
• Get the five nearest stores within 10 km
db.stores.find({
location : {
$near : {
$geometry :...
Stores – Indices
• { "storeId" : 1 }, { "unique" : true }
• { "name" : 1 }
• { "address.zip" : 1 }
• { "location" : "2dsph...
> db.inventory.findOne()
{
"_id": "5354869f300487d20b2b011d",
"storeId": "store0",
"location": [-86.95444, 33.40178],
"pro...
Inventory – Sample Queries
• Get all items in a store
db.inventory.find({ storeId : "store100" })
• Get quantity for an it...
Inventory – Sample Queries
• Get quantity for a sku at a store
db.inventory.find(
{
"storeId" : "store100",
"productId" : ...
Inventory – Sample Update
• Increment / decrement inventory for an item at a
store
db.inventory.update(
{
"storeId" : "sto...
Inventory – Sample Aggregations
• Aggregate total quantity for a product
db.inventory.aggregate( [
{ $match : { productId ...
Inventory – Sample Aggregations
• Aggregate total quantity for a store
db.inventory.aggregate( [
{ $match : { storeId : "s...
Inventory – Sample Aggregations
• Aggregate total quantity for a store
db.inventory.aggregate( [
{ $match : { storeId : "s...
Inventory – Sample Geo-Query
• Get inventory for an item near a point
db.runCommand( {
geoNear : "inventory",
near : {
typ...
Inventory – Sample Geo-Query
• Get closest store with available sku
db.runCommand( {
geoNear : "inventory",
near : {
type ...
Inventory – Sample Geo-
Aggregation
• Get count of inventory for an item near a point
db.inventory.aggregate([
{ $geoNear:...
Inventory – Sample Indices
• { storeId : 1 }
• { productId : 1, storeId : 1 }
• { productId : 1, location : "2dsphere" }
•...
Horizontal Scaling
Inventory – Technical Decisions
Store
Inventory
Schema
Indexing
Shard
East
Shard
Central
Shard
West
East DC
Inventory – Sharding Topology
West DC Central DC
Legacy
Inventory
Primary
Prim...
Inventory – Shard Key
• Choose shard key
– { storeId : 1 } ?
– { productId : 1, storeId : 1 }?
– { storeId : 1, productId ...
Inventory – Shard Tags
• Set up shard tags
– sh.addShardTag("shard0000", "west")
– sh.addShardTag("shard0001", "central")
...
Senior Consulting Engineer, MongoDB
Norman Graham
#MongoDBWorld
Thank You
Retail Reference Architecture Part 2: Real-Time, Geo Distributed Inventory
Upcoming SlideShare
Loading in...5
×

Retail Reference Architecture Part 2: Real-Time, Geo Distributed Inventory

986

Published on

During this session we will cover the best practices for implementing a real-time inventory with MongoDB. This includes properly model quantities and stores to avoid large numbers of documents being indexed, how to efficiently use geo-indexing to find the closest store with a specific item available and how to run aggregation to gather interesting inventory stats. We will also cover operational considerations, like how to make inventory queries and updates from anywhere be low-latency and resilient to network partitions via tag-aware sharding.

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
986
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
64
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Retail Reference Architecture Part 2: Real-Time, Geo Distributed Inventory

  1. 1. Senior Consulting Engineer, MongoDB Norman Graham #MongoDBWorld Retail Reference Architecture: Real-Time, Geo-Distributed Inventory
  2. 2. Inventory Inventory MongoDB External Inventory Internal Inventory Regional Inventory Purchase Orders Fulfillment Promotions
  3. 3. Inventory – Traditional Architecture Relational DB System of Records Nightly Batches Analytics, Aggregations, Reports Caching Layer Field Inventory Internal & External Apps Point-in-time Loads
  4. 4. Inventory – Opportunities Missed • Can’t reliably detect availability • Can't redirect purchasers to in-store pickup • Can’t do intra-day replenishment • Degraded customer experience • Higher internal expense
  5. 5. Inventory – Principles • Single view of the inventory • Used by most services and channels • Read-dominated workload • Local, real-time writes • Bulk writes for refresh • Geographically distributed • Horizontally scalable
  6. 6. Requirement Challenge MongoDB Single view of inventory Ensure availability of inventory information on all channels and services Developer-friendly, document-oriented storage High volume, low latency reads Anytime, anywhere access to inventory data without overloading the system of record Fast, indexed reads Local reads Horizontal scaling Bulk updates, intra-day deltas Provide window-in-time consistency for highly available services Bulk writes Fast, in-place updates Horizontal scaling Rapid application development cycles Deliver new services rapidly to capture new opportunities Flexible schema Rich query language Agile-friendly iterations Inventory – Requirements
  7. 7. Inventory – Target Architecture Relational DB System of Records Analytics, Aggregations, Reports Field Inventory Internal & External Apps Inventory Assortments Shipments Audits Products Stores Point-in-time Loads Nightly Refresh Real-time Updates
  8. 8. Horizontal Scaling Inventory – Technical Decisions Store Inventory Schema Indexing
  9. 9. Inventory – Collections Stores Inventory Products Audits AssortmentsShipments
  10. 10. > db.stores.findOne() { "_id" : ObjectId("53549fd3e4b0aaf5d6d07f35"), "className" : "catalog.Store", "storeId" : "store0", "name" : "Bessemer store", "address" : { "addr1" : "1st Main St", "city" : "Bessemer", "state" : "AL", "zip" : "12345", "country" : "US" }, "location" : [ -86.95444, 33.40178 ], ... } Stores – Sample Document
  11. 11. Stores – Sample Queries • Get a store by storeId db.stores.find({ "storeId" : "store0" }) • Get a store by zip code db.stores.find({ "address.zip" : "12345" })
  12. 12. What’s near me?
  13. 13. Stores – Sample Geo Queries • Get nearby stores sorted by distance db.runCommand({ geoNear : "stores", near : { type : "Point", coordinates : [-82.8006, 40.0908] }, maxDistance : 10000.0, spherical : true })
  14. 14. Stores – Sample Geo Queries • Get the five nearest stores within 10 km db.stores.find({ location : { $near : { $geometry : { type : "Point", coordinates : [-82.80, 40.09] }, $maxDistance : 10000.0 } } }).limit(5)
  15. 15. Stores – Indices • { "storeId" : 1 }, { "unique" : true } • { "name" : 1 } • { "address.zip" : 1 } • { "location" : "2dsphere" }
  16. 16. > db.inventory.findOne() { "_id": "5354869f300487d20b2b011d", "storeId": "store0", "location": [-86.95444, 33.40178], "productId": "p0", "vars": [ { "sku": "sku1", "q": 14 }, { "sku": "sku3", "q": 7 }, { "sku": "sku7", "q": 32 }, { "sku": "sku14", "q": 65 }, ... ] } Inventory – Sample Document
  17. 17. Inventory – Sample Queries • Get all items in a store db.inventory.find({ storeId : "store100" }) • Get quantity for an item at a store db.inventory.find({ "storeId" : "store100", "productId" : "p200" })
  18. 18. Inventory – Sample Queries • Get quantity for a sku at a store db.inventory.find( { "storeId" : "store100", "productId" : "p200", "vars.sku" : "sku11736" }, { "vars.$" : 1 } )
  19. 19. Inventory – Sample Update • Increment / decrement inventory for an item at a store db.inventory.update( { "storeId" : "store100", "productId" : "p200", "vars.sku" : "sku11736" }, { "$inc" : { "vars.$.q" : 20 } } )
  20. 20. Inventory – Sample Aggregations • Aggregate total quantity for a product db.inventory.aggregate( [ { $match : { productId : "p200" } }, { $unwind : "$vars" }, { $group : { _id : "result", count : { $sum : "$vars.q" } } } ] ) { "_id" : "result", "count" : 101752 }
  21. 21. Inventory – Sample Aggregations • Aggregate total quantity for a store db.inventory.aggregate( [ { $match : { storeId : "store100" } }, { $unwind : "$vars" }, { $match : { "vars.q" : { $gt : 0 } } }, { $group : { _id : "result", count : { $sum : 1 } } } ] ) { "_id" : "result", "count" : 29347 }
  22. 22. Inventory – Sample Aggregations • Aggregate total quantity for a store db.inventory.aggregate( [ { $match : { storeId : "store100" } }, { $unwind : "$vars" }, { $group : { _id : "result", count : { $sum : "$vars.q" } } } ] ) { "_id" : "result", "count" : 29347 }
  23. 23. Inventory – Sample Geo-Query • Get inventory for an item near a point db.runCommand( { geoNear : "inventory", near : { type : "Point", coordinates : [-82.8006, 40.0908] }, maxDistance : 10000.0, spherical : true, limit : 10, query : { "productId" : "p200", "vars.sku" : "sku11736" } } )
  24. 24. Inventory – Sample Geo-Query • Get closest store with available sku db.runCommand( { geoNear : "inventory", near : { type : "Point", coordinates : [-82.800672, 40.090844] }, maxDistance : 10000.0, spherical : true, limit : 1, query : { productId : "p200", vars : { $elemMatch : { sku : "sku11736", q : { $gt : 0 } } } } } )
  25. 25. Inventory – Sample Geo- Aggregation • Get count of inventory for an item near a point db.inventory.aggregate([ { $geoNear: { near : { type : "Point", coordinates : [-82.800672, 40.090844] }, distanceField: "distance", maxDistance: 10000.0, spherical : true, query: { productId: "p200", vars : { $elemMatch : { sku : "sku11736", q : {$gt : 0} } } }, includeLocs: "dist.location", num: 5 } }, { $unwind: "$vars" }, { $match: { "vars.sku" : "sku11736" } }, { $group: { _id: "result", count: {$sum: "$vars.q"} } }])
  26. 26. Inventory – Sample Indices • { storeId : 1 } • { productId : 1, storeId : 1 } • { productId : 1, location : "2dsphere" } • Why not "vars.sku"? – { productId : 1, storeId : 1, "vars.sku" : 1 }
  27. 27. Horizontal Scaling Inventory – Technical Decisions Store Inventory Schema Indexing
  28. 28. Shard East Shard Central Shard West East DC Inventory – Sharding Topology West DC Central DC Legacy Inventory Primary Primary Primary
  29. 29. Inventory – Shard Key • Choose shard key – { storeId : 1 } ? – { productId : 1, storeId : 1 }? – { storeId : 1, productId : 1 } ? • Set up sharding – sh.enableSharding("inventoryDB") – sh.shardCollection( "inventoryDB.inventory", { storeId : 1, productId : 1 })
  30. 30. Inventory – Shard Tags • Set up shard tags – sh.addShardTag("shard0000", "west") – sh.addShardTag("shard0001", "central") – sh.addShardTag("shard0002", "east") • Set up tag ranges – sh.addTagRange("inventoryDB.inventory", { storeId : 0 }, { storeId : 100}, "west" ) – sh.addTagRange("inventoryDB.inventory", { storeId : 100 }, { storeId : 200 }, "central" ) – sh.addTagRange("inventoryDB.inventory", { storeId : 200 }, { storeId : 300 }, "east" )
  31. 31. Senior Consulting Engineer, MongoDB Norman Graham #MongoDBWorld Thank You
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×