• Save
Augmenting RDBMS with MongoDB for ecommerce
Upcoming SlideShare
Loading in...5
×
 

Augmenting RDBMS with MongoDB for ecommerce

on

  • 11,557 views

Steve Francia, VP of Engineering at OpenSky a NYC based social commerce company, on how OpenSky augments using RDBMS with MongoDB to develop the next ecommerce platform. ...

Steve Francia, VP of Engineering at OpenSky a NYC based social commerce company, on how OpenSky augments using RDBMS with MongoDB to develop the next ecommerce platform.

OpenSky utilizes both traditional SQL solutions and combines them with NoSQL to overcome the limitations of each, increase development speed and scale quickly.

Statistics

Views

Total Views
11,557
Views on SlideShare
6,608
Embed Views
4,949

Actions

Likes
41
Downloads
0
Comments
2

22 Embeds 4,949

http://spf13.com 1886
http://www.abstra.cc 1480
http://www.nosqldatabases.com 527
http://judydba.tistory.com 516
http://justinhileman.info 449
http://feeds.feedburner.com 21
http://www.sfexception.com 18
http://localhost 7
http://www.linkedin.com 6
http://www.slideshare.net 6
http://flavors.me 5
http://paper.li 5
https://twitter.com 4
http://translate.googleusercontent.com 4
http://webcache.googleusercontent.com 3
http://www.hanrss.com 3
http://localhost:3000 2
url_unknown 2
http://72.30.186.176 2
http://supportfiles.scribeseo.com 1
http://www.twylah.com 1
http://beta.justinhileman.info 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • actually, just the first 1/3 of it. \n
  • \n
  • Ironically this is how magento solves the performance problems associated with EAV, by caching the data into insanely wide tables.\n
  • \n
  • \n
  • \n
  • Can’t create a FK as each set references a different table. “Key” really made of attribute table name id and attribute table name\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Whenever you use a inter system coordination you need to implement your own atomic checks in the application... But SOAP does have transactions.. so not quite accurate. \n\nkyle idea... but we are fairly atomic with authorize.net\n\n
  • Whenever you use a inter system coordination you need to implement your own atomic checks in the application... But SOAP does have transactions.. so not quite accurate. \n\nkyle idea... but we are fairly atomic with authorize.net\n\n
  • atomicity, consistency, isolation, durability.\n\n
  • atomicity, consistency, isolation, durability.\n\n
  • Mongo has a grip of atomic operations: set, unset, etc.\n
  • Mongo has a grip of atomic operations: set, unset, etc.\n
  • Mongo has a grip of atomic operations: set, unset, etc.\n
  • \n
  • \n
  • \n
  • update( { where }, { values }, upsert?, multiple? )\n\n\n
  • update( { where }, { values }, upsert?, multiple? )\n\n\n
  • update( { where }, { values }, upsert?, multiple? )\n\n\n
  • update( { where }, { values }, upsert?, multiple? )\n\n\n
  • update( { where }, { values }, upsert?, multiple? )\n\n\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • lemme show you an example\n
  • lemme show you an example\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Imagine what would happen if everyone tried to access the same record at the same time. Just think of all those spinning while loops :)\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • And I’ll show you how OpenSky does it.\n
  • \n
  • Since we really like MongoDB, we want to keep as much of our data in Mongo as possible.\n
  • \n
  • \n
  • \n
  • Mind if I tell you a story?\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • But this sounds like it could get COMPLICATED...\n
  • But this sounds like it could get COMPLICATED...\n
  • But this sounds like it could get COMPLICATED...\n
  • But this sounds like it could get COMPLICATED...\n
  • But this sounds like it could get COMPLICATED...\n
  • But this sounds like it could get COMPLICATED...\n
  • But this sounds like it could get COMPLICATED...\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Given that split, we just happen to have the most boring SQL schema ever\n
  • This is pretty much it.\n\nIt goes on for a few more lines, with a few other properties flattened onto the order table. \n
  • \n
  • \n
  • Back to the schema for a second.\n\n- Product ID here is a fake foreign key.\n- Inventory is a real integer.\n\nThat’s all there is to this table.\n
  • \n
  • \n
  • \n
  • And here’s why we like Doctrine so much.\n
  • And here’s why we like Doctrine so much.\n
  • And here’s why we like Doctrine so much.\n
  • This will look a bit like when I bought those shoes.\n
  • This will look a bit like when I bought those shoes.\n
  • This will look a bit like when I bought those shoes.\n
  • This will look a bit like when I bought those shoes.\n
  • This will look a bit like when I bought those shoes.\n
  • This will look a bit like when I bought those shoes.\n
  • This will look a bit like when I bought those shoes.\n
  • \n
  • \n
  • The interesting parts here are the annotations.\n\nIf you don’t speak PHP annotation, this stores a document with two properties—ID and title—in the `products` collection of a Mongo database.\n
  • \n
  • \n
  • Did you notice the property named `product`? That’s not just a reference to another document, that’s a reference to an entirely different database paradigm.\n\nCheck out the setter:\n
  • This is key: we set both the product id and a reference to the product itself.\n
  • When this document is saved in Mongo, the productId will end up in the database, but the product reference will disappear.\n
  • \n
  • This is one of those listeners I was telling you about. At a high level:\n\n1. Every time an Order is loaded from the database, this listener is called.\n2. The listener gets the Order’s product id, and creates a Doctrine proxy object.\n3. It uses magick (e.g. reflection) to set the product property of the order to this new proxy.\n
  • Here’s our inter-db relationship in action.\n\nNote that the product is lazily loaded from MongoDB. Because $product is a proxy, we don’t actually query Mongo until we try to access a property of $product (in this case the title).\n
  • \n
  • \n
  • \n

Augmenting RDBMS with MongoDB for ecommerce Augmenting RDBMS with MongoDB for ecommerce Presentation Transcript

  • Augmenting RDBMS with MongoDB for e-commerce
  • My name isSteve Francia @spf13
  • • 15+ years building e-commerce• Long time open source contributor• Entrepreneur• Hacker, father, husband, skate punk
  • My name isJustin Hileman @bobthecow
  • • 10+ years making the Internet awesomer• Open Source contributor• Vespa rider, swing dancer, coder, standardista
  • We work for OpenSky http://shopopensky.com
  • OpenSky is a new way to shopOpenSky connects you with innovators,trendsetters and tastemakers.You choosethe ones you like and each week they inviteyou to their private online sales.
  • OpenSky Loves Open Source• PHP 5.3• Apache2• Symfony2• Doctrine2• jQuery• Mule• HornetQ• MongoDB• nginx• varnish
  • We contribute to manyopen source projects and pioneer innovative solutions using them
  • OpenSky was the firste-commerce site built on MongoDB... also the first e-commerce site built on Symfony2
  • Why NoSQL for e-commerce?Using the right solution for each situation
  • Data dilemma of e-commerce Pick One
  • Data dilemma of e-commerce Pick One• Stick to one vertical (Sane schema)
  • Data dilemma of e-commerce Pick One• Stick to one vertical (Sane schema)• Flexibility (Insane schema)
  • Sane schema
  • Sane schema• Works ... for a while
  • Sane schema• Works ... for a while• Fine for a few types of products
  • Sane schema• Works ... for a while• Fine for a few types of products• Not possible when more product types introduced
  • Let’s Use an Example
  • Let’s Use an Example How about we start with books
  • Book Product SchemaProduct {id:sku: General Productproduct dimensions:shipping weight: attributesMSRP:price:description:...author: Orson Scott Cardtitle: Enders Gamebinding: Hardcoverpublication date: July 15, 1994 Book Specificpublisher name: Tor Science Fiction attributesnumber of pages: 352ISBN: 0812550706language: English...
  • Seems simple enough
  • Seems simple enoughWhat happens when we add another vertical... say music albums
  • Album Product SchemaProduct {id:sku: General Productproduct dimensions: attributes stay theshipping weight:MSRP: sameprice:description:...artist: MxPxtitle: Panic Album Specificrelease date: June 7, 2005 attributes arelabel: Side One Dummytrack listing: [ The Darkest ... differentlanguage: Englishformat: CD...
  • Okay, it’s getting hairy butis still manageable, right?
  • Okay, it’s getting hairy butis still manageable, right? Now the business want to sell jeans
  • Jeans Product SchemaProduct {id: General Productsku:product dimensions: attributes stay theshipping weight: sameMSRP:price:description:...brand: Luckygender: Mens Jeans specificmake: Vintage attributes are totallystyle: Straight Cut different ... and notlength: 34width: 34 consistent acrosscolor: Hipster brands & makematerial: Cotten Blend...
  • Now we’re screwed
  • We need a flexibleschema in RDBMS
  • We need a flexibleschema in RDBMS We got this ... right?
  • Many approachesdealing with unknownunknowns in RDBMS
  • Many approachesdealing with unknownunknowns in RDBMS None work well
  • EAV as popularized by Magento“For purposes of flexibility, the Magneto database heavily utilizesan Entity-Attribute-Value (EAV) data model.As is often the case, the cost of flexibility is complexity -Magento is no exception.The process of manipulating data in Magento is often more“involved” than that typically experienced using traditionalrelational tables.” - Varien
  • EAV• Crazy SQL queries• Hundreds of joins in a query... or• Hundreds of queries joined in the application• No database enforced integrity
  • Did I say crazy SQL(this is a single query)
  • Did I say crazy SQL(this is a single query)You may have trouble reading this in the back
  • Selecting a single product
  • Single Table Inheritance (insanely wide tables)• No data integrity enforcement• Only can use FK for common elements• Very wasteful (but disk is cheap!)• Can’t effectively index
  • Generic Columns• No data integrity enforcement• No data type enforcement• Only can use FK for common elements• Wasteful (but disk is cheap!)• Can’t index
  • Serialized in Blob• Not searchable• No integrity• All the disadvantages of a document store, but none of the advantages• Never should be used• One exception is Oracle XML which operates similar to a document store
  • Concrete Table Inheritance (a table for each product attribute set)• Allows for data integrity• Querying across attribute sets quite hard to do (lots of joins, OR statements and full table scanning)• New table needs to be created for each new attribute set
  • Class table inheritance (single product table, each attribute set in own table)• Likely best solution within the constraint of SQL• Supports data type enforcement• No data integrity enforcement• Easy querying across categories (for browse pages) since common data in single table• Every set needs a new table• Requires a ton of forsight, as changes are very complicated
  • MongoDB to the Rescue
  • MongoDB to the Rescue• Flexible (and sane) Schema
  • MongoDB to the Rescue• Flexible (and sane) Schema• Easily searchable
  • MongoDB to the Rescue• Flexible (and sane) Schema• Easily searchable• Easily accessible
  • MongoDB to the Rescue• Flexible (and sane) Schema• Easily searchable• Easily accessible• Fast
  • Flexible schema
  • { { sku: "00e8da9c", sku: "00e8da9d", type: "Audio Album", type: "Film", title: "Hoss", title: "The Matrix", description: "by Lagwagon", description: "Set in the 22nd century, Th asin: "B0000007QG", asin: "B000P0J0AQ", shipping: { shipping: { weight: 6, weight: 6, dimensions: { dimensions: { width: 10, width: 10, height: 10, height: 10, depth: 1 depth: 1 }, }, }, }, pricing: { pricing: { list: 1000, list: 1200, retail: 800, retail: 1100, savings: 200, savings: 100, pct_savings: 20 pct_savings: 8.5 }, }, details: { details: { title: "Hoss", title: "The Matrix",
  • pct_savings: 20 pct_savings: 8.5}, },details: { details: { title: "Hoss", title: "The Matrix", artist: "Lagwagon", director: [ "Andy Wachowski", "Larry Wa genre: [ "Punk", "Hardcore", "Indie Rock" ], [ "Andy Wachowski", "Larry Wach writer: label: "Fat Wreck Chords", actor: [ "Keanu Reeves" , "Lawrence Fis number_of_discs: 1, genre: [ "Science Fiction", "Action" ], issue_date: "November 21, 1995", number_of_discs: 1, format: "CD", issue_date: "May 15 2007", alternate_formats: [ Vinyl, MP3 ],original_release_date: "1999", tracks: [ disc_format: "DVD", "Kids Dont Like To Share", rating: "R", "Violins", alternate_formats: [ VHS, Bluray ], "Name Dropping", run_time: "136", "Bombs Away", studio: "Warner Bros", "Move The Car", language: "English", "Sleep", format: [ "AC-3", "Closed-captioned", " "Sick", aspect_ratio: "1.66:1" "Rifle", }, "Weak", } "Black Eye", "Bro Dependent", "Razor Burn", "Shaving Your Head", "Ride The Snake", ],
  • Queries
  • db.products.find( { name: "The Matrix" } );
  • db.products.find( { name: "The Matrix" } ); { "_id": ObjectId("4d8ad78b46b731a22943d3d3"), "sku": "00e8da9d", "type": "Film", "name": "The Matrix", "description": "Set in the 22nd century, The Matrix...", "asin": "B000P0J0AQ", "shipping": { "weight": 6, "dimensions": { "width": 10, "height": 10, "depth": 1 } }, "pricing": {
  • db.products.find( { details.actor: "Groucho Marx" } );
  • db.products.find( { details.actor: "Groucho Marx" } ); }, "pricing": { "list": 1000, "retail": 800, "savings": 200, "pct_savings": 20 }, "details": { "title": "A Night at the Opera", "director": "Sam Wood", "actor": ["Groucho Marx", "Chico Marx", "Harpo Marx"], "genre": "Comedy", "number_of_discs": 1, "issue_date": "May 4 2004", "original_release_date": "1935", "disc_format": "DVD",
  • db.products.find( { details.genre: "Jazz", details.format: "CD"} );
  • db.products.find( { details.genre: "Jazz", details.format: "CD"} ); "list": 1200, "retail": 1100, "savings": 100, "pct_savings": 8 }, "details": { "title": "A Love Supreme [Original Recording Reissued]", "artist": "John Coltrane", "genre": ["Jazz", "General"], "format": "CD", "label": "Impulse Records", "number_of_discs": 1, "issue_date": "December 9, 1964", "alternate_formats": ["Vinyl", "MP3"], "tracks": [ "A Love Supreme Part I: Acknowledgement",
  • db.products.find( { details.actor: { $all: [James Stewart, Donna Reed] }} );
  • db.products.find( { details.actor: { $all: [James Stewart, Donna Reed] }} ); }, "details": { "title": "Its a Wonderful Life", "director": "Frank Capra", "actor": ["James Stewart", "Donna Reed", "Lionel Barrymore"], "writer": [ "Frank Capra", "Albert Hackett", "Frances Goodrich", "Jo Swerling", "Michael Wilson" ], "genre": "Drama", "number_of_discs": 1, "issue_date": "Oct 31 2006", "original_release_date": "1947",
  • Wanna Play?• grab products.js from http://github.com/spf13/mongoProducts• mongo --shell products.js• > use mongoProducts
  • Embedded documents are great for orders• Ordered items need to be fixed at the time of purchase• Embed them right in the orderdb.order.find( { items.sku: 00e8da9f } );db.order.find( { items.details.actor: James Stewart} ).count();
  • Why not NoSQL?Using the right solution for each situation
  • Data (like people) arereally sensitive when it comes to money
  • Stricter datarequirements for $$
  • Stricter data requirements for $$• For financial systems any data inconsistency is unacceptable
  • Stricter data requirements for $$• For financial systems any data inconsistency is unacceptable• Perhaps you’ve heard of ACID?
  • What about ACID?
  • What about ACID?Q: Is MongoDB ACID?
  • What about ACID?Q: Is MongoDB ACID?A: Kinda
  • Atomicity
  • Atomicity• MongoDB does atomic writes
  • Atomicity• MongoDB does atomic writes ... for single document changesets
  • Atomicity• MongoDB does atomic writes ... for single document changesets• $set, $unset, $inc, $push, $pushAll, $pull, $pullAll, $bit
  • Consistency
  • Consistency• MongoDB can enforce unique keys
  • Consistency• MongoDB can enforce unique keys ... but only on keys shared by every document in the collection
  • Consistency• MongoDB can enforce unique keys ... but only on keys shared by every document in the collection• MongoDB cant enforce referential integrity
  • Isolation
  • Isolation• // Pseudo-isolated updates db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );
  • Isolation• // Pseudo-isolated updates db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );• // Isolated updates db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false , true );
  • Isolation• // Pseudo-isolated updates db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );• // Isolated updates db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false , true );• But there are caveats...
  • Isolation• // Pseudo-isolated updates db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );• // Isolated updates db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false , true );• But there are caveats... • Despite the $atomic keyword, this is not an atomic update, since atomicity implies “all or nothing”
  • Isolation• // Pseudo-isolated updates db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );• // Isolated updates db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false , true );• But there are caveats... • Despite the $atomic keyword, this is not an atomic update, since atomicity implies “all or nothing” • An isolated update can only act on a single collection. Multi- collection updates are not transactional, thus not isolatable.
  • Durability
  • Durability• Mongo has this one covered
  • What doesMongoDB Support?
  • • Atomic single document writes • If you need atomic writes across multi-document transactions dont use Mongo • Many e-commerce transactions could be accomplished within a single document write
  • • Atomic single document writes • If you need atomic writes across multi-document transactions dont use Mongo • Many e-commerce transactions could be accomplished within a single document write• Unique indexes • This only works on keys used by the entire collection
  • • Atomic single document writes • If you need atomic writes across multi-document transactions dont use Mongo • Many e-commerce transactions could be accomplished within a single document write• Unique indexes • This only works on keys used by the entire collection• Isolated (not atomic) single collection updates. • Mongo does not support locking • There are ways to work around this
  • • Atomic single document writes • If you need atomic writes across multi-document transactions dont use Mongo • Many e-commerce transactions could be accomplished within a single document write• Unique indexes • This only works on keys used by the entire collection• Isolated (not atomic) single collection updates. • Mongo does not support locking • There are ways to work around this• It’s durable
  • There are ways toguarantee ACID properties in inconsistent databases
  • There are ways toguarantee ACID properties in inconsistent databases (or, as we call them, consistency impaired databases)
  • Optimistic concurrency
  • Optimistic concurrency• Read the current state of a product
  • Optimistic concurrency• Read the current state of a product• Make your changes with the assertion that your product has the same state as it did when you last read it
  • Optimistic concurrency in MongoDB
  • Optimistic concurrency in MongoDBWe’ll use an update-if-current strategy.
  • Optimistic concurrency in MongoDBWe’ll use an update-if-current strategy.This example is straight from the documentation:
  • Optimistic concurrency in MongoDB We’ll use an update-if-current strategy. This example is straight from the documentation:> t = db.inventory> p = t.findOne({sku:abc})> t.update({_id:p._id, qty:p.qty}, {$inc: {qty: -1}});> db.$cmd.findOne({getlasterror:1});{"err" : , "updatedExisting" : true , "n" : 1 , "ok" : 1}// it worked
  • Optimistic concurrency in MongoDB We’ll use an update-if-current strategy. This example is straight from the documentation:> t = db.inventory> p = t.findOne({sku:abc})> t.update({_id:p._id, qty:p.qty}, {$inc: {qty: -1}});> db.$cmd.findOne({getlasterror:1});{"err" : , "updatedExisting" : true , "n" : 1 , "ok" : 1}// it worked ... If that didnt work, try again until it does.
  • Optimistic concurrency• Read the current state of a product.• Make your changes with the assertion that your product has the same state as it did when you last read it.
  • Optimistic concurrency• Read the current state of a product.• Make your changes with the assertion that your product has the same state as it did when you last read it.• Its possible to use OCC to bootstrap pessimistic concurrency and fake row level locking
  • Optimistic concurrency• Read the current state of a product.• Make your changes with the assertion that your product has the same state as it did when you last read it.• Its possible to use OCC to bootstrap pessimistic concurrency and fake row level locking ... ask me about this some time
  • Optimistic concurrency control assumes anenvironment with low data contention
  • OCC works great forcompanies like Amazon• Amazon has a long-tail catalog• A long tail catalog lends itself well to optimistic concurrency, because it has low data contention
  • OCC fails miserably for
  • OCC fails miserably for• eBay
  • OCC fails miserably for• eBay• Gilt
  • OCC fails miserably for• eBay• Gilt• Groupon
  • OCC fails miserably for• eBay• Gilt• Groupon• OpenSky
  • OCC fails miserably for• eBay• Gilt• Groupon• OpenSky• Living Social
  • OCC fails miserably for• eBay• Gilt• Groupon• OpenSky• Living Social• InsertFlashSaleSiteOfTheMinute
  • Flash sales and auctionsare defined by high data contention
  • Flash sales and auctionsare defined by high data contention• The model doesnt work otherwise
  • Flash sales and auctionsare defined by high data contention• The model doesnt work otherwise• They cant afford to be optimistic
  • Can we use pessimistic concurrency with a distributed NoSQL database?
  • Yep.
  • BlendingNoSQL & RDBMSUsing the right solution for each situation
  • Our goal is to put as much in Mongo as possible• What makes more sense in RDBMS? • Inventory • Orders
  • Inventory requires• Row level locking (or table level locking)
  • Orders require• Row level locking (or table level locking)• Atomic writes (inventory decremented)• Transactions (3rd party processing)
  • Inventory & checkout transactions
  • Commerce is ACID In Real Life
  • 1. I go to Barneys and see a pair of shoes I just have to buy.
  • 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).
  • 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).3. I take them up to the cash register and purchase them:
  • 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).3. I take them up to the cash register and purchase them: • Store inventory has been manually decremented.
  • 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).3. I take them up to the cash register and purchase them: • Store inventory has been manually decremented. • I pay for them with my trusty AmEx.
  • 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).3. I take them up to the cash register and purchase them: • Store inventory has been manually decremented. • I pay for them with my trusty AmEx.4. If all goes according to plan, I walk out of the store.
  • 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).3. I take them up to the cash register and purchase them: • Store inventory has been manually decremented. • I pay for them with my trusty AmEx.4. If all goes according to plan, I walk out of the store.5. If my card was declined, the shoes are “rolled back”
  • 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).3. I take them up to the cash register and purchase them: • Store inventory has been manually decremented. • I pay for them with my trusty AmEx.4. If all goes according to plan, I walk out of the store.5. If my card was declined, the shoes are “rolled back” ... out onto the shelves and sold to the next customer who wants them.
  • We follow the samemodel for e-commerce
  • 1. Select a product.
  • 1. Select a product.2. Lock the row or table and confirm inventory.
  • 1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product:
  • 1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product: • Decrement product inventory
  • 1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product: • Decrement product inventory • Process payment
  • 1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product: • Decrement product inventory • Process payment4. Commit the transaction.
  • 1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product: • Decrement product inventory • Process payment4. Commit the transaction.5. Roll back if anything went wrong.
  • Doctrine (ORM/ODM) to the rescue
  • Doctrine (ORM/ODM) to the rescue It would be possible without them, but were not that masochistic
  • Data we store in SQL• Order• Order/Shipment• Order/Transaction• Inventory
  • Data we store in MongoDB
  • Data we store in MongoDB• User • Event• Product • TaxRate• Product/Sellable • ... and then I got tired of typing them in• Address • Just imagine this list has• Cart 40 more classes• CreditCard • ...
  • We have themost boring SQL schema ever
  • CREATE TABLE `product_inventory` ( `product_id` char(32) NOT NULL, `inventory` int(11) NOT NULL DEFAULT 0, PRIMARY KEY (`product_id`));CREATE TABLE `sellable_inventory` ( `sellable_id` char(32) NOT NULL, `inventory` int(11) NOT NULL DEFAULT 0, PRIMARY KEY (`sellable_id`));CREATE TABLE `orders` ( `id` int(11) NOT NULL AUTO_INCREMENT, `userId` char(32) NOT NULL, `shippingName` varchar(255) DEFAULT NULL, `shippingAddress1` varchar(255) DEFAULT NULL, `shippingAddress2` varchar(255) DEFAULT NULL, `shippingCity` varchar(255) DEFAULT NULL, `shippingState` varchar(2) DEFAULT NULL, `shippingZip` varchar(255) DEFAULT NULL, `billingName` varchar(255) DEFAULT NULL, `billingAddress1` varchar(255) DEFAULT NULL, `billingAddress2` varchar(255) DEFAULT NULL, `billingCity` varchar(255) DEFAULT NULL,
  • Wait. How does inventory live in SQL?Isn’t that a property in one of your Mongo collections?
  • I thought you’d never ask!
  • CREATE TABLE `product_inventory` ( `product_id` char(32) NOT NULL, `inventory` int(11) NOT NULL DEFAULT 0, PRIMARY KEY (`product_id`));CREATE TABLE `sellable_inventory` ( `sellable_id` char(32) NOT NULL, `inventory` int(11) NOT NULL DEFAULT 0, PRIMARY KEY (`sellable_id`));CREATE TABLE `orders` ( `id` int(11) NOT NULL AUTO_INCREMENT, `userId` char(32) NOT NULL, `shippingName` varchar(255) DEFAULT NULL, `shippingAddress1` varchar(255) DEFAULT NULL, `shippingAddress2` varchar(255) DEFAULT NULL, `shippingCity` varchar(255) DEFAULT NULL, `shippingState` varchar(2) DEFAULT NULL, `shippingZip` varchar(255) DEFAULT NULL, `billingName` varchar(255) DEFAULT NULL, `billingAddress1` varchar(255) DEFAULT NULL, `billingAddress2` varchar(255) DEFAULT NULL, `billingCity` varchar(255) DEFAULT NULL,
  • Inventory is transient• Product::$inventory is effectively a transient property• Note how I said “effectively”? ... we cheat and persist our transient property to MongoDB as well• We can do this because we never really trust the value stored in Mongo
  • Accuracy is only important when there’s contention
  • Accuracy is only important when there’s contention• For display, sorting and alerts, we can use the value stashed in MongoDB • It’s faster • It’s accurate enough
  • Accuracy is only important when there’s contention• For display, sorting and alerts, we can use the value stashed in MongoDB • It’s faster • It’s accurate enough• For financial transactions, we want the security and comfort of our RDBMS.
  • We keep inventory in sync with listeners
  • We keep inventory in sync with listeners• Every time a new product is created, its inventory is inserted in SQL
  • We keep inventory in sync with listeners• Every time a new product is created, its inventory is inserted in SQL• Every time an order is placed, inventory is verified and decremented
  • We keep inventory in sync with listeners• Every time a new product is created, its inventory is inserted in SQL• Every time an order is placed, inventory is verified and decremented• Whenever the SQL inventory changes, it is saved to MongoDB as well
  • Be careful what you lock
  • Be careful what you lock1. Acquire inventory row lock and begin transaction2. Check current product inventory3. Decrement product inventory4. Write the Order to SQL5. Update affected MongoDB documents6. Commit the transaction7. Release product inventory lock
  • Making MongoDBand RDBMS relations play nice
  • Products aredocuments stored in MongoDB
  • /** @mongodb:Document(collection="products") */class Product{ /** @mongodb:Id */ private $id; /** @mongodb:String */ private $title; public function getId() { return $this->id; } public function getTitle() { return $this->title; } public function setTitle($title) { $this->title = $title; }}
  • Orders are entitiesstored in an RDBMS
  • /** * @orm:Entity * @orm:Table(name="orders") * @orm:HasLifecycleCallbacks */class Order{ /** * @orm:Id @orm:Column(type="integer") * @orm:GeneratedValue(strategy="AUTO") */ private $id; /** * @orm:Column(type="string") */ private $productId; /** * @var DocumentsProduct */ private $product; // ...}
  • So how does an RDBMS have areference to something outside the database?
  • Setting the Productclass Order { // ... public function setProduct(Product $product) { $this->productId = $product->getId(); $this->product = $product; }}
  • • $productId is mapped and persisted• $product which stores the Product instance is not a persistent entity property
  • Retrieving ourproduct later
  • OrderPostLoadListeneruse DoctrineORMEventLifecycleEventArgs;class OrderPostLoadListener{ public function postLoad(LifecycleEventArgs $eventArgs) { // get the order entity $order = $eventArgs->getEntity(); // get odm reference to order.product_id $productId = $order->getProductId(); $product = $this->dm->getReference(MyBundle:DocumentProduct, $productId); // set the product on the order $em = $eventArgs->getEntityManager(); $productReflProp = $em->getClassMetadata(MyBundle:EntityOrder) ->reflClass->getProperty(product); $productReflProp->setAccessible(true); $productReflProp->setValue($order, $product); }}
  • All Together Now// Create a new product and order$product = new Product();$product->setTitle(Test Product);$dm->persist($product);$dm->flush();$order = new Order();$order->setProduct($product);$em->persist($order);$em->flush();// Find the order later$order = $em->find(Order, $order->getId());// Instance of an uninitialized product proxy$product = $order->getProduct();// Initializes proxy and queries the monogodb databaseecho "Order Title: " . $product->getTitle();print_r($order);
  • Read more about this techniqueJon Wage, one of OpenSky’s engineers, firstwrote about this technique on his personalblog: http://jwage.comYou can read the full article here:http://jwage.com/2010/08/25/blending-the-doctrine-orm-and-mongodb-odm/
  • Questions? http://spf13.com @spf13 http://justinhileman.com @bobthecow http://shopopensky.comPS: We’re hiring!! Contact us at jobs@shopopensky.com