Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Blending MongoDB   with RDBMS    for e-commerce
My name isSteve Francia    @spf13
• 15+ years building e-commerce• Long time open source contributor• Entrepreneur• Hacker, father, husband, skate punk• VP ...
My name isJustin Hileman    @bobthecow
• 10+ years making the Internet  awesomer• Open Source contributor• Vespa rider, swing dancer,  coder, standardista• Softw...
We work for OpenSky     http://opensky.com
OpenSky is  a new way to shopOpenSky connects you with innovators,trendsetters and tastemakers.You choosethe ones you like...
OpenSky Loves             Open Source•   PHP 5.3•   Apache2•   Symfony2•   Doctrine2•   jQuery•   Mule•   HornetQ•   Mongo...
We contribute to manyopen source projects and   pioneer innovative  solutions using them
OpenSky was the firste-commerce site built    on MongoDB... also the first e-commerce site built on Symfony2
Why NoSQL for  e-commerce?Using the right solution for each situation
Data dilemma of e-commerce     Pick One
Data dilemma of      e-commerce                 Pick One• Stick to one vertical (Sane schema)
Data dilemma of      e-commerce                 Pick One• Stick to one vertical (Sane schema)• Flexibility (Insane schema)
Sane schema
Sane schema• Works ... for a while
Sane schema• Works ... for a while• Fine for a few types of products
Sane schema• Works ... for a while• Fine for a few types of products• Not possible when more product types  introduced
Let’s Use an Example
Let’s Use an Example   How about we start with books
Book Product SchemaProduct {id:sku:                                    General Productproduct dimensions:shipping weight: ...
Seems simple enough
Seems simple enoughWhat happens when we add another vertical...            say music albums
Album Product SchemaProduct {id:sku:                               General Productproduct dimensions:                attri...
Okay, it’s getting hairy butis still manageable, right?
Okay, it’s getting hairy butis still manageable, right?    Now the business want to sell jeans
Jeans Product SchemaProduct {id:                           General Productsku:product dimensions:                         ...
Now we’re screwed
We need a flexibleschema in RDBMS
We need a flexibleschema in RDBMS    We got this ... right?
Many approachesdealing with unknownunknowns in RDBMS
Many approachesdealing with unknownunknowns in RDBMS      None work well
EAV             as popularized by Magento“For purposes of flexibility, the Magneto database heavily utilizesan Entity-Attri...
EAV•   Crazy SQL queries•   Hundreds of joins in a query...    or•   Hundreds of queries joined in    the application•   N...
Did I say crazy SQL(this is a single query)
Did I say crazy SQL(this is a single query)You may have trouble reading this in the back
Selecting a single product
Single Table Inheritance            (insanely wide tables)•   No data integrity enforcement•   Only can use FK for common ...
Generic Columns•   No data integrity enforcement•   No data type enforcement•   Only can use FK for common    elements•   ...
Serialized in Blob•   Not searchable•   No integrity•   All the disadvantages of a document    store, but none of the adva...
Concrete Table Inheritance    (a table for each product attribute set)•   Allows for data integrity•   Querying across att...
Class table inheritance                  (single product table,             each attribute set in own table)•   Likely bes...
MongoDB to the   Rescue
MongoDB to the        Rescue• Flexible (and sane) Schema
MongoDB to the        Rescue• Flexible (and sane) Schema• Easily searchable
MongoDB to the        Rescue• Flexible (and sane) Schema• Easily searchable• Easily accessible
MongoDB to the        Rescue• Flexible (and sane) Schema• Easily searchable• Easily accessible• Fast
Flexible schema
{                                 {    sku: "00e8da9c",                  sku: "00e8da9d",    type: "Audio Album",         ...
pct_savings: 20                      pct_savings: 8.5},                                   },details: {                    ...
Queries
db.products.find( { name: "The Matrix" } );
db.products.find( { name: "The Matrix" } ); {     "_id": ObjectId("4d8ad78b46b731a22943d3d3"),     "sku": "00e8da9d",     ...
db.products.find( { details.actor: "Groucho Marx" } );
db.products.find( { details.actor: "Groucho Marx" } ); }, "pricing": {     "list": 1000,     "retail": 800,     "savings":...
db.products.find( {     details.genre: "Jazz", details.format: "CD"} );
db.products.find( {     details.genre: "Jazz", details.format: "CD"} );     "list": 1200,     "retail": 1100,     "savings...
db.products.find( { details.actor:     { $all: [James Stewart, Donna Reed] }} );
db.products.find( { details.actor:     { $all: [James Stewart, Donna Reed] }} ); }, "details": {     "title": "Its a Wonde...
Wanna Play?•   grab products.js from    http://github.com/spf13/mongoProducts•   mongo --shell products.js•   > use mongoP...
Embedded documents are great for orders• Ordered items need to be fixed at the time  of purchase• Embed them right in the o...
Why not NoSQL?Using the right solution for each situation
Data (like people) arereally sensitive when it   comes to money
Stricter datarequirements for $$
Stricter data   requirements for $$• For financial systems any data inconsistency  is unacceptable
Stricter data   requirements for $$• For financial systems any data inconsistency  is unacceptable• Perhaps you’ve heard of...
What about ACID?
What about ACID?Q: Is MongoDB ACID?
What about ACID?Q: Is MongoDB ACID?A: Kinda
Atomicity
Atomicity• MongoDB does atomic writes
Atomicity• MongoDB does atomic writes  ... for single document changesets
Atomicity• MongoDB does atomic writes    ... for single document changesets•   $set, $unset, $inc, $push,    $pushAll, $pu...
Consistency
Consistency• MongoDB can enforce unique keys
Consistency• MongoDB can enforce unique keys  ... but only on keys shared by every  document in the collection
Consistency• MongoDB can enforce unique keys  ... but only on keys shared by every  document in the collection• MongoDB ca...
Isolation
Isolation•   // Pseudo-isolated updates    db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );
Isolation•   // Pseudo-isolated updates    db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );•   // Isolate...
Isolation•   // Pseudo-isolated updates    db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );•   // Isolate...
Isolation•   // Pseudo-isolated updates    db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );•   // Isolate...
Isolation•   // Pseudo-isolated updates    db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );•   // Isolate...
Durability
Durability• Mongo has this one covered
What doesMongoDB Support?
•   Atomic single document writes    •   If you need atomic writes across multi-document        transactions dont use Mong...
•   Atomic single document writes    •   If you need atomic writes across multi-document        transactions dont use Mong...
•   Atomic single document writes    •   If you need atomic writes across multi-document        transactions dont use Mong...
•   Atomic single document writes    •   If you need atomic writes across multi-document        transactions dont use Mong...
There are ways toguarantee ACID properties in inconsistent databases
There are ways toguarantee ACID properties in inconsistent databases (or, as we call them, consistency impaired databases)
Optimistic concurrency
Optimistic concurrency• Read the current state of a product
Optimistic concurrency• Read the current state of a product• Make your changes with the assertion that  your product has t...
Optimistic concurrency    in MongoDB
Optimistic concurrency    in MongoDBWe’ll use an update-if-current strategy.
Optimistic concurrency    in MongoDBWe’ll use an update-if-current strategy.This example is straight from the documentation:
Optimistic concurrency    in MongoDB    We’ll use an update-if-current strategy.    This example is straight from the docu...
Optimistic concurrency    in MongoDB    We’ll use an update-if-current strategy.    This example is straight from the docu...
Optimistic concurrency• Read the current state of a product.• Make your changes with the assertion that  your product has ...
Optimistic concurrency• Read the current state of a product.• Make your changes with the assertion that    your product ha...
Optimistic concurrency• Read the current state of a product.• Make your changes with the assertion that    your product ha...
Optimistic concurrency control assumes anenvironment with low   data contention
OCC works great forcompanies like Amazon• Amazon has a long-tail catalog• A long tail catalog lends itself well to  optimi...
OCC fails miserably for
OCC fails miserably for• eBay
OCC fails miserably for• eBay• Gilt
OCC fails miserably for• eBay• Gilt• Groupon
OCC fails miserably for• eBay• Gilt• Groupon• OpenSky
OCC fails miserably for• eBay• Gilt• Groupon• OpenSky• Living Social
OCC fails miserably for• eBay• Gilt• Groupon• OpenSky• Living Social• InsertFlashSaleSiteOfTheMinute
Flash sales and auctionsare defined by high data       contention
Flash sales and auctionsare defined by high data       contention• The model doesnt work otherwise
Flash sales and auctionsare defined by high data       contention• The model doesnt work otherwise• They cant afford to be ...
Can we use pessimistic concurrency with a distributed NoSQL       database?
Yep.
BlendingNoSQL & RDBMSUsing the right solution for each situation
Our goal is to put as much  in Mongo as possible• What makes more sense in RDBMS? • Inventory • Orders
Inventory requires• Row level locking (or table level locking)
Orders require• Row level locking (or table level locking)• Atomic writes (inventory decremented)• Transactions (3rd party...
Inventory & checkout     transactions
Commerce is ACID   In Real Life
1. I go to Barneys and see a pair of shoes I just have to   buy.
1. I go to Barneys and see a pair of shoes I just have to   buy.2. I call “dibs” (by grabbing them off the shelf).
1. I go to Barneys and see a pair of shoes I just have to   buy.2. I call “dibs” (by grabbing them off the shelf).3. I tak...
1. I go to Barneys and see a pair of shoes I just have to   buy.2. I call “dibs” (by grabbing them off the shelf).3. I tak...
1. I go to Barneys and see a pair of shoes I just have to   buy.2. I call “dibs” (by grabbing them off the shelf).3. I tak...
1. I go to Barneys and see a pair of shoes I just have to   buy.2. I call “dibs” (by grabbing them off the shelf).3. I tak...
1. I go to Barneys and see a pair of shoes I just have to   buy.2. I call “dibs” (by grabbing them off the shelf).3. I tak...
1. I go to Barneys and see a pair of shoes I just have to   buy.2. I call “dibs” (by grabbing them off the shelf).3. I tak...
We follow the samemodel for e-commerce
1. Select a product.
1. Select a product.2. Lock the row or table and confirm inventory.
1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product:
1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product:  •   Decrement product inventory
1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product:  •   Decrement product inventor...
1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product:  •   Decrement product inventor...
1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product:  •   Decrement product inventor...
Doctrine (ORM/ODM)    to the rescue
Doctrine (ORM/ODM)    to the rescue   It would be possible without them,      but were not that masochistic
Data we store in SQL• Order• Order/Shipment• Order/Transaction• Inventory
Data we store in  MongoDB
Data we store in             MongoDB•   User               •   Event•   Product            •   TaxRate•   Product/Sellable...
We have themost boring SQL  schema ever
CREATE TABLE `product_inventory` (   `product_id` char(32) NOT NULL,   `inventory` int(11) NOT NULL DEFAULT 0,   PRIMARY K...
Wait. How does inventory live in SQL?Isn’t that a property in one of your Mongo collections?
I thought you’d   never ask!
CREATE TABLE `product_inventory` (   `product_id` char(32) NOT NULL,   `inventory` int(11) NOT NULL DEFAULT 0,   PRIMARY K...
Inventory is transient•   Product::$inventory is effectively a    transient property• Note how I said “effectively”? ... w...
Accuracy is only important when there’s contention
Accuracy is only important when there’s contention• For display, sorting and alerts, we can use  the value stashed in Mong...
Accuracy is only important when there’s contention• For display, sorting and alerts, we can use  the value stashed in Mong...
We keep inventory in sync with listeners
We keep inventory in sync with listeners• Every time a new product is created, its  inventory is inserted in SQL
We keep inventory in sync with listeners• Every time a new product is created, its  inventory is inserted in SQL• Every ti...
We keep inventory in sync with listeners• Every time a new product is created, its  inventory is inserted in SQL• Every ti...
Be careful what you lock
Be careful what you lock1. Acquire inventory row lock and begin transaction2. Check current product inventory3. Decrement ...
Making MongoDBand RDBMS relations      play nice
Products aredocuments stored  in MongoDB
/** @mongodb:Document(collection="products") */class Product{    /** @mongodb:Id */    private $id;    /** @mongodb:String...
Orders are entitiesstored in an RDBMS
/** * @orm:Entity * @orm:Table(name="orders") * @orm:HasLifecycleCallbacks */class Order{    /**     * @orm:Id @orm:Column...
So how does an     RDBMS have areference to something outside the database?
Setting the Productclass Order {    // ...    public function setProduct(Product $product)    {        $this->productId = ...
•   $productId is mapped and persisted•   $product which stores the Product    instance is not a persistent entity property
Retrieving ourproduct later
OrderPostLoadListeneruse DoctrineORMEventLifecycleEventArgs;class OrderPostLoadListener{    public function postLoad(Lifec...
All Together Now// Create a new product and order$product = new Product();$product->setTitle(Test Product);$dm->persist($p...
Read more about       this techniqueJon Wage, one of OpenSky’s engineers, firstwrote about this technique on his personalbl...
Questions?             http://spf13.com                @spf13             http://justinhileman.com                @bobthec...
Blending MongoDB and RDBMS for ecommerce
Blending MongoDB and RDBMS for ecommerce
Blending MongoDB and RDBMS for ecommerce
Blending MongoDB and RDBMS for ecommerce
Upcoming SlideShare
Loading in …5
×

Blending MongoDB and RDBMS for ecommerce

12,181 views

Published on

Blending MongoDB and RDBMS for ecommerce

  1. 1. Blending MongoDB with RDBMS for e-commerce
  2. 2. My name isSteve Francia @spf13
  3. 3. • 15+ years building e-commerce• Long time open source contributor• Entrepreneur• Hacker, father, husband, skate punk• VP Engineering @ OpenSky
  4. 4. My name isJustin Hileman @bobthecow
  5. 5. • 10+ years making the Internet awesomer• Open Source contributor• Vespa rider, swing dancer, coder, standardista• Software Engineer @ OpenSky
  6. 6. We work for OpenSky http://opensky.com
  7. 7. OpenSky is a new way to shopOpenSky connects you with innovators,trendsetters and tastemakers.You choosethe ones you like and each week they inviteyou to their private online sales.
  8. 8. OpenSky Loves Open Source• PHP 5.3• Apache2• Symfony2• Doctrine2• jQuery• Mule• HornetQ• MongoDB• nginx• varnish
  9. 9. We contribute to manyopen source projects and pioneer innovative solutions using them
  10. 10. OpenSky was the firste-commerce site built on MongoDB... also the first e-commerce site built on Symfony2
  11. 11. Why NoSQL for e-commerce?Using the right solution for each situation
  12. 12. Data dilemma of e-commerce Pick One
  13. 13. Data dilemma of e-commerce Pick One• Stick to one vertical (Sane schema)
  14. 14. Data dilemma of e-commerce Pick One• Stick to one vertical (Sane schema)• Flexibility (Insane schema)
  15. 15. Sane schema
  16. 16. Sane schema• Works ... for a while
  17. 17. Sane schema• Works ... for a while• Fine for a few types of products
  18. 18. Sane schema• Works ... for a while• Fine for a few types of products• Not possible when more product types introduced
  19. 19. Let’s Use an Example
  20. 20. Let’s Use an Example How about we start with books
  21. 21. Book Product SchemaProduct {id:sku: General Productproduct dimensions:shipping weight: attributesMSRP:price:description:...author: Orson Scott Cardtitle: Enders Gamebinding: Hardcoverpublication date: July 15, 1994 Book Specificpublisher name: Tor Science Fiction attributesnumber of pages: 352ISBN: 0812550706language: English...
  22. 22. Seems simple enough
  23. 23. Seems simple enoughWhat happens when we add another vertical... say music albums
  24. 24. Album Product SchemaProduct {id:sku: General Productproduct dimensions: attributes stay theshipping weight:MSRP: sameprice:description:...artist: MxPxtitle: Panic Album Specificrelease date: June 7, 2005 attributes arelabel: Side One Dummytrack listing: [ The Darkest ... differentlanguage: Englishformat: CD...
  25. 25. Okay, it’s getting hairy butis still manageable, right?
  26. 26. Okay, it’s getting hairy butis still manageable, right? Now the business want to sell jeans
  27. 27. Jeans Product SchemaProduct {id: General Productsku:product dimensions: attributes stay theshipping weight: sameMSRP:price:description:...brand: Luckygender: Mens Jeans specificmake: Vintage attributes are totallystyle: Straight Cut different ... and notlength: 34width: 34 consistent acrosscolor: Hipster brands & makematerial: Cotten Blend...
  28. 28. Now we’re screwed
  29. 29. We need a flexibleschema in RDBMS
  30. 30. We need a flexibleschema in RDBMS We got this ... right?
  31. 31. Many approachesdealing with unknownunknowns in RDBMS
  32. 32. Many approachesdealing with unknownunknowns in RDBMS None work well
  33. 33. EAV as popularized by Magento“For purposes of flexibility, the Magneto database heavily utilizesan Entity-Attribute-Value (EAV) data model.As is often the case, the cost of flexibility is complexity -Magento is no exception.The process of manipulating data in Magento is often more“involved” than that typically experienced using traditionalrelational tables.” - Varien
  34. 34. EAV• Crazy SQL queries• Hundreds of joins in a query... or• Hundreds of queries joined in the application• No database enforced integrity
  35. 35. Did I say crazy SQL(this is a single query)
  36. 36. Did I say crazy SQL(this is a single query)You may have trouble reading this in the back
  37. 37. Selecting a single product
  38. 38. Single Table Inheritance (insanely wide tables)• No data integrity enforcement• Only can use FK for common elements• Very wasteful (but disk is cheap!)• Can’t effectively index
  39. 39. Generic Columns• No data integrity enforcement• No data type enforcement• Only can use FK for common elements• Wasteful (but disk is cheap!)• Can’t index
  40. 40. Serialized in Blob• Not searchable• No integrity• All the disadvantages of a document store, but none of the advantages• Never should be used• One exception is Oracle XML which operates similar to a document store
  41. 41. Concrete Table Inheritance (a table for each product attribute set)• Allows for data integrity• Querying across attribute sets quite hard to do (lots of joins, OR statements and full table scanning)• New table needs to be created for each new attribute set
  42. 42. Class table inheritance (single product table, each attribute set in own table)• Likely best solution within the constraint of SQL• Supports data type enforcement• No data integrity enforcement• Easy querying across categories (for browse pages) since common data in single table• Every set needs a new table• Requires a ton of forsight, as changes are very complicated
  43. 43. MongoDB to the Rescue
  44. 44. MongoDB to the Rescue• Flexible (and sane) Schema
  45. 45. MongoDB to the Rescue• Flexible (and sane) Schema• Easily searchable
  46. 46. MongoDB to the Rescue• Flexible (and sane) Schema• Easily searchable• Easily accessible
  47. 47. MongoDB to the Rescue• Flexible (and sane) Schema• Easily searchable• Easily accessible• Fast
  48. 48. Flexible schema
  49. 49. { { sku: "00e8da9c", sku: "00e8da9d", type: "Audio Album", type: "Film", title: "Hoss", title: "The Matrix", description: "by Lagwagon", description: "Set in the 22nd century, Th asin: "B0000007QG", asin: "B000P0J0AQ", shipping: { shipping: { weight: 6, weight: 6, dimensions: { dimensions: { width: 10, width: 10, height: 10, height: 10, depth: 1 depth: 1 }, }, }, }, pricing: { pricing: { list: 1000, list: 1200, retail: 800, retail: 1100, savings: 200, savings: 100, pct_savings: 20 pct_savings: 8.5 }, }, details: { details: { title: "Hoss", title: "The Matrix",
  50. 50. pct_savings: 20 pct_savings: 8.5}, },details: { details: { title: "Hoss", title: "The Matrix", artist: "Lagwagon", director: [ "Andy Wachowski", "Larry Wa genre: [ "Punk", "Hardcore", "Indie Rock" ], [ "Andy Wachowski", "Larry Wach writer: label: "Fat Wreck Chords", actor: [ "Keanu Reeves" , "Lawrence Fis number_of_discs: 1, genre: [ "Science Fiction", "Action" ], issue_date: "November 21, 1995", number_of_discs: 1, format: "CD", issue_date: "May 15 2007", alternate_formats: [ Vinyl, MP3 ],original_release_date: "1999", tracks: [ disc_format: "DVD", "Kids Dont Like To Share", rating: "R", "Violins", alternate_formats: [ VHS, Bluray ], "Name Dropping", run_time: "136", "Bombs Away", studio: "Warner Bros", "Move The Car", language: "English", "Sleep", format: [ "AC-3", "Closed-captioned", " "Sick", aspect_ratio: "1.66:1" "Rifle", }, "Weak", } "Black Eye", "Bro Dependent", "Razor Burn", "Shaving Your Head", "Ride The Snake", ],
  51. 51. Queries
  52. 52. db.products.find( { name: "The Matrix" } );
  53. 53. db.products.find( { name: "The Matrix" } ); { "_id": ObjectId("4d8ad78b46b731a22943d3d3"), "sku": "00e8da9d", "type": "Film", "name": "The Matrix", "description": "Set in the 22nd century, The Matrix...", "asin": "B000P0J0AQ", "shipping": { "weight": 6, "dimensions": { "width": 10, "height": 10, "depth": 1 } }, "pricing": {
  54. 54. db.products.find( { details.actor: "Groucho Marx" } );
  55. 55. db.products.find( { details.actor: "Groucho Marx" } ); }, "pricing": { "list": 1000, "retail": 800, "savings": 200, "pct_savings": 20 }, "details": { "title": "A Night at the Opera", "director": "Sam Wood", "actor": ["Groucho Marx", "Chico Marx", "Harpo Marx"], "genre": "Comedy", "number_of_discs": 1, "issue_date": "May 4 2004", "original_release_date": "1935", "disc_format": "DVD",
  56. 56. db.products.find( { details.genre: "Jazz", details.format: "CD"} );
  57. 57. db.products.find( { details.genre: "Jazz", details.format: "CD"} ); "list": 1200, "retail": 1100, "savings": 100, "pct_savings": 8 }, "details": { "title": "A Love Supreme [Original Recording Reissued]", "artist": "John Coltrane", "genre": ["Jazz", "General"], "format": "CD", "label": "Impulse Records", "number_of_discs": 1, "issue_date": "December 9, 1964", "alternate_formats": ["Vinyl", "MP3"], "tracks": [ "A Love Supreme Part I: Acknowledgement",
  58. 58. db.products.find( { details.actor: { $all: [James Stewart, Donna Reed] }} );
  59. 59. db.products.find( { details.actor: { $all: [James Stewart, Donna Reed] }} ); }, "details": { "title": "Its a Wonderful Life", "director": "Frank Capra", "actor": ["James Stewart", "Donna Reed", "Lionel Barrymore"], "writer": [ "Frank Capra", "Albert Hackett", "Frances Goodrich", "Jo Swerling", "Michael Wilson" ], "genre": "Drama", "number_of_discs": 1, "issue_date": "Oct 31 2006", "original_release_date": "1947",
  60. 60. Wanna Play?• grab products.js from http://github.com/spf13/mongoProducts• mongo --shell products.js• > use mongoProducts
  61. 61. Embedded documents are great for orders• Ordered items need to be fixed at the time of purchase• Embed them right in the orderdb.order.find( { items.sku: 00e8da9f } );db.order.find( { items.details.actor: James Stewart} ).count();
  62. 62. Why not NoSQL?Using the right solution for each situation
  63. 63. Data (like people) arereally sensitive when it comes to money
  64. 64. Stricter datarequirements for $$
  65. 65. Stricter data requirements for $$• For financial systems any data inconsistency is unacceptable
  66. 66. Stricter data requirements for $$• For financial systems any data inconsistency is unacceptable• Perhaps you’ve heard of ACID?
  67. 67. What about ACID?
  68. 68. What about ACID?Q: Is MongoDB ACID?
  69. 69. What about ACID?Q: Is MongoDB ACID?A: Kinda
  70. 70. Atomicity
  71. 71. Atomicity• MongoDB does atomic writes
  72. 72. Atomicity• MongoDB does atomic writes ... for single document changesets
  73. 73. Atomicity• MongoDB does atomic writes ... for single document changesets• $set, $unset, $inc, $push, $pushAll, $pull, $pullAll, $bit
  74. 74. Consistency
  75. 75. Consistency• MongoDB can enforce unique keys
  76. 76. Consistency• MongoDB can enforce unique keys ... but only on keys shared by every document in the collection
  77. 77. Consistency• MongoDB can enforce unique keys ... but only on keys shared by every document in the collection• MongoDB cant enforce referential integrity
  78. 78. Isolation
  79. 79. Isolation• // Pseudo-isolated updates db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );
  80. 80. Isolation• // Pseudo-isolated updates db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );• // Isolated updates db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false , true );
  81. 81. Isolation• // Pseudo-isolated updates db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );• // Isolated updates db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false , true );• But there are caveats...
  82. 82. Isolation• // Pseudo-isolated updates db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );• // Isolated updates db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false , true );• But there are caveats... • Despite the $atomic keyword, this is not an atomic update, since atomicity implies “all or nothing”
  83. 83. Isolation• // Pseudo-isolated updates db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );• // Isolated updates db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false , true );• But there are caveats... • Despite the $atomic keyword, this is not an atomic update, since atomicity implies “all or nothing” • An isolated update can only act on a single collection. Multi- collection updates are not transactional, thus not isolatable.
  84. 84. Durability
  85. 85. Durability• Mongo has this one covered
  86. 86. What doesMongoDB Support?
  87. 87. • Atomic single document writes • If you need atomic writes across multi-document transactions dont use Mongo • Many e-commerce transactions could be accomplished within a single document write
  88. 88. • Atomic single document writes • If you need atomic writes across multi-document transactions dont use Mongo • Many e-commerce transactions could be accomplished within a single document write• Unique indexes • This only works on keys used by the entire collection
  89. 89. • Atomic single document writes • If you need atomic writes across multi-document transactions dont use Mongo • Many e-commerce transactions could be accomplished within a single document write• Unique indexes • This only works on keys used by the entire collection• Isolated (not atomic) single collection updates. • Mongo does not support locking • There are ways to work around this
  90. 90. • Atomic single document writes • If you need atomic writes across multi-document transactions dont use Mongo • Many e-commerce transactions could be accomplished within a single document write• Unique indexes • This only works on keys used by the entire collection• Isolated (not atomic) single collection updates. • Mongo does not support locking • There are ways to work around this• It’s durable
  91. 91. There are ways toguarantee ACID properties in inconsistent databases
  92. 92. There are ways toguarantee ACID properties in inconsistent databases (or, as we call them, consistency impaired databases)
  93. 93. Optimistic concurrency
  94. 94. Optimistic concurrency• Read the current state of a product
  95. 95. Optimistic concurrency• Read the current state of a product• Make your changes with the assertion that your product has the same state as it did when you last read it
  96. 96. Optimistic concurrency in MongoDB
  97. 97. Optimistic concurrency in MongoDBWe’ll use an update-if-current strategy.
  98. 98. Optimistic concurrency in MongoDBWe’ll use an update-if-current strategy.This example is straight from the documentation:
  99. 99. Optimistic concurrency in MongoDB We’ll use an update-if-current strategy. This example is straight from the documentation:> t = db.inventory> p = t.findOne({sku:abc})> t.update({_id:p._id, qty:p.qty}, {$inc: {qty: -1}});> db.$cmd.findOne({getlasterror:1});{"err" : , "updatedExisting" : true , "n" : 1 , "ok" : 1}// it worked
  100. 100. Optimistic concurrency in MongoDB We’ll use an update-if-current strategy. This example is straight from the documentation:> t = db.inventory> p = t.findOne({sku:abc})> t.update({_id:p._id, qty:p.qty}, {$inc: {qty: -1}});> db.$cmd.findOne({getlasterror:1});{"err" : , "updatedExisting" : true , "n" : 1 , "ok" : 1}// it worked ... If that didnt work, try again until it does.
  101. 101. Optimistic concurrency• Read the current state of a product.• Make your changes with the assertion that your product has the same state as it did when you last read it.
  102. 102. Optimistic concurrency• Read the current state of a product.• Make your changes with the assertion that your product has the same state as it did when you last read it.• Its possible to use OCC to bootstrap pessimistic concurrency and fake row level locking
  103. 103. Optimistic concurrency• Read the current state of a product.• Make your changes with the assertion that your product has the same state as it did when you last read it.• Its possible to use OCC to bootstrap pessimistic concurrency and fake row level locking ... ask me about this some time
  104. 104. Optimistic concurrency control assumes anenvironment with low data contention
  105. 105. OCC works great forcompanies like Amazon• Amazon has a long-tail catalog• A long tail catalog lends itself well to optimistic concurrency, because it has low data contention
  106. 106. OCC fails miserably for
  107. 107. OCC fails miserably for• eBay
  108. 108. OCC fails miserably for• eBay• Gilt
  109. 109. OCC fails miserably for• eBay• Gilt• Groupon
  110. 110. OCC fails miserably for• eBay• Gilt• Groupon• OpenSky
  111. 111. OCC fails miserably for• eBay• Gilt• Groupon• OpenSky• Living Social
  112. 112. OCC fails miserably for• eBay• Gilt• Groupon• OpenSky• Living Social• InsertFlashSaleSiteOfTheMinute
  113. 113. Flash sales and auctionsare defined by high data contention
  114. 114. Flash sales and auctionsare defined by high data contention• The model doesnt work otherwise
  115. 115. Flash sales and auctionsare defined by high data contention• The model doesnt work otherwise• They cant afford to be optimistic
  116. 116. Can we use pessimistic concurrency with a distributed NoSQL database?
  117. 117. Yep.
  118. 118. BlendingNoSQL & RDBMSUsing the right solution for each situation
  119. 119. Our goal is to put as much in Mongo as possible• What makes more sense in RDBMS? • Inventory • Orders
  120. 120. Inventory requires• Row level locking (or table level locking)
  121. 121. Orders require• Row level locking (or table level locking)• Atomic writes (inventory decremented)• Transactions (3rd party processing)
  122. 122. Inventory & checkout transactions
  123. 123. Commerce is ACID In Real Life
  124. 124. 1. I go to Barneys and see a pair of shoes I just have to buy.
  125. 125. 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).
  126. 126. 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).3. I take them up to the cash register and purchase them:
  127. 127. 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).3. I take them up to the cash register and purchase them: • Store inventory has been manually decremented.
  128. 128. 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).3. I take them up to the cash register and purchase them: • Store inventory has been manually decremented. • I pay for them with my trusty AmEx.
  129. 129. 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).3. I take them up to the cash register and purchase them: • Store inventory has been manually decremented. • I pay for them with my trusty AmEx.4. If all goes according to plan, I walk out of the store.
  130. 130. 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).3. I take them up to the cash register and purchase them: • Store inventory has been manually decremented. • I pay for them with my trusty AmEx.4. If all goes according to plan, I walk out of the store.5. If my card was declined, the shoes are “rolled back”
  131. 131. 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).3. I take them up to the cash register and purchase them: • Store inventory has been manually decremented. • I pay for them with my trusty AmEx.4. If all goes according to plan, I walk out of the store.5. If my card was declined, the shoes are “rolled back” ... out onto the shelves and sold to the next customer who wants them.
  132. 132. We follow the samemodel for e-commerce
  133. 133. 1. Select a product.
  134. 134. 1. Select a product.2. Lock the row or table and confirm inventory.
  135. 135. 1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product:
  136. 136. 1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product: • Decrement product inventory
  137. 137. 1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product: • Decrement product inventory • Process payment
  138. 138. 1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product: • Decrement product inventory • Process payment4. Commit the transaction.
  139. 139. 1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product: • Decrement product inventory • Process payment4. Commit the transaction.5. Roll back if anything went wrong.
  140. 140. Doctrine (ORM/ODM) to the rescue
  141. 141. Doctrine (ORM/ODM) to the rescue It would be possible without them, but were not that masochistic
  142. 142. Data we store in SQL• Order• Order/Shipment• Order/Transaction• Inventory
  143. 143. Data we store in MongoDB
  144. 144. Data we store in MongoDB• User • Event• Product • TaxRate• Product/Sellable • ... and then I got tired of typing them in• Address • Just imagine this list has• Cart 40 more classes• CreditCard • ...
  145. 145. We have themost boring SQL schema ever
  146. 146. CREATE TABLE `product_inventory` ( `product_id` char(32) NOT NULL, `inventory` int(11) NOT NULL DEFAULT 0, PRIMARY KEY (`product_id`));CREATE TABLE `sellable_inventory` ( `sellable_id` char(32) NOT NULL, `inventory` int(11) NOT NULL DEFAULT 0, PRIMARY KEY (`sellable_id`));CREATE TABLE `orders` ( `id` int(11) NOT NULL AUTO_INCREMENT, `userId` char(32) NOT NULL, `shippingName` varchar(255) DEFAULT NULL, `shippingAddress1` varchar(255) DEFAULT NULL, `shippingAddress2` varchar(255) DEFAULT NULL, `shippingCity` varchar(255) DEFAULT NULL, `shippingState` varchar(2) DEFAULT NULL, `shippingZip` varchar(255) DEFAULT NULL, `billingName` varchar(255) DEFAULT NULL, `billingAddress1` varchar(255) DEFAULT NULL, `billingAddress2` varchar(255) DEFAULT NULL, `billingCity` varchar(255) DEFAULT NULL,
  147. 147. Wait. How does inventory live in SQL?Isn’t that a property in one of your Mongo collections?
  148. 148. I thought you’d never ask!
  149. 149. CREATE TABLE `product_inventory` ( `product_id` char(32) NOT NULL, `inventory` int(11) NOT NULL DEFAULT 0, PRIMARY KEY (`product_id`));CREATE TABLE `sellable_inventory` ( `sellable_id` char(32) NOT NULL, `inventory` int(11) NOT NULL DEFAULT 0, PRIMARY KEY (`sellable_id`));CREATE TABLE `orders` ( `id` int(11) NOT NULL AUTO_INCREMENT, `userId` char(32) NOT NULL, `shippingName` varchar(255) DEFAULT NULL, `shippingAddress1` varchar(255) DEFAULT NULL, `shippingAddress2` varchar(255) DEFAULT NULL, `shippingCity` varchar(255) DEFAULT NULL, `shippingState` varchar(2) DEFAULT NULL, `shippingZip` varchar(255) DEFAULT NULL, `billingName` varchar(255) DEFAULT NULL, `billingAddress1` varchar(255) DEFAULT NULL, `billingAddress2` varchar(255) DEFAULT NULL, `billingCity` varchar(255) DEFAULT NULL,
  150. 150. Inventory is transient• Product::$inventory is effectively a transient property• Note how I said “effectively”? ... we cheat and persist our transient property to MongoDB as well• We can do this because we never really trust the value stored in Mongo
  151. 151. Accuracy is only important when there’s contention
  152. 152. Accuracy is only important when there’s contention• For display, sorting and alerts, we can use the value stashed in MongoDB • It’s faster • It’s accurate enough
  153. 153. Accuracy is only important when there’s contention• For display, sorting and alerts, we can use the value stashed in MongoDB • It’s faster • It’s accurate enough• For financial transactions, we want the security and comfort of our RDBMS.
  154. 154. We keep inventory in sync with listeners
  155. 155. We keep inventory in sync with listeners• Every time a new product is created, its inventory is inserted in SQL
  156. 156. We keep inventory in sync with listeners• Every time a new product is created, its inventory is inserted in SQL• Every time an order is placed, inventory is verified and decremented
  157. 157. We keep inventory in sync with listeners• Every time a new product is created, its inventory is inserted in SQL• Every time an order is placed, inventory is verified and decremented• Whenever the SQL inventory changes, it is saved to MongoDB as well
  158. 158. Be careful what you lock
  159. 159. Be careful what you lock1. Acquire inventory row lock and begin transaction2. Check current product inventory3. Decrement product inventory4. Write the Order to SQL5. Update affected MongoDB documents6. Commit the transaction7. Release product inventory lock
  160. 160. Making MongoDBand RDBMS relations play nice
  161. 161. Products aredocuments stored in MongoDB
  162. 162. /** @mongodb:Document(collection="products") */class Product{ /** @mongodb:Id */ private $id; /** @mongodb:String */ private $title; public function getId() { return $this->id; } public function getTitle() { return $this->title; } public function setTitle($title) { $this->title = $title; }}
  163. 163. Orders are entitiesstored in an RDBMS
  164. 164. /** * @orm:Entity * @orm:Table(name="orders") * @orm:HasLifecycleCallbacks */class Order{ /** * @orm:Id @orm:Column(type="integer") * @orm:GeneratedValue(strategy="AUTO") */ private $id; /** * @orm:Column(type="string") */ private $productId; /** * @var DocumentsProduct */ private $product; // ...}
  165. 165. So how does an RDBMS have areference to something outside the database?
  166. 166. Setting the Productclass Order { // ... public function setProduct(Product $product) { $this->productId = $product->getId(); $this->product = $product; }}
  167. 167. • $productId is mapped and persisted• $product which stores the Product instance is not a persistent entity property
  168. 168. Retrieving ourproduct later
  169. 169. OrderPostLoadListeneruse DoctrineORMEventLifecycleEventArgs;class OrderPostLoadListener{ public function postLoad(LifecycleEventArgs $eventArgs) { // get the order entity $order = $eventArgs->getEntity(); // get odm reference to order.product_id $productId = $order->getProductId(); $product = $this->dm->getReference(MyBundle:DocumentProduct, $productId); // set the product on the order $em = $eventArgs->getEntityManager(); $productReflProp = $em->getClassMetadata(MyBundle:EntityOrder) ->reflClass->getProperty(product); $productReflProp->setAccessible(true); $productReflProp->setValue($order, $product); }}
  170. 170. All Together Now// Create a new product and order$product = new Product();$product->setTitle(Test Product);$dm->persist($product);$dm->flush();$order = new Order();$order->setProduct($product);$em->persist($order);$em->flush();// Find the order later$order = $em->find(Order, $order->getId());// Instance of an uninitialized product proxy$product = $order->getProduct();// Initializes proxy and queries the monogodb databaseecho "Order Title: " . $product->getTitle();print_r($order);
  171. 171. Read more about this techniqueJon Wage, one of OpenSky’s engineers, firstwrote about this technique on his personalblog: http://jwage.comYou can read the full article here:http://jwage.com/2010/08/25/blending-the-doctrine-orm-and-mongodb-odm/
  172. 172. Questions? http://spf13.com @spf13 http://justinhileman.com @bobthecow http://opensky.comPS: We’re hiring!! Contact us at jobs@opensky.com

×