Augmenting RDBMS  with MongoDB    for e-commerce
My name isSteve Francia    @spf13
• 15+ years building e-commerce• Long time open source contributor• Entrepreneur• Hacker, father, husband, skate punk
My name isJustin Hileman    @bobthecow
• 10+ years making the Internet  awesomer• Open Source contributor• Vespa rider, swing dancer,  coder, standardista
We work for OpenSky    http://shopopensky.com
OpenSky is  a new way to shopOpenSky connects you with innovators,trendsetters and tastemakers.You choosethe ones you like...
OpenSky Loves             Open Source•   PHP 5.3•   Apache2•   Symfony2•   Doctrine2•   jQuery•   Mule•   HornetQ•   Mongo...
We contribute to manyopen source projects and   pioneer innovative  solutions using them
OpenSky was the firste-commerce site built    on MongoDB... also the first e-commerce site built on Symfony2
Why NoSQL for  e-commerce?Using the right solution for each situation
Data dilemma of e-commerce     Pick One
Data dilemma of      e-commerce                 Pick One• Stick to one vertical (Sane schema)
Data dilemma of      e-commerce                 Pick One• Stick to one vertical (Sane schema)• Flexibility (Insane schema)
Sane schema
Sane schema• Works ... for a while
Sane schema• Works ... for a while• Fine for a few types of products
Sane schema• Works ... for a while• Fine for a few types of products• Not possible when more product types  introduced
Let’s Use an Example
Let’s Use an Example   How about we start with books
Book Product SchemaProduct {id:sku:                                    General Productproduct dimensions:shipping weight: ...
Seems simple enough
Seems simple enoughWhat happens when we add another vertical...            say music albums
Album Product SchemaProduct {id:sku:                               General Productproduct dimensions:                attri...
Okay, it’s getting hairy butis still manageable, right?
Okay, it’s getting hairy butis still manageable, right?    Now the business want to sell jeans
Jeans Product SchemaProduct {id:                           General Productsku:product dimensions:                         ...
Now we’re screwed
We need a flexibleschema in RDBMS
We need a flexibleschema in RDBMS    We got this ... right?
Many approachesdealing with unknownunknowns in RDBMS
Many approachesdealing with unknownunknowns in RDBMS      None work well
EAV             as popularized by Magento“For purposes of flexibility, the Magneto database heavily utilizesan Entity-Attri...
EAV•   Crazy SQL queries•   Hundreds of joins in a query...    or•   Hundreds of queries joined in    the application•   N...
Did I say crazy SQL(this is a single query)
Did I say crazy SQL(this is a single query)You may have trouble reading this in the back
Selecting a single product
Single Table Inheritance            (insanely wide tables)•   No data integrity enforcement•   Only can use FK for common ...
Generic Columns•   No data integrity enforcement•   No data type enforcement•   Only can use FK for common    elements•   ...
Serialized in Blob•   Not searchable•   No integrity•   All the disadvantages of a document    store, but none of the adva...
Concrete Table Inheritance    (a table for each product attribute set)•   Allows for data integrity•   Querying across att...
Class table inheritance                  (single product table,             each attribute set in own table)•   Likely bes...
MongoDB to the   Rescue
MongoDB to the        Rescue• Flexible (and sane) Schema
MongoDB to the        Rescue• Flexible (and sane) Schema• Easily searchable
MongoDB to the        Rescue• Flexible (and sane) Schema• Easily searchable• Easily accessible
MongoDB to the        Rescue• Flexible (and sane) Schema• Easily searchable• Easily accessible• Fast
Flexible schema
{                                 {    sku: "00e8da9c",                  sku: "00e8da9d",    type: "Audio Album",         ...
pct_savings: 20                      pct_savings: 8.5},                                   },details: {                    ...
Queries
db.products.find( { name: "The Matrix" } );
db.products.find( { name: "The Matrix" } ); {     "_id": ObjectId("4d8ad78b46b731a22943d3d3"),     "sku": "00e8da9d",     ...
db.products.find( { details.actor: "Groucho Marx" } );
db.products.find( { details.actor: "Groucho Marx" } ); }, "pricing": {     "list": 1000,     "retail": 800,     "savings":...
db.products.find( {     details.genre: "Jazz", details.format: "CD"} );
db.products.find( {     details.genre: "Jazz", details.format: "CD"} );     "list": 1200,     "retail": 1100,     "savings...
db.products.find( { details.actor:     { $all: [James Stewart, Donna Reed] }} );
db.products.find( { details.actor:     { $all: [James Stewart, Donna Reed] }} ); }, "details": {     "title": "Its a Wonde...
Wanna Play?•   grab products.js from    http://github.com/spf13/mongoProducts•   mongo --shell products.js•   > use mongoP...
Embedded documents are great for orders• Ordered items need to be fixed at the time  of purchase• Embed them right in the o...
Why not NoSQL?Using the right solution for each situation
Data (like people) arereally sensitive when it   comes to money
Stricter datarequirements for $$
Stricter data   requirements for $$• For financial systems any data inconsistency  is unacceptable
Stricter data   requirements for $$• For financial systems any data inconsistency  is unacceptable• Perhaps you’ve heard of...
What about ACID?
What about ACID?Q: Is MongoDB ACID?
What about ACID?Q: Is MongoDB ACID?A: Kinda
Atomicity
Atomicity• MongoDB does atomic writes
Atomicity• MongoDB does atomic writes  ... for single document changesets
Atomicity• MongoDB does atomic writes    ... for single document changesets•   $set, $unset, $inc, $push,    $pushAll, $pu...
Consistency
Consistency• MongoDB can enforce unique keys
Consistency• MongoDB can enforce unique keys  ... but only on keys shared by every  document in the collection
Consistency• MongoDB can enforce unique keys  ... but only on keys shared by every  document in the collection• MongoDB ca...
Isolation
Isolation•   // Pseudo-isolated updates    db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );
Isolation•   // Pseudo-isolated updates    db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );•   // Isolate...
Isolation•   // Pseudo-isolated updates    db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );•   // Isolate...
Isolation•   // Pseudo-isolated updates    db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );•   // Isolate...
Isolation•   // Pseudo-isolated updates    db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );•   // Isolate...
Durability
Durability• Mongo has this one covered
What doesMongoDB Support?
•   Atomic single document writes    •   If you need atomic writes across multi-document        transactions dont use Mong...
•   Atomic single document writes    •   If you need atomic writes across multi-document        transactions dont use Mong...
•   Atomic single document writes    •   If you need atomic writes across multi-document        transactions dont use Mong...
•   Atomic single document writes    •   If you need atomic writes across multi-document        transactions dont use Mong...
There are ways toguarantee ACID properties in inconsistent databases
There are ways toguarantee ACID properties in inconsistent databases (or, as we call them, consistency impaired databases)
Optimistic concurrency
Optimistic concurrency• Read the current state of a product
Optimistic concurrency• Read the current state of a product• Make your changes with the assertion that  your product has t...
Optimistic concurrency    in MongoDB
Optimistic concurrency    in MongoDBWe’ll use an update-if-current strategy.
Optimistic concurrency    in MongoDBWe’ll use an update-if-current strategy.This example is straight from the documentation:
Optimistic concurrency    in MongoDB    We’ll use an update-if-current strategy.    This example is straight from the docu...
Optimistic concurrency    in MongoDB    We’ll use an update-if-current strategy.    This example is straight from the docu...
Optimistic concurrency• Read the current state of a product.• Make your changes with the assertion that  your product has ...
Optimistic concurrency• Read the current state of a product.• Make your changes with the assertion that    your product ha...
Optimistic concurrency• Read the current state of a product.• Make your changes with the assertion that    your product ha...
Optimistic concurrency control assumes anenvironment with low   data contention
OCC works great forcompanies like Amazon• Amazon has a long-tail catalog• A long tail catalog lends itself well to  optimi...
OCC fails miserably for
OCC fails miserably for• eBay
OCC fails miserably for• eBay• Gilt
OCC fails miserably for• eBay• Gilt• Groupon
OCC fails miserably for• eBay• Gilt• Groupon• OpenSky
OCC fails miserably for• eBay• Gilt• Groupon• OpenSky• Living Social
OCC fails miserably for• eBay• Gilt• Groupon• OpenSky• Living Social• InsertFlashSaleSiteOfTheMinute
Flash sales and auctionsare defined by high data       contention
Flash sales and auctionsare defined by high data       contention• The model doesnt work otherwise
Flash sales and auctionsare defined by high data       contention• The model doesnt work otherwise• They cant afford to be ...
Can we use pessimistic concurrency with a distributed NoSQL       database?
Yep.
BlendingNoSQL & RDBMSUsing the right solution for each situation
Our goal is to put as much  in Mongo as possible• What makes more sense in RDBMS? • Inventory • Orders
Inventory requires• Row level locking (or table level locking)
Orders require• Row level locking (or table level locking)• Atomic writes (inventory decremented)• Transactions (3rd party...
Inventory & checkout     transactions
Commerce is ACID   In Real Life
1. I go to Barneys and see a pair of shoes I just have to   buy.
1. I go to Barneys and see a pair of shoes I just have to   buy.2. I call “dibs” (by grabbing them off the shelf).
1. I go to Barneys and see a pair of shoes I just have to   buy.2. I call “dibs” (by grabbing them off the shelf).3. I tak...
1. I go to Barneys and see a pair of shoes I just have to   buy.2. I call “dibs” (by grabbing them off the shelf).3. I tak...
1. I go to Barneys and see a pair of shoes I just have to   buy.2. I call “dibs” (by grabbing them off the shelf).3. I tak...
1. I go to Barneys and see a pair of shoes I just have to   buy.2. I call “dibs” (by grabbing them off the shelf).3. I tak...
1. I go to Barneys and see a pair of shoes I just have to   buy.2. I call “dibs” (by grabbing them off the shelf).3. I tak...
1. I go to Barneys and see a pair of shoes I just have to   buy.2. I call “dibs” (by grabbing them off the shelf).3. I tak...
We follow the samemodel for e-commerce
1. Select a product.
1. Select a product.2. Lock the row or table and confirm inventory.
1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product:
1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product:  •   Decrement product inventory
1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product:  •   Decrement product inventor...
1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product:  •   Decrement product inventor...
1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product:  •   Decrement product inventor...
Doctrine (ORM/ODM)    to the rescue
Doctrine (ORM/ODM)    to the rescue   It would be possible without them,      but were not that masochistic
Data we store in SQL• Order• Order/Shipment• Order/Transaction• Inventory
Data we store in  MongoDB
Data we store in             MongoDB•   User               •   Event•   Product            •   TaxRate•   Product/Sellable...
We have themost boring SQL  schema ever
CREATE TABLE `product_inventory` (   `product_id` char(32) NOT NULL,   `inventory` int(11) NOT NULL DEFAULT 0,   PRIMARY K...
Wait. How does inventory live in SQL?Isn’t that a property in one of your Mongo collections?
I thought you’d   never ask!
CREATE TABLE `product_inventory` (   `product_id` char(32) NOT NULL,   `inventory` int(11) NOT NULL DEFAULT 0,   PRIMARY K...
Inventory is transient•   Product::$inventory is effectively a    transient property• Note how I said “effectively”? ... w...
Accuracy is only important when there’s contention
Accuracy is only important when there’s contention• For display, sorting and alerts, we can use  the value stashed in Mong...
Accuracy is only important when there’s contention• For display, sorting and alerts, we can use  the value stashed in Mong...
We keep inventory in sync with listeners
We keep inventory in sync with listeners• Every time a new product is created, its  inventory is inserted in SQL
We keep inventory in sync with listeners• Every time a new product is created, its  inventory is inserted in SQL• Every ti...
We keep inventory in sync with listeners• Every time a new product is created, its  inventory is inserted in SQL• Every ti...
Be careful what you lock
Be careful what you lock1. Acquire inventory row lock and begin transaction2. Check current product inventory3. Decrement ...
Making MongoDBand RDBMS relations      play nice
Products aredocuments stored  in MongoDB
/** @mongodb:Document(collection="products") */class Product{    /** @mongodb:Id */    private $id;    /** @mongodb:String...
Orders are entitiesstored in an RDBMS
/** * @orm:Entity * @orm:Table(name="orders") * @orm:HasLifecycleCallbacks */class Order{    /**     * @orm:Id @orm:Column...
So how does an     RDBMS have areference to something outside the database?
Setting the Productclass Order {    // ...    public function setProduct(Product $product)    {        $this->productId = ...
•   $productId is mapped and persisted•   $product which stores the Product    instance is not a persistent entity property
Retrieving ourproduct later
OrderPostLoadListeneruse DoctrineORMEventLifecycleEventArgs;class OrderPostLoadListener{    public function postLoad(Lifec...
All Together Now// Create a new product and order$product = new Product();$product->setTitle(Test Product);$dm->persist($p...
Read more about       this techniqueJon Wage, one of OpenSky’s engineers, firstwrote about this technique on his personalbl...
Questions?   http://spf13.com      @spf13   http://justinhileman.com      @bobthecow   http://shopopensky.comPS: We’re hir...
Augmenting RDBMS with MongoDB for ecommerce
Augmenting RDBMS with MongoDB for ecommerce
Augmenting RDBMS with MongoDB for ecommerce
Augmenting RDBMS with MongoDB for ecommerce
Upcoming SlideShare
Loading in...5
×

Augmenting RDBMS with MongoDB for ecommerce

12,228

Published on

Steve Francia, VP of Engineering at OpenSky a NYC based social commerce company, on how OpenSky augments using RDBMS with MongoDB to develop the next ecommerce platform.

OpenSky utilizes both traditional SQL solutions and combines them with NoSQL to overcome the limitations of each, increase development speed and scale quickly.

Published in: Technology
2 Comments
44 Likes
Statistics
Notes
No Downloads
Views
Total Views
12,228
On Slideshare
0
From Embeds
0
Number of Embeds
14
Actions
Shares
0
Downloads
0
Comments
2
Likes
44
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • actually, just the first 1/3 of it. \n
  • \n
  • Ironically this is how magento solves the performance problems associated with EAV, by caching the data into insanely wide tables.\n
  • \n
  • \n
  • \n
  • Can’t create a FK as each set references a different table. “Key” really made of attribute table name id and attribute table name\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Whenever you use a inter system coordination you need to implement your own atomic checks in the application... But SOAP does have transactions.. so not quite accurate. \n\nkyle idea... but we are fairly atomic with authorize.net\n\n
  • Whenever you use a inter system coordination you need to implement your own atomic checks in the application... But SOAP does have transactions.. so not quite accurate. \n\nkyle idea... but we are fairly atomic with authorize.net\n\n
  • atomicity, consistency, isolation, durability.\n\n
  • atomicity, consistency, isolation, durability.\n\n
  • Mongo has a grip of atomic operations: set, unset, etc.\n
  • Mongo has a grip of atomic operations: set, unset, etc.\n
  • Mongo has a grip of atomic operations: set, unset, etc.\n
  • \n
  • \n
  • \n
  • update( { where }, { values }, upsert?, multiple? )\n\n\n
  • update( { where }, { values }, upsert?, multiple? )\n\n\n
  • update( { where }, { values }, upsert?, multiple? )\n\n\n
  • update( { where }, { values }, upsert?, multiple? )\n\n\n
  • update( { where }, { values }, upsert?, multiple? )\n\n\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • lemme show you an example\n
  • lemme show you an example\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Imagine what would happen if everyone tried to access the same record at the same time. Just think of all those spinning while loops :)\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • And I’ll show you how OpenSky does it.\n
  • \n
  • Since we really like MongoDB, we want to keep as much of our data in Mongo as possible.\n
  • \n
  • \n
  • \n
  • Mind if I tell you a story?\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • But this sounds like it could get COMPLICATED...\n
  • But this sounds like it could get COMPLICATED...\n
  • But this sounds like it could get COMPLICATED...\n
  • But this sounds like it could get COMPLICATED...\n
  • But this sounds like it could get COMPLICATED...\n
  • But this sounds like it could get COMPLICATED...\n
  • But this sounds like it could get COMPLICATED...\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Given that split, we just happen to have the most boring SQL schema ever\n
  • This is pretty much it.\n\nIt goes on for a few more lines, with a few other properties flattened onto the order table. \n
  • \n
  • \n
  • Back to the schema for a second.\n\n- Product ID here is a fake foreign key.\n- Inventory is a real integer.\n\nThat’s all there is to this table.\n
  • \n
  • \n
  • \n
  • And here’s why we like Doctrine so much.\n
  • And here’s why we like Doctrine so much.\n
  • And here’s why we like Doctrine so much.\n
  • This will look a bit like when I bought those shoes.\n
  • This will look a bit like when I bought those shoes.\n
  • This will look a bit like when I bought those shoes.\n
  • This will look a bit like when I bought those shoes.\n
  • This will look a bit like when I bought those shoes.\n
  • This will look a bit like when I bought those shoes.\n
  • This will look a bit like when I bought those shoes.\n
  • \n
  • \n
  • The interesting parts here are the annotations.\n\nIf you don’t speak PHP annotation, this stores a document with two properties—ID and title—in the `products` collection of a Mongo database.\n
  • \n
  • \n
  • Did you notice the property named `product`? That’s not just a reference to another document, that’s a reference to an entirely different database paradigm.\n\nCheck out the setter:\n
  • This is key: we set both the product id and a reference to the product itself.\n
  • When this document is saved in Mongo, the productId will end up in the database, but the product reference will disappear.\n
  • \n
  • This is one of those listeners I was telling you about. At a high level:\n\n1. Every time an Order is loaded from the database, this listener is called.\n2. The listener gets the Order’s product id, and creates a Doctrine proxy object.\n3. It uses magick (e.g. reflection) to set the product property of the order to this new proxy.\n
  • Here’s our inter-db relationship in action.\n\nNote that the product is lazily loaded from MongoDB. Because $product is a proxy, we don’t actually query Mongo until we try to access a property of $product (in this case the title).\n
  • \n
  • \n
  • \n
  • Augmenting RDBMS with MongoDB for ecommerce

    1. 1. Augmenting RDBMS with MongoDB for e-commerce
    2. 2. My name isSteve Francia @spf13
    3. 3. • 15+ years building e-commerce• Long time open source contributor• Entrepreneur• Hacker, father, husband, skate punk
    4. 4. My name isJustin Hileman @bobthecow
    5. 5. • 10+ years making the Internet awesomer• Open Source contributor• Vespa rider, swing dancer, coder, standardista
    6. 6. We work for OpenSky http://shopopensky.com
    7. 7. OpenSky is a new way to shopOpenSky connects you with innovators,trendsetters and tastemakers.You choosethe ones you like and each week they inviteyou to their private online sales.
    8. 8. OpenSky Loves Open Source• PHP 5.3• Apache2• Symfony2• Doctrine2• jQuery• Mule• HornetQ• MongoDB• nginx• varnish
    9. 9. We contribute to manyopen source projects and pioneer innovative solutions using them
    10. 10. OpenSky was the firste-commerce site built on MongoDB... also the first e-commerce site built on Symfony2
    11. 11. Why NoSQL for e-commerce?Using the right solution for each situation
    12. 12. Data dilemma of e-commerce Pick One
    13. 13. Data dilemma of e-commerce Pick One• Stick to one vertical (Sane schema)
    14. 14. Data dilemma of e-commerce Pick One• Stick to one vertical (Sane schema)• Flexibility (Insane schema)
    15. 15. Sane schema
    16. 16. Sane schema• Works ... for a while
    17. 17. Sane schema• Works ... for a while• Fine for a few types of products
    18. 18. Sane schema• Works ... for a while• Fine for a few types of products• Not possible when more product types introduced
    19. 19. Let’s Use an Example
    20. 20. Let’s Use an Example How about we start with books
    21. 21. Book Product SchemaProduct {id:sku: General Productproduct dimensions:shipping weight: attributesMSRP:price:description:...author: Orson Scott Cardtitle: Enders Gamebinding: Hardcoverpublication date: July 15, 1994 Book Specificpublisher name: Tor Science Fiction attributesnumber of pages: 352ISBN: 0812550706language: English...
    22. 22. Seems simple enough
    23. 23. Seems simple enoughWhat happens when we add another vertical... say music albums
    24. 24. Album Product SchemaProduct {id:sku: General Productproduct dimensions: attributes stay theshipping weight:MSRP: sameprice:description:...artist: MxPxtitle: Panic Album Specificrelease date: June 7, 2005 attributes arelabel: Side One Dummytrack listing: [ The Darkest ... differentlanguage: Englishformat: CD...
    25. 25. Okay, it’s getting hairy butis still manageable, right?
    26. 26. Okay, it’s getting hairy butis still manageable, right? Now the business want to sell jeans
    27. 27. Jeans Product SchemaProduct {id: General Productsku:product dimensions: attributes stay theshipping weight: sameMSRP:price:description:...brand: Luckygender: Mens Jeans specificmake: Vintage attributes are totallystyle: Straight Cut different ... and notlength: 34width: 34 consistent acrosscolor: Hipster brands & makematerial: Cotten Blend...
    28. 28. Now we’re screwed
    29. 29. We need a flexibleschema in RDBMS
    30. 30. We need a flexibleschema in RDBMS We got this ... right?
    31. 31. Many approachesdealing with unknownunknowns in RDBMS
    32. 32. Many approachesdealing with unknownunknowns in RDBMS None work well
    33. 33. EAV as popularized by Magento“For purposes of flexibility, the Magneto database heavily utilizesan Entity-Attribute-Value (EAV) data model.As is often the case, the cost of flexibility is complexity -Magento is no exception.The process of manipulating data in Magento is often more“involved” than that typically experienced using traditionalrelational tables.” - Varien
    34. 34. EAV• Crazy SQL queries• Hundreds of joins in a query... or• Hundreds of queries joined in the application• No database enforced integrity
    35. 35. Did I say crazy SQL(this is a single query)
    36. 36. Did I say crazy SQL(this is a single query)You may have trouble reading this in the back
    37. 37. Selecting a single product
    38. 38. Single Table Inheritance (insanely wide tables)• No data integrity enforcement• Only can use FK for common elements• Very wasteful (but disk is cheap!)• Can’t effectively index
    39. 39. Generic Columns• No data integrity enforcement• No data type enforcement• Only can use FK for common elements• Wasteful (but disk is cheap!)• Can’t index
    40. 40. Serialized in Blob• Not searchable• No integrity• All the disadvantages of a document store, but none of the advantages• Never should be used• One exception is Oracle XML which operates similar to a document store
    41. 41. Concrete Table Inheritance (a table for each product attribute set)• Allows for data integrity• Querying across attribute sets quite hard to do (lots of joins, OR statements and full table scanning)• New table needs to be created for each new attribute set
    42. 42. Class table inheritance (single product table, each attribute set in own table)• Likely best solution within the constraint of SQL• Supports data type enforcement• No data integrity enforcement• Easy querying across categories (for browse pages) since common data in single table• Every set needs a new table• Requires a ton of forsight, as changes are very complicated
    43. 43. MongoDB to the Rescue
    44. 44. MongoDB to the Rescue• Flexible (and sane) Schema
    45. 45. MongoDB to the Rescue• Flexible (and sane) Schema• Easily searchable
    46. 46. MongoDB to the Rescue• Flexible (and sane) Schema• Easily searchable• Easily accessible
    47. 47. MongoDB to the Rescue• Flexible (and sane) Schema• Easily searchable• Easily accessible• Fast
    48. 48. Flexible schema
    49. 49. { { sku: "00e8da9c", sku: "00e8da9d", type: "Audio Album", type: "Film", title: "Hoss", title: "The Matrix", description: "by Lagwagon", description: "Set in the 22nd century, Th asin: "B0000007QG", asin: "B000P0J0AQ", shipping: { shipping: { weight: 6, weight: 6, dimensions: { dimensions: { width: 10, width: 10, height: 10, height: 10, depth: 1 depth: 1 }, }, }, }, pricing: { pricing: { list: 1000, list: 1200, retail: 800, retail: 1100, savings: 200, savings: 100, pct_savings: 20 pct_savings: 8.5 }, }, details: { details: { title: "Hoss", title: "The Matrix",
    50. 50. pct_savings: 20 pct_savings: 8.5}, },details: { details: { title: "Hoss", title: "The Matrix", artist: "Lagwagon", director: [ "Andy Wachowski", "Larry Wa genre: [ "Punk", "Hardcore", "Indie Rock" ], [ "Andy Wachowski", "Larry Wach writer: label: "Fat Wreck Chords", actor: [ "Keanu Reeves" , "Lawrence Fis number_of_discs: 1, genre: [ "Science Fiction", "Action" ], issue_date: "November 21, 1995", number_of_discs: 1, format: "CD", issue_date: "May 15 2007", alternate_formats: [ Vinyl, MP3 ],original_release_date: "1999", tracks: [ disc_format: "DVD", "Kids Dont Like To Share", rating: "R", "Violins", alternate_formats: [ VHS, Bluray ], "Name Dropping", run_time: "136", "Bombs Away", studio: "Warner Bros", "Move The Car", language: "English", "Sleep", format: [ "AC-3", "Closed-captioned", " "Sick", aspect_ratio: "1.66:1" "Rifle", }, "Weak", } "Black Eye", "Bro Dependent", "Razor Burn", "Shaving Your Head", "Ride The Snake", ],
    51. 51. Queries
    52. 52. db.products.find( { name: "The Matrix" } );
    53. 53. db.products.find( { name: "The Matrix" } ); { "_id": ObjectId("4d8ad78b46b731a22943d3d3"), "sku": "00e8da9d", "type": "Film", "name": "The Matrix", "description": "Set in the 22nd century, The Matrix...", "asin": "B000P0J0AQ", "shipping": { "weight": 6, "dimensions": { "width": 10, "height": 10, "depth": 1 } }, "pricing": {
    54. 54. db.products.find( { details.actor: "Groucho Marx" } );
    55. 55. db.products.find( { details.actor: "Groucho Marx" } ); }, "pricing": { "list": 1000, "retail": 800, "savings": 200, "pct_savings": 20 }, "details": { "title": "A Night at the Opera", "director": "Sam Wood", "actor": ["Groucho Marx", "Chico Marx", "Harpo Marx"], "genre": "Comedy", "number_of_discs": 1, "issue_date": "May 4 2004", "original_release_date": "1935", "disc_format": "DVD",
    56. 56. db.products.find( { details.genre: "Jazz", details.format: "CD"} );
    57. 57. db.products.find( { details.genre: "Jazz", details.format: "CD"} ); "list": 1200, "retail": 1100, "savings": 100, "pct_savings": 8 }, "details": { "title": "A Love Supreme [Original Recording Reissued]", "artist": "John Coltrane", "genre": ["Jazz", "General"], "format": "CD", "label": "Impulse Records", "number_of_discs": 1, "issue_date": "December 9, 1964", "alternate_formats": ["Vinyl", "MP3"], "tracks": [ "A Love Supreme Part I: Acknowledgement",
    58. 58. db.products.find( { details.actor: { $all: [James Stewart, Donna Reed] }} );
    59. 59. db.products.find( { details.actor: { $all: [James Stewart, Donna Reed] }} ); }, "details": { "title": "Its a Wonderful Life", "director": "Frank Capra", "actor": ["James Stewart", "Donna Reed", "Lionel Barrymore"], "writer": [ "Frank Capra", "Albert Hackett", "Frances Goodrich", "Jo Swerling", "Michael Wilson" ], "genre": "Drama", "number_of_discs": 1, "issue_date": "Oct 31 2006", "original_release_date": "1947",
    60. 60. Wanna Play?• grab products.js from http://github.com/spf13/mongoProducts• mongo --shell products.js• > use mongoProducts
    61. 61. Embedded documents are great for orders• Ordered items need to be fixed at the time of purchase• Embed them right in the orderdb.order.find( { items.sku: 00e8da9f } );db.order.find( { items.details.actor: James Stewart} ).count();
    62. 62. Why not NoSQL?Using the right solution for each situation
    63. 63. Data (like people) arereally sensitive when it comes to money
    64. 64. Stricter datarequirements for $$
    65. 65. Stricter data requirements for $$• For financial systems any data inconsistency is unacceptable
    66. 66. Stricter data requirements for $$• For financial systems any data inconsistency is unacceptable• Perhaps you’ve heard of ACID?
    67. 67. What about ACID?
    68. 68. What about ACID?Q: Is MongoDB ACID?
    69. 69. What about ACID?Q: Is MongoDB ACID?A: Kinda
    70. 70. Atomicity
    71. 71. Atomicity• MongoDB does atomic writes
    72. 72. Atomicity• MongoDB does atomic writes ... for single document changesets
    73. 73. Atomicity• MongoDB does atomic writes ... for single document changesets• $set, $unset, $inc, $push, $pushAll, $pull, $pullAll, $bit
    74. 74. Consistency
    75. 75. Consistency• MongoDB can enforce unique keys
    76. 76. Consistency• MongoDB can enforce unique keys ... but only on keys shared by every document in the collection
    77. 77. Consistency• MongoDB can enforce unique keys ... but only on keys shared by every document in the collection• MongoDB cant enforce referential integrity
    78. 78. Isolation
    79. 79. Isolation• // Pseudo-isolated updates db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );
    80. 80. Isolation• // Pseudo-isolated updates db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );• // Isolated updates db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false , true );
    81. 81. Isolation• // Pseudo-isolated updates db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );• // Isolated updates db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false , true );• But there are caveats...
    82. 82. Isolation• // Pseudo-isolated updates db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );• // Isolated updates db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false , true );• But there are caveats... • Despite the $atomic keyword, this is not an atomic update, since atomicity implies “all or nothing”
    83. 83. Isolation• // Pseudo-isolated updates db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );• // Isolated updates db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false , true );• But there are caveats... • Despite the $atomic keyword, this is not an atomic update, since atomicity implies “all or nothing” • An isolated update can only act on a single collection. Multi- collection updates are not transactional, thus not isolatable.
    84. 84. Durability
    85. 85. Durability• Mongo has this one covered
    86. 86. What doesMongoDB Support?
    87. 87. • Atomic single document writes • If you need atomic writes across multi-document transactions dont use Mongo • Many e-commerce transactions could be accomplished within a single document write
    88. 88. • Atomic single document writes • If you need atomic writes across multi-document transactions dont use Mongo • Many e-commerce transactions could be accomplished within a single document write• Unique indexes • This only works on keys used by the entire collection
    89. 89. • Atomic single document writes • If you need atomic writes across multi-document transactions dont use Mongo • Many e-commerce transactions could be accomplished within a single document write• Unique indexes • This only works on keys used by the entire collection• Isolated (not atomic) single collection updates. • Mongo does not support locking • There are ways to work around this
    90. 90. • Atomic single document writes • If you need atomic writes across multi-document transactions dont use Mongo • Many e-commerce transactions could be accomplished within a single document write• Unique indexes • This only works on keys used by the entire collection• Isolated (not atomic) single collection updates. • Mongo does not support locking • There are ways to work around this• It’s durable
    91. 91. There are ways toguarantee ACID properties in inconsistent databases
    92. 92. There are ways toguarantee ACID properties in inconsistent databases (or, as we call them, consistency impaired databases)
    93. 93. Optimistic concurrency
    94. 94. Optimistic concurrency• Read the current state of a product
    95. 95. Optimistic concurrency• Read the current state of a product• Make your changes with the assertion that your product has the same state as it did when you last read it
    96. 96. Optimistic concurrency in MongoDB
    97. 97. Optimistic concurrency in MongoDBWe’ll use an update-if-current strategy.
    98. 98. Optimistic concurrency in MongoDBWe’ll use an update-if-current strategy.This example is straight from the documentation:
    99. 99. Optimistic concurrency in MongoDB We’ll use an update-if-current strategy. This example is straight from the documentation:> t = db.inventory> p = t.findOne({sku:abc})> t.update({_id:p._id, qty:p.qty}, {$inc: {qty: -1}});> db.$cmd.findOne({getlasterror:1});{"err" : , "updatedExisting" : true , "n" : 1 , "ok" : 1}// it worked
    100. 100. Optimistic concurrency in MongoDB We’ll use an update-if-current strategy. This example is straight from the documentation:> t = db.inventory> p = t.findOne({sku:abc})> t.update({_id:p._id, qty:p.qty}, {$inc: {qty: -1}});> db.$cmd.findOne({getlasterror:1});{"err" : , "updatedExisting" : true , "n" : 1 , "ok" : 1}// it worked ... If that didnt work, try again until it does.
    101. 101. Optimistic concurrency• Read the current state of a product.• Make your changes with the assertion that your product has the same state as it did when you last read it.
    102. 102. Optimistic concurrency• Read the current state of a product.• Make your changes with the assertion that your product has the same state as it did when you last read it.• Its possible to use OCC to bootstrap pessimistic concurrency and fake row level locking
    103. 103. Optimistic concurrency• Read the current state of a product.• Make your changes with the assertion that your product has the same state as it did when you last read it.• Its possible to use OCC to bootstrap pessimistic concurrency and fake row level locking ... ask me about this some time
    104. 104. Optimistic concurrency control assumes anenvironment with low data contention
    105. 105. OCC works great forcompanies like Amazon• Amazon has a long-tail catalog• A long tail catalog lends itself well to optimistic concurrency, because it has low data contention
    106. 106. OCC fails miserably for
    107. 107. OCC fails miserably for• eBay
    108. 108. OCC fails miserably for• eBay• Gilt
    109. 109. OCC fails miserably for• eBay• Gilt• Groupon
    110. 110. OCC fails miserably for• eBay• Gilt• Groupon• OpenSky
    111. 111. OCC fails miserably for• eBay• Gilt• Groupon• OpenSky• Living Social
    112. 112. OCC fails miserably for• eBay• Gilt• Groupon• OpenSky• Living Social• InsertFlashSaleSiteOfTheMinute
    113. 113. Flash sales and auctionsare defined by high data contention
    114. 114. Flash sales and auctionsare defined by high data contention• The model doesnt work otherwise
    115. 115. Flash sales and auctionsare defined by high data contention• The model doesnt work otherwise• They cant afford to be optimistic
    116. 116. Can we use pessimistic concurrency with a distributed NoSQL database?
    117. 117. Yep.
    118. 118. BlendingNoSQL & RDBMSUsing the right solution for each situation
    119. 119. Our goal is to put as much in Mongo as possible• What makes more sense in RDBMS? • Inventory • Orders
    120. 120. Inventory requires• Row level locking (or table level locking)
    121. 121. Orders require• Row level locking (or table level locking)• Atomic writes (inventory decremented)• Transactions (3rd party processing)
    122. 122. Inventory & checkout transactions
    123. 123. Commerce is ACID In Real Life
    124. 124. 1. I go to Barneys and see a pair of shoes I just have to buy.
    125. 125. 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).
    126. 126. 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).3. I take them up to the cash register and purchase them:
    127. 127. 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).3. I take them up to the cash register and purchase them: • Store inventory has been manually decremented.
    128. 128. 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).3. I take them up to the cash register and purchase them: • Store inventory has been manually decremented. • I pay for them with my trusty AmEx.
    129. 129. 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).3. I take them up to the cash register and purchase them: • Store inventory has been manually decremented. • I pay for them with my trusty AmEx.4. If all goes according to plan, I walk out of the store.
    130. 130. 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).3. I take them up to the cash register and purchase them: • Store inventory has been manually decremented. • I pay for them with my trusty AmEx.4. If all goes according to plan, I walk out of the store.5. If my card was declined, the shoes are “rolled back”
    131. 131. 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).3. I take them up to the cash register and purchase them: • Store inventory has been manually decremented. • I pay for them with my trusty AmEx.4. If all goes according to plan, I walk out of the store.5. If my card was declined, the shoes are “rolled back” ... out onto the shelves and sold to the next customer who wants them.
    132. 132. We follow the samemodel for e-commerce
    133. 133. 1. Select a product.
    134. 134. 1. Select a product.2. Lock the row or table and confirm inventory.
    135. 135. 1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product:
    136. 136. 1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product: • Decrement product inventory
    137. 137. 1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product: • Decrement product inventory • Process payment
    138. 138. 1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product: • Decrement product inventory • Process payment4. Commit the transaction.
    139. 139. 1. Select a product.2. Lock the row or table and confirm inventory.3. Purchase the product: • Decrement product inventory • Process payment4. Commit the transaction.5. Roll back if anything went wrong.
    140. 140. Doctrine (ORM/ODM) to the rescue
    141. 141. Doctrine (ORM/ODM) to the rescue It would be possible without them, but were not that masochistic
    142. 142. Data we store in SQL• Order• Order/Shipment• Order/Transaction• Inventory
    143. 143. Data we store in MongoDB
    144. 144. Data we store in MongoDB• User • Event• Product • TaxRate• Product/Sellable • ... and then I got tired of typing them in• Address • Just imagine this list has• Cart 40 more classes• CreditCard • ...
    145. 145. We have themost boring SQL schema ever
    146. 146. CREATE TABLE `product_inventory` ( `product_id` char(32) NOT NULL, `inventory` int(11) NOT NULL DEFAULT 0, PRIMARY KEY (`product_id`));CREATE TABLE `sellable_inventory` ( `sellable_id` char(32) NOT NULL, `inventory` int(11) NOT NULL DEFAULT 0, PRIMARY KEY (`sellable_id`));CREATE TABLE `orders` ( `id` int(11) NOT NULL AUTO_INCREMENT, `userId` char(32) NOT NULL, `shippingName` varchar(255) DEFAULT NULL, `shippingAddress1` varchar(255) DEFAULT NULL, `shippingAddress2` varchar(255) DEFAULT NULL, `shippingCity` varchar(255) DEFAULT NULL, `shippingState` varchar(2) DEFAULT NULL, `shippingZip` varchar(255) DEFAULT NULL, `billingName` varchar(255) DEFAULT NULL, `billingAddress1` varchar(255) DEFAULT NULL, `billingAddress2` varchar(255) DEFAULT NULL, `billingCity` varchar(255) DEFAULT NULL,
    147. 147. Wait. How does inventory live in SQL?Isn’t that a property in one of your Mongo collections?
    148. 148. I thought you’d never ask!
    149. 149. CREATE TABLE `product_inventory` ( `product_id` char(32) NOT NULL, `inventory` int(11) NOT NULL DEFAULT 0, PRIMARY KEY (`product_id`));CREATE TABLE `sellable_inventory` ( `sellable_id` char(32) NOT NULL, `inventory` int(11) NOT NULL DEFAULT 0, PRIMARY KEY (`sellable_id`));CREATE TABLE `orders` ( `id` int(11) NOT NULL AUTO_INCREMENT, `userId` char(32) NOT NULL, `shippingName` varchar(255) DEFAULT NULL, `shippingAddress1` varchar(255) DEFAULT NULL, `shippingAddress2` varchar(255) DEFAULT NULL, `shippingCity` varchar(255) DEFAULT NULL, `shippingState` varchar(2) DEFAULT NULL, `shippingZip` varchar(255) DEFAULT NULL, `billingName` varchar(255) DEFAULT NULL, `billingAddress1` varchar(255) DEFAULT NULL, `billingAddress2` varchar(255) DEFAULT NULL, `billingCity` varchar(255) DEFAULT NULL,
    150. 150. Inventory is transient• Product::$inventory is effectively a transient property• Note how I said “effectively”? ... we cheat and persist our transient property to MongoDB as well• We can do this because we never really trust the value stored in Mongo
    151. 151. Accuracy is only important when there’s contention
    152. 152. Accuracy is only important when there’s contention• For display, sorting and alerts, we can use the value stashed in MongoDB • It’s faster • It’s accurate enough
    153. 153. Accuracy is only important when there’s contention• For display, sorting and alerts, we can use the value stashed in MongoDB • It’s faster • It’s accurate enough• For financial transactions, we want the security and comfort of our RDBMS.
    154. 154. We keep inventory in sync with listeners
    155. 155. We keep inventory in sync with listeners• Every time a new product is created, its inventory is inserted in SQL
    156. 156. We keep inventory in sync with listeners• Every time a new product is created, its inventory is inserted in SQL• Every time an order is placed, inventory is verified and decremented
    157. 157. We keep inventory in sync with listeners• Every time a new product is created, its inventory is inserted in SQL• Every time an order is placed, inventory is verified and decremented• Whenever the SQL inventory changes, it is saved to MongoDB as well
    158. 158. Be careful what you lock
    159. 159. Be careful what you lock1. Acquire inventory row lock and begin transaction2. Check current product inventory3. Decrement product inventory4. Write the Order to SQL5. Update affected MongoDB documents6. Commit the transaction7. Release product inventory lock
    160. 160. Making MongoDBand RDBMS relations play nice
    161. 161. Products aredocuments stored in MongoDB
    162. 162. /** @mongodb:Document(collection="products") */class Product{ /** @mongodb:Id */ private $id; /** @mongodb:String */ private $title; public function getId() { return $this->id; } public function getTitle() { return $this->title; } public function setTitle($title) { $this->title = $title; }}
    163. 163. Orders are entitiesstored in an RDBMS
    164. 164. /** * @orm:Entity * @orm:Table(name="orders") * @orm:HasLifecycleCallbacks */class Order{ /** * @orm:Id @orm:Column(type="integer") * @orm:GeneratedValue(strategy="AUTO") */ private $id; /** * @orm:Column(type="string") */ private $productId; /** * @var DocumentsProduct */ private $product; // ...}
    165. 165. So how does an RDBMS have areference to something outside the database?
    166. 166. Setting the Productclass Order { // ... public function setProduct(Product $product) { $this->productId = $product->getId(); $this->product = $product; }}
    167. 167. • $productId is mapped and persisted• $product which stores the Product instance is not a persistent entity property
    168. 168. Retrieving ourproduct later
    169. 169. OrderPostLoadListeneruse DoctrineORMEventLifecycleEventArgs;class OrderPostLoadListener{ public function postLoad(LifecycleEventArgs $eventArgs) { // get the order entity $order = $eventArgs->getEntity(); // get odm reference to order.product_id $productId = $order->getProductId(); $product = $this->dm->getReference(MyBundle:DocumentProduct, $productId); // set the product on the order $em = $eventArgs->getEntityManager(); $productReflProp = $em->getClassMetadata(MyBundle:EntityOrder) ->reflClass->getProperty(product); $productReflProp->setAccessible(true); $productReflProp->setValue($order, $product); }}
    170. 170. All Together Now// Create a new product and order$product = new Product();$product->setTitle(Test Product);$dm->persist($product);$dm->flush();$order = new Order();$order->setProduct($product);$em->persist($order);$em->flush();// Find the order later$order = $em->find(Order, $order->getId());// Instance of an uninitialized product proxy$product = $order->getProduct();// Initializes proxy and queries the monogodb databaseecho "Order Title: " . $product->getTitle();print_r($order);
    171. 171. Read more about this techniqueJon Wage, one of OpenSky’s engineers, firstwrote about this technique on his personalblog: http://jwage.comYou can read the full article here:http://jwage.com/2010/08/25/blending-the-doctrine-orm-and-mongodb-odm/
    172. 172. Questions? http://spf13.com @spf13 http://justinhileman.com @bobthecow http://shopopensky.comPS: We’re hiring!! Contact us at jobs@shopopensky.com

    ×