MongoDB  E-commerceand Transactions
My name isSteve Francia     @spf13
• 15+ years building the  internet• Father, husband, skateboarder• Chief Solutions Architect @  10gen• Author of upcoming ...
Before 10gen Iworked    for     http://opensky.com
OpenSky was the firste-commerce site built    on MongoDB ... also the first e-commerce site built on                  Symfony2
Introduction to   MongoDB
Why MongoDB?
MongoDB Goals• Open source• Designed for today • Today’s hardware / environments • Today’s challenges• Great developer exp...
• Company behind MongoDB • AGPL license, own copyrights, engineering    team  • support, consulting, commercial license   ...
A bit of history
1974The relational database is created
1979
1979   1982-1996
1979   1982-1996   1995
Computers in 1995•Pentium 100 mhz•10base T•16 MB ram•200 MB HD
Cloud in 1995(Windows 95 cloud wallpaper)
Cell Phones in 2011•Dual core 1.5 Ghz•WiFi 802.11n (300+ Mbps)•1 GB ram•64GB Solid State
How about a DBdesigned for today?
MongoDB philosophy• Keep functionality when we can (key/  value stores are great, but we need more)• Non-relational (no jo...
MongoDB is:          Application      Document                           Oriented                           { author: “ste...
Under the hood•Written in C++•Runs on nearly anything•Data serialized to BSON•Extensive use of memory-mapped files
Database Landscape
This has led    some to say“MongoDB has the bestfeatures of key/ valuesstores, document databasesand relational databases ...
Use Cases
CMS / BlogNeeds:• Business needed modern data store for rapid development and  scaleSolution:• Use PHP & MongoDBResults:• ...
Photo Meta-DataProblem:• Business needed more flexibility than Oracle could deliverSolution:• Use MongoDB instead of Oracle...
Customer AnalyticsProblem:• Deal with massive data volume across all customer sitesSolution:• Use MongoDB to replace Googl...
Online DictionaryProblem:• MySQL could not scale to handle their 5B+ documentsSolution:• Switched from MySQL to MongoDBRes...
E-commerceProblem:• Multi-vertical E-commerce impossible to model (efficiently) in  RDBMSSolution:• Switched from MySQL to ...
Tons morePretty much if you can use a RDMBS or Key/Value             MongoDB is a great fit
In Good Company
Why NoSQL for   e-commerce?Using the right solution for each situation
Data dilemma of  e-commerce     Pick One
Data dilemma of     e-commerce              Pick One•Stick to one vertical (Sane schema)•Flexibility (Insane schema)
Sane schema
Sane schema•Works ... for a while•Fine for a few types of products•Not possible when more product types  introduced
Let’s Use an Example
Let’s Use an Example   How about we start with books
Book Product SchemaProduct {id:sku:                                    General Productproduct dimensions:shipping weight: ...
Seems simple enoughWhat happens when we add another vertical...             say music albums
Album Product SchemaProduct {id:sku:                               General Productproduct dimensions:                attri...
Okay, it’s getting  hairy but is stillmanageable, right?  Now the business want to sell jeans
Jeans Product SchemaProduct {id:                           General Productsku:                          attributes stay th...
Now we’re screwed
Now we’re screwed
We need a flexibleschema in RDBMS
We need a flexibleschema in RDBMS    We got this ... right?
Many approachesdealing with unknown unknowns in RDBMS
Many approachesdealing with unknown unknowns in RDBMS      None work well
EAV         as popularized by Magento“For purposes of flexibility, the Magento database heavilyutilizes an Entity-Attribute...
EAV• Crazy SQL queries• Hundreds of joins in a  query... or• Hundreds of queries joined  in the application• No database e...
Did I say crazy SQL(this is a single query)
Did I say crazy SQL(this is a single query)You may have trouble reading this in the back
Selecting a single     product
Single Table Inheritance         (insanely wide tables)• No data integrity enforcement• Only can use FK for common  elemen...
Generic Columns• No data integrity enforcement• No data type enforcement• Only can use FK for common  elements• Wasteful (...
Serialized in Blob• Not searchable• No integrity• All the disadvantages of a  document store, but none of the  advantages•...
Concrete Table Inheritance  (a table for each product attribute set)• Allows for data integrity• Querying across attribute...
Class table inheritance              (single product table,         each attribute set in own table)• Likely best of SQL w...
MongoDB to the   Rescue
MongoDB to the      Rescue•Flexible (and sane) Schema
MongoDB to the      Rescue•Flexible (and sane) Schema•Easily searchable
MongoDB to the      Rescue•Flexible (and sane) Schema•Easily searchable•Easily accessible
MongoDB to the      Rescue•Flexible (and sane) Schema•Easily searchable•Easily accessible•Fast
Flexible schema
{                                 {    sku: "00e8da9c",                  sku: "00e8da9d",    type: "Audio Album",         ...
pct_savings: 20                      pct_savings: 8.5},                                   },details: {                    ...
Queries
db.products.find( { name: "The Matrix" } );
db.products.find( { name: "The Matrix" } ); {     "_id": ObjectId("4d8ad78b46b731a22943d3d3"),     "sku": "00e8da9d",     ...
db.products.find( { details.actor: "Groucho Marx" } );
db.products.find( { details.actor: "Groucho Marx" } ); }, "pricing": {     "list": 1000,     "retail": 800,     "savings":...
db.products.find( {     details.genre: "Jazz", details.format: "CD"} );
db.products.find( {     details.genre: "Jazz", details.format: "CD"} );     "list": 1200,     "retail": 1100,     "savings...
db.products.find( { details.actor:     { $all: [James Stewart, Donna Reed] }} );
db.products.find( { details.actor:     { $all: [James Stewart, Donna Reed] }} ); }, "details": {     "title": "Its a Wonde...
Wanna Play?• grab products.js from  http://github.com/spf13/  mongoProducts• mongo   --shell products.js• > use   mongoPro...
Embedded documents are great for orders•Ordered items need to be fixed at the  time of purchase•Embed them right in the ord...
What about     transactions?Using the right solution for each situation
Data (like people) arereally sensitive when  it comes to money
Stricter datarequirements for $$
Stricter data requirements for $$•For financial systems any data  inconsistency is unacceptable
Stricter data requirements for $$•For financial systems any data  inconsistency is unacceptable•Perhaps you’ve heard of ACID?
What about ACID?
What about ACID?Q: Is MongoDB ACID?
What about ACID?Q: Is MongoDB ACID?A: Kinda
Atomicity
Atomicity•MongoDB does atomic writes
Atomicity•MongoDB does atomic writes  ... for single document changesets
Atomicity•MongoDB does atomic writes  ... for single document changesets• $set, $unset, $inc, $push,  $pushAll, $pull, $pu...
Consistency
Consistency•MongoDB can enforce unique keys
Consistency•MongoDB can enforce unique keys•MongoDB cant enforce referential  integrity
Isolation
Isolation•   // Pseudo-isolated updates    db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );
Isolation•   // Pseudo-isolated updates    db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );•   // Isolate...
Isolation•   // Pseudo-isolated updates    db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );•   // Isolate...
Isolation•   // Pseudo-isolated updates    db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );•   // Isolate...
Isolation•   // Pseudo-isolated updates    db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );•   // Isolate...
Isolation•   // Pseudo-isolated updates    db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );•   // Isolate...
Durability
Durability•Mongo has this one covered
What doesMongoDB Support?
• Atomic single document writes • If you need atomic writes across multi-document     transactions dont use Mongo  • Many ...
• Atomic single document writes • If you need atomic writes across multi-document     transactions dont use Mongo  • Many ...
• Atomic single document writes • If you need atomic writes across multi-document     transactions dont use Mongo  • Many ...
• Atomic single document writes • If you need atomic writes across multi-document     transactions dont use Mongo  • Many ...
There are ways to   guarantee ACIDproperties in MongoDB Here are 2 good approaches useful for       E-commerce transactions
OptimisticConcurrency
Optimistic       Concurrency•Read the current state of a product
Optimistic       Concurrency•Read the current state of a product•Make your changes with the assertion  that your product h...
Optimistic concurrency     in MongoDB
Optimistic concurrency     in MongoDB We’ll use an update-if-current strategy.
Optimistic concurrency     in MongoDB We’ll use an update-if-current strategy. This example is straight from the documenta...
Optimistic concurrency     in MongoDB    We’ll use an update-if-current strategy.    This example is straight from the doc...
Optimistic concurrency      in MongoDB      We’ll use an update-if-current strategy.      This example is straight from th...
Optimistic       concurrency•Read the current state of a product.•Make your changes with the assertion  that your product ...
Optimistic         concurrency•Read the current state of a product.•Make your changes with the assertion    that your prod...
Optimistic concurrency control assumes anenvironment with low   data contention
OCC works great forcompanies like Amazon•Amazon has a long-tail catalog•A long tail catalog lends itself well to  optimist...
OCC fails miserably        for
OCC fails miserably        for•eBay
OCC fails miserably        for•eBay•Gilt
OCC fails miserably         for•eBay•Gilt•Groupon
OCC fails miserably         for•eBay•Gilt•Groupon•OpenSky
OCC fails miserably             for•eBay•Gilt•Groupon•OpenSky•Living Social
OCC fails miserably             for•eBay•Gilt•Groupon•OpenSky•Living Social•InsertFlashSaleSiteOfTheMinute
Flash sales andauctions are defined by high data contention
Flash sales andauctions are defined by high data contention•The model doesnt work otherwise
Flash sales andauctions are defined by high data contention•The model doesnt work otherwise•They cant afford to be optimistic
Flash sales andauctions are defined by high data contention•The model doesnt work otherwise•They cant afford to be optimist...
What about high  contentionenvironments?
If we can avoidconcurrency we’ve    got it made
Commerce is ACID   In Real Life
1. I go to Barneys and see a pair of shoes I just have to   buy.2. I call “dibs” (by grabbing them off the shelf).3. I tak...
All of this is   accomplishedwithout concurrency
Each item can only be held by a consumer
We follow the samemodel for e-commerce
1. Select a product.
1. Select a product.2. Update the document to hold inventory.
1. Select a product.2. Update the document to hold inventory. • Store inventory has been    decremented.
1. Select a product.2. Update the document to hold inventory. • Store inventory has been    decremented.3. Purchase the pr...
1. Select a product.2. Update the document to hold inventory. • Store inventory has been    decremented.3. Purchase the pr...
1. Select a product.2. Update the document to hold inventory. • Store inventory has been    decremented.3. Purchase the pr...
MongoDB e-commerce    transactions
MongoDB e-commerce    transactions• Each Item (not SKU) has it’s own document• Document consists of... • a reference to th...
Transactionsin MongoDB
Transactions      in MongoDBWe’ll use a simple update statementhere.
Transactions            in MongoDB We’ll use a simple update statement here.> t = db.inventory> sku = sku.findOne({sku:abc...
Transactions              in MongoDB   We’ll use a simple update statement   here.  > t = db.inventory  > sku = sku.findOn...
Cart in Cart Action
Cart in Cart ActionAn added benefit, it can easily provideinventory hold in cart.
Cart in Cart Action An added benefit, it can easily provide inventory hold in cart.> t = db.inventory> sku = sku.findOne({s...
Cart in Cart Action An added benefit, it can easily provide inventory hold in cart.> t = db.inventory> sku = sku.findOne({s...
http://spf13.com                                http://github.com/spf13                                @spf13           Qu...
MongoDB, E-commerce and Transactions
MongoDB, E-commerce and Transactions
MongoDB, E-commerce and Transactions
MongoDB, E-commerce and Transactions
MongoDB, E-commerce and Transactions
MongoDB, E-commerce and Transactions
Upcoming SlideShare
Loading in...5
×

MongoDB, E-commerce and Transactions

58,051

Published on

Why MongoDB is the perfect solution for the challenges of E-Commerce.

Published in: Technology, Business
6 Comments
86 Likes
Statistics
Notes
No Downloads
Views
Total Views
58,051
On Slideshare
0
From Embeds
0
Number of Embeds
18
Actions
Shares
0
Downloads
769
Comments
6
Likes
86
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Remember in 1995 there were around 10,000 websites. Mosiac, Lynx, Mozilla (pre netscape) and IE 2.0 were the only web browsers. \nApache (Dec ’95), Java (’96), PHP (June ’95), and .net didn’t exist yet. Linux just barely (1.0 in ’94)\n
  • Remember in 1995 there were around 10,000 websites. Mosiac, Lynx, Mozilla (pre netscape) and IE 2.0 were the only web browsers. \nApache (Dec ’95), Java (’96), PHP (June ’95), and .net didn’t exist yet. Linux just barely (1.0 in ’94)\n
  • Remember in 1995 there were around 10,000 websites. Mosiac, Lynx, Mozilla (pre netscape) and IE 2.0 were the only web browsers. \nApache (Dec ’95), Java (’96), PHP (June ’95), and .net didn’t exist yet. Linux just barely (1.0 in ’94)\n
  • \n
  • \n
  • \n
  • \n
  • By reducing transactional semantics the db provides, one can still solve an interesting set of problems where performance is very important, and horizontal scaling then becomes easier.\n\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • actually, just the first 1/3 of it. \n
  • \n
  • Ironically this is how magento solves the performance problems associated with EAV, by caching the data into insanely wide tables.\n
  • \n
  • \n
  • \n
  • Can’t create a FK as each set references a different table. “Key” really made of attribute table name id and attribute table name\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Whenever you use a inter system coordination you need to implement your own atomic checks in the application... But SOAP does have transactions.. so not quite accurate. \n\nkyle idea... but we are fairly atomic with authorize.net\n\natomicity, consistency, isolation, durability.\n\n
  • Whenever you use a inter system coordination you need to implement your own atomic checks in the application... But SOAP does have transactions.. so not quite accurate. \n\nkyle idea... but we are fairly atomic with authorize.net\n\natomicity, consistency, isolation, durability.\n\n
  • atomicity, consistency, isolation, durability.\n\n
  • atomicity, consistency, isolation, durability.\n\n
  • Mongo has a grip of atomic operations: set, unset, etc.\n
  • Mongo has a grip of atomic operations: set, unset, etc.\n
  • Mongo has a grip of atomic operations: set, unset, etc.\n
  • \n
  • \n
  • update( { where }, { values }, upsert?, multiple? )\n\nIsolated is not atomic. Atomic implies that there is an all-or-nothing semantic to the update; this is not possible with more than one document. Isolated just means than you are the only one writing when the update is done; this means each update is done without any interference from any other.\n\nMongoDB supports atomic operations on single documents.  MongoDB does not support traditional locking and complex transactions for a number of reasons:\nFirst, in sharded environments, distributed locks could be expensive and slow.  Mongo DB's goal is to be lightweight and fast.\nWe dislike the concept of deadlocks.  We want the system to be simple and predictable without these sort of surprises.\nWe want Mongo DB to work well for realtime problems.  If an operation may execute which locks large amounts of data, it might stop some small light queries for an extended period of time.  (We don't claim Mongo DB is perfect yet in regards to being "real-time", but we certainly think locking would make it even harder.)\n\n\n\n
  • update( { where }, { values }, upsert?, multiple? )\n\nIsolated is not atomic. Atomic implies that there is an all-or-nothing semantic to the update; this is not possible with more than one document. Isolated just means than you are the only one writing when the update is done; this means each update is done without any interference from any other.\n\nMongoDB supports atomic operations on single documents.  MongoDB does not support traditional locking and complex transactions for a number of reasons:\nFirst, in sharded environments, distributed locks could be expensive and slow.  Mongo DB's goal is to be lightweight and fast.\nWe dislike the concept of deadlocks.  We want the system to be simple and predictable without these sort of surprises.\nWe want Mongo DB to work well for realtime problems.  If an operation may execute which locks large amounts of data, it might stop some small light queries for an extended period of time.  (We don't claim Mongo DB is perfect yet in regards to being "real-time", but we certainly think locking would make it even harder.)\n\n\n\n
  • update( { where }, { values }, upsert?, multiple? )\n\nIsolated is not atomic. Atomic implies that there is an all-or-nothing semantic to the update; this is not possible with more than one document. Isolated just means than you are the only one writing when the update is done; this means each update is done without any interference from any other.\n\nMongoDB supports atomic operations on single documents.  MongoDB does not support traditional locking and complex transactions for a number of reasons:\nFirst, in sharded environments, distributed locks could be expensive and slow.  Mongo DB's goal is to be lightweight and fast.\nWe dislike the concept of deadlocks.  We want the system to be simple and predictable without these sort of surprises.\nWe want Mongo DB to work well for realtime problems.  If an operation may execute which locks large amounts of data, it might stop some small light queries for an extended period of time.  (We don't claim Mongo DB is perfect yet in regards to being "real-time", but we certainly think locking would make it even harder.)\n\n\n\n
  • update( { where }, { values }, upsert?, multiple? )\n\nIsolated is not atomic. Atomic implies that there is an all-or-nothing semantic to the update; this is not possible with more than one document. Isolated just means than you are the only one writing when the update is done; this means each update is done without any interference from any other.\n\nMongoDB supports atomic operations on single documents.  MongoDB does not support traditional locking and complex transactions for a number of reasons:\nFirst, in sharded environments, distributed locks could be expensive and slow.  Mongo DB's goal is to be lightweight and fast.\nWe dislike the concept of deadlocks.  We want the system to be simple and predictable without these sort of surprises.\nWe want Mongo DB to work well for realtime problems.  If an operation may execute which locks large amounts of data, it might stop some small light queries for an extended period of time.  (We don't claim Mongo DB is perfect yet in regards to being "real-time", but we certainly think locking would make it even harder.)\n\n\n\n
  • update( { where }, { values }, upsert?, multiple? )\n\nIsolated is not atomic. Atomic implies that there is an all-or-nothing semantic to the update; this is not possible with more than one document. Isolated just means than you are the only one writing when the update is done; this means each update is done without any interference from any other.\n\nMongoDB supports atomic operations on single documents.  MongoDB does not support traditional locking and complex transactions for a number of reasons:\nFirst, in sharded environments, distributed locks could be expensive and slow.  Mongo DB's goal is to be lightweight and fast.\nWe dislike the concept of deadlocks.  We want the system to be simple and predictable without these sort of surprises.\nWe want Mongo DB to work well for realtime problems.  If an operation may execute which locks large amounts of data, it might stop some small light queries for an extended period of time.  (We don't claim Mongo DB is perfect yet in regards to being "real-time", but we certainly think locking would make it even harder.)\n\n\n\n
  • update( { where }, { values }, upsert?, multiple? )\n\nIsolated is not atomic. Atomic implies that there is an all-or-nothing semantic to the update; this is not possible with more than one document. Isolated just means than you are the only one writing when the update is done; this means each update is done without any interference from any other.\n\nMongoDB supports atomic operations on single documents.  MongoDB does not support traditional locking and complex transactions for a number of reasons:\nFirst, in sharded environments, distributed locks could be expensive and slow.  Mongo DB's goal is to be lightweight and fast.\nWe dislike the concept of deadlocks.  We want the system to be simple and predictable without these sort of surprises.\nWe want Mongo DB to work well for realtime problems.  If an operation may execute which locks large amounts of data, it might stop some small light queries for an extended period of time.  (We don't claim Mongo DB is perfect yet in regards to being "real-time", but we certainly think locking would make it even harder.)\n\n\n\n
  • \n
  • \n
  • MongoDB supports atomic operations on single documents.  MongoDB does not support traditional locking and complex transactions for a number of reasons:\nFirst, in sharded environments, distributed locks could be expensive and slow.  Mongo DB's goal is to be lightweight and fast.\nWe dislike the concept of deadlocks.  We want the system to be simple and predictable without these sort of surprises.\nWe want Mongo DB to work well for realtime problems.  If an operation may execute which locks large amounts of data, it might stop some small light queries for an extended period of time.  (We don't claim Mongo DB is perfect yet in regards to being "real-time", but we certainly think locking would make it even harder.)\n\n
  • MongoDB supports atomic operations on single documents.  MongoDB does not support traditional locking and complex transactions for a number of reasons:\nFirst, in sharded environments, distributed locks could be expensive and slow.  Mongo DB's goal is to be lightweight and fast.\nWe dislike the concept of deadlocks.  We want the system to be simple and predictable without these sort of surprises.\nWe want Mongo DB to work well for realtime problems.  If an operation may execute which locks large amounts of data, it might stop some small light queries for an extended period of time.  (We don't claim Mongo DB is perfect yet in regards to being "real-time", but we certainly think locking would make it even harder.)\n\n
  • MongoDB supports atomic operations on single documents.  MongoDB does not support traditional locking and complex transactions for a number of reasons:\nFirst, in sharded environments, distributed locks could be expensive and slow.  Mongo DB's goal is to be lightweight and fast.\nWe dislike the concept of deadlocks.  We want the system to be simple and predictable without these sort of surprises.\nWe want Mongo DB to work well for realtime problems.  If an operation may execute which locks large amounts of data, it might stop some small light queries for an extended period of time.  (We don't claim Mongo DB is perfect yet in regards to being "real-time", but we certainly think locking would make it even harder.)\n\n
  • MongoDB supports atomic operations on single documents.  MongoDB does not support traditional locking and complex transactions for a number of reasons:\nFirst, in sharded environments, distributed locks could be expensive and slow.  Mongo DB's goal is to be lightweight and fast.\nWe dislike the concept of deadlocks.  We want the system to be simple and predictable without these sort of surprises.\nWe want Mongo DB to work well for realtime problems.  If an operation may execute which locks large amounts of data, it might stop some small light queries for an extended period of time.  (We don't claim Mongo DB is perfect yet in regards to being "real-time", but we certainly think locking would make it even harder.)\n\n
  • \n
  • lemme show you an example\n
  • lemme show you an example\n
  • or instead of qty, use an version_id.\n object id / md5 as a version. \n
  • or instead of qty, use an version_id.\n object id / md5 as a version. \n
  • or instead of qty, use an version_id.\n object id / md5 as a version. \n
  • or instead of qty, use an version_id.\n object id / md5 as a version. \n
  • \n
  • Imagine what would happen if everyone tried to access the same record at the same time. Just think of all those spinning while loops :)\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Mind if I tell you a story?\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • By using a single document we avoid any need for complicated transactions. \n Any number of concurrent read operations are allowed, but typically only one write operation (although some write operations yield and in the future more concurrency will be added). The write lock acquisition is greedy: a pending write lock acquisition will prevent further read lock acquisitions until fulfilled.\n
  • By using a single document we avoid any need for complicated transactions. \n Any number of concurrent read operations are allowed, but typically only one write operation (although some write operations yield and in the future more concurrency will be added). The write lock acquisition is greedy: a pending write lock acquisition will prevent further read lock acquisitions until fulfilled.\n
  • By using a single document we avoid any need for complicated transactions. \n Any number of concurrent read operations are allowed, but typically only one write operation (although some write operations yield and in the future more concurrency will be added). The write lock acquisition is greedy: a pending write lock acquisition will prevent further read lock acquisitions until fulfilled.\n
  • By using a single document we avoid any need for complicated transactions. \n Any number of concurrent read operations are allowed, but typically only one write operation (although some write operations yield and in the future more concurrency will be added). The write lock acquisition is greedy: a pending write lock acquisition will prevent further read lock acquisitions until fulfilled.\n
  • By using a single document we avoid any need for complicated transactions. \n Any number of concurrent read operations are allowed, but typically only one write operation (although some write operations yield and in the future more concurrency will be added). The write lock acquisition is greedy: a pending write lock acquisition will prevent further read lock acquisitions until fulfilled.\n
  • By using a single document we avoid any need for complicated transactions. \n Any number of concurrent read operations are allowed, but typically only one write operation (although some write operations yield and in the future more concurrency will be added). The write lock acquisition is greedy: a pending write lock acquisition will prevent further read lock acquisitions until fulfilled.\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Inventory can be provided by using count. Can be cached as value on the sku.\nAs the items themselves are atomic, the order need not be to reserve inventory.\n
  • Inventory can be provided by using count. Can be cached as value on the sku.\nAs the items themselves are atomic, the order need not be to reserve inventory.\n
  • Inventory can be provided by using count. Can be cached as value on the sku.\nAs the items themselves are atomic, the order need not be to reserve inventory.\n
  • remember by default, update only updates 1 document and the operation is atomic on that document.\n\n
  • remember by default, update only updates 1 document and the operation is atomic on that document.\n\n
  • remember by default, update only updates 1 document and the operation is atomic on that document.\n\n
  • \n
  • \n
  • MongoDB, E-commerce and Transactions

    1. 1. MongoDB E-commerceand Transactions
    2. 2. My name isSteve Francia @spf13
    3. 3. • 15+ years building the internet• Father, husband, skateboarder• Chief Solutions Architect @ 10gen• Author of upcoming O’Reilly publication “MongoDB and PHP”
    4. 4. Before 10gen Iworked for http://opensky.com
    5. 5. OpenSky was the firste-commerce site built on MongoDB ... also the first e-commerce site built on Symfony2
    6. 6. Introduction to MongoDB
    7. 7. Why MongoDB?
    8. 8. MongoDB Goals• Open source• Designed for today • Today’s hardware / environments • Today’s challenges• Great developer experience• Reliable• Scalable
    9. 9. • Company behind MongoDB • AGPL license, own copyrights, engineering team • support, consulting, commercial license revenue• Management • Google/DoubleClick, Oracle, Apple, NetApp • Funding: Sequoia, Union Square, Flybridge • Offices in NYC, Redwood Shores & London • 60+ employees
    10. 10. A bit of history
    11. 11. 1974The relational database is created
    12. 12. 1979
    13. 13. 1979 1982-1996
    14. 14. 1979 1982-1996 1995
    15. 15. Computers in 1995•Pentium 100 mhz•10base T•16 MB ram•200 MB HD
    16. 16. Cloud in 1995(Windows 95 cloud wallpaper)
    17. 17. Cell Phones in 2011•Dual core 1.5 Ghz•WiFi 802.11n (300+ Mbps)•1 GB ram•64GB Solid State
    18. 18. How about a DBdesigned for today?
    19. 19. MongoDB philosophy• Keep functionality when we can (key/ value stores are great, but we need more)• Non-relational (no joins) makes scaling horizontally practical• Document data models are good• Database technology should run anywhere VMs, cloud, metal, etc
    20. 20. MongoDB is: Application Document Oriented { author: “steve”, High date: new Date(), text: “About MongoDB...”,Performance tags: [“tech”, “database”]} Fully Consistent Horizontally Scalable
    21. 21. Under the hood•Written in C++•Runs on nearly anything•Data serialized to BSON•Extensive use of memory-mapped files
    22. 22. Database Landscape
    23. 23. This has led some to say“MongoDB has the bestfeatures of key/ valuesstores, document databasesand relational databases inone. John Nunemaker
    24. 24. Use Cases
    25. 25. CMS / BlogNeeds:• Business needed modern data store for rapid development and scaleSolution:• Use PHP & MongoDBResults:• Real time statistics• All data, images, etc stored together, easy access, easy deployment, easy high availability• No need for complex migrations• Enabled very rapid development and growth
    26. 26. Photo Meta-DataProblem:• Business needed more flexibility than Oracle could deliverSolution:• Use MongoDB instead of OracleResults:• Developed application in one sprint cycle• 500% cost reduction compared to Oracle• 900% performance improvement compared to Oracle
    27. 27. Customer AnalyticsProblem:• Deal with massive data volume across all customer sitesSolution:• Use MongoDB to replace Google Analytics / Omniture optionsResults:• Less than one week to build prototype and prove business case• Rapid deployment of new features
    28. 28. Online DictionaryProblem:• MySQL could not scale to handle their 5B+ documentsSolution:• Switched from MySQL to MongoDBResults:• Massive simplification of code base• Eliminated need for external caching system• 20x performance improvement over MySQL
    29. 29. E-commerceProblem:• Multi-vertical E-commerce impossible to model (efficiently) in RDBMSSolution:• Switched from MySQL to MongoDBResults:• Massive simplification of code base• Rapidly build, halving time to market (and cost)• Eliminated need for external caching system• 50x+ improvement over MySQL
    30. 30. Tons morePretty much if you can use a RDMBS or Key/Value MongoDB is a great fit
    31. 31. In Good Company
    32. 32. Why NoSQL for e-commerce?Using the right solution for each situation
    33. 33. Data dilemma of e-commerce Pick One
    34. 34. Data dilemma of e-commerce Pick One•Stick to one vertical (Sane schema)•Flexibility (Insane schema)
    35. 35. Sane schema
    36. 36. Sane schema•Works ... for a while•Fine for a few types of products•Not possible when more product types introduced
    37. 37. Let’s Use an Example
    38. 38. Let’s Use an Example How about we start with books
    39. 39. Book Product SchemaProduct {id:sku: General Productproduct dimensions:shipping weight: attributesMSRP:price:description:...author: Orson Scott Cardtitle: Enders Gamebinding: Hardcoverpublication date: July 15, 1994 Book Specificpublisher name: Tor Science Fiction attributesnumber of pages: 352ISBN: 0812550706language: English...
    40. 40. Seems simple enoughWhat happens when we add another vertical... say music albums
    41. 41. Album Product SchemaProduct {id:sku: General Productproduct dimensions: attributes stayshipping weight:MSRP: the sameprice:description:...artist: MxPxtitle: Panic Album Specificrelease date: June 7, 2005label: Side One Dummy attributes aretrack listing: [ The Darkest ... differentlanguage: Englishformat: CD...
    42. 42. Okay, it’s getting hairy but is stillmanageable, right? Now the business want to sell jeans
    43. 43. Jeans Product SchemaProduct {id: General Productsku: attributes stay theproduct dimensions: sameshipping weight:MSRP:price:description:...brand: Lucky Jeans specificgender: Mens attributes aremake: Vintage totally different ...style: Straight Cutlength: 34 and not consistentwidth: 34 across brands &color: Hipster makematerial: Cotten Blend...
    44. 44. Now we’re screwed
    45. 45. Now we’re screwed
    46. 46. We need a flexibleschema in RDBMS
    47. 47. We need a flexibleschema in RDBMS We got this ... right?
    48. 48. Many approachesdealing with unknown unknowns in RDBMS
    49. 49. Many approachesdealing with unknown unknowns in RDBMS None work well
    50. 50. EAV as popularized by Magento“For purposes of flexibility, the Magento database heavilyutilizes an Entity-Attribute-Value (EAV) data model.As is often the case, the cost of flexibility is complexity -Magento is no exception.The process of manipulating data in Magento is oftenmore “involved” than that typically experienced usingtraditional relational tables.” - Varien
    51. 51. EAV• Crazy SQL queries• Hundreds of joins in a query... or• Hundreds of queries joined in the application• No database enforced integrity
    52. 52. Did I say crazy SQL(this is a single query)
    53. 53. Did I say crazy SQL(this is a single query)You may have trouble reading this in the back
    54. 54. Selecting a single product
    55. 55. Single Table Inheritance (insanely wide tables)• No data integrity enforcement• Only can use FK for common elements• Very wasteful (but disk is cheap!)• Can’t effectively index
    56. 56. Generic Columns• No data integrity enforcement• No data type enforcement• Only can use FK for common elements• Wasteful (but disk is cheap!)• Can’t index
    57. 57. Serialized in Blob• Not searchable• No integrity• All the disadvantages of a document store, but none of the advantages• Never should be used• One exception is Oracle XML which operates similar to a document store
    58. 58. Concrete Table Inheritance (a table for each product attribute set)• Allows for data integrity• Querying across attribute sets quite hard to do (lots of joins, OR statements and full table scanning)• New table needs to be created for each new attribute set
    59. 59. Class table inheritance (single product table, each attribute set in own table)• Likely best of SQL within the constraint solution• Supports data type enforcement• No data integrity enforcement• Easybrowse pages) since (for querying across categories common data in single table• Every set needs a new table• Requiresare very complicated changes a ton of forsight, as
    60. 60. MongoDB to the Rescue
    61. 61. MongoDB to the Rescue•Flexible (and sane) Schema
    62. 62. MongoDB to the Rescue•Flexible (and sane) Schema•Easily searchable
    63. 63. MongoDB to the Rescue•Flexible (and sane) Schema•Easily searchable•Easily accessible
    64. 64. MongoDB to the Rescue•Flexible (and sane) Schema•Easily searchable•Easily accessible•Fast
    65. 65. Flexible schema
    66. 66. { { sku: "00e8da9c", sku: "00e8da9d", type: "Audio Album", type: "Film", title: "Hoss", title: "The Matrix", description: "by Lagwagon", description: "Set in the 22nd century, Th asin: "B0000007QG", asin: "B000P0J0AQ", shipping: { shipping: { weight: 6, weight: 6, dimensions: { dimensions: { width: 10, width: 10, height: 10, height: 10, depth: 1 depth: 1 }, }, }, }, pricing: { pricing: { list: 1000, list: 1200, retail: 800, retail: 1100, savings: 200, savings: 100, pct_savings: 20 pct_savings: 8.5 }, }, details: { details: { title: "Hoss", title: "The Matrix",
    67. 67. pct_savings: 20 pct_savings: 8.5}, },details: { details: { title: "Hoss", title: "The Matrix", artist: "Lagwagon", director: [ "Andy Wachowski", "Larry Wa genre: [ "Punk", "Hardcore", "Indie Rock" ], [ "Andy Wachowski", "Larry Wach writer: label: "Fat Wreck Chords", actor: [ "Keanu Reeves" , "Lawrence Fis number_of_discs: 1, genre: [ "Science Fiction", "Action" ], issue_date: "November 21, 1995", number_of_discs: 1, format: "CD", issue_date: "May 15 2007", alternate_formats: [ Vinyl, MP3 ],original_release_date: "1999", tracks: [ disc_format: "DVD", "Kids Dont Like To Share", rating: "R", "Violins", alternate_formats: [ VHS, Bluray ], "Name Dropping", run_time: "136", "Bombs Away", studio: "Warner Bros", "Move The Car", language: "English", "Sleep", format: [ "AC-3", "Closed-captioned", " "Sick", aspect_ratio: "1.66:1" "Rifle", }, "Weak", } "Black Eye", "Bro Dependent", "Razor Burn", "Shaving Your Head", "Ride The Snake", ],
    68. 68. Queries
    69. 69. db.products.find( { name: "The Matrix" } );
    70. 70. db.products.find( { name: "The Matrix" } ); { "_id": ObjectId("4d8ad78b46b731a22943d3d3"), "sku": "00e8da9d", "type": "Film", "name": "The Matrix", "description": "Set in the 22nd century, The Matrix...", "asin": "B000P0J0AQ", "shipping": { "weight": 6, "dimensions": { "width": 10, "height": 10, "depth": 1 } }, "pricing": {
    71. 71. db.products.find( { details.actor: "Groucho Marx" } );
    72. 72. db.products.find( { details.actor: "Groucho Marx" } ); }, "pricing": { "list": 1000, "retail": 800, "savings": 200, "pct_savings": 20 }, "details": { "title": "A Night at the Opera", "director": "Sam Wood", "actor": ["Groucho Marx", "Chico Marx", "Harpo Marx"], "genre": "Comedy", "number_of_discs": 1, "issue_date": "May 4 2004", "original_release_date": "1935", "disc_format": "DVD",
    73. 73. db.products.find( { details.genre: "Jazz", details.format: "CD"} );
    74. 74. db.products.find( { details.genre: "Jazz", details.format: "CD"} ); "list": 1200, "retail": 1100, "savings": 100, "pct_savings": 8 }, "details": { "title": "A Love Supreme [Original Recording Reissued]", "artist": "John Coltrane", "genre": ["Jazz", "General"], "format": "CD", "label": "Impulse Records", "number_of_discs": 1, "issue_date": "December 9, 1964", "alternate_formats": ["Vinyl", "MP3"], "tracks": [ "A Love Supreme Part I: Acknowledgement",
    75. 75. db.products.find( { details.actor: { $all: [James Stewart, Donna Reed] }} );
    76. 76. db.products.find( { details.actor: { $all: [James Stewart, Donna Reed] }} ); }, "details": { "title": "Its a Wonderful Life", "director": "Frank Capra", "actor": ["James Stewart", "Donna Reed", "Lionel Barrymore"], "writer": [ "Frank Capra", "Albert Hackett", "Frances Goodrich", "Jo Swerling", "Michael Wilson" ], "genre": "Drama", "number_of_discs": 1, "issue_date": "Oct 31 2006", "original_release_date": "1947",
    77. 77. Wanna Play?• grab products.js from http://github.com/spf13/ mongoProducts• mongo --shell products.js• > use mongoProducts
    78. 78. Embedded documents are great for orders•Ordered items need to be fixed at the time of purchase•Embed them right in the orderdb.order.find( { items.sku: 00e8da9f } );db.order.find( { items.details.actor: James Stewart} ).count();
    79. 79. What about transactions?Using the right solution for each situation
    80. 80. Data (like people) arereally sensitive when it comes to money
    81. 81. Stricter datarequirements for $$
    82. 82. Stricter data requirements for $$•For financial systems any data inconsistency is unacceptable
    83. 83. Stricter data requirements for $$•For financial systems any data inconsistency is unacceptable•Perhaps you’ve heard of ACID?
    84. 84. What about ACID?
    85. 85. What about ACID?Q: Is MongoDB ACID?
    86. 86. What about ACID?Q: Is MongoDB ACID?A: Kinda
    87. 87. Atomicity
    88. 88. Atomicity•MongoDB does atomic writes
    89. 89. Atomicity•MongoDB does atomic writes ... for single document changesets
    90. 90. Atomicity•MongoDB does atomic writes ... for single document changesets• $set, $unset, $inc, $push, $pushAll, $pull, $pullAll, $bit
    91. 91. Consistency
    92. 92. Consistency•MongoDB can enforce unique keys
    93. 93. Consistency•MongoDB can enforce unique keys•MongoDB cant enforce referential integrity
    94. 94. Isolation
    95. 95. Isolation• // Pseudo-isolated updates db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );
    96. 96. Isolation• // Pseudo-isolated updates db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );• // Isolated updates db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false , true );
    97. 97. Isolation• // Pseudo-isolated updates db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );• // Isolated updates db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false , true );• But there are caveats...
    98. 98. Isolation• // Pseudo-isolated updates db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );• // Isolated updates db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false , true );• But there are caveats... • Despite the $atomic keyword, this is not an atomic update, since atomicity implies “all or nothing”
    99. 99. Isolation• // Pseudo-isolated updates db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );• // Isolated updates db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false , true );• But there are caveats... • Despite the $atomic keyword, this is not an atomic update, since atomicity implies “all or nothing” • $atomic here means update is done without an interference from any other operation (isolated)
    100. 100. Isolation• // Pseudo-isolated updates db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );• // Isolated updates db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false , true );• But there are caveats... • Despite the $atomic keyword, this is not an atomic update, since atomicity implies “all or nothing” • $atomic here means update is done without an interference from any other operation (isolated) • An isolated update can only act on a single collection. Multi- collection updates are not transactional, thus not isolatable.
    101. 101. Durability
    102. 102. Durability•Mongo has this one covered
    103. 103. What doesMongoDB Support?
    104. 104. • Atomic single document writes • If you need atomic writes across multi-document transactions dont use Mongo • Many if not most e-commerce transactions could be accomplished within a single document write
    105. 105. • Atomic single document writes • If you need atomic writes across multi-document transactions dont use Mongo • Many if not most e-commerce transactions could be accomplished within a single document write• Unique indexes • This only works on keys used by the entire collection
    106. 106. • Atomic single document writes • If you need atomic writes across multi-document transactions dont use Mongo • Many if not most e-commerce transactions could be accomplished within a single document write• Unique indexes • This only works on keys used by the entire collection• Isolated (not atomic) single collection updates. • Mongo does not support locking • There are ways to work around this
    107. 107. • Atomic single document writes • If you need atomic writes across multi-document transactions dont use Mongo • Many if not most e-commerce transactions could be accomplished within a single document write• Unique indexes • This only works on keys used by the entire collection• Isolated (not atomic) single collection updates. • Mongo does not support locking • There are ways to work around this• It’s durable
    108. 108. There are ways to guarantee ACIDproperties in MongoDB Here are 2 good approaches useful for E-commerce transactions
    109. 109. OptimisticConcurrency
    110. 110. Optimistic Concurrency•Read the current state of a product
    111. 111. Optimistic Concurrency•Read the current state of a product•Make your changes with the assertion that your product has the same state as it did when you last read it
    112. 112. Optimistic concurrency in MongoDB
    113. 113. Optimistic concurrency in MongoDB We’ll use an update-if-current strategy.
    114. 114. Optimistic concurrency in MongoDB We’ll use an update-if-current strategy. This example is straight from the documentation:
    115. 115. Optimistic concurrency in MongoDB We’ll use an update-if-current strategy. This example is straight from the documentation:> t = db.inventory> p = t.findOne({sku:abc})> t.update({_id:p._id, qty:p.qty}, {$inc: {qty: -1}});> db.$cmd.findOne({getlasterror:1});{"err" : , "updatedExisting" : true , "n" : 1 , "ok" : 1}// it worked
    116. 116. Optimistic concurrency in MongoDB We’ll use an update-if-current strategy. This example is straight from the documentation: > t = db.inventory > p = t.findOne({sku:abc}) > t.update({_id:p._id, qty:p.qty}, {$inc: {qty: -1}}); > db.$cmd.findOne({getlasterror:1}); {"err" : , "updatedExisting" : true , "n" : 1 , "ok" : 1} // it worked... If that didnt work, try again until it does.
    117. 117. Optimistic concurrency•Read the current state of a product.•Make your changes with the assertion that your product has the same state as it did when you last read it.
    118. 118. Optimistic concurrency•Read the current state of a product.•Make your changes with the assertion that your product has the same state as it did when you last read it.• It is also possible to use OCC to bootstrap pessimistic concurrency and fake row level locking
    119. 119. Optimistic concurrency control assumes anenvironment with low data contention
    120. 120. OCC works great forcompanies like Amazon•Amazon has a long-tail catalog•A long tail catalog lends itself well to optimistic concurrency, because it has low data contention
    121. 121. OCC fails miserably for
    122. 122. OCC fails miserably for•eBay
    123. 123. OCC fails miserably for•eBay•Gilt
    124. 124. OCC fails miserably for•eBay•Gilt•Groupon
    125. 125. OCC fails miserably for•eBay•Gilt•Groupon•OpenSky
    126. 126. OCC fails miserably for•eBay•Gilt•Groupon•OpenSky•Living Social
    127. 127. OCC fails miserably for•eBay•Gilt•Groupon•OpenSky•Living Social•InsertFlashSaleSiteOfTheMinute
    128. 128. Flash sales andauctions are defined by high data contention
    129. 129. Flash sales andauctions are defined by high data contention•The model doesnt work otherwise
    130. 130. Flash sales andauctions are defined by high data contention•The model doesnt work otherwise•They cant afford to be optimistic
    131. 131. Flash sales andauctions are defined by high data contention•The model doesnt work otherwise•They cant afford to be optimistic•Order really matters
    132. 132. What about high contentionenvironments?
    133. 133. If we can avoidconcurrency we’ve got it made
    134. 134. Commerce is ACID In Real Life
    135. 135. 1. I go to Barneys and see a pair of shoes I just have to buy.2. I call “dibs” (by grabbing them off the shelf).3. I take them up to the cash register and purchase them: • Store inventory has been manually decremented. • I pay for them with my trusty AmEx.4. If all goes according to plan, I walk out of the store.5. If my card was declined, the shoes are “rolled back” ... out onto the shelves and sold to the next customer who wants them.
    136. 136. All of this is accomplishedwithout concurrency
    137. 137. Each item can only be held by a consumer
    138. 138. We follow the samemodel for e-commerce
    139. 139. 1. Select a product.
    140. 140. 1. Select a product.2. Update the document to hold inventory.
    141. 141. 1. Select a product.2. Update the document to hold inventory. • Store inventory has been decremented.
    142. 142. 1. Select a product.2. Update the document to hold inventory. • Store inventory has been decremented.3. Purchase the product(s)
    143. 143. 1. Select a product.2. Update the document to hold inventory. • Store inventory has been decremented.3. Purchase the product(s) • Process payment
    144. 144. 1. Select a product.2. Update the document to hold inventory. • Store inventory has been decremented.3. Purchase the product(s) • Process payment4. Roll back if anything went wrong.
    145. 145. MongoDB e-commerce transactions
    146. 146. MongoDB e-commerce transactions• Each Item (not SKU) has it’s own document• Document consists of... • a reference to the SKU (product) • a state ( available / sold / ... ) • potentially other data (timestamp, order ref)
    147. 147. Transactionsin MongoDB
    148. 148. Transactions in MongoDBWe’ll use a simple update statementhere.
    149. 149. Transactions in MongoDB We’ll use a simple update statement here.> t = db.inventory> sku = sku.findOne({sku:abc})> t.update({ref_id:sku._id, state: available}, {$set:{state: ordered}});> db.$cmd.findOne({getlasterror:1});{"err" : , "updatedExisting" : true , "n" : 1 , "ok" : 1}// it worked
    150. 150. Transactions in MongoDB We’ll use a simple update statement here. > t = db.inventory > sku = sku.findOne({sku:abc}) > t.update({ref_id:sku._id, state: available}, {$set: {state: ordered}}); > db.$cmd.findOne({getlasterror:1}); {"err" : , "updatedExisting" : true , "n" : 1 , "ok" : 1} // it worked... If that didnt work, no inventory available
    151. 151. Cart in Cart Action
    152. 152. Cart in Cart ActionAn added benefit, it can easily provideinventory hold in cart.
    153. 153. Cart in Cart Action An added benefit, it can easily provide inventory hold in cart.> t = db.inventory> sku = sku.findOne({sku:abc})> t.update({ref_id:sku._id, state: available}, {$set:{state: in cart}});> db.$cmd.findOne({getlasterror:1});{"err" : , "updatedExisting" : true , "n" : 1 , "ok" : 1}// it worked
    154. 154. Cart in Cart Action An added benefit, it can easily provide inventory hold in cart.> t = db.inventory> sku = sku.findOne({sku:abc})> t.update({ref_id:sku._id, state: available}, {$set:{state: in cart}});> db.$cmd.findOne({getlasterror:1});{"err" : , "updatedExisting" : true , "n" : 1 , "ok" : 1}// it worked just like reality, each item is either available, in a cart, or purchased
    155. 155. http://spf13.com http://github.com/spf13 @spf13 Questions? download at mongodb.orgPS: We’re hiring!! Contact us at jobs@10gen.com
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×