SlideShare a Scribd company logo
Rick Copeland @rick446
Arborian Consulting, LLC
   Now a consultant, but formerly…

     Software engineer at SourceForge, early adopter of
     MongoDB (version 0.8)

     Wrote the SQLAlchemy book (I love SQL when it’s
     used well)

     Mainly write Python now, but have done C++, C#,
     Java, Javascript, VHDL, Verilog, …
   You can do it with an RDBMS as long as you…
     Don’t use joins
     Don’t use transactions
     Use read-only slaves
     Use memcached
     Denormalize your data
     Use custom sharding/partitioning
     Do a lot of vertical scaling
      ▪ (we’re going to need a bigger box)
+1
Year
Scaling with MongoDB
   Use documents to improve locality

   Optimize your indexes

   Be aware of your working set

   Scaling your disks

   Replication for fault-tolerance and read scaling

   Sharding for read and write scaling
Relational (SQL)   MongoDB
Database           Database              Dynamic
                                          Typing
Table              Collection            B-tree
                                     (range-based)
Index              Index
Row                Document
                                          Think JSON
Column             Field
                                 Primitive types +
                                arrays, documents
{
    title: "Slides for Scaling with MongoDB",
    author: "Rick Copeland",
    date: ISODate("20012-02-29T19:30:00Z"),
    text: "My slides are available on speakerdeck.com",
    comments: [
      { author: "anonymous",
         date: ISODate("20012-02-29T19:30:01Z"),
        text: "Fristpsot!" },
      { author: "mark”,
        date: ISODate("20012-02-29T19:45:23Z"),
        text: "Nice slides" } ] }
                                                 Embed comment data in
                                                  blog post document
Seek = 5+ ms   Read = really really fast
Post
                Comment
Author
Post

Author

Comment
Comment
 Comment
 Comment
  Comment
Find where x equals 7

1   2    3   4   5   6   7




        Looked at 7 objects
Find where x
equals 7         4



        2                6



    1        3       5       7




            Looked at 3 objects
Entire index
must fit in
RAM
Only small
 portion in
      RAM
   Working set =
     sizeof(frequently used data)
     + sizeof(frequently used indexes)

   Right-aligned indexes reduce working set size

   Working set should fit in available RAM for best
    performance

   Page faults are the biggest cause of performance
    loss in MongoDB
>db.foo.stats()
                                      Data Size
{
  "ns" : "test.foo",
  "count" : 1338330,
  "size" : 46915928,                                 Average doc size
  "avgObjSize" : 35.05557523181876,
  "storageSize" : 86092032,
  "numExtents" : 12,
  "nindexes" : 2,                        Size on disk (or RAM!)
  "lastExtentSize" : 20872960,
  "paddingFactor" : 1,
  "flags" : 0,                                    Size of all indexes
  "totalIndexSize" : 99860480,
  "indexSizes" : {
    "_id_" : 55877632,
    "x_1" : 43982848},
  "ok" : 1                                        Size of each index
}
~200 seeks / second
~200 seeks / second   ~200 seeks / second   ~200 seeks / second


   Faster, but less reliable
~400 seeks / second   ~400 seeks / second   ~400 seeks / second


   Faster and more reliable ($$$ though)
   Old and busted  master/slave replication

   The new hotness  replica sets with automatic
    failover
                 Read / Write       Primary




                       Read       Secondary




                       Read       Secondary
   Primary handles all
    writes

   Application optionally
    sends reads to slaves

   Heartbeat manages
    automatic failover
   Special collection (the oplog) records operations
    idempotently

   Secondaries read from primary oplog and replay
    operations locally

   Space is preallocated and fixed for the oplog
{
"ts" : Timestamp(1317653790000, 2),
                                     Insert
"h" : -6022751846629753359,
"op" : "i",
"ns" : "confoo.People",                  Collection name
"o" : {
"_id" : ObjectId("4e89cd1e0364241932324269"),
"first" : "Rick",
"last" : "Copeland”
   }
}                                                   Object to insert
   Use heartbeat signal to detect failure

   When primary can’t be reached, elect a new one

   Replica that’s the most up-to-date is chosen

   If there is skew, changes not on new primary are
    saved to a .bson file for manual reconciliation

   Application can require data to be replicated to a
    majority to ensure this doesn’t happen
   Priority
     Slower nodes with lower priority
     Backup or read-only nodes to never be primary

   slaveDelay
     Fat-finger protection

   Data center awareness and tagging
     Application can ensure complex replication
     guarantees
   Reads scale nicely
     As long as the working set fits in RAM
     … and you don’t mind eventual consistency


   Sharding to the rescue!
     Automatically partitioned data sets
     Scale writes and reads
     Automatic load balancing between the shards
Configuration
             MongoS        MongoS
                                           Config 1    Config 2             Config 3




Shard 1               Shard 2       Shard 3               Shard 4
 0..10                 10..20        20..30                30..40


   Primary               Primary       Primary                    Primary




 Secondary             Secondary     Secondary              Secondary




 Secondary             Secondary     Secondary              Secondary
   Sharding is per-collection and range-based

   The highest-impact choice (and hardest to
    change decision) you make is the shard key
     Random keys: good for writes, bad for reads
     Right-aligned index: bad for writes
     Small # of discrete keys: very bad
     Ideal: balance writes, make reads routable by mongos
     Optimal shard key selection is hard
Primary Data Center               Secondary Data Center


 Shard 1           Shard 1                    Shard 1
Priority 1        Priority 1                 Priority 0



 Shard 2           Shard 2                    Shard 2
Priority 1        Priority 1                 Priority 0



 Shard 3           Shard 3                    Shard 3
                             RS3
Priority 1        Priority 1                 Priority 0



Config 1          Config 2                   Config 3
   Writes and reads both scale (with good choice of
    shard key)

   Reads scale while remaining strongly consistent

   Partitioning ensures you get more usable RAM

   Pitfall: don’t wait too long to add capacity
Rick Copeland @rick446
Arborian Consulting, LLC

More Related Content

What's hot

MongoDB Pros and Cons
MongoDB Pros and ConsMongoDB Pros and Cons
MongoDB Pros and Cons
johnrjenson
 
MongoDB Capacity Planning
MongoDB Capacity PlanningMongoDB Capacity Planning
MongoDB Capacity Planning
Norberto Leite
 
Running MongoDB 3.0 on AWS
Running MongoDB 3.0 on AWSRunning MongoDB 3.0 on AWS
Running MongoDB 3.0 on AWS
MongoDB
 
Migrating to MongoDB: Best Practices
Migrating to MongoDB: Best PracticesMigrating to MongoDB: Best Practices
Migrating to MongoDB: Best Practices
MongoDB
 
Sharding Methods for MongoDB
Sharding Methods for MongoDBSharding Methods for MongoDB
Sharding Methods for MongoDB
MongoDB
 
3 scenarios when to use MongoDB!
3 scenarios when to use MongoDB!3 scenarios when to use MongoDB!
3 scenarios when to use MongoDB!
Edureka!
 
Webinar: Deploying MongoDB to Production in Data Centers and the Cloud
Webinar: Deploying MongoDB to Production in Data Centers and the CloudWebinar: Deploying MongoDB to Production in Data Centers and the Cloud
Webinar: Deploying MongoDB to Production in Data Centers and the Cloud
MongoDB
 
Benchmarking Top NoSQL Databases: Apache Cassandra, Apache HBase and MongoDB
Benchmarking Top NoSQL Databases: Apache Cassandra, Apache HBase and MongoDBBenchmarking Top NoSQL Databases: Apache Cassandra, Apache HBase and MongoDB
Benchmarking Top NoSQL Databases: Apache Cassandra, Apache HBase and MongoDB
Athiq Ahamed
 
Introduction to Sharding
Introduction to ShardingIntroduction to Sharding
Introduction to Sharding
MongoDB
 
A New MongoDB Sharding Architecture for Higher Availability and Better Resour...
A New MongoDB Sharding Architecture for Higher Availability and Better Resour...A New MongoDB Sharding Architecture for Higher Availability and Better Resour...
A New MongoDB Sharding Architecture for Higher Availability and Better Resour...
leifwalsh
 
MongoDB Deployment Checklist
MongoDB Deployment ChecklistMongoDB Deployment Checklist
MongoDB Deployment Checklist
MongoDB
 
Everything You Need to Know About Sharding
Everything You Need to Know About ShardingEverything You Need to Know About Sharding
Everything You Need to Know About Sharding
MongoDB
 
Strengths and Weaknesses of MongoDB
Strengths and Weaknesses of MongoDBStrengths and Weaknesses of MongoDB
Strengths and Weaknesses of MongoDB
lehresman
 
Evolution of MonogDB Sharding and Its Best Practices - Ranjith A - Mydbops Team
Evolution of MonogDB Sharding and Its Best Practices - Ranjith A - Mydbops TeamEvolution of MonogDB Sharding and Its Best Practices - Ranjith A - Mydbops Team
Evolution of MonogDB Sharding and Its Best Practices - Ranjith A - Mydbops Team
Mydbops
 
Efficient in situ processing of various storage types on apache tajo
Efficient in situ processing of various storage types on apache tajoEfficient in situ processing of various storage types on apache tajo
Efficient in situ processing of various storage types on apache tajo
Hyunsik Choi
 
Mongo db
Mongo dbMongo db
Mongo db
Akshay Mathur
 
Capacity Planning
Capacity PlanningCapacity Planning
Capacity Planning
MongoDB
 
Hardware Provisioning for MongoDB
Hardware Provisioning for MongoDBHardware Provisioning for MongoDB
Hardware Provisioning for MongoDB
MongoDB
 
Challenges with MongoDB
Challenges with MongoDBChallenges with MongoDB
Challenges with MongoDB
Stone Gao
 
Sharding
ShardingSharding
Sharding
MongoDB
 

What's hot (20)

MongoDB Pros and Cons
MongoDB Pros and ConsMongoDB Pros and Cons
MongoDB Pros and Cons
 
MongoDB Capacity Planning
MongoDB Capacity PlanningMongoDB Capacity Planning
MongoDB Capacity Planning
 
Running MongoDB 3.0 on AWS
Running MongoDB 3.0 on AWSRunning MongoDB 3.0 on AWS
Running MongoDB 3.0 on AWS
 
Migrating to MongoDB: Best Practices
Migrating to MongoDB: Best PracticesMigrating to MongoDB: Best Practices
Migrating to MongoDB: Best Practices
 
Sharding Methods for MongoDB
Sharding Methods for MongoDBSharding Methods for MongoDB
Sharding Methods for MongoDB
 
3 scenarios when to use MongoDB!
3 scenarios when to use MongoDB!3 scenarios when to use MongoDB!
3 scenarios when to use MongoDB!
 
Webinar: Deploying MongoDB to Production in Data Centers and the Cloud
Webinar: Deploying MongoDB to Production in Data Centers and the CloudWebinar: Deploying MongoDB to Production in Data Centers and the Cloud
Webinar: Deploying MongoDB to Production in Data Centers and the Cloud
 
Benchmarking Top NoSQL Databases: Apache Cassandra, Apache HBase and MongoDB
Benchmarking Top NoSQL Databases: Apache Cassandra, Apache HBase and MongoDBBenchmarking Top NoSQL Databases: Apache Cassandra, Apache HBase and MongoDB
Benchmarking Top NoSQL Databases: Apache Cassandra, Apache HBase and MongoDB
 
Introduction to Sharding
Introduction to ShardingIntroduction to Sharding
Introduction to Sharding
 
A New MongoDB Sharding Architecture for Higher Availability and Better Resour...
A New MongoDB Sharding Architecture for Higher Availability and Better Resour...A New MongoDB Sharding Architecture for Higher Availability and Better Resour...
A New MongoDB Sharding Architecture for Higher Availability and Better Resour...
 
MongoDB Deployment Checklist
MongoDB Deployment ChecklistMongoDB Deployment Checklist
MongoDB Deployment Checklist
 
Everything You Need to Know About Sharding
Everything You Need to Know About ShardingEverything You Need to Know About Sharding
Everything You Need to Know About Sharding
 
Strengths and Weaknesses of MongoDB
Strengths and Weaknesses of MongoDBStrengths and Weaknesses of MongoDB
Strengths and Weaknesses of MongoDB
 
Evolution of MonogDB Sharding and Its Best Practices - Ranjith A - Mydbops Team
Evolution of MonogDB Sharding and Its Best Practices - Ranjith A - Mydbops TeamEvolution of MonogDB Sharding and Its Best Practices - Ranjith A - Mydbops Team
Evolution of MonogDB Sharding and Its Best Practices - Ranjith A - Mydbops Team
 
Efficient in situ processing of various storage types on apache tajo
Efficient in situ processing of various storage types on apache tajoEfficient in situ processing of various storage types on apache tajo
Efficient in situ processing of various storage types on apache tajo
 
Mongo db
Mongo dbMongo db
Mongo db
 
Capacity Planning
Capacity PlanningCapacity Planning
Capacity Planning
 
Hardware Provisioning for MongoDB
Hardware Provisioning for MongoDBHardware Provisioning for MongoDB
Hardware Provisioning for MongoDB
 
Challenges with MongoDB
Challenges with MongoDBChallenges with MongoDB
Challenges with MongoDB
 
Sharding
ShardingSharding
Sharding
 

Viewers also liked

Custom Courseware Development
Custom Courseware DevelopmentCustom Courseware Development
Custom Courseware Development
CommLab India – Rapid eLearning Solutions
 
2017 Volvo S60 Brochure | Orange County Volvo
2017 Volvo S60 Brochure | Orange County Volvo2017 Volvo S60 Brochure | Orange County Volvo
2017 Volvo S60 Brochure | Orange County Volvo
Volvo Cars Mission Viejo
 
Containerization and palletization
Containerization and palletizationContainerization and palletization
Containerization and palletization
Amar Ashish Shrivastava
 
How to configure static nat on cisco routers
How to configure static nat on cisco routersHow to configure static nat on cisco routers
How to configure static nat on cisco routers
IT Tech
 
Assessment for learning meeting april 29th 2014
Assessment for learning meeting april 29th 2014Assessment for learning meeting april 29th 2014
Assessment for learning meeting april 29th 2014
Mr Bounab Samir
 
Global IT Consulting Market
Global IT Consulting MarketGlobal IT Consulting Market
Global IT Consulting Market
Joyjeet Dan
 
Best practices multichannel-integration
Best practices multichannel-integrationBest practices multichannel-integration
Best practices multichannel-integration
Giuseppe Monserrato
 
Dantes Inferno Study Guide
Dantes Inferno Study GuideDantes Inferno Study Guide
Dantes Inferno Study Guide
followthelamb
 
Finding the best Radio Network Planning and Radio Network Optimization software
Finding the best Radio Network Planning and Radio Network Optimization softwareFinding the best Radio Network Planning and Radio Network Optimization software
Finding the best Radio Network Planning and Radio Network Optimization software
Muhammad Waqas Akram
 
Temperature Transducer
Temperature TransducerTemperature Transducer
Temperature Transducer
AIT
 
Camels approach
Camels approachCamels approach
Camels approach
Vishal Parmar
 

Viewers also liked (11)

Custom Courseware Development
Custom Courseware DevelopmentCustom Courseware Development
Custom Courseware Development
 
2017 Volvo S60 Brochure | Orange County Volvo
2017 Volvo S60 Brochure | Orange County Volvo2017 Volvo S60 Brochure | Orange County Volvo
2017 Volvo S60 Brochure | Orange County Volvo
 
Containerization and palletization
Containerization and palletizationContainerization and palletization
Containerization and palletization
 
How to configure static nat on cisco routers
How to configure static nat on cisco routersHow to configure static nat on cisco routers
How to configure static nat on cisco routers
 
Assessment for learning meeting april 29th 2014
Assessment for learning meeting april 29th 2014Assessment for learning meeting april 29th 2014
Assessment for learning meeting april 29th 2014
 
Global IT Consulting Market
Global IT Consulting MarketGlobal IT Consulting Market
Global IT Consulting Market
 
Best practices multichannel-integration
Best practices multichannel-integrationBest practices multichannel-integration
Best practices multichannel-integration
 
Dantes Inferno Study Guide
Dantes Inferno Study GuideDantes Inferno Study Guide
Dantes Inferno Study Guide
 
Finding the best Radio Network Planning and Radio Network Optimization software
Finding the best Radio Network Planning and Radio Network Optimization softwareFinding the best Radio Network Planning and Radio Network Optimization software
Finding the best Radio Network Planning and Radio Network Optimization software
 
Temperature Transducer
Temperature TransducerTemperature Transducer
Temperature Transducer
 
Camels approach
Camels approachCamels approach
Camels approach
 

Similar to Scaling with MongoDB

Scaling with MongoDB
Scaling with MongoDBScaling with MongoDB
Scaling with MongoDB
MongoDB
 
Optimizing MongoDB: Lessons Learned at Localytics
Optimizing MongoDB: Lessons Learned at LocalyticsOptimizing MongoDB: Lessons Learned at Localytics
Optimizing MongoDB: Lessons Learned at Localytics
andrew311
 
Python mongo db-training-europython-2011
Python mongo db-training-europython-2011Python mongo db-training-europython-2011
Python mongo db-training-europython-2011
Andreas Jung
 
MongoDB 3.0
MongoDB 3.0 MongoDB 3.0
MongoDB 3.0
Victoria Malaya
 
MongoDB Knowledge Shareing
MongoDB Knowledge ShareingMongoDB Knowledge Shareing
MongoDB Knowledge Shareing
Philip Zhong
 
Intro to mongo db
Intro to mongo dbIntro to mongo db
Intro to mongo db
Chi Lee
 
2012 phoenix mug
2012 phoenix mug2012 phoenix mug
2012 phoenix mug
Paul Pedersen
 
MongoDB Auto-Sharding at Mongo Seattle
MongoDB Auto-Sharding at Mongo SeattleMongoDB Auto-Sharding at Mongo Seattle
MongoDB Auto-Sharding at Mongo Seattle
MongoDB
 
Webinar: Understanding Storage for Performance and Data Safety
Webinar: Understanding Storage for Performance and Data SafetyWebinar: Understanding Storage for Performance and Data Safety
Webinar: Understanding Storage for Performance and Data Safety
MongoDB
 
MongoDB for Time Series Data Part 3: Sharding
MongoDB for Time Series Data Part 3: ShardingMongoDB for Time Series Data Part 3: Sharding
MongoDB for Time Series Data Part 3: Sharding
MongoDB
 
MongoDB is the MashupDB
MongoDB is the MashupDBMongoDB is the MashupDB
MongoDB is the MashupDB
Wynn Netherland
 
DBVersity MongoDB Online Training Presentations
DBVersity MongoDB Online Training PresentationsDBVersity MongoDB Online Training Presentations
DBVersity MongoDB Online Training Presentations
Srinivas Mutyala
 
MongoDB for Time Series Data: Sharding
MongoDB for Time Series Data: ShardingMongoDB for Time Series Data: Sharding
MongoDB for Time Series Data: Sharding
MongoDB
 
Mongo db japan
Mongo db japanMongo db japan
Mongo db japan
rogerbodamer
 
2014 05-07-fr - add dev series - session 6 - deploying your application-2
2014 05-07-fr - add dev series - session 6 - deploying your application-22014 05-07-fr - add dev series - session 6 - deploying your application-2
2014 05-07-fr - add dev series - session 6 - deploying your application-2
MongoDB
 
Cassandra for Sysadmins
Cassandra for SysadminsCassandra for Sysadmins
Cassandra for Sysadmins
Nathan Milford
 
MongoDB: Optimising for Performance, Scale & Analytics
MongoDB: Optimising for Performance, Scale & AnalyticsMongoDB: Optimising for Performance, Scale & Analytics
MongoDB: Optimising for Performance, Scale & Analytics
Server Density
 
Deployment Preparedness
Deployment Preparedness Deployment Preparedness
Deployment Preparedness
MongoDB
 
OSDC 2012 | Scaling with MongoDB by Ross Lawley
OSDC 2012 | Scaling with MongoDB by Ross LawleyOSDC 2012 | Scaling with MongoDB by Ross Lawley
OSDC 2012 | Scaling with MongoDB by Ross Lawley
NETWAYS
 
Sedna XML Database: Memory Management
Sedna XML Database: Memory ManagementSedna XML Database: Memory Management
Sedna XML Database: Memory Management
Ivan Shcheklein
 

Similar to Scaling with MongoDB (20)

Scaling with MongoDB
Scaling with MongoDBScaling with MongoDB
Scaling with MongoDB
 
Optimizing MongoDB: Lessons Learned at Localytics
Optimizing MongoDB: Lessons Learned at LocalyticsOptimizing MongoDB: Lessons Learned at Localytics
Optimizing MongoDB: Lessons Learned at Localytics
 
Python mongo db-training-europython-2011
Python mongo db-training-europython-2011Python mongo db-training-europython-2011
Python mongo db-training-europython-2011
 
MongoDB 3.0
MongoDB 3.0 MongoDB 3.0
MongoDB 3.0
 
MongoDB Knowledge Shareing
MongoDB Knowledge ShareingMongoDB Knowledge Shareing
MongoDB Knowledge Shareing
 
Intro to mongo db
Intro to mongo dbIntro to mongo db
Intro to mongo db
 
2012 phoenix mug
2012 phoenix mug2012 phoenix mug
2012 phoenix mug
 
MongoDB Auto-Sharding at Mongo Seattle
MongoDB Auto-Sharding at Mongo SeattleMongoDB Auto-Sharding at Mongo Seattle
MongoDB Auto-Sharding at Mongo Seattle
 
Webinar: Understanding Storage for Performance and Data Safety
Webinar: Understanding Storage for Performance and Data SafetyWebinar: Understanding Storage for Performance and Data Safety
Webinar: Understanding Storage for Performance and Data Safety
 
MongoDB for Time Series Data Part 3: Sharding
MongoDB for Time Series Data Part 3: ShardingMongoDB for Time Series Data Part 3: Sharding
MongoDB for Time Series Data Part 3: Sharding
 
MongoDB is the MashupDB
MongoDB is the MashupDBMongoDB is the MashupDB
MongoDB is the MashupDB
 
DBVersity MongoDB Online Training Presentations
DBVersity MongoDB Online Training PresentationsDBVersity MongoDB Online Training Presentations
DBVersity MongoDB Online Training Presentations
 
MongoDB for Time Series Data: Sharding
MongoDB for Time Series Data: ShardingMongoDB for Time Series Data: Sharding
MongoDB for Time Series Data: Sharding
 
Mongo db japan
Mongo db japanMongo db japan
Mongo db japan
 
2014 05-07-fr - add dev series - session 6 - deploying your application-2
2014 05-07-fr - add dev series - session 6 - deploying your application-22014 05-07-fr - add dev series - session 6 - deploying your application-2
2014 05-07-fr - add dev series - session 6 - deploying your application-2
 
Cassandra for Sysadmins
Cassandra for SysadminsCassandra for Sysadmins
Cassandra for Sysadmins
 
MongoDB: Optimising for Performance, Scale & Analytics
MongoDB: Optimising for Performance, Scale & AnalyticsMongoDB: Optimising for Performance, Scale & Analytics
MongoDB: Optimising for Performance, Scale & Analytics
 
Deployment Preparedness
Deployment Preparedness Deployment Preparedness
Deployment Preparedness
 
OSDC 2012 | Scaling with MongoDB by Ross Lawley
OSDC 2012 | Scaling with MongoDB by Ross LawleyOSDC 2012 | Scaling with MongoDB by Ross Lawley
OSDC 2012 | Scaling with MongoDB by Ross Lawley
 
Sedna XML Database: Memory Management
Sedna XML Database: Memory ManagementSedna XML Database: Memory Management
Sedna XML Database: Memory Management
 

More from Rick Copeland

Python Functions (PyAtl Beginners Night)
Python Functions (PyAtl Beginners Night)Python Functions (PyAtl Beginners Night)
Python Functions (PyAtl Beginners Night)
Rick Copeland
 
Schema Design at Scale
Schema Design at ScaleSchema Design at Scale
Schema Design at Scale
Rick Copeland
 
Building Your First MongoDB Application
Building Your First MongoDB ApplicationBuilding Your First MongoDB Application
Building Your First MongoDB Application
Rick Copeland
 
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and MingRapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
Rick Copeland
 
Chef on MongoDB and Pyramid
Chef on MongoDB and PyramidChef on MongoDB and Pyramid
Chef on MongoDB and Pyramid
Rick Copeland
 
Chef on Python and MongoDB
Chef on Python and MongoDBChef on Python and MongoDB
Chef on Python and MongoDB
Rick Copeland
 
Real-Time Python Web: Gevent and Socket.io
Real-Time Python Web: Gevent and Socket.ioReal-Time Python Web: Gevent and Socket.io
Real-Time Python Web: Gevent and Socket.io
Rick Copeland
 
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and MingRapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
Rick Copeland
 
Realtime Analytics Using MongoDB, Python, Gevent, and ZeroMQ
Realtime Analytics Using MongoDB, Python, Gevent, and ZeroMQRealtime Analytics Using MongoDB, Python, Gevent, and ZeroMQ
Realtime Analytics Using MongoDB, Python, Gevent, and ZeroMQ
Rick Copeland
 
Rapid, Scalable Web Development with MongoDB, Ming, and Python
Rapid, Scalable Web Development with MongoDB, Ming, and PythonRapid, Scalable Web Development with MongoDB, Ming, and Python
Rapid, Scalable Web Development with MongoDB, Ming, and Python
Rick Copeland
 
Allura - an Open Source MongoDB Based Document Oriented SourceForge
Allura - an Open Source MongoDB Based Document Oriented SourceForgeAllura - an Open Source MongoDB Based Document Oriented SourceForge
Allura - an Open Source MongoDB Based Document Oriented SourceForge
Rick Copeland
 
MongoATL: How Sourceforge is Using MongoDB
MongoATL: How Sourceforge is Using MongoDBMongoATL: How Sourceforge is Using MongoDB
MongoATL: How Sourceforge is Using MongoDB
Rick Copeland
 

More from Rick Copeland (12)

Python Functions (PyAtl Beginners Night)
Python Functions (PyAtl Beginners Night)Python Functions (PyAtl Beginners Night)
Python Functions (PyAtl Beginners Night)
 
Schema Design at Scale
Schema Design at ScaleSchema Design at Scale
Schema Design at Scale
 
Building Your First MongoDB Application
Building Your First MongoDB ApplicationBuilding Your First MongoDB Application
Building Your First MongoDB Application
 
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and MingRapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
 
Chef on MongoDB and Pyramid
Chef on MongoDB and PyramidChef on MongoDB and Pyramid
Chef on MongoDB and Pyramid
 
Chef on Python and MongoDB
Chef on Python and MongoDBChef on Python and MongoDB
Chef on Python and MongoDB
 
Real-Time Python Web: Gevent and Socket.io
Real-Time Python Web: Gevent and Socket.ioReal-Time Python Web: Gevent and Socket.io
Real-Time Python Web: Gevent and Socket.io
 
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and MingRapid and Scalable Development with MongoDB, PyMongo, and Ming
Rapid and Scalable Development with MongoDB, PyMongo, and Ming
 
Realtime Analytics Using MongoDB, Python, Gevent, and ZeroMQ
Realtime Analytics Using MongoDB, Python, Gevent, and ZeroMQRealtime Analytics Using MongoDB, Python, Gevent, and ZeroMQ
Realtime Analytics Using MongoDB, Python, Gevent, and ZeroMQ
 
Rapid, Scalable Web Development with MongoDB, Ming, and Python
Rapid, Scalable Web Development with MongoDB, Ming, and PythonRapid, Scalable Web Development with MongoDB, Ming, and Python
Rapid, Scalable Web Development with MongoDB, Ming, and Python
 
Allura - an Open Source MongoDB Based Document Oriented SourceForge
Allura - an Open Source MongoDB Based Document Oriented SourceForgeAllura - an Open Source MongoDB Based Document Oriented SourceForge
Allura - an Open Source MongoDB Based Document Oriented SourceForge
 
MongoATL: How Sourceforge is Using MongoDB
MongoATL: How Sourceforge is Using MongoDBMongoATL: How Sourceforge is Using MongoDB
MongoATL: How Sourceforge is Using MongoDB
 

Recently uploaded

Patch Tuesday de julio
Patch Tuesday de julioPatch Tuesday de julio
Patch Tuesday de julio
Ivanti
 
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptxMAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
janagijoythi
 
UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...
UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...
UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...
FIDO Alliance
 
kk vathada _digital transformation frameworks_2024.pdf
kk vathada _digital transformation frameworks_2024.pdfkk vathada _digital transformation frameworks_2024.pdf
kk vathada _digital transformation frameworks_2024.pdf
KIRAN KV
 
Camunda Chapter NY Meetup July 2024.pptx
Camunda Chapter NY Meetup July 2024.pptxCamunda Chapter NY Meetup July 2024.pptx
Camunda Chapter NY Meetup July 2024.pptx
ZachWylie3
 
Generative AI Reasoning Tech Talk - July 2024
Generative AI Reasoning Tech Talk - July 2024Generative AI Reasoning Tech Talk - July 2024
Generative AI Reasoning Tech Talk - July 2024
siddu769252
 
Improving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning ContentImproving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning Content
Enterprise Knowledge
 
How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...
DianaGray10
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
maigasapphire
 
Redefining Cybersecurity with AI Capabilities
Redefining Cybersecurity with AI CapabilitiesRedefining Cybersecurity with AI Capabilities
Redefining Cybersecurity with AI Capabilities
Priyanka Aash
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
bhumivarma35300
 
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
alexjohnson7307
 
Step-By-Step Process to Develop a Mobile App From Scratch
Step-By-Step Process to Develop a Mobile App From ScratchStep-By-Step Process to Develop a Mobile App From Scratch
Step-By-Step Process to Develop a Mobile App From Scratch
softsuave
 
Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
shyamraj55
 
Finetuning GenAI For Hacking and Defending
Finetuning GenAI For Hacking and DefendingFinetuning GenAI For Hacking and Defending
Finetuning GenAI For Hacking and Defending
Priyanka Aash
 
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdfLeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
SelfMade bd
 
Mule Experience Hub and Release Channel with Java 17
Mule Experience Hub and Release Channel with Java 17Mule Experience Hub and Release Channel with Java 17
Mule Experience Hub and Release Channel with Java 17
Bhajan Mehta
 
Semantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software DevelopmentSemantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software Development
Baishakhi Ray
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
DianaGray10
 
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
FIDO Alliance
 

Recently uploaded (20)

Patch Tuesday de julio
Patch Tuesday de julioPatch Tuesday de julio
Patch Tuesday de julio
 
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptxMAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
MAKE MONEY ONLINE Unlock Your Income Potential Today.pptx
 
UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...
UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...
UX Webinar Series: Drive Revenue and Decrease Costs with Passkeys for Consume...
 
kk vathada _digital transformation frameworks_2024.pdf
kk vathada _digital transformation frameworks_2024.pdfkk vathada _digital transformation frameworks_2024.pdf
kk vathada _digital transformation frameworks_2024.pdf
 
Camunda Chapter NY Meetup July 2024.pptx
Camunda Chapter NY Meetup July 2024.pptxCamunda Chapter NY Meetup July 2024.pptx
Camunda Chapter NY Meetup July 2024.pptx
 
Generative AI Reasoning Tech Talk - July 2024
Generative AI Reasoning Tech Talk - July 2024Generative AI Reasoning Tech Talk - July 2024
Generative AI Reasoning Tech Talk - July 2024
 
Improving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning ContentImproving Learning Content Efficiency with Reusable Learning Content
Improving Learning Content Efficiency with Reusable Learning Content
 
How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...How UiPath Discovery Suite supports identification of Agentic Process Automat...
How UiPath Discovery Suite supports identification of Agentic Process Automat...
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
 
Redefining Cybersecurity with AI Capabilities
Redefining Cybersecurity with AI CapabilitiesRedefining Cybersecurity with AI Capabilities
Redefining Cybersecurity with AI Capabilities
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
 
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
leewayhertz.com-Generative AI tech stack Frameworks infrastructure models and...
 
Step-By-Step Process to Develop a Mobile App From Scratch
Step-By-Step Process to Develop a Mobile App From ScratchStep-By-Step Process to Develop a Mobile App From Scratch
Step-By-Step Process to Develop a Mobile App From Scratch
 
Integrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecaseIntegrating Kafka with MuleSoft 4 and usecase
Integrating Kafka with MuleSoft 4 and usecase
 
Finetuning GenAI For Hacking and Defending
Finetuning GenAI For Hacking and DefendingFinetuning GenAI For Hacking and Defending
Finetuning GenAI For Hacking and Defending
 
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdfLeadMagnet IQ Review:  Unlock the Secret to Effortless Traffic and Leads.pdf
LeadMagnet IQ Review: Unlock the Secret to Effortless Traffic and Leads.pdf
 
Mule Experience Hub and Release Channel with Java 17
Mule Experience Hub and Release Channel with Java 17Mule Experience Hub and Release Channel with Java 17
Mule Experience Hub and Release Channel with Java 17
 
Semantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software DevelopmentSemantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software Development
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
 
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
UX Webinar Series: Essentials for Adopting Passkeys as the Foundation of your...
 

Scaling with MongoDB

  • 2. Now a consultant, but formerly…  Software engineer at SourceForge, early adopter of MongoDB (version 0.8)  Wrote the SQLAlchemy book (I love SQL when it’s used well)  Mainly write Python now, but have done C++, C#, Java, Javascript, VHDL, Verilog, …
  • 3. You can do it with an RDBMS as long as you…  Don’t use joins  Don’t use transactions  Use read-only slaves  Use memcached  Denormalize your data  Use custom sharding/partitioning  Do a lot of vertical scaling ▪ (we’re going to need a bigger box)
  • 6. Use documents to improve locality  Optimize your indexes  Be aware of your working set  Scaling your disks  Replication for fault-tolerance and read scaling  Sharding for read and write scaling
  • 7. Relational (SQL) MongoDB Database Database Dynamic Typing Table Collection B-tree (range-based) Index Index Row Document Think JSON Column Field Primitive types + arrays, documents
  • 8. { title: "Slides for Scaling with MongoDB", author: "Rick Copeland", date: ISODate("20012-02-29T19:30:00Z"), text: "My slides are available on speakerdeck.com", comments: [ { author: "anonymous", date: ISODate("20012-02-29T19:30:01Z"), text: "Fristpsot!" }, { author: "mark”, date: ISODate("20012-02-29T19:45:23Z"), text: "Nice slides" } ] } Embed comment data in blog post document
  • 9. Seek = 5+ ms Read = really really fast
  • 10. Post Comment Author
  • 12. Find where x equals 7 1 2 3 4 5 6 7 Looked at 7 objects
  • 13. Find where x equals 7 4 2 6 1 3 5 7 Looked at 3 objects
  • 16. Working set =  sizeof(frequently used data)  + sizeof(frequently used indexes)  Right-aligned indexes reduce working set size  Working set should fit in available RAM for best performance  Page faults are the biggest cause of performance loss in MongoDB
  • 17. >db.foo.stats() Data Size { "ns" : "test.foo", "count" : 1338330, "size" : 46915928, Average doc size "avgObjSize" : 35.05557523181876, "storageSize" : 86092032, "numExtents" : 12, "nindexes" : 2, Size on disk (or RAM!) "lastExtentSize" : 20872960, "paddingFactor" : 1, "flags" : 0, Size of all indexes "totalIndexSize" : 99860480, "indexSizes" : { "_id_" : 55877632, "x_1" : 43982848}, "ok" : 1 Size of each index }
  • 18. ~200 seeks / second
  • 19. ~200 seeks / second ~200 seeks / second ~200 seeks / second  Faster, but less reliable
  • 20. ~400 seeks / second ~400 seeks / second ~400 seeks / second  Faster and more reliable ($$$ though)
  • 21. Old and busted  master/slave replication  The new hotness  replica sets with automatic failover Read / Write Primary Read Secondary Read Secondary
  • 22. Primary handles all writes  Application optionally sends reads to slaves  Heartbeat manages automatic failover
  • 23. Special collection (the oplog) records operations idempotently  Secondaries read from primary oplog and replay operations locally  Space is preallocated and fixed for the oplog
  • 24. { "ts" : Timestamp(1317653790000, 2), Insert "h" : -6022751846629753359, "op" : "i", "ns" : "confoo.People", Collection name "o" : { "_id" : ObjectId("4e89cd1e0364241932324269"), "first" : "Rick", "last" : "Copeland” } } Object to insert
  • 25. Use heartbeat signal to detect failure  When primary can’t be reached, elect a new one  Replica that’s the most up-to-date is chosen  If there is skew, changes not on new primary are saved to a .bson file for manual reconciliation  Application can require data to be replicated to a majority to ensure this doesn’t happen
  • 26. Priority  Slower nodes with lower priority  Backup or read-only nodes to never be primary  slaveDelay  Fat-finger protection  Data center awareness and tagging  Application can ensure complex replication guarantees
  • 27. Reads scale nicely  As long as the working set fits in RAM  … and you don’t mind eventual consistency  Sharding to the rescue!  Automatically partitioned data sets  Scale writes and reads  Automatic load balancing between the shards
  • 28. Configuration MongoS MongoS Config 1 Config 2 Config 3 Shard 1 Shard 2 Shard 3 Shard 4 0..10 10..20 20..30 30..40 Primary Primary Primary Primary Secondary Secondary Secondary Secondary Secondary Secondary Secondary Secondary
  • 29. Sharding is per-collection and range-based  The highest-impact choice (and hardest to change decision) you make is the shard key  Random keys: good for writes, bad for reads  Right-aligned index: bad for writes  Small # of discrete keys: very bad  Ideal: balance writes, make reads routable by mongos  Optimal shard key selection is hard
  • 30. Primary Data Center Secondary Data Center Shard 1 Shard 1 Shard 1 Priority 1 Priority 1 Priority 0 Shard 2 Shard 2 Shard 2 Priority 1 Priority 1 Priority 0 Shard 3 Shard 3 Shard 3 RS3 Priority 1 Priority 1 Priority 0 Config 1 Config 2 Config 3
  • 31. Writes and reads both scale (with good choice of shard key)  Reads scale while remaining strongly consistent  Partitioning ensures you get more usable RAM  Pitfall: don’t wait too long to add capacity

Editor's Notes

  1. You’d like to just ‘add capacity’ but you end up having to buy a bigger serverBuild your own infrastructure and you pay more for less as you scaleThe cloud can help with this, but only up to a point; what happens when you’re using the largest instance? Time to rearchitect.
  2. There are a lot of features that make RDBMSs attractiveBut as we scale we need to turn off a lot of them to get performance increasesWe end up with something that scales, but it’s hard to use
  3. RAM functions as a cacheReplication ends up caching documents in multiple locationsSharding makes sure documents only have one ‘home’
  4. A single shard is a replica setMongoS is a router that determines where reads and writes goDocuments is ‘chunked’ into ranges. Chunks can be split and migrated to other servers based on load.Configuration servers persist location of particular shard key ranges Cluster is alive when one or more config servers are down, but there can be no migration