SlideShare a Scribd company logo
1 of 29
Download to read offline
From Dev to Prod with Sharded Mongo DB Clusters
May 2013
Johan Andersson
Severalnines AB
johan@severalnines.com
Topics
¤  Best Practice Deployment
¤  Keeping it running
¤  Scaling with Shards
Note: Most features mentioned in this presentation are
applicable only on MongoDB 2.2 or later.
2
Copyright 2011 Severalnines AB
3
Copyright 2011 Severalnines AB
“An OS check-up a day keeps downtime away”
Ancient Chinese wisdom
Distributed Computing 101
¤  Network latency is not 0
¤  Network is unreliable
¤  Network bandwidth is not infinite
¤  CPUs are not infinitely fast
¤  Disks are not infinitely fast
¤  Free RAM decreases fast
¤  Virtualization does not make the underlying hardware more
powerful
4
Copyright 2011 Severalnines AB
MongoDB Concepts
¤  Mongos
¤  Query routers, routes queries to shards
¤  Caches data from Config Servers
¤  Config Servers
¤  Records which shard holds what chunks
¤  Shard Servers
¤  Stores the actual data
5
Copyright 2011 Severalnines AB
Developer Setup
¤  A good test setup for a developer:
¤  One or three config Servers
¤  Thee shard servers
¤  One or more Mongos
¤  But do set it up as a shard server so you can validate queries
(explain) to make sure they behave as intended.
¤  Using correct indexes
¤  Vagrant image -
http://alexyu.se/content/2013/03/vagrant-box-severalnines-
database-cluster-deployment-scripts
6
Copyright 2011 Severalnines AB
Test Setup
¤  Recommendations
¤  One or three config Servers
¤  >1 shard - thee shard servers
¤  One or more Mongos
¤  Stay close to your production system.
¤  Two shards (even if prod use one only) allows you to:
¤  Verify queries use the correct indexes
¤  Check if one or all shards are hit
¤  Extremely useful when adding shards to Prod to avoid
performance regression.
7
Copyright 2011 Severalnines AB
Production Setup
¤  Start with a good and sound architecture from day one!
¤  A sharded architecture will allow you to scale easily
¤  Avoid un-necessary craziness and risk to switch a non-
sharded setup to a sharded
¤  You do want three copies of your data
8
Copyright 2011 Severalnines AB
Production Setup
9
Copyright 2011 Severalnines AB
PRIMARY SECONDARY
repl
Production Setup
10
Copyright 2011 Severalnines AB
Hard to scale, too crowded
Go for Best Practice instead
PRIMARY SECONDARY
repl
Production Setup
11
Copyright 2011 Severalnines AB
…
Shard0 Shard1 ShardN
…
Production Setup
¤  Allows you to scale - Add more shards and mongos easily
¤  It is proven – used by many large organizations
¤  Deploy…
¤  Mongos
¤  Config servers
¤  Shard servers (mongod --shardsvr)
¤  … on individual servers
12
Copyright 2011 Severalnines AB
When processes fails
¤  Config Server process fails
¤  If one config server fails, no chunk migration between shards
¤  No chunk splitting
¤  Shards are still read/writeable
¤  Mongos fails
¤  You will have many
¤  Shard server fails
¤  If PRIMARY fails à One SECONDARY will become PRIMARY
¤  If SECONDADY fails à Restart and it will resync with PRIMARY
¤  Restart the failed process!
¤  Troll the logs to see it starts without errors
13
Copyright 2011 Severalnines AB
Keeping it running
¤  RAM is a scarce resource – keep as much as possible in it!
¤  Use short field names in the Documents to reduce size:
¤  { first_name:’not good’}
¤  {fn:’better’}
¤  Important when you have many Documents
¤  Compact collections:
¤  Run on SECONDARY
¤  db.mycollection.runCommand(“compact”)
¤  Stop PRIMARY, SECONDARY becomes new PRIMARY.
14
Copyright 2011 Severalnines AB
Keeping it running
¤  Replication Lag
¤  Producer / Consumer problem
¤  Temporary or Permanent Problem?
¤  Permanent problem
¤  Expunge the oplog?
¤  Magic wand?
¤  Disk, network, or load-dependent problem?
¤  Remedy:
¤  Faster disks: Improve disk subsystem -> IOPS or SSD, RAID configuration
¤  Faster CPUs
¤  Can you reduce writes to the shard(s)?
¤  Scale with shards
15
Copyright 2011 Severalnines AB
Keeping it running
¤  Page Faults
¤  MongoDB uses mmap:ed files, allocated from the VMM address space.
Mmapped files does not need to fit in RAM
¤  OS manages what pages are mapped in RAM and not
¤  Accessing data on a page not in RAM cause a page fault
¤  Data will be loaded from DISK
¤  DISK is slow
¤  Performance will degrade
¤  If the number of PFs is constant, no worry, else if increasing, problem in the
horizon.
¤  Remedy:
¤  Increase RAM in server (Active working set should fit in RAM)
¤  Scale with shards
¤  Add SSD
16
Copyright 2011 Severalnines AB
Not cool
17
Copyright 2011 Severalnines AB
Not cool
18
Copyright 2011 Severalnines AB
You are done for!
19
Copyright 2011 Severalnines AB
Avoid the situation
¤  Keep data set in RAM
¤  Use fast io-subsystem since OS is managing the paging
¤  SSD
¤  IOPS on EC2
¤  Make sure the Host OS (Virtualized env) does not swap.
¤  Lock mongod’s to CPUs, check /etc/interrupts, bind to CPUs
not handling ETH interrupts, avoids ctx switching.
¤  Add shards before its too late
¤  More disks/CPUs,RAM to handle the load
Copyright 2011 Severalnines AB
20
Scaling with Shards
¤  Shard Keys
¤  When to scale?
¤  Add a shard
21
Copyright 2011 Severalnines AB
Database Size
¤  db.stats()
¤  Size of one db = dbStats.dataSize + dbStats.indexSize
¤  Does it fit in RAM?
¤  Plan ahead:
¤  avgObjSize x expectedNoOfDocuments
¤  Does it fit in RAM?
¤  How many documents can you store?
22
Copyright 2011 Severalnines AB
Working set
¤  Working Set Analyzer in 2.4 helps:
!"workingSet" : {!
! !"note" : "thisIsAnEstimate",!
! !"pagesInMemory" : 33688, //4KB pg_size!
! !"computationTimeMicros" : 15283,!
! !"overSeconds" : 1923 //dist pg_new to pg_oldest

},!
¤  Trend overSeconds and pagesInMemory
23
Copyright 2011 Severalnines AB
Shard Key
¤  Collections must have a Shard Key
¤  The shard key must be indexed
¤  Hash-based or Range based Sharding
¤  Hash-bashed great for exact match
¤  Subscriber/user databases
¤  Queries must use the shard key
¤  If not, all shards will be queried à slow
¤  Check and verify with .explain()
24
Copyright 2011 Severalnines AB
Adding Shards
¤  Adding a new Shard is easy:
¤  Deploy a new Replica-set:
¤  PRIMARY - 192.168.0.151
SECONDARY - 192.168.0.152
SECONDARY - 192.168.0.153
¤  Register the Replica Set:
¤  sh.addShard(“myshard2/192.168.0.151:27017”);
¤  Finding the time to add shards is harder
¤  Rebalancing costs and query performance will suffer due to
chunk migration.
¤  Do it off-peak if possible
¤  MongoDB will automatically rebalance the shards.
25
Copyright 2011 Severalnines AB
Q&A
26
Copyright 2011 Severalnines AB
Resources
27
Copyright 2011 Severalnines AB
¤  MongoDB Configurator
http://www.severalnines.com/mongodb-configurator/
¤  -ClusterControl for MongoDB
http://www.severalnines.com/resources/cmon-mongodb-
cluster-management-monitoring-tool
¤  Installing ClusterControl on top of existing MongoDB Cluster
http://www.severalnines.com/blog/install-clustercontrol-top-
existing-mongodb-sharded-cluster
¤  Vagrant image -
http://alexyu.se/content/2013/03/vagrant-box-severalnines-
database-cluster-deployment-scripts
Follow us on
28
Copyright 2011 Severalnines AB
¤  Twitter : @severalnines
¤  Facebook: www.facebook.com/severalnines
¤  Slideshare : www.slideshare.net/severalnines
¤  Linked-in: www.linkedin.com/company/severalnines
Thank you!
29
Copyright 2011 Severalnines AB

More Related Content

What's hot

Date-tiered Compaction Policy for Time-series Data
Date-tiered Compaction Policy for Time-series DataDate-tiered Compaction Policy for Time-series Data
Date-tiered Compaction Policy for Time-series DataHBaseCon
 
MongoDB San Francisco 2013: Basic Sharding in MongoDB presented by Brandon Bl...
MongoDB San Francisco 2013: Basic Sharding in MongoDB presented by Brandon Bl...MongoDB San Francisco 2013: Basic Sharding in MongoDB presented by Brandon Bl...
MongoDB San Francisco 2013: Basic Sharding in MongoDB presented by Brandon Bl...MongoDB
 
Cassandra TK 2014 - Large Nodes
Cassandra TK 2014 - Large NodesCassandra TK 2014 - Large Nodes
Cassandra TK 2014 - Large Nodesaaronmorton
 
Understanding and tuning WiredTiger, the new high performance database engine...
Understanding and tuning WiredTiger, the new high performance database engine...Understanding and tuning WiredTiger, the new high performance database engine...
Understanding and tuning WiredTiger, the new high performance database engine...Ontico
 
Managing Cassandra at Scale by Al Tobey
Managing Cassandra at Scale by Al TobeyManaging Cassandra at Scale by Al Tobey
Managing Cassandra at Scale by Al TobeyDataStax Academy
 
Basic Sharding in MongoDB presented by Shaun Verch
Basic Sharding in MongoDB presented by Shaun VerchBasic Sharding in MongoDB presented by Shaun Verch
Basic Sharding in MongoDB presented by Shaun VerchMongoDB
 
Scaling Cassandra for Big Data
Scaling Cassandra for Big DataScaling Cassandra for Big Data
Scaling Cassandra for Big DataDataStax Academy
 
Cassandra Summit 2014: Lesser Known Features of Cassandra 2.1
Cassandra Summit 2014: Lesser Known Features of Cassandra 2.1Cassandra Summit 2014: Lesser Known Features of Cassandra 2.1
Cassandra Summit 2014: Lesser Known Features of Cassandra 2.1DataStax Academy
 
Optimizing MongoDB: Lessons Learned at Localytics
Optimizing MongoDB: Lessons Learned at LocalyticsOptimizing MongoDB: Lessons Learned at Localytics
Optimizing MongoDB: Lessons Learned at LocalyticsBenjamin Darfler
 
Lessons Learned on Java Tuning for Our Cassandra Clusters (Carlos Monroy, Kne...
Lessons Learned on Java Tuning for Our Cassandra Clusters (Carlos Monroy, Kne...Lessons Learned on Java Tuning for Our Cassandra Clusters (Carlos Monroy, Kne...
Lessons Learned on Java Tuning for Our Cassandra Clusters (Carlos Monroy, Kne...DataStax
 
Keynote: Apache HBase at Yahoo! Scale
Keynote: Apache HBase at Yahoo! ScaleKeynote: Apache HBase at Yahoo! Scale
Keynote: Apache HBase at Yahoo! ScaleHBaseCon
 
Sharding with MongoDB (Eliot Horowitz)
Sharding with MongoDB (Eliot Horowitz)Sharding with MongoDB (Eliot Horowitz)
Sharding with MongoDB (Eliot Horowitz)MongoSF
 
Performance tuning - A key to successful cassandra migration
Performance tuning - A key to successful cassandra migrationPerformance tuning - A key to successful cassandra migration
Performance tuning - A key to successful cassandra migrationRamkumar Nottath
 
Troubleshooting redis
Troubleshooting redisTroubleshooting redis
Troubleshooting redisDaeMyung Kang
 
Evolution of MonogDB Sharding and Its Best Practices - Ranjith A - Mydbops Team
Evolution of MonogDB Sharding and Its Best Practices - Ranjith A - Mydbops TeamEvolution of MonogDB Sharding and Its Best Practices - Ranjith A - Mydbops Team
Evolution of MonogDB Sharding and Its Best Practices - Ranjith A - Mydbops TeamMydbops
 
A Detailed Look At cassandra.yaml (Edward Capriolo, The Last Pickle) | Cassan...
A Detailed Look At cassandra.yaml (Edward Capriolo, The Last Pickle) | Cassan...A Detailed Look At cassandra.yaml (Edward Capriolo, The Last Pickle) | Cassan...
A Detailed Look At cassandra.yaml (Edward Capriolo, The Last Pickle) | Cassan...DataStax
 
Mongodb - Scaling write performance
Mongodb - Scaling write performanceMongodb - Scaling write performance
Mongodb - Scaling write performanceDaum DNA
 
Tales from Taming the Long Tail
Tales from Taming the Long TailTales from Taming the Long Tail
Tales from Taming the Long TailHBaseCon
 
Deploying postgre sql on amazon ec2
Deploying postgre sql on amazon ec2 Deploying postgre sql on amazon ec2
Deploying postgre sql on amazon ec2 Denish Patel
 

What's hot (20)

Date-tiered Compaction Policy for Time-series Data
Date-tiered Compaction Policy for Time-series DataDate-tiered Compaction Policy for Time-series Data
Date-tiered Compaction Policy for Time-series Data
 
MongoDB San Francisco 2013: Basic Sharding in MongoDB presented by Brandon Bl...
MongoDB San Francisco 2013: Basic Sharding in MongoDB presented by Brandon Bl...MongoDB San Francisco 2013: Basic Sharding in MongoDB presented by Brandon Bl...
MongoDB San Francisco 2013: Basic Sharding in MongoDB presented by Brandon Bl...
 
Cassandra TK 2014 - Large Nodes
Cassandra TK 2014 - Large NodesCassandra TK 2014 - Large Nodes
Cassandra TK 2014 - Large Nodes
 
Understanding and tuning WiredTiger, the new high performance database engine...
Understanding and tuning WiredTiger, the new high performance database engine...Understanding and tuning WiredTiger, the new high performance database engine...
Understanding and tuning WiredTiger, the new high performance database engine...
 
Managing Cassandra at Scale by Al Tobey
Managing Cassandra at Scale by Al TobeyManaging Cassandra at Scale by Al Tobey
Managing Cassandra at Scale by Al Tobey
 
Basic Sharding in MongoDB presented by Shaun Verch
Basic Sharding in MongoDB presented by Shaun VerchBasic Sharding in MongoDB presented by Shaun Verch
Basic Sharding in MongoDB presented by Shaun Verch
 
Scaling Cassandra for Big Data
Scaling Cassandra for Big DataScaling Cassandra for Big Data
Scaling Cassandra for Big Data
 
Cassandra Summit 2014: Lesser Known Features of Cassandra 2.1
Cassandra Summit 2014: Lesser Known Features of Cassandra 2.1Cassandra Summit 2014: Lesser Known Features of Cassandra 2.1
Cassandra Summit 2014: Lesser Known Features of Cassandra 2.1
 
Optimizing MongoDB: Lessons Learned at Localytics
Optimizing MongoDB: Lessons Learned at LocalyticsOptimizing MongoDB: Lessons Learned at Localytics
Optimizing MongoDB: Lessons Learned at Localytics
 
Lessons Learned on Java Tuning for Our Cassandra Clusters (Carlos Monroy, Kne...
Lessons Learned on Java Tuning for Our Cassandra Clusters (Carlos Monroy, Kne...Lessons Learned on Java Tuning for Our Cassandra Clusters (Carlos Monroy, Kne...
Lessons Learned on Java Tuning for Our Cassandra Clusters (Carlos Monroy, Kne...
 
Keynote: Apache HBase at Yahoo! Scale
Keynote: Apache HBase at Yahoo! ScaleKeynote: Apache HBase at Yahoo! Scale
Keynote: Apache HBase at Yahoo! Scale
 
Sharding with MongoDB (Eliot Horowitz)
Sharding with MongoDB (Eliot Horowitz)Sharding with MongoDB (Eliot Horowitz)
Sharding with MongoDB (Eliot Horowitz)
 
Performance tuning - A key to successful cassandra migration
Performance tuning - A key to successful cassandra migrationPerformance tuning - A key to successful cassandra migration
Performance tuning - A key to successful cassandra migration
 
Really Big Elephants: PostgreSQL DW
Really Big Elephants: PostgreSQL DWReally Big Elephants: PostgreSQL DW
Really Big Elephants: PostgreSQL DW
 
Troubleshooting redis
Troubleshooting redisTroubleshooting redis
Troubleshooting redis
 
Evolution of MonogDB Sharding and Its Best Practices - Ranjith A - Mydbops Team
Evolution of MonogDB Sharding and Its Best Practices - Ranjith A - Mydbops TeamEvolution of MonogDB Sharding and Its Best Practices - Ranjith A - Mydbops Team
Evolution of MonogDB Sharding and Its Best Practices - Ranjith A - Mydbops Team
 
A Detailed Look At cassandra.yaml (Edward Capriolo, The Last Pickle) | Cassan...
A Detailed Look At cassandra.yaml (Edward Capriolo, The Last Pickle) | Cassan...A Detailed Look At cassandra.yaml (Edward Capriolo, The Last Pickle) | Cassan...
A Detailed Look At cassandra.yaml (Edward Capriolo, The Last Pickle) | Cassan...
 
Mongodb - Scaling write performance
Mongodb - Scaling write performanceMongodb - Scaling write performance
Mongodb - Scaling write performance
 
Tales from Taming the Long Tail
Tales from Taming the Long TailTales from Taming the Long Tail
Tales from Taming the Long Tail
 
Deploying postgre sql on amazon ec2
Deploying postgre sql on amazon ec2 Deploying postgre sql on amazon ec2
Deploying postgre sql on amazon ec2
 

Similar to Development to Production with Sharded MongoDB Clusters

Ceph Day Shanghai - SSD/NVM Technology Boosting Ceph Performance
Ceph Day Shanghai - SSD/NVM Technology Boosting Ceph Performance Ceph Day Shanghai - SSD/NVM Technology Boosting Ceph Performance
Ceph Day Shanghai - SSD/NVM Technology Boosting Ceph Performance Ceph Community
 
Getting The Most Out Of Your Flash/SSDs
Getting The Most Out Of Your Flash/SSDsGetting The Most Out Of Your Flash/SSDs
Getting The Most Out Of Your Flash/SSDsAerospike, Inc.
 
Ceph Community Talk on High-Performance Solid Sate Ceph
Ceph Community Talk on High-Performance Solid Sate Ceph Ceph Community Talk on High-Performance Solid Sate Ceph
Ceph Community Talk on High-Performance Solid Sate Ceph Ceph Community
 
Severalnines Training: MySQL® Cluster - Part IX
Severalnines Training: MySQL® Cluster - Part IXSeveralnines Training: MySQL® Cluster - Part IX
Severalnines Training: MySQL® Cluster - Part IXSeveralnines
 
Delivering Apache Hadoop for the Modern Data Architecture
Delivering Apache Hadoop for the Modern Data Architecture Delivering Apache Hadoop for the Modern Data Architecture
Delivering Apache Hadoop for the Modern Data Architecture Hortonworks
 
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...Odinot Stanislas
 
Performance Whack-a-Mole Tutorial (pgCon 2009)
Performance Whack-a-Mole Tutorial (pgCon 2009) Performance Whack-a-Mole Tutorial (pgCon 2009)
Performance Whack-a-Mole Tutorial (pgCon 2009) PostgreSQL Experts, Inc.
 
SQLintersection keynote a tale of two teams
SQLintersection keynote a tale of two teamsSQLintersection keynote a tale of two teams
SQLintersection keynote a tale of two teamsSumeet Bansal
 
Rails Conf Europe 2007 Notes
Rails Conf  Europe 2007  NotesRails Conf  Europe 2007  Notes
Rails Conf Europe 2007 NotesRoss Lawley
 
Why does my choice of storage matter with cassandra?
Why does my choice of storage matter with cassandra?Why does my choice of storage matter with cassandra?
Why does my choice of storage matter with cassandra?Johnny Miller
 
Using ТРСС to study Firebird performance
Using ТРСС to study Firebird performanceUsing ТРСС to study Firebird performance
Using ТРСС to study Firebird performanceMind The Firebird
 
VMworld 2013: Just Because You Could, Doesn't Mean You Should: Lessons Learne...
VMworld 2013: Just Because You Could, Doesn't Mean You Should: Lessons Learne...VMworld 2013: Just Because You Could, Doesn't Mean You Should: Lessons Learne...
VMworld 2013: Just Because You Could, Doesn't Mean You Should: Lessons Learne...VMworld
 
QCon2016--Drive Best Spark Performance on AI
QCon2016--Drive Best Spark Performance on AIQCon2016--Drive Best Spark Performance on AI
QCon2016--Drive Best Spark Performance on AILex Yu
 
Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...
Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...
Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...Severalnines
 
Presentation architecting a cloud infrastructure
Presentation   architecting a cloud infrastructurePresentation   architecting a cloud infrastructure
Presentation architecting a cloud infrastructuresolarisyourep
 
Presentation architecting a cloud infrastructure
Presentation   architecting a cloud infrastructurePresentation   architecting a cloud infrastructure
Presentation architecting a cloud infrastructurexKinAnx
 

Similar to Development to Production with Sharded MongoDB Clusters (20)

Ceph Day Shanghai - SSD/NVM Technology Boosting Ceph Performance
Ceph Day Shanghai - SSD/NVM Technology Boosting Ceph Performance Ceph Day Shanghai - SSD/NVM Technology Boosting Ceph Performance
Ceph Day Shanghai - SSD/NVM Technology Boosting Ceph Performance
 
Getting The Most Out Of Your Flash/SSDs
Getting The Most Out Of Your Flash/SSDsGetting The Most Out Of Your Flash/SSDs
Getting The Most Out Of Your Flash/SSDs
 
Ceph Community Talk on High-Performance Solid Sate Ceph
Ceph Community Talk on High-Performance Solid Sate Ceph Ceph Community Talk on High-Performance Solid Sate Ceph
Ceph Community Talk on High-Performance Solid Sate Ceph
 
Severalnines Training: MySQL® Cluster - Part IX
Severalnines Training: MySQL® Cluster - Part IXSeveralnines Training: MySQL® Cluster - Part IX
Severalnines Training: MySQL® Cluster - Part IX
 
Delivering Apache Hadoop for the Modern Data Architecture
Delivering Apache Hadoop for the Modern Data Architecture Delivering Apache Hadoop for the Modern Data Architecture
Delivering Apache Hadoop for the Modern Data Architecture
 
Ceph
CephCeph
Ceph
 
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
 
Performance Whack-a-Mole Tutorial (pgCon 2009)
Performance Whack-a-Mole Tutorial (pgCon 2009) Performance Whack-a-Mole Tutorial (pgCon 2009)
Performance Whack-a-Mole Tutorial (pgCon 2009)
 
SQLintersection keynote a tale of two teams
SQLintersection keynote a tale of two teamsSQLintersection keynote a tale of two teams
SQLintersection keynote a tale of two teams
 
Rails Conf Europe 2007 Notes
Rails Conf  Europe 2007  NotesRails Conf  Europe 2007  Notes
Rails Conf Europe 2007 Notes
 
Why does my choice of storage matter with cassandra?
Why does my choice of storage matter with cassandra?Why does my choice of storage matter with cassandra?
Why does my choice of storage matter with cassandra?
 
Using ТРСС to study Firebird performance
Using ТРСС to study Firebird performanceUsing ТРСС to study Firebird performance
Using ТРСС to study Firebird performance
 
5 Steps to PostgreSQL Performance
5 Steps to PostgreSQL Performance5 Steps to PostgreSQL Performance
5 Steps to PostgreSQL Performance
 
Five steps perform_2009 (1)
Five steps perform_2009 (1)Five steps perform_2009 (1)
Five steps perform_2009 (1)
 
VMworld 2013: Just Because You Could, Doesn't Mean You Should: Lessons Learne...
VMworld 2013: Just Because You Could, Doesn't Mean You Should: Lessons Learne...VMworld 2013: Just Because You Could, Doesn't Mean You Should: Lessons Learne...
VMworld 2013: Just Because You Could, Doesn't Mean You Should: Lessons Learne...
 
QCon2016--Drive Best Spark Performance on AI
QCon2016--Drive Best Spark Performance on AIQCon2016--Drive Best Spark Performance on AI
QCon2016--Drive Best Spark Performance on AI
 
Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...
Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...
Webinar slides: The Holy Grail Webinar: Become a MySQL DBA - Database Perform...
 
Presentation architecting a cloud infrastructure
Presentation   architecting a cloud infrastructurePresentation   architecting a cloud infrastructure
Presentation architecting a cloud infrastructure
 
Presentation architecting a cloud infrastructure
Presentation   architecting a cloud infrastructurePresentation   architecting a cloud infrastructure
Presentation architecting a cloud infrastructure
 
MySQL Tuning
MySQL TuningMySQL Tuning
MySQL Tuning
 

More from Severalnines

Cloud's future runs through Sovereign DBaaS
Cloud's future runs through Sovereign DBaaSCloud's future runs through Sovereign DBaaS
Cloud's future runs through Sovereign DBaaSSeveralnines
 
Tips to drive maria db cluster performance for nextcloud
Tips to drive maria db cluster performance for nextcloudTips to drive maria db cluster performance for nextcloud
Tips to drive maria db cluster performance for nextcloudSeveralnines
 
Working with the Moodle Database: The Basics
Working with the Moodle Database: The BasicsWorking with the Moodle Database: The Basics
Working with the Moodle Database: The BasicsSeveralnines
 
SysAdmin Working from Home? Tips to Automate MySQL, MariaDB, Postgres & MongoDB
SysAdmin Working from Home? Tips to Automate MySQL, MariaDB, Postgres & MongoDBSysAdmin Working from Home? Tips to Automate MySQL, MariaDB, Postgres & MongoDB
SysAdmin Working from Home? Tips to Automate MySQL, MariaDB, Postgres & MongoDBSeveralnines
 
(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...
(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...
(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...Severalnines
 
Webinar slides: How to Migrate from Oracle DB to MariaDB
Webinar slides: How to Migrate from Oracle DB to MariaDBWebinar slides: How to Migrate from Oracle DB to MariaDB
Webinar slides: How to Migrate from Oracle DB to MariaDBSeveralnines
 
Webinar slides: How to Automate & Manage PostgreSQL with ClusterControl
Webinar slides: How to Automate & Manage PostgreSQL with ClusterControlWebinar slides: How to Automate & Manage PostgreSQL with ClusterControl
Webinar slides: How to Automate & Manage PostgreSQL with ClusterControlSeveralnines
 
Webinar slides: How to Manage Replication Failover Processes for MySQL, Maria...
Webinar slides: How to Manage Replication Failover Processes for MySQL, Maria...Webinar slides: How to Manage Replication Failover Processes for MySQL, Maria...
Webinar slides: How to Manage Replication Failover Processes for MySQL, Maria...Severalnines
 
Webinar slides: Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB wi...
Webinar slides: Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB wi...Webinar slides: Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB wi...
Webinar slides: Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB wi...Severalnines
 
Disaster Recovery Planning for MySQL & MariaDB
Disaster Recovery Planning for MySQL & MariaDBDisaster Recovery Planning for MySQL & MariaDB
Disaster Recovery Planning for MySQL & MariaDBSeveralnines
 
MariaDB Performance Tuning Crash Course
MariaDB Performance Tuning Crash CourseMariaDB Performance Tuning Crash Course
MariaDB Performance Tuning Crash CourseSeveralnines
 
Performance Tuning Cheat Sheet for MongoDB
Performance Tuning Cheat Sheet for MongoDBPerformance Tuning Cheat Sheet for MongoDB
Performance Tuning Cheat Sheet for MongoDBSeveralnines
 
Advanced MySql Data-at-Rest Encryption in Percona Server
Advanced MySql Data-at-Rest Encryption in Percona ServerAdvanced MySql Data-at-Rest Encryption in Percona Server
Advanced MySql Data-at-Rest Encryption in Percona ServerSeveralnines
 
Polyglot Persistence Utilizing Open Source Databases as a Swiss Pocket Knife
Polyglot Persistence Utilizing Open Source Databases as a Swiss Pocket KnifePolyglot Persistence Utilizing Open Source Databases as a Swiss Pocket Knife
Polyglot Persistence Utilizing Open Source Databases as a Swiss Pocket KnifeSeveralnines
 
Webinar slides: Free Monitoring (on Steroids) for MySQL, MariaDB, PostgreSQL ...
Webinar slides: Free Monitoring (on Steroids) for MySQL, MariaDB, PostgreSQL ...Webinar slides: Free Monitoring (on Steroids) for MySQL, MariaDB, PostgreSQL ...
Webinar slides: Free Monitoring (on Steroids) for MySQL, MariaDB, PostgreSQL ...Severalnines
 
Webinar slides: An Introduction to Performance Monitoring for PostgreSQL
Webinar slides: An Introduction to Performance Monitoring for PostgreSQLWebinar slides: An Introduction to Performance Monitoring for PostgreSQL
Webinar slides: An Introduction to Performance Monitoring for PostgreSQLSeveralnines
 
Webinar slides: Our Guide to MySQL & MariaDB Performance Tuning
Webinar slides: Our Guide to MySQL & MariaDB Performance TuningWebinar slides: Our Guide to MySQL & MariaDB Performance Tuning
Webinar slides: Our Guide to MySQL & MariaDB Performance TuningSeveralnines
 
Webinar slides: Migrating to Galera Cluster for MySQL and MariaDB
Webinar slides: Migrating to Galera Cluster for MySQL and MariaDBWebinar slides: Migrating to Galera Cluster for MySQL and MariaDB
Webinar slides: Migrating to Galera Cluster for MySQL and MariaDBSeveralnines
 
Webinar slides: How to Measure Database Availability?
Webinar slides: How to Measure Database Availability?Webinar slides: How to Measure Database Availability?
Webinar slides: How to Measure Database Availability?Severalnines
 
Webinar slides: Designing Open Source Databases for High Availability
Webinar slides: Designing Open Source Databases for High AvailabilityWebinar slides: Designing Open Source Databases for High Availability
Webinar slides: Designing Open Source Databases for High AvailabilitySeveralnines
 

More from Severalnines (20)

Cloud's future runs through Sovereign DBaaS
Cloud's future runs through Sovereign DBaaSCloud's future runs through Sovereign DBaaS
Cloud's future runs through Sovereign DBaaS
 
Tips to drive maria db cluster performance for nextcloud
Tips to drive maria db cluster performance for nextcloudTips to drive maria db cluster performance for nextcloud
Tips to drive maria db cluster performance for nextcloud
 
Working with the Moodle Database: The Basics
Working with the Moodle Database: The BasicsWorking with the Moodle Database: The Basics
Working with the Moodle Database: The Basics
 
SysAdmin Working from Home? Tips to Automate MySQL, MariaDB, Postgres & MongoDB
SysAdmin Working from Home? Tips to Automate MySQL, MariaDB, Postgres & MongoDBSysAdmin Working from Home? Tips to Automate MySQL, MariaDB, Postgres & MongoDB
SysAdmin Working from Home? Tips to Automate MySQL, MariaDB, Postgres & MongoDB
 
(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...
(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...
(slides) Polyglot persistence: utilizing open source databases as a Swiss poc...
 
Webinar slides: How to Migrate from Oracle DB to MariaDB
Webinar slides: How to Migrate from Oracle DB to MariaDBWebinar slides: How to Migrate from Oracle DB to MariaDB
Webinar slides: How to Migrate from Oracle DB to MariaDB
 
Webinar slides: How to Automate & Manage PostgreSQL with ClusterControl
Webinar slides: How to Automate & Manage PostgreSQL with ClusterControlWebinar slides: How to Automate & Manage PostgreSQL with ClusterControl
Webinar slides: How to Automate & Manage PostgreSQL with ClusterControl
 
Webinar slides: How to Manage Replication Failover Processes for MySQL, Maria...
Webinar slides: How to Manage Replication Failover Processes for MySQL, Maria...Webinar slides: How to Manage Replication Failover Processes for MySQL, Maria...
Webinar slides: How to Manage Replication Failover Processes for MySQL, Maria...
 
Webinar slides: Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB wi...
Webinar slides: Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB wi...Webinar slides: Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB wi...
Webinar slides: Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB wi...
 
Disaster Recovery Planning for MySQL & MariaDB
Disaster Recovery Planning for MySQL & MariaDBDisaster Recovery Planning for MySQL & MariaDB
Disaster Recovery Planning for MySQL & MariaDB
 
MariaDB Performance Tuning Crash Course
MariaDB Performance Tuning Crash CourseMariaDB Performance Tuning Crash Course
MariaDB Performance Tuning Crash Course
 
Performance Tuning Cheat Sheet for MongoDB
Performance Tuning Cheat Sheet for MongoDBPerformance Tuning Cheat Sheet for MongoDB
Performance Tuning Cheat Sheet for MongoDB
 
Advanced MySql Data-at-Rest Encryption in Percona Server
Advanced MySql Data-at-Rest Encryption in Percona ServerAdvanced MySql Data-at-Rest Encryption in Percona Server
Advanced MySql Data-at-Rest Encryption in Percona Server
 
Polyglot Persistence Utilizing Open Source Databases as a Swiss Pocket Knife
Polyglot Persistence Utilizing Open Source Databases as a Swiss Pocket KnifePolyglot Persistence Utilizing Open Source Databases as a Swiss Pocket Knife
Polyglot Persistence Utilizing Open Source Databases as a Swiss Pocket Knife
 
Webinar slides: Free Monitoring (on Steroids) for MySQL, MariaDB, PostgreSQL ...
Webinar slides: Free Monitoring (on Steroids) for MySQL, MariaDB, PostgreSQL ...Webinar slides: Free Monitoring (on Steroids) for MySQL, MariaDB, PostgreSQL ...
Webinar slides: Free Monitoring (on Steroids) for MySQL, MariaDB, PostgreSQL ...
 
Webinar slides: An Introduction to Performance Monitoring for PostgreSQL
Webinar slides: An Introduction to Performance Monitoring for PostgreSQLWebinar slides: An Introduction to Performance Monitoring for PostgreSQL
Webinar slides: An Introduction to Performance Monitoring for PostgreSQL
 
Webinar slides: Our Guide to MySQL & MariaDB Performance Tuning
Webinar slides: Our Guide to MySQL & MariaDB Performance TuningWebinar slides: Our Guide to MySQL & MariaDB Performance Tuning
Webinar slides: Our Guide to MySQL & MariaDB Performance Tuning
 
Webinar slides: Migrating to Galera Cluster for MySQL and MariaDB
Webinar slides: Migrating to Galera Cluster for MySQL and MariaDBWebinar slides: Migrating to Galera Cluster for MySQL and MariaDB
Webinar slides: Migrating to Galera Cluster for MySQL and MariaDB
 
Webinar slides: How to Measure Database Availability?
Webinar slides: How to Measure Database Availability?Webinar slides: How to Measure Database Availability?
Webinar slides: How to Measure Database Availability?
 
Webinar slides: Designing Open Source Databases for High Availability
Webinar slides: Designing Open Source Databases for High AvailabilityWebinar slides: Designing Open Source Databases for High Availability
Webinar slides: Designing Open Source Databases for High Availability
 

Recently uploaded

الأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهلهالأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهلهMohamed Sweelam
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform EngineeringMarcus Vechiato
 
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)Samir Dash
 
Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data SciencePaolo Missier
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch TuesdayIvanti
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!Memoori
 
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptxCyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptxMasterG
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...ScyllaDB
 
Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMKumar Satyam
 
How to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfHow to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfdanishmna97
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsLeah Henrickson
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfalexjohnson7307
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxFIDO Alliance
 
ChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps ProductivityChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps ProductivityVictorSzoltysek
 
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe中 央社
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptxFIDO Alliance
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
AI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by AnitarajAI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by AnitarajAnitaRaj43
 

Recently uploaded (20)

الأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهلهالأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهله
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
 
Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data Science
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptxCyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
 
Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDM
 
How to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfHow to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cf
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdf
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
 
ChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps ProductivityChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps Productivity
 
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
AI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by AnitarajAI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by Anitaraj
 

Development to Production with Sharded MongoDB Clusters

  • 1. From Dev to Prod with Sharded Mongo DB Clusters May 2013 Johan Andersson Severalnines AB johan@severalnines.com
  • 2. Topics ¤  Best Practice Deployment ¤  Keeping it running ¤  Scaling with Shards Note: Most features mentioned in this presentation are applicable only on MongoDB 2.2 or later. 2 Copyright 2011 Severalnines AB
  • 3. 3 Copyright 2011 Severalnines AB “An OS check-up a day keeps downtime away” Ancient Chinese wisdom
  • 4. Distributed Computing 101 ¤  Network latency is not 0 ¤  Network is unreliable ¤  Network bandwidth is not infinite ¤  CPUs are not infinitely fast ¤  Disks are not infinitely fast ¤  Free RAM decreases fast ¤  Virtualization does not make the underlying hardware more powerful 4 Copyright 2011 Severalnines AB
  • 5. MongoDB Concepts ¤  Mongos ¤  Query routers, routes queries to shards ¤  Caches data from Config Servers ¤  Config Servers ¤  Records which shard holds what chunks ¤  Shard Servers ¤  Stores the actual data 5 Copyright 2011 Severalnines AB
  • 6. Developer Setup ¤  A good test setup for a developer: ¤  One or three config Servers ¤  Thee shard servers ¤  One or more Mongos ¤  But do set it up as a shard server so you can validate queries (explain) to make sure they behave as intended. ¤  Using correct indexes ¤  Vagrant image - http://alexyu.se/content/2013/03/vagrant-box-severalnines- database-cluster-deployment-scripts 6 Copyright 2011 Severalnines AB
  • 7. Test Setup ¤  Recommendations ¤  One or three config Servers ¤  >1 shard - thee shard servers ¤  One or more Mongos ¤  Stay close to your production system. ¤  Two shards (even if prod use one only) allows you to: ¤  Verify queries use the correct indexes ¤  Check if one or all shards are hit ¤  Extremely useful when adding shards to Prod to avoid performance regression. 7 Copyright 2011 Severalnines AB
  • 8. Production Setup ¤  Start with a good and sound architecture from day one! ¤  A sharded architecture will allow you to scale easily ¤  Avoid un-necessary craziness and risk to switch a non- sharded setup to a sharded ¤  You do want three copies of your data 8 Copyright 2011 Severalnines AB
  • 9. Production Setup 9 Copyright 2011 Severalnines AB PRIMARY SECONDARY repl
  • 10. Production Setup 10 Copyright 2011 Severalnines AB Hard to scale, too crowded Go for Best Practice instead PRIMARY SECONDARY repl
  • 11. Production Setup 11 Copyright 2011 Severalnines AB … Shard0 Shard1 ShardN …
  • 12. Production Setup ¤  Allows you to scale - Add more shards and mongos easily ¤  It is proven – used by many large organizations ¤  Deploy… ¤  Mongos ¤  Config servers ¤  Shard servers (mongod --shardsvr) ¤  … on individual servers 12 Copyright 2011 Severalnines AB
  • 13. When processes fails ¤  Config Server process fails ¤  If one config server fails, no chunk migration between shards ¤  No chunk splitting ¤  Shards are still read/writeable ¤  Mongos fails ¤  You will have many ¤  Shard server fails ¤  If PRIMARY fails à One SECONDARY will become PRIMARY ¤  If SECONDADY fails à Restart and it will resync with PRIMARY ¤  Restart the failed process! ¤  Troll the logs to see it starts without errors 13 Copyright 2011 Severalnines AB
  • 14. Keeping it running ¤  RAM is a scarce resource – keep as much as possible in it! ¤  Use short field names in the Documents to reduce size: ¤  { first_name:’not good’} ¤  {fn:’better’} ¤  Important when you have many Documents ¤  Compact collections: ¤  Run on SECONDARY ¤  db.mycollection.runCommand(“compact”) ¤  Stop PRIMARY, SECONDARY becomes new PRIMARY. 14 Copyright 2011 Severalnines AB
  • 15. Keeping it running ¤  Replication Lag ¤  Producer / Consumer problem ¤  Temporary or Permanent Problem? ¤  Permanent problem ¤  Expunge the oplog? ¤  Magic wand? ¤  Disk, network, or load-dependent problem? ¤  Remedy: ¤  Faster disks: Improve disk subsystem -> IOPS or SSD, RAID configuration ¤  Faster CPUs ¤  Can you reduce writes to the shard(s)? ¤  Scale with shards 15 Copyright 2011 Severalnines AB
  • 16. Keeping it running ¤  Page Faults ¤  MongoDB uses mmap:ed files, allocated from the VMM address space. Mmapped files does not need to fit in RAM ¤  OS manages what pages are mapped in RAM and not ¤  Accessing data on a page not in RAM cause a page fault ¤  Data will be loaded from DISK ¤  DISK is slow ¤  Performance will degrade ¤  If the number of PFs is constant, no worry, else if increasing, problem in the horizon. ¤  Remedy: ¤  Increase RAM in server (Active working set should fit in RAM) ¤  Scale with shards ¤  Add SSD 16 Copyright 2011 Severalnines AB
  • 17. Not cool 17 Copyright 2011 Severalnines AB
  • 18. Not cool 18 Copyright 2011 Severalnines AB
  • 19. You are done for! 19 Copyright 2011 Severalnines AB
  • 20. Avoid the situation ¤  Keep data set in RAM ¤  Use fast io-subsystem since OS is managing the paging ¤  SSD ¤  IOPS on EC2 ¤  Make sure the Host OS (Virtualized env) does not swap. ¤  Lock mongod’s to CPUs, check /etc/interrupts, bind to CPUs not handling ETH interrupts, avoids ctx switching. ¤  Add shards before its too late ¤  More disks/CPUs,RAM to handle the load Copyright 2011 Severalnines AB 20
  • 21. Scaling with Shards ¤  Shard Keys ¤  When to scale? ¤  Add a shard 21 Copyright 2011 Severalnines AB
  • 22. Database Size ¤  db.stats() ¤  Size of one db = dbStats.dataSize + dbStats.indexSize ¤  Does it fit in RAM? ¤  Plan ahead: ¤  avgObjSize x expectedNoOfDocuments ¤  Does it fit in RAM? ¤  How many documents can you store? 22 Copyright 2011 Severalnines AB
  • 23. Working set ¤  Working Set Analyzer in 2.4 helps: !"workingSet" : {! ! !"note" : "thisIsAnEstimate",! ! !"pagesInMemory" : 33688, //4KB pg_size! ! !"computationTimeMicros" : 15283,! ! !"overSeconds" : 1923 //dist pg_new to pg_oldest
 },! ¤  Trend overSeconds and pagesInMemory 23 Copyright 2011 Severalnines AB
  • 24. Shard Key ¤  Collections must have a Shard Key ¤  The shard key must be indexed ¤  Hash-based or Range based Sharding ¤  Hash-bashed great for exact match ¤  Subscriber/user databases ¤  Queries must use the shard key ¤  If not, all shards will be queried à slow ¤  Check and verify with .explain() 24 Copyright 2011 Severalnines AB
  • 25. Adding Shards ¤  Adding a new Shard is easy: ¤  Deploy a new Replica-set: ¤  PRIMARY - 192.168.0.151 SECONDARY - 192.168.0.152 SECONDARY - 192.168.0.153 ¤  Register the Replica Set: ¤  sh.addShard(“myshard2/192.168.0.151:27017”); ¤  Finding the time to add shards is harder ¤  Rebalancing costs and query performance will suffer due to chunk migration. ¤  Do it off-peak if possible ¤  MongoDB will automatically rebalance the shards. 25 Copyright 2011 Severalnines AB
  • 27. Resources 27 Copyright 2011 Severalnines AB ¤  MongoDB Configurator http://www.severalnines.com/mongodb-configurator/ ¤  -ClusterControl for MongoDB http://www.severalnines.com/resources/cmon-mongodb- cluster-management-monitoring-tool ¤  Installing ClusterControl on top of existing MongoDB Cluster http://www.severalnines.com/blog/install-clustercontrol-top- existing-mongodb-sharded-cluster ¤  Vagrant image - http://alexyu.se/content/2013/03/vagrant-box-severalnines- database-cluster-deployment-scripts
  • 28. Follow us on 28 Copyright 2011 Severalnines AB ¤  Twitter : @severalnines ¤  Facebook: www.facebook.com/severalnines ¤  Slideshare : www.slideshare.net/severalnines ¤  Linked-in: www.linkedin.com/company/severalnines
  • 29. Thank you! 29 Copyright 2011 Severalnines AB