SlideShare a Scribd company logo

Sharding in MongoDB 4.2 #what_is_new

MongoDB 4.2 comes GA soon delivering some amazing new features on multiple areas. In this talk, we will focus on changes related to sharded clusters. We are going to cover distributed transactions & mutable shard keys providing examples that will reveal the internals of those new features. We will provide best practices around the new sharding features and we will cover other minor changes related to it.

1 of 38
Download to read offline
Sharding in MongoDB 4.2
#what_is_new
Antonios Giannopoulos
DBA @ ObjectRocket by Rackspace
Connect:linkedin.com/in/antonis/ Follow:@iamantonios
1
Introduction
www.objectrocket.com
2
Antonios Giannopoulos
Database troubleshooter aka troublemaker
@ObjectRocket
Troubleshoot: MongoDB, CockroachDB &
Postgres
Troublemaking: All the things
Overview
• Change the shard key value
• Distributed transactions
• Split Chunks
• Balancer
• Connection Pool
www.objectrocket.com
3
www.objectrocket.com
4
Before we start, presentation examples are based on the following:
Use case: A virtual Bank
Customers
db.bank.ensureIndex({region:1,iban:1});
db.bank.insert({_id:1, name:"Antonios",amount:100, region:"Europe",country:"GR", iban:"GR001"});
db.bank.insert({_id:2, name:"Alex",amount:100, region:"Europe",country:"UK", iban:"UK001"});
db.bank.insert({_id:3, name:"Jon",amount:100, region:"Pacific",country:"AU", iban:"AU001"});
db.bank.insert({_id:4, name:"Jason",amount:100, region:"America", country:"US", iban:"US001"});
Sharded on {region, iban} – iban is unique
sh.shardCollection("percona.bank",{region:1,iban:1});
Two Shards rs01, rs02 – With three Zones
sh.addShardTag("rs02", "Europe");
sh.addShardTag("rs01","America");
sh.addShardTag("rs02", "RestoftheWorld");
sh.addTagRange("percona.bank", { region: "Europe" }, { region: "Europe1" }, "Europe");
sh.addTagRange("percona.bank", { region: "America" }, { region: "America1" }, "America");
sh.addTagRange("percona.bank", { region: "Pacific" }, { region: "Pacific1" }, "RestoftheWorld");
A good shard key must…
www.objectrocket.com
5
Be immutable...
Let’s examine this throughout an example:
sh.shardCollection("percona.bank", {region:1,iban:1});
example... continued
www.objectrocket.com
6
rs01 rs0
2
sh.addShardTag("rs01","America");
sh.addShardTag("rs02", "Europe");
sh.addShardTag("rs02", "RestoftheWorld");
{region:”America”} {region:”Europe”}
{region:”Pacific”
Customers from America go to rs01
Customers from Europe go to rs02
Move a customer from America to Europe
requires document relocation

Recommended

Upgrading to MongoDB 4.0 from older versions
Upgrading to MongoDB 4.0 from older versionsUpgrading to MongoDB 4.0 from older versions
Upgrading to MongoDB 4.0 from older versionsAntonios Giannopoulos
 
How to upgrade to MongoDB 4.0 - Percona Europe 2018
How to upgrade to MongoDB 4.0 - Percona Europe 2018How to upgrade to MongoDB 4.0 - Percona Europe 2018
How to upgrade to MongoDB 4.0 - Percona Europe 2018Antonios Giannopoulos
 
Managing data and operation distribution in MongoDB
Managing data and operation distribution in MongoDBManaging data and operation distribution in MongoDB
Managing data and operation distribution in MongoDBAntonios Giannopoulos
 
Using MongoDB with Kafka - Use Cases and Best Practices
Using MongoDB with Kafka -  Use Cases and Best PracticesUsing MongoDB with Kafka -  Use Cases and Best Practices
Using MongoDB with Kafka - Use Cases and Best PracticesAntonios Giannopoulos
 
New Indexing and Aggregation Pipeline Capabilities in MongoDB 4.2
New Indexing and Aggregation Pipeline Capabilities in MongoDB 4.2New Indexing and Aggregation Pipeline Capabilities in MongoDB 4.2
New Indexing and Aggregation Pipeline Capabilities in MongoDB 4.2Antonios Giannopoulos
 
Elastic 101 tutorial - Percona Europe 2018
Elastic 101 tutorial - Percona Europe 2018 Elastic 101 tutorial - Percona Europe 2018
Elastic 101 tutorial - Percona Europe 2018 Antonios Giannopoulos
 

More Related Content

What's hot

MongoDB - Sharded Cluster Tutorial
MongoDB - Sharded Cluster TutorialMongoDB - Sharded Cluster Tutorial
MongoDB - Sharded Cluster TutorialJason Terpko
 
MongoDB - External Authentication
MongoDB - External AuthenticationMongoDB - External Authentication
MongoDB - External AuthenticationJason Terpko
 
Triggers In MongoDB
Triggers In MongoDBTriggers In MongoDB
Triggers In MongoDBJason Terpko
 
MongoDB: Comparing WiredTiger In-Memory Engine to Redis
MongoDB: Comparing WiredTiger In-Memory Engine to RedisMongoDB: Comparing WiredTiger In-Memory Engine to Redis
MongoDB: Comparing WiredTiger In-Memory Engine to RedisJason Terpko
 
MongoDB – Sharded cluster tutorial - Percona Europe 2017
MongoDB – Sharded cluster tutorial - Percona Europe 2017MongoDB – Sharded cluster tutorial - Percona Europe 2017
MongoDB – Sharded cluster tutorial - Percona Europe 2017Antonios Giannopoulos
 
MongoDB Chunks - Distribution, Splitting, and Merging
MongoDB Chunks - Distribution, Splitting, and MergingMongoDB Chunks - Distribution, Splitting, and Merging
MongoDB Chunks - Distribution, Splitting, and MergingJason Terpko
 
Like loggly using open source
Like loggly using open sourceLike loggly using open source
Like loggly using open sourceThomas Alrin
 
Managing Data and Operation Distribution In MongoDB
Managing Data and Operation Distribution In MongoDBManaging Data and Operation Distribution In MongoDB
Managing Data and Operation Distribution In MongoDBJason Terpko
 
Sessionization with Spark streaming
Sessionization with Spark streamingSessionization with Spark streaming
Sessionization with Spark streamingRamūnas Urbonas
 
Redis modules 101
Redis modules 101Redis modules 101
Redis modules 101Dvir Volk
 
Percona Live 2017 ­- Sharded cluster tutorial
Percona Live 2017 ­- Sharded cluster tutorialPercona Live 2017 ­- Sharded cluster tutorial
Percona Live 2017 ­- Sharded cluster tutorialAntonios Giannopoulos
 
Using Apache Spark to Solve Sessionization Problem in Batch and Streaming
Using Apache Spark to Solve Sessionization Problem in Batch and StreamingUsing Apache Spark to Solve Sessionization Problem in Batch and Streaming
Using Apache Spark to Solve Sessionization Problem in Batch and StreamingDatabricks
 
Extend Redis with Modules
Extend Redis with ModulesExtend Redis with Modules
Extend Redis with ModulesItamar Haber
 
Redis - for duplicate detection on real time stream
Redis - for duplicate detection on real time streamRedis - for duplicate detection on real time stream
Redis - for duplicate detection on real time streamCodemotion
 
MongoDB Scalability Best Practices
MongoDB Scalability Best PracticesMongoDB Scalability Best Practices
MongoDB Scalability Best PracticesJason Terpko
 
Apache Spark Structured Streaming + Apache Kafka = ♡
Apache Spark Structured Streaming + Apache Kafka = ♡Apache Spark Structured Streaming + Apache Kafka = ♡
Apache Spark Structured Streaming + Apache Kafka = ♡Bartosz Konieczny
 
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::ClientBuilding Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::ClientMike Friedman
 

What's hot (20)

MongoDB - Sharded Cluster Tutorial
MongoDB - Sharded Cluster TutorialMongoDB - Sharded Cluster Tutorial
MongoDB - Sharded Cluster Tutorial
 
MongoDB - External Authentication
MongoDB - External AuthenticationMongoDB - External Authentication
MongoDB - External Authentication
 
Triggers In MongoDB
Triggers In MongoDBTriggers In MongoDB
Triggers In MongoDB
 
MongoDB: Comparing WiredTiger In-Memory Engine to Redis
MongoDB: Comparing WiredTiger In-Memory Engine to RedisMongoDB: Comparing WiredTiger In-Memory Engine to Redis
MongoDB: Comparing WiredTiger In-Memory Engine to Redis
 
MongoDB – Sharded cluster tutorial - Percona Europe 2017
MongoDB – Sharded cluster tutorial - Percona Europe 2017MongoDB – Sharded cluster tutorial - Percona Europe 2017
MongoDB – Sharded cluster tutorial - Percona Europe 2017
 
MongoDB Chunks - Distribution, Splitting, and Merging
MongoDB Chunks - Distribution, Splitting, and MergingMongoDB Chunks - Distribution, Splitting, and Merging
MongoDB Chunks - Distribution, Splitting, and Merging
 
Like loggly using open source
Like loggly using open sourceLike loggly using open source
Like loggly using open source
 
Managing Data and Operation Distribution In MongoDB
Managing Data and Operation Distribution In MongoDBManaging Data and Operation Distribution In MongoDB
Managing Data and Operation Distribution In MongoDB
 
Sessionization with Spark streaming
Sessionization with Spark streamingSessionization with Spark streaming
Sessionization with Spark streaming
 
Fluentd meetup
Fluentd meetupFluentd meetup
Fluentd meetup
 
Redis modules 101
Redis modules 101Redis modules 101
Redis modules 101
 
Percona Live 2017 ­- Sharded cluster tutorial
Percona Live 2017 ­- Sharded cluster tutorialPercona Live 2017 ­- Sharded cluster tutorial
Percona Live 2017 ­- Sharded cluster tutorial
 
Fluentd vs. Logstash for OpenStack Log Management
Fluentd vs. Logstash for OpenStack Log ManagementFluentd vs. Logstash for OpenStack Log Management
Fluentd vs. Logstash for OpenStack Log Management
 
Prometheus Storage
Prometheus StoragePrometheus Storage
Prometheus Storage
 
Using Apache Spark to Solve Sessionization Problem in Batch and Streaming
Using Apache Spark to Solve Sessionization Problem in Batch and StreamingUsing Apache Spark to Solve Sessionization Problem in Batch and Streaming
Using Apache Spark to Solve Sessionization Problem in Batch and Streaming
 
Extend Redis with Modules
Extend Redis with ModulesExtend Redis with Modules
Extend Redis with Modules
 
Redis - for duplicate detection on real time stream
Redis - for duplicate detection on real time streamRedis - for duplicate detection on real time stream
Redis - for duplicate detection on real time stream
 
MongoDB Scalability Best Practices
MongoDB Scalability Best PracticesMongoDB Scalability Best Practices
MongoDB Scalability Best Practices
 
Apache Spark Structured Streaming + Apache Kafka = ♡
Apache Spark Structured Streaming + Apache Kafka = ♡Apache Spark Structured Streaming + Apache Kafka = ♡
Apache Spark Structured Streaming + Apache Kafka = ♡
 
Building Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::ClientBuilding Scalable, Distributed Job Queues with Redis and Redis::Client
Building Scalable, Distributed Job Queues with Redis and Redis::Client
 

Similar to Sharding in MongoDB 4.2 #what_is_new

How sitecore depends on mongo db for scalability and performance, and what it...
How sitecore depends on mongo db for scalability and performance, and what it...How sitecore depends on mongo db for scalability and performance, and what it...
How sitecore depends on mongo db for scalability and performance, and what it...Antonios Giannopoulos
 
MongoDB Auto-Sharding at Mongo Seattle
MongoDB Auto-Sharding at Mongo SeattleMongoDB Auto-Sharding at Mongo Seattle
MongoDB Auto-Sharding at Mongo SeattleMongoDB
 
MongoDB Days Silicon Valley: MongoDB and the Hadoop Connector
MongoDB Days Silicon Valley: MongoDB and the Hadoop ConnectorMongoDB Days Silicon Valley: MongoDB and the Hadoop Connector
MongoDB Days Silicon Valley: MongoDB and the Hadoop ConnectorMongoDB
 
Get expertise with mongo db
Get expertise with mongo dbGet expertise with mongo db
Get expertise with mongo dbAmit Thakkar
 
Lessons from running AppSync in prod
Lessons from running AppSync in prodLessons from running AppSync in prod
Lessons from running AppSync in prodYan Cui
 
Introduction to trader bots with Python
Introduction to trader bots with PythonIntroduction to trader bots with Python
Introduction to trader bots with Pythonroskakori
 
OrientDB for real & Web App development
OrientDB for real & Web App developmentOrientDB for real & Web App development
OrientDB for real & Web App developmentLuca Garulli
 
OSCON 2011 Learning CouchDB
OSCON 2011 Learning CouchDBOSCON 2011 Learning CouchDB
OSCON 2011 Learning CouchDBBradley Holt
 
MongoDB- Crud Operation
MongoDB- Crud OperationMongoDB- Crud Operation
MongoDB- Crud OperationEdureka!
 
Hands-on Lab: Migrating Oracle to PostgreSQL
Hands-on Lab: Migrating Oracle to PostgreSQL Hands-on Lab: Migrating Oracle to PostgreSQL
Hands-on Lab: Migrating Oracle to PostgreSQL Amazon Web Services
 
MongoDB World 2018: Building a New Transactional Model
MongoDB World 2018: Building a New Transactional ModelMongoDB World 2018: Building a New Transactional Model
MongoDB World 2018: Building a New Transactional ModelMongoDB
 
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...Databricks
 
Webinar Slides: Migrating to Galera Cluster
Webinar Slides: Migrating to Galera ClusterWebinar Slides: Migrating to Galera Cluster
Webinar Slides: Migrating to Galera ClusterSeveralnines
 
PostgreSQL as seen by Rubyists (Kaigi on Rails 2022)
PostgreSQL as seen by Rubyists (Kaigi on Rails 2022)PostgreSQL as seen by Rubyists (Kaigi on Rails 2022)
PostgreSQL as seen by Rubyists (Kaigi on Rails 2022)Андрей Новиков
 
MongoDB 3.2 - a giant leap. What’s new?
MongoDB 3.2 - a giant leap. What’s new?MongoDB 3.2 - a giant leap. What’s new?
MongoDB 3.2 - a giant leap. What’s new?Binary Studio
 
Grokking #9: Building a real-time and offline editing service with Couchbase
Grokking #9: Building a real-time and offline editing service with CouchbaseGrokking #9: Building a real-time and offline editing service with Couchbase
Grokking #9: Building a real-time and offline editing service with CouchbaseOliver N
 
mloc.js 2014 - JavaScript and the browser as a platform for game development
mloc.js 2014 - JavaScript and the browser as a platform for game developmentmloc.js 2014 - JavaScript and the browser as a platform for game development
mloc.js 2014 - JavaScript and the browser as a platform for game developmentDavid Galeano
 
MongoDB Knowledge Shareing
MongoDB Knowledge ShareingMongoDB Knowledge Shareing
MongoDB Knowledge ShareingPhilip Zhong
 

Similar to Sharding in MongoDB 4.2 #what_is_new (20)

How sitecore depends on mongo db for scalability and performance, and what it...
How sitecore depends on mongo db for scalability and performance, and what it...How sitecore depends on mongo db for scalability and performance, and what it...
How sitecore depends on mongo db for scalability and performance, and what it...
 
MongoDB Auto-Sharding at Mongo Seattle
MongoDB Auto-Sharding at Mongo SeattleMongoDB Auto-Sharding at Mongo Seattle
MongoDB Auto-Sharding at Mongo Seattle
 
MongoDB Days Silicon Valley: MongoDB and the Hadoop Connector
MongoDB Days Silicon Valley: MongoDB and the Hadoop ConnectorMongoDB Days Silicon Valley: MongoDB and the Hadoop Connector
MongoDB Days Silicon Valley: MongoDB and the Hadoop Connector
 
Get expertise with mongo db
Get expertise with mongo dbGet expertise with mongo db
Get expertise with mongo db
 
Lessons from running AppSync in prod
Lessons from running AppSync in prodLessons from running AppSync in prod
Lessons from running AppSync in prod
 
Introduction to trader bots with Python
Introduction to trader bots with PythonIntroduction to trader bots with Python
Introduction to trader bots with Python
 
OrientDB for real & Web App development
OrientDB for real & Web App developmentOrientDB for real & Web App development
OrientDB for real & Web App development
 
OSCON 2011 Learning CouchDB
OSCON 2011 Learning CouchDBOSCON 2011 Learning CouchDB
OSCON 2011 Learning CouchDB
 
MongoDB- Crud Operation
MongoDB- Crud OperationMongoDB- Crud Operation
MongoDB- Crud Operation
 
Hands-on Lab: Migrating Oracle to PostgreSQL
Hands-on Lab: Migrating Oracle to PostgreSQL Hands-on Lab: Migrating Oracle to PostgreSQL
Hands-on Lab: Migrating Oracle to PostgreSQL
 
MongoDB World 2018: Building a New Transactional Model
MongoDB World 2018: Building a New Transactional ModelMongoDB World 2018: Building a New Transactional Model
MongoDB World 2018: Building a New Transactional Model
 
A Brief MongoDB Intro
A Brief MongoDB IntroA Brief MongoDB Intro
A Brief MongoDB Intro
 
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
 
Webinar Slides: Migrating to Galera Cluster
Webinar Slides: Migrating to Galera ClusterWebinar Slides: Migrating to Galera Cluster
Webinar Slides: Migrating to Galera Cluster
 
PostgreSQL as seen by Rubyists (Kaigi on Rails 2022)
PostgreSQL as seen by Rubyists (Kaigi on Rails 2022)PostgreSQL as seen by Rubyists (Kaigi on Rails 2022)
PostgreSQL as seen by Rubyists (Kaigi on Rails 2022)
 
MongoDB 3.2 - a giant leap. What’s new?
MongoDB 3.2 - a giant leap. What’s new?MongoDB 3.2 - a giant leap. What’s new?
MongoDB 3.2 - a giant leap. What’s new?
 
Grokking #9: Building a real-time and offline editing service with Couchbase
Grokking #9: Building a real-time and offline editing service with CouchbaseGrokking #9: Building a real-time and offline editing service with Couchbase
Grokking #9: Building a real-time and offline editing service with Couchbase
 
mloc.js 2014 - JavaScript and the browser as a platform for game development
mloc.js 2014 - JavaScript and the browser as a platform for game developmentmloc.js 2014 - JavaScript and the browser as a platform for game development
mloc.js 2014 - JavaScript and the browser as a platform for game development
 
MongoDB Knowledge Shareing
MongoDB Knowledge ShareingMongoDB Knowledge Shareing
MongoDB Knowledge Shareing
 
Mongo-Drupal
Mongo-DrupalMongo-Drupal
Mongo-Drupal
 

Recently uploaded

Implementing Docker Containers with Windows Server 2019
Implementing Docker Containers with Windows Server 2019Implementing Docker Containers with Windows Server 2019
Implementing Docker Containers with Windows Server 2019VICTOR MAESTRE RAMIREZ
 
Agile & Scrum, Certified Scrum Master! Crash Course
Agile & Scrum,  Certified Scrum Master! Crash CourseAgile & Scrum,  Certified Scrum Master! Crash Course
Agile & Scrum, Certified Scrum Master! Crash CourseRohan Chandane
 
Orion Context Broker introduction 20240227
Orion Context Broker introduction 20240227Orion Context Broker introduction 20240227
Orion Context Broker introduction 20240227Fermin Galan
 
How AI is preventing account fraud at web scale
How AI is preventing account fraud at web scaleHow AI is preventing account fraud at web scale
How AI is preventing account fraud at web scaleAmir Moghimi
 
Open Source vs Closed Source LLMs. Pros and Cons
Open Source vs Closed Source LLMs. Pros and ConsOpen Source vs Closed Source LLMs. Pros and Cons
Open Source vs Closed Source LLMs. Pros and ConsSprings
 
Automation for Bonterra Impact Management (fka Apricot)
Automation for Bonterra Impact Management (fka Apricot)Automation for Bonterra Impact Management (fka Apricot)
Automation for Bonterra Impact Management (fka Apricot)Jeffrey Haguewood
 
killingcamp longest common subsequence.pdf
killingcamp longest common subsequence.pdfkillingcamp longest common subsequence.pdf
killingcamp longest common subsequence.pdfssuser82c38d
 
Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)Dmitry Zinoviev
 
No more Dockerfiles? Buildpacks to help you ship your image!
No more Dockerfiles? Buildpacks to help you ship your image!No more Dockerfiles? Buildpacks to help you ship your image!
No more Dockerfiles? Buildpacks to help you ship your image!Anthony Dahanne
 
Role of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptxRole of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptxMindInventory
 
LLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flowLLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flowNaoki (Neo) SATO
 
killingcamp 광고삽입문제 풀이, killingcamp 광고삽입문제 풀이
killingcamp 광고삽입문제 풀이, killingcamp 광고삽입문제 풀이killingcamp 광고삽입문제 풀이, killingcamp 광고삽입문제 풀이
killingcamp 광고삽입문제 풀이, killingcamp 광고삽입문제 풀이ssuser82c38d
 
killing camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdfkilling camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdfssuser82c38d
 
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...Alluxio, Inc.
 
Welcome to AltTask - the nexus where innovation converges with empowerment!
Welcome to AltTask - the nexus where innovation converges with empowerment!Welcome to AltTask - the nexus where innovation converges with empowerment!
Welcome to AltTask - the nexus where innovation converges with empowerment!alttaskcom
 
The Top Outages of 2023: Analyses and Takeaways
The Top Outages of 2023: Analyses and TakeawaysThe Top Outages of 2023: Analyses and Takeaways
The Top Outages of 2023: Analyses and TakeawaysThousandEyes
 
Cybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdfCybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdfCIOWomenMagazine
 
killing camp week 6 problem - maximal matrix.pdf
killing camp week 6 problem - maximal matrix.pdfkilling camp week 6 problem - maximal matrix.pdf
killing camp week 6 problem - maximal matrix.pdfssuser82c38d
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flinkconfluent
 

Recently uploaded (20)

Implementing Docker Containers with Windows Server 2019
Implementing Docker Containers with Windows Server 2019Implementing Docker Containers with Windows Server 2019
Implementing Docker Containers with Windows Server 2019
 
Agile & Scrum, Certified Scrum Master! Crash Course
Agile & Scrum,  Certified Scrum Master! Crash CourseAgile & Scrum,  Certified Scrum Master! Crash Course
Agile & Scrum, Certified Scrum Master! Crash Course
 
eLearning Content Development Company Code and Pixels.pdf
eLearning Content Development Company Code and Pixels.pdfeLearning Content Development Company Code and Pixels.pdf
eLearning Content Development Company Code and Pixels.pdf
 
Orion Context Broker introduction 20240227
Orion Context Broker introduction 20240227Orion Context Broker introduction 20240227
Orion Context Broker introduction 20240227
 
How AI is preventing account fraud at web scale
How AI is preventing account fraud at web scaleHow AI is preventing account fraud at web scale
How AI is preventing account fraud at web scale
 
Open Source vs Closed Source LLMs. Pros and Cons
Open Source vs Closed Source LLMs. Pros and ConsOpen Source vs Closed Source LLMs. Pros and Cons
Open Source vs Closed Source LLMs. Pros and Cons
 
Automation for Bonterra Impact Management (fka Apricot)
Automation for Bonterra Impact Management (fka Apricot)Automation for Bonterra Impact Management (fka Apricot)
Automation for Bonterra Impact Management (fka Apricot)
 
killingcamp longest common subsequence.pdf
killingcamp longest common subsequence.pdfkillingcamp longest common subsequence.pdf
killingcamp longest common subsequence.pdf
 
Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)
 
No more Dockerfiles? Buildpacks to help you ship your image!
No more Dockerfiles? Buildpacks to help you ship your image!No more Dockerfiles? Buildpacks to help you ship your image!
No more Dockerfiles? Buildpacks to help you ship your image!
 
Role of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptxRole of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptx
 
LLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flowLLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flow
 
killingcamp 광고삽입문제 풀이, killingcamp 광고삽입문제 풀이
killingcamp 광고삽입문제 풀이, killingcamp 광고삽입문제 풀이killingcamp 광고삽입문제 풀이, killingcamp 광고삽입문제 풀이
killingcamp 광고삽입문제 풀이, killingcamp 광고삽입문제 풀이
 
killing camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdfkilling camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdf
 
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...
 
Welcome to AltTask - the nexus where innovation converges with empowerment!
Welcome to AltTask - the nexus where innovation converges with empowerment!Welcome to AltTask - the nexus where innovation converges with empowerment!
Welcome to AltTask - the nexus where innovation converges with empowerment!
 
The Top Outages of 2023: Analyses and Takeaways
The Top Outages of 2023: Analyses and TakeawaysThe Top Outages of 2023: Analyses and Takeaways
The Top Outages of 2023: Analyses and Takeaways
 
Cybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdfCybersecurity Measures For Remote Workers.pdf
Cybersecurity Measures For Remote Workers.pdf
 
killing camp week 6 problem - maximal matrix.pdf
killing camp week 6 problem - maximal matrix.pdfkilling camp week 6 problem - maximal matrix.pdf
killing camp week 6 problem - maximal matrix.pdf
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flink
 

Sharding in MongoDB 4.2 #what_is_new

  • 1. Sharding in MongoDB 4.2 #what_is_new Antonios Giannopoulos DBA @ ObjectRocket by Rackspace Connect:linkedin.com/in/antonis/ Follow:@iamantonios 1
  • 2. Introduction www.objectrocket.com 2 Antonios Giannopoulos Database troubleshooter aka troublemaker @ObjectRocket Troubleshoot: MongoDB, CockroachDB & Postgres Troublemaking: All the things
  • 3. Overview • Change the shard key value • Distributed transactions • Split Chunks • Balancer • Connection Pool www.objectrocket.com 3
  • 4. www.objectrocket.com 4 Before we start, presentation examples are based on the following: Use case: A virtual Bank Customers db.bank.ensureIndex({region:1,iban:1}); db.bank.insert({_id:1, name:"Antonios",amount:100, region:"Europe",country:"GR", iban:"GR001"}); db.bank.insert({_id:2, name:"Alex",amount:100, region:"Europe",country:"UK", iban:"UK001"}); db.bank.insert({_id:3, name:"Jon",amount:100, region:"Pacific",country:"AU", iban:"AU001"}); db.bank.insert({_id:4, name:"Jason",amount:100, region:"America", country:"US", iban:"US001"}); Sharded on {region, iban} – iban is unique sh.shardCollection("percona.bank",{region:1,iban:1}); Two Shards rs01, rs02 – With three Zones sh.addShardTag("rs02", "Europe"); sh.addShardTag("rs01","America"); sh.addShardTag("rs02", "RestoftheWorld"); sh.addTagRange("percona.bank", { region: "Europe" }, { region: "Europe1" }, "Europe"); sh.addTagRange("percona.bank", { region: "America" }, { region: "America1" }, "America"); sh.addTagRange("percona.bank", { region: "Pacific" }, { region: "Pacific1" }, "RestoftheWorld");
  • 5. A good shard key must… www.objectrocket.com 5 Be immutable... Let’s examine this throughout an example: sh.shardCollection("percona.bank", {region:1,iban:1});
  • 6. example... continued www.objectrocket.com 6 rs01 rs0 2 sh.addShardTag("rs01","America"); sh.addShardTag("rs02", "Europe"); sh.addShardTag("rs02", "RestoftheWorld"); {region:”America”} {region:”Europe”} {region:”Pacific” Customers from America go to rs01 Customers from Europe go to rs02 Move a customer from America to Europe requires document relocation
  • 7. A good shard key must… Be immutable… www.objectrocket.com 7
  • 8. A good shard key is mutable… www.objectrocket.com 8
  • 9. shard key is mutable, unless… www.objectrocket.com 9 Unless the shard key field is the immutable _id field You miss the full shard key in the query
  • 10. shard key is mutable, unless… www.objectrocket.com 10 If the shard key modification does not result in moving the document to another shard, you can specify multiple shard key modification in the bulk operation.
  • 11. shard key is mutable, unless… www.objectrocket.com 11 the shard key modification does not result in moving the document to another shard, you can specify multiple shard key modification in the bulk operation.
  • 12. Change the shard key… (?) www.objectrocket.com 12 You can’t change the fields of the shard key  …but you can re-purpose it  For example, shard key {client_id:1} Bucketing: {client_id:”000”} to {client_id:”000-2019”} Locality: {client_id: “US-000”} , {client_id:”UK-000”} Completely repurpose: A field name is what the application think it is!!!
  • 13. Distributed Transactions • Implementation • Examples • Considerations www.objectrocket.com 13
  • 14. Distributed Transactions www.objectrocket.com 14 In MongoDB operations on a single document are atomic. MongoDB 4.0 supports multi-document transactions on replica sets (WiredTiger only) MongoDB 4.2 supports distributed transactions, which adds support for multi-document transactions on sharded clusters Change the value of the shard key is nothing more than a distributed transaction Transactions on any distributed system are challenging (anyone disagrees?) One of the biggest challenges is the “All or nothing”
  • 15. How Transactions work… www.objectrocket.com 15 If the transaction touches only one shard, behavior is similar to a replica-set transaction
  • 16. One shard involved… continued www.objectrocket.com 16
  • 17. How Transactions work… www.objectrocket.com 17 If the trx touches more than one shard: behavior is similar to a two phase commit On every distributed transaction a shard acts as Coordinator A distributed transaction has two states: the Prepare and the Commit state - Prepare state guarantees the ability to Commit - All shards must prepare a transaction (w:majority) before Commit - If any shard fails to Prepare, then no shard will Commit - Coordinator is responsible for the ack of Prepare and Commit - Prepared Transactions held in memory, and Replication makes them durable Confused… Let see an example
  • 18. 2+ shards involved… continued www.objectrocket.com 18 *Zones: Europe and America are on different shards
  • 19. How Transactions work… rs01 rs0 2 (1)update({EU},{$inc:{amount:50}}) (2)update({US},{$inc:{amount:-50}}) Both (1) & (2) are now in cache 1 1 2 2 C Shard becomes coordinator (C) Coordinator say prepare (Succeeds) (1) & (2) are written in the oplog Coordinator say commit (Succeeds) (1) & (2) are written in the storage and become visible
  • 20. 2+ shards involved… continued www.objectrocket.com 20 The first statements picks a coordinator (first update in our case)
  • 21. 2+ shards involved… continued www.objectrocket.com 21 Coordinator says:: Lets prepare Oplog entries from rs01 and rs02
  • 22. 2+ shards involved… continued www.objectrocket.com 22 Coordinator says: Lets commit (Coordinator’s oplog)
  • 23. 2+ shards involved… continued www.objectrocket.com 23 Coordinator says: Lets commit, Oplog entries from rs01 and rs02
  • 24. Transactions & the oplog… www.objectrocket.com 24 The 16MB limit removed in 4.2 Transactions break into a chain of events prevOptime : connects the chain partialTnx: create the chain *The oplog entries are truncated
  • 25. Considerations www.objectrocket.com 25 db.adminCommand( { setFeatureCompatibilityVersion: “4.2” } ) You will need the latest drivers writeConcernMajorityJournalDefault must be set to true Set maxTimeMS on commit, else it would default transactionLifetimeLimitSeconds Chunk migrations: A chunk migration waits for transaction lock on chunks documents If a chunk migration is ongoing transaction may fail db.serverStatus().shardingStatistics.countDonorMoveChunkLockTimeout
  • 26. Considerations … continued www.objectrocket.com 26 Multi shard transactions will fail, if an arbiter is in place:
  • 27. Considerations … continued www.objectrocket.com 27 There are restrictions on certain operators - Same restrictions as 4.0 with the addition, - You cannot write to capped collections. - You cannot specify killCursors as the first operation in a transaction. Outside Reads During Commit - Read concern snapshot wait for all writes of a transaction to be visible. - Other read concerns (local or majority) do not wait for all writes of a transaction to be visible but instead read the before-transaction version of the documents available. Reconsider backup strategy (mongodump)
  • 28. Considerations… Failovers www.objectrocket.com 28 Elections: - Majority commit or Failed to prepare Startup Recovery: - Consistent point in time -> noted on prepare trx table -> Recover -> Check if any prepared trx needs to be applied - Prepare transactions are immutable - Conflicts handled by the Primary - Reads are not allowed while recovering Initial sync – same as startup recovery Rollback: - Rollback to stable timestamp WT-3387 - Move to Common point with prepare trx table - After Common point act as Primary
  • 29. Performance www.objectrocket.com 29 Single shard transactions should have the same cost as replica-set transactions Multi shard transactions are more expensive compared to ReplicaSet ones’ Transactions saved in cache – more RAM may needed Remote shards may slow down due to network latency Don’t give up on the MongoDB data modeling Use transactions whenever is absolutely necessary Try to hit as less shards as possible Read many , Write one is optimized
  • 30. Miscellaneous Changes • Chunk Split • Balancer • Connection Pool www.objectrocket.com 30
  • 31. Responsible for AutoSplit… rs01 rs0 2 Prior to 4.2 : Mongos In 4.2: The responsibility passed to Shards SERVER-9287 - Each mongos keeps its own statistics - May lead to jumbo chunks - May lead into too many split requests - Especially with high number of mongos
  • 32. Balancer www.objectrocket.com 32 The balancerStart command and the mongo shell helper methods sh.startBalancer() and sh.setBalancerState(true) also enable auto-splitting for the sharded cluster. To disable auto-splitting when the balancer is enabled, you can use sh.disableAutoSplit(). The balancerStop command and the mongo shell helper methods sh.stopBalancer() and sh.setBalancerState(false) also disable auto-splitting for the sharded cluster. To enable auto-splitting when the balancer is disabled, you can use sh.enableAutoSplit() The mongo methods sh.enableBalancing(namespace) & sh.disableBalancing(namespace) have no affect on the auto-splitting.
  • 33. Mongos Connection Pool www.objectrocket.com 33 ShardingTaskExecutorPoolReplicaSetMatching: determines the minimum size limit of the mongos instance’s connection pools to the sharded cluster’s replica set secondaries. db.adminCommand( { setParameter: 1, ShardingTaskExecutorPoolReplicaSetMatching: <value>} ) ,where <value>: - matchPrimaryNode : the minimum size limit of each secondary of that replica set is equal to the size of its connection pool to the primary. - matchBusiestNode : the minimum size limit is the largest among the active connections counts to the primary and each secondary members. - Disabled : the minimum number of connections in the mongos instance’s connection pool to each secondary is equal to the ShardingTaskExecutorPoolMinSize.
  • 34. Recap & Takeways www.objectrocket.com 34 o The shard key value is mutable o Transactions are supported on sharded clusters o On a single shard same performance as Replset transactions o On multiple shards there is a performance overhead o Transaction 16MiB limit lifted o Split is now running on the shards
  • 37. www.objectrocket.com 37 We’re Hiring! Looking to join a dynamic & innovative team? https://www.objectrocket.com/careers/
  • 38. Thank you! 38 Address: 9001 N Interstate Hwy 35 #150, Austin, TX 78753 Support: US Toll free: 1-855-722-8165 UK Toll free +448081686840 support@objectrocket.com Sales: 1-888-440-3242 sales@objectrocket.com www.objectrocket.com