Database replication involves keeping identical copies of data on different servers to provide redundancy and minimize downtime. Replication is recommended for databases in production from the start. A MongoDB replica set consists of a primary server that handles client requests and secondary servers that copy the primary's data. Replica sets can include up to 50 members with 7 voting members and use an oplog to replicate operations from the primary to secondaries. For elections and writes to succeed, a majority of voting members must be reachable.
MongoDB WiredTiger Internals: Journey To TransactionsMydbops
MongoDB has adapted transaction feature (ACID Properties) in MongoDB 4.0. This talk focuses on the internals of how MongoDB adapted the ACID properties with Weird Tiger Engine. Weird tiger offers more future possibilities for MongoDB. This tech talk was presented at Mydbops Database Meetup on 27-04-2019 by Manosh Malai Senior Devops/NoSQL Consultant with Mydbops and Ranjith Database Administrator with Mydbops.
MongoDB World 2019: MongoDB Read Isolation: Making Your Reads Clean, Committe...MongoDB
Isolation, the I in ACID, determines how/when the changes made by one operation become visible to another. Relational databases provide four isolation levels (uncommitted, committed, repeatable reads, and serialiable) to enable the trade off of performance versus the level of cross operation change visibility. In contrast, MongoDB’s isolation levels are controlled by using readConcerns and transactions. This talk will describe how the relational isolation levels compare to MongoDB’s isolation guarantees, how you configure MongoDB to provide the desired isolation level, and the performance implications.
Video available here: http://vivu.tv/portal/archive.jsp?flow=783-586-4282&id=1270584002677
We all know that MongoDB is one of the most flexible and feature-rich databases available. In this webinar we'll discuss how you can leverage this feature set and maintain high performance with your project's massive data sets and high loads. We'll cover how indexes can be designed to optimize the performance of MongoDB. We'll also discuss tips for diagnosing and fixing performance issues should they arise.
MongoDB WiredTiger Internals: Journey To TransactionsMydbops
MongoDB has adapted transaction feature (ACID Properties) in MongoDB 4.0. This talk focuses on the internals of how MongoDB adapted the ACID properties with Weird Tiger Engine. Weird tiger offers more future possibilities for MongoDB. This tech talk was presented at Mydbops Database Meetup on 27-04-2019 by Manosh Malai Senior Devops/NoSQL Consultant with Mydbops and Ranjith Database Administrator with Mydbops.
MongoDB World 2019: MongoDB Read Isolation: Making Your Reads Clean, Committe...MongoDB
Isolation, the I in ACID, determines how/when the changes made by one operation become visible to another. Relational databases provide four isolation levels (uncommitted, committed, repeatable reads, and serialiable) to enable the trade off of performance versus the level of cross operation change visibility. In contrast, MongoDB’s isolation levels are controlled by using readConcerns and transactions. This talk will describe how the relational isolation levels compare to MongoDB’s isolation guarantees, how you configure MongoDB to provide the desired isolation level, and the performance implications.
Video available here: http://vivu.tv/portal/archive.jsp?flow=783-586-4282&id=1270584002677
We all know that MongoDB is one of the most flexible and feature-rich databases available. In this webinar we'll discuss how you can leverage this feature set and maintain high performance with your project's massive data sets and high loads. We'll cover how indexes can be designed to optimize the performance of MongoDB. We'll also discuss tips for diagnosing and fixing performance issues should they arise.
The presentation covers improvements made to the redo logs in MySQL 8.0 and their impact on the MySQL performance and Operations. This covers the MySQL version still MySQL 8.0.30.
Slidedeck presented at http://devternity.com/ around MongoDB internals. We review the usage patterns of MongoDB, the different storage engines and persistency models as well has the definition of documents and general data structures.
MySQL has multiple timeouts variables to control its operations. This presentation focus on the purpose of each timeout variables and how it can be used.
Whether you're a MongoDB professional or totally new to document databases, our MongoDB performance success factors & evaluation framework has something for you,
Curious about MongoDB performance?
Mydbops CTO, Manosh Malai illustrates the secret sauce for MongoDB performance best practices & analysis tool.
This talk is focused on tuning analysing and optimizing MongoDB query and index with the use of Database Profiler and "explain()" function.
Also, performance of database can also be impacted by configuring the underline ( Linux ) OS with some recommended settings which do not come by default.
Meta/Facebook's database serving social workloads is running on top of MyRocks (MySQL on RocksDB). This means our performance and reliability depends a lot on RocksDB. Not just MyRocks, but also we have other important systems running on top of RocksDB. We have learned many lessons from operating and debugging RocksDB at scale.
In this session, we will offer an overview of RocksDB, key differences from InnoDB, and share a few interesting lessons learned from production.
This presentation covers MySQL data encryption at disk. How to encrypt all tablespaces and MySQL related files for the compliances ? The new releases in MySQL 8.0 take care of the encryption of the system tablespace and supporting tables unlike MySQL 5.7.
This tutorial will guide you through the many considerations when deploying a sharded cluster. We will cover the services that make up a sharded cluster, configuration recommendations for these services, shard key selection, use cases, and how data is managed within a sharded cluster. Maintaining a sharded cluster also has its challenges. We will review these challenges and how you can prevent them with proper design or ways to resolve them if they exist today. There will be lab sessions at the end of some chapters so please have your laptops with you.
Using all of the high availability options in MariaDBMariaDB plc
MariaDB provides a number of high availability options, including replication with automatic failover and multi-master clustering. In this session Wagner Bianchi, Principal Remote DBA, provides a comprehensive overview of the high availability features in MariaDB, highlights their impact on consistency and performance, discusses advanced failover strategies and introduces new features such as casual reads and transparent connection failover.
Webinar: MariaDB 10.11 key features overview for DBAs
Orgnised by Vettabase
27 April 2023
Amongst other topics:
- Long ALTER TABLES now don’t cause replicas to lag
- InnoDB configuration is now more dynamic, and certain important variables can be modified without a restart
- Populating an empty table is now much faster
- New data types: UUID, INET4, INET6
- SFORMAT() function, NATURAL_KEY_SORT() function
ClickHouse Materialized Views: The Magic ContinuesAltinity Ltd
Slides for the webinar, presented on February 26, 2020
By Robert Hodges, Altinity CEO
Materialized views are the killer feature of ClickHouse, and the Altinity 2019 webinar on how they work was very popular. Join this updated webinar to learn how to use materialized views to speed up queries hundreds of times. We'll cover basic design, last point queries, using TTLs to drop source data, counting unique values, and other useful tricks. Finally, we'll cover recent improvements that make materialized views more useful than ever.
MongoDB World 2015 - A Technical Introduction to WiredTigerWiredTiger
MongoDB 3.0 introduces a new pluggable storage engine API and a new storage engine called WiredTiger. The engineering team behind WiredTiger team has a long and distinguished career, having architected and built Berkeley DB, now the world's most widely used embedded database. In this talk we will describe our original design goals for WiredTiger, including considerations we made for heavily threaded hardware, large on-chip caches, and SSD storage. We'll also look at some of the latch-free and non-blocking algorithms we've implemented, as well as other techniques that improve scaling, overall throughput and latency. Finally, we'll take a look at some of the features we hope to incorporate into WiredTiger and MongoDB in the future.
The presentation covers improvements made to the redo logs in MySQL 8.0 and their impact on the MySQL performance and Operations. This covers the MySQL version still MySQL 8.0.30.
Slidedeck presented at http://devternity.com/ around MongoDB internals. We review the usage patterns of MongoDB, the different storage engines and persistency models as well has the definition of documents and general data structures.
MySQL has multiple timeouts variables to control its operations. This presentation focus on the purpose of each timeout variables and how it can be used.
Whether you're a MongoDB professional or totally new to document databases, our MongoDB performance success factors & evaluation framework has something for you,
Curious about MongoDB performance?
Mydbops CTO, Manosh Malai illustrates the secret sauce for MongoDB performance best practices & analysis tool.
This talk is focused on tuning analysing and optimizing MongoDB query and index with the use of Database Profiler and "explain()" function.
Also, performance of database can also be impacted by configuring the underline ( Linux ) OS with some recommended settings which do not come by default.
Meta/Facebook's database serving social workloads is running on top of MyRocks (MySQL on RocksDB). This means our performance and reliability depends a lot on RocksDB. Not just MyRocks, but also we have other important systems running on top of RocksDB. We have learned many lessons from operating and debugging RocksDB at scale.
In this session, we will offer an overview of RocksDB, key differences from InnoDB, and share a few interesting lessons learned from production.
This presentation covers MySQL data encryption at disk. How to encrypt all tablespaces and MySQL related files for the compliances ? The new releases in MySQL 8.0 take care of the encryption of the system tablespace and supporting tables unlike MySQL 5.7.
This tutorial will guide you through the many considerations when deploying a sharded cluster. We will cover the services that make up a sharded cluster, configuration recommendations for these services, shard key selection, use cases, and how data is managed within a sharded cluster. Maintaining a sharded cluster also has its challenges. We will review these challenges and how you can prevent them with proper design or ways to resolve them if they exist today. There will be lab sessions at the end of some chapters so please have your laptops with you.
Using all of the high availability options in MariaDBMariaDB plc
MariaDB provides a number of high availability options, including replication with automatic failover and multi-master clustering. In this session Wagner Bianchi, Principal Remote DBA, provides a comprehensive overview of the high availability features in MariaDB, highlights their impact on consistency and performance, discusses advanced failover strategies and introduces new features such as casual reads and transparent connection failover.
Webinar: MariaDB 10.11 key features overview for DBAs
Orgnised by Vettabase
27 April 2023
Amongst other topics:
- Long ALTER TABLES now don’t cause replicas to lag
- InnoDB configuration is now more dynamic, and certain important variables can be modified without a restart
- Populating an empty table is now much faster
- New data types: UUID, INET4, INET6
- SFORMAT() function, NATURAL_KEY_SORT() function
ClickHouse Materialized Views: The Magic ContinuesAltinity Ltd
Slides for the webinar, presented on February 26, 2020
By Robert Hodges, Altinity CEO
Materialized views are the killer feature of ClickHouse, and the Altinity 2019 webinar on how they work was very popular. Join this updated webinar to learn how to use materialized views to speed up queries hundreds of times. We'll cover basic design, last point queries, using TTLs to drop source data, counting unique values, and other useful tricks. Finally, we'll cover recent improvements that make materialized views more useful than ever.
MongoDB World 2015 - A Technical Introduction to WiredTigerWiredTiger
MongoDB 3.0 introduces a new pluggable storage engine API and a new storage engine called WiredTiger. The engineering team behind WiredTiger team has a long and distinguished career, having architected and built Berkeley DB, now the world's most widely used embedded database. In this talk we will describe our original design goals for WiredTiger, including considerations we made for heavily threaded hardware, large on-chip caches, and SSD storage. We'll also look at some of the latch-free and non-blocking algorithms we've implemented, as well as other techniques that improve scaling, overall throughput and latency. Finally, we'll take a look at some of the features we hope to incorporate into WiredTiger and MongoDB in the future.
This talk was given by Bridget Kromhout and Mike Hobbs at MinneBar 8, April 6th 2013.
We’ve been using MongoDB on EC2 for about a year now; our production deployment includes a 12-node cluster (4 shards x 3 replica sets) as well as several other non-sharded replica sets. (The sharded cluster used to be bigger, which, as it turns out, isn’t always better.)
Join us to benefit from our lessons learned, discuss what works and what doesn’t, and marvel at oddities we’ve encountered.
Evolution of MongoDB Replicaset and Its Best PracticesMydbops
There are several exciting and long-awaited features released from MongoDB 4.0. He will focus on the prime features, the kind of problem it solves, and the best practices for deploying replica sets.
Optimizing the queries you send to the database can greatly increase the database's performance, but what do you know about all those strange MySQL variables that can be played with to get even more power from the database? Join me as we go over some of the basics of the various MySQL settings you can twitch, tweak and massage to get the most out of your MySQL server.
Configuring MongoDB HA Replica Set on AWS EC2ShepHertz
It has always been a tedious task to choose the right configuration for MongoDB on AWS EC2
It is always challenging and takes a lots of time to make your system Production Ready.
Here is a quick guide on how to setup MongoDB on AWS EC2.
Redundancy and high availability are the basis for all production deployments. With MongoDB this can be achieved by deploying replica set. In this slides we are exploring how the replication works with MongoDB, why you should use replication, what are the features and go over different deployment use cases. At the end we are comparing some features with MySQL replication and what are the differences between the two
Replication, Durability, and Disaster RecoverySteven Francia
This session introduces the basic components of high availability before going into a deep dive on MongoDB replication. We'll explore some of the advanced capabilities with MongoDB replication and best practices to ensure data durability and redundancy. We'll also look at various deployment scenarios and disaster recovery configurations.
Literature Review Basics and Understanding Reference Management.pptxDr Ramhari Poudyal
Three-day training on academic research focuses on analytical tools at United Technical College, supported by the University Grant Commission, Nepal. 24-26 May 2024
Online aptitude test management system project report.pdfKamal Acharya
The purpose of on-line aptitude test system is to take online test in an efficient manner and no time wasting for checking the paper. The main objective of on-line aptitude test system is to efficiently evaluate the candidate thoroughly through a fully automated system that not only saves lot of time but also gives fast results. For students they give papers according to their convenience and time and there is no need of using extra thing like paper, pen etc. This can be used in educational institutions as well as in corporate world. Can be used anywhere any time as it is a web based application (user Location doesn’t matter). No restriction that examiner has to be present when the candidate takes the test.
Every time when lecturers/professors need to conduct examinations they have to sit down think about the questions and then create a whole new set of questions for each and every exam. In some cases the professor may want to give an open book online exam that is the student can take the exam any time anywhere, but the student might have to answer the questions in a limited time period. The professor may want to change the sequence of questions for every student. The problem that a student has is whenever a date for the exam is declared the student has to take it and there is no way he can take it at some other time. This project will create an interface for the examiner to create and store questions in a repository. It will also create an interface for the student to take examinations at his convenience and the questions and/or exams may be timed. Thereby creating an application which can be used by examiners and examinee’s simultaneously.
Examination System is very useful for Teachers/Professors. As in the teaching profession, you are responsible for writing question papers. In the conventional method, you write the question paper on paper, keep question papers separate from answers and all this information you have to keep in a locker to avoid unauthorized access. Using the Examination System you can create a question paper and everything will be written to a single exam file in encrypted format. You can set the General and Administrator password to avoid unauthorized access to your question paper. Every time you start the examination, the program shuffles all the questions and selects them randomly from the database, which reduces the chances of memorizing the questions.
KuberTENes Birthday Bash Guadalajara - K8sGPT first impressionsVictor Morales
K8sGPT is a tool that analyzes and diagnoses Kubernetes clusters. This presentation was used to share the requirements and dependencies to deploy K8sGPT in a local environment.
ACEP Magazine edition 4th launched on 05.06.2024Rahul
This document provides information about the third edition of the magazine "Sthapatya" published by the Association of Civil Engineers (Practicing) Aurangabad. It includes messages from current and past presidents of ACEP, memories and photos from past ACEP events, information on life time achievement awards given by ACEP, and a technical article on concrete maintenance, repairs and strengthening. The document highlights activities of ACEP and provides a technical educational article for members.
Harnessing WebAssembly for Real-time Stateless Streaming PipelinesChristina Lin
Traditionally, dealing with real-time data pipelines has involved significant overhead, even for straightforward tasks like data transformation or masking. However, in this talk, we’ll venture into the dynamic realm of WebAssembly (WASM) and discover how it can revolutionize the creation of stateless streaming pipelines within a Kafka (Redpanda) broker. These pipelines are adept at managing low-latency, high-data-volume scenarios.
2. Replication
Process of keeping IDENTICAL COPY
of your data on different servers.
Replication offers
Minimum Downtime
in case of hardware failure, software
failure or maintenance
4. Replica Set
MongoDB’s Terminology for Replication
A replica set is a group of servers with one primary,
the server taking client requests, and multiple
secondaries, servers that keep copies of the primary’s
data.
5. Give A Go To
MongoDB’s Replica Set
In
60 Seconds!
#ToBeTriedOnlyAtHome
6. Step 1: Setup Replica Set
$: mongo --nodb
$: myReplica = new ReplSetTest({nodes: 3});
$: myReplica.startSet();
$: myReplica.initiate();
8. Step 2: Connect To Primary
$: mongo --port 31000
testReplSet:PRIMARY> use surryhills_db
testReplSet:PRIMARY> db.offices.insert({name: ‘MongoDB’});
testReplSet:PRIMARY> show dbs
testReplSet:PRIMARY> show collections
testReplSet:PRIMARY> db.offices.find().pretty()
9. Pro Tip :]
Auto Format Query Response
Add
DBQuery.prototype._prettyShell = true;
to your ~/.mongorc.js
10. Step 3: Connect To A Secondary
$: mongo --port 31001
testReplSet:SECONDARY> db.slaveOk()
testReplSet:SECONDARY> show dbs
testReplSet:SECONDARY> use surryhills_db
testReplSet:SECONDARY> show collections
testReplSet:SECONDARY> db.offices.find()
15. Auto Connect To Current Primary
$: mongo --host myReplica/my-host:31000,my-host:31001,my-host:31002
16. Replica Set
Good To Knows
1. Always start your instance with `replSet`
2. You can’t initiate a replica set with more than
one member with existing data on it.
3. If one of set members has data, you must run
rs.initiate() against that member
4. `system.replset` collection on `local` database
of all members contains set configuration.
5. Replica sets can have up to 50 members, 7 voting
6. By default, MongoDB reads from primary
17. Replica Set Reconfiguration
Add new member
$: mongo --port 31000 # current primary
$: rs.add(‘my-host:31003’) # accepts object as well.
19. Replica Set Reconfiguration
Update existing member
$: mongo --port 31000 # current primary
$: config = rs.config()
$: config.members[2].priority = 0;
$: rs.reconfig(config);
20. How Replication Works
MongoDB keeps a log of every write operations that
primary does in database
Logs are saved in “oplog.rs” collection of “local” db
Secondaries apply changes by reading from oplog.rs and
applieing to their data
Secondaries maintain their own “oplog.rs” as well.
21. “oplog.rs” Facts
“oplog.rs” is a capped collection which means it can
save limited amount of operation logs.
Bulk operations fill “oplog” quickly. A bulk removal of
1K documents adds 1K documents to “oplog”
It’s important to have a “oplog.rs” to keep logs for
atleast last 24 hrs.
22. “Stale” Secondary :(
Secondary misses a copy of last operation it applied in
syncing source.
Secondary queries other members to find another
source to continue syncing.
Secondary missed some operations permanently.
Admin must resyncs from scratch.
23. Heartbeat
Members PING each other every 2 seconds.
Members should
reply within 10 seconds
otherwise they’re counted
as unreachable.
24. Change in replica set configuration
Current primary steps down
Replica Set Election
When It Happens?
Current primary is unreachable
26. You need a majority of voting
members to elect a primary.
A primary can only stay
primary so long as it can reach
a majority of voting members.
Majority
27. Replica Set with two members.
Three Common
Replica Set
Design Mistakes :(
Even number of members in two data center.
A set with more than one arbiter
28. Keep a majority of members in preferred
data center and rest in the other one.
Two Common
Replica Set
Design Patterns :)
Keep even number of members in two data
center and a tie breaker in third one.
29. Meet Arbiter
Budget Friendly Member Of Replica Set
Does not hold data
It’s a lightweight process
Can vote on elections
30. Start An Arbiter Process
$: mongod --port 31005 --replSet myReplica --
nojournal --smallfiles
Arbiter does not save data, no need for journaling
Don’t waste disk space and use --smallfiles
31. Add An Arbiter To Replica Set
$: rs.addArb(‘arbiter-host-name:port’);
32. Arbiter
Good To Know
You can’t change an arbiter to non-arbiter member
If possible, use normal members instead of arbiters
At most you need one arbiter in your set
Do not add an arbiter to a set with odd members
36. Rollback Process
1. Former primary reverts write operations
2. Saves affected documents into /rollback directory
3. Admin browses documents using mongorestore
4. Admin applies changes to the current primary
37. Durability
Write Concerns And Replica Set
1. Unacknowledged
2. Acknowledged
3. Journaled
4. Replica Acknowledged
5. Custom Writer Concern
43. Replica Set &
Mixed Storage Engines
You can have both
WiredTiger & MMAPv1
● Due to migration
● Due to different use cases
44. Keen To Go Further?
Register for M102: MONGODB FOR DBAS course!
Starts tomorrow!
It’s awesome!
https://university.mongodb.com/courses/M102/about
45. Keen To Go Further?
docs.mongodb.org/manual/replication
MongoDB
The Definite Guide
46. Stay In Touch
● Follow me on Twitter
@mehdivk
● Add me to your Linkedin network https://au.
linkedin.com/in/valikhani
● Subscribe to my blog
blog.mehdivk.net
● Send me an email
valikhani@sirangweb.com