MongoDB Backups and PITR

MongoDB Backups and PITR
Igor Donchovski
May-23-2023
© Pythian Services Inc 2023 | Confidential|

About me
---
Name: Igor Donchovski
Company: Pythian
Email: donchovski@pythian.com
Current_Role: Principle Consultant
Education:
- Type: Master
Name: FINKI
Graduated: 2013
University: UKIM
Certiﬁcates:
- MongoDB: C100DBA
- Oracle:
- MySQL 5.7 Professional
- OCI Foundations Associate
- HashiCorp: Terraform Associate (002)
- Google: PCA
Social:
- Network: LinkedIn
url: https://mk.linkedin.com/in/igorle
- Network: Twitter
url: https://twitter.com/igorle
© Pythian Services Inc 2023 | Conﬁdential | 2

25
Years in Business
450+
Experts across every Data Domain & Technology
400+
Global Customers
_______________
Gold Partner
40+ Certifications
5 Competencies,
Incl. Data Analytics
and Data Platform
_______________
Silver Partner
50+ Certifications
Migration Factory
Certified Apps
Hosting
_______________
Advanced Partner
1O0+ Certs/Accreds
Migration & DevOps
Competencies
Pythian overview
________________
Premier Partner
120+ Certs/Creds
6 Specializations, MSP
Badge, Partner &
Technical Advisory Bds
Pythian maximizes the value of your data estate by delivering advanced on-prem, hybrid, cloud, and
multi-cloud solutions and solving your toughest data and analytics challenges.
_______________
Select Partner
10+ Certs/Accreds
_______________
Platinum Partner
60+ Certifications
Advisory Board
Member
© Pythian Services Inc 2023 | Confidential | 3

Overview
• Why Backups
• RTO and RPO
• Types of backups
• MongoDB speciﬁcs
• PITR
• ChatGPT feedback
• QA

Why Backups

Why Backups
High Availability
Disast r Re ov ry
Scalability
Elasticity
Auto ation
Performance

● Power outage
● Software issues
● Hardware failure
● User error
● Network outages
● Natural disaster
● Other
Why Backups
High Availability
Scalability
Elasticity
Disast r Re ov ry
Performance

● RTO - recovery time objective
○ How long it will take to recover?
● RPO - recovery point objective
○ How much data can we lose?
Why Backups
TIME
Disaster
RTO
RPO
LOST DATA DOWNTIME

Types of Backups

Types of Backups
Logical
• Easy to implement
• Human readable
• Slow for larger dataset
• Might affect database performance
• mongodump (Binary export. Does not contain the index data)
• mongoexport (json or csv export for collection)
• Works with Replica sets and Sharded clusters

Types of Backups
Physical
• Moderate to implement
• Cold or hot
• Filesystem copy (cp, rsync)
• LVM Snapshot
• Cloud based disk snapshot
• Percona backup for MongoDB
• MongoDB Cloud manager or Ops manager

● Sharded clusters
● Replica sets
Types of Backups - MongoDB

● Replica sets
Types of Backups - MongoDB
● Sharded clusters

Logical Backups

Logical Backup
• mongodump
mongodump --host mongodb1.example.net --port 27017 --username user --password pass --oplog
--out /backup/mongodump-2023-05-23
Optional
--db Speciﬁes the database to backup. Does not work with --oplog
--collection Speciﬁes the collection to backup
--query Limits the documents returned in the output
--gzip Compresses the output
--numParallelCollections=N (No of collections to export in parallel. Default to 4)
• mongorestore
mongorestore --host mongodb1.example.net --port 27017 --username user --password pass
--OplogReplay --oplogLimit <timestamp> /backup/mongodump-2023-05-23

Logical Backup
• mongoexport
mongoexport --host mongodb1.example.net --port 27017 --username user --password pass --db
marketing --collection contacts --out /backups/contacts.json
Optional
--csv Changes the export format to CSV
--fields Specifies a field or fields to include in the export
--query Limits the documents returned in the output
• mongoimport
mongoimport --host mongodb1.example.net --port 27017 --username user --password pass --db
marketing --collection contacts --file /backups/contacts.json

Physical Backups

Physical Backups
● Run on a Secondary node for replica set (Hidden, Delayed)
● Lock the database for writes
> db.fsyncLock()
● Take backup from the disk[s] relevant for mongo
● Grab the latest oplog position (optional)
● Unlock the database for writes
> db.fsyncUnlock()
● Copy the backup to offsite location

Physical Backups
Filesystem copy (cp, rsync)
● Stop mongod process
● Rsync or cp the ﬁles to external location
$ rsync -av --partial --inplace --append --progress
user@mongodb1.example.net:/backups/mongodb
● Start mongod process
● Might take a while for large datasets

Physical Backups
LVM snapshot
● Hot backup on running mongod
> db.fsyncLock()
lvcreate -L5G -s -n mongosnap_23may2023 /dev/vgdiskdata/lvmongo
➔ /dev/vgdiskdata/lvmongo - Logical volume path
➔ lvmongo - LV Name
➔ -L Size of the snapshot
➔ -s It’s a snapshot
➔ -n Name for the snapshot

Physical Backups
LVM snapshot
mkdir /tmp/mongosnap
mount -t xfs -o nouuid,ro /dev/vgdiskdata/mongosnap_23may2023 /tmp/mongosnap/
> db.fsyncUnlock()
tar -czf /backups/mongodb_backup_$(date '+%Y%m%d%H%M').tar.gz --absolute-names
/tmp/mongosnap
umount /tmp/mongosnap
lvremove /dev/vgdiskdata/mongosnap_23may2023

Physical Backups
Cloud based disk snapshot
AWS
aws ec2 create-snapshot --volume-id VOLUME-ID --description "This is my mongo
volume snapshot"
GCP
gcloud compute snapshots create SNAPSHOT_NAME --source-disk SOURCE_DISK
--source-disk-zone SOURCE_DISK_ZONE
Azure
az snapshot create -g ResourceGroup --source MONGO-DISK-ID --name mongoDisk-bak

Physical Backups
Percona backup for MongoDB
● Logical and physical backups and restores
● Selective logical backups and restores with PITR
● Works for both sharded clusters and non-sharded replica sets
● Can be used with any S3-compatible storage
● Support for Microsoft Azure Blob storage
● Supports ﬁlesystem storage type for locally mounted remote ﬁlesystem
backup servers

Physical Backups
PBM Architecture
● pbm-agent is a process running on every mongod node within the cluster or a replica
set that performs backup and restore operations
● pbm CLI is a command-line utility that instructs pbm-agents to perform an operation
● PBM Control collections are special collections in MongoDB that store the
conﬁguration data and backup states
● Remote backup storage as either s3-compatible or ﬁlesystem type storage
pbm backup --type=logical|physical --compression=gzip

Physical Backups
MongoDB Cloud manager backup

Physical Backups
MongoDB Cloud manager backup
● Requires monthly subscription based on snapshot size, schedule, and retention policy
● Install the backup agent and conﬁgure in cloud.mongodb.com
● Optional namespace ﬁlter to exclude collections from the backup
● Point in time recovery with low overhead
● Data transfer to Cloud manager datacenters is encrypted
● Stores at least 3 copies of your data in at least 2 geographic locations to ensure
redundancy
● Only works with deployments of up to 100k collections
● Snapshot lowest frequency of 6 hours

Physical Backups
MongoDB Ops manager backup
● Must run with MongoDB enterprise
● Works with deployments of up to 100k
collections
● Replica set, sharded cluster, and shard
names within the same project must be
unique

PITR

PITR
TIME
RTO
LOST DATA DOWNTIME
Full backup
RPO
Disaster

PITR
Incremental backups
TIME
RTO
Full backup LOST DATA DOWNTIME
Inc backup
Disaster

PITR
Incremental backups
● RPO ~ 0
TIME
RTO
Full backup LOST
DATA
DOWNTIME
Inc 1 Inc 2 Inc 3 Inc n
Disaster

PITR
Full backup: lvm snapshot + Incremental backups using the Oplog
root @mongodb: ~# mongo - u < username > -p < password > --port < port > --quiet backup_oplog_ts.js > oplog_position
cat backup_oplog_ts.js
var local = db.getSiblingDB('local');
var last = local['oplog.rs'].find().sort({
'$natural': -1
}).limit(1)[0];
var result = {};
if (last != null) {
result = {
position: last['ts']
};
}
print(JSON.stringify(result));
TIME
Full backup
"$timestamp": {"t": 1666355398,"i": 1}

PITR
Oplog backup using mongodump
Inc 1 start position
{"position": {"$timestamp": {"t": 1666355398,"i": 1}}}
mongodump --quiet -u < username > -p < password > --port < port > --authenticationDatabase = admin - d local - c oplog.rs --query='{"ts": {
"$gt": {"$timestamp":{"t":1666355398,"i":1}}}}' -o/backups/mongo_20230523/incremental_1
Inc 2 start position (end of Inc 1)
bsondump --quiet /backup/mongo_20230523/incremental_1/local/oplog.rs.bson | tail -1 | jq -c .ts
{"$timestamp": {"t": 1666616998, "i": 1 }}
TIME
Full backup
Inc 1

PITR
Inc 2
mongodump --quiet -u < username > -p < password > --port < port > --authenticationDatabase=admin -d local -c oplog.rs --query='{"ts": { "$gt":
{"$timestamp":{"t":1666616998,"i":1}}}}' -o /backups/mongo_20230523/incremental_2
Inc 3 start position (end of Inc 2)
bsondump --quiet/backup/mongo_20230523/incremental_2/local/oplog.rs.bson | tail -1 | jq -c .ts
{"$timestamp": {"t": 1666617939,"i": 1}}
TIME
Full backup

PITR
Inc 3
mongodump --quiet -u < username > -p < password > --port < port > --authenticationDatabase=admin -d local -c oplog.rs --query='{"ts": { "$gt":
{"$timestamp":{"t":1666617939,"i":1}}}}' -o /backups/mongo_20230523/incremental_3
Inc n position: (Inc(n) = Between Inc(n-1) and Full backup(m) )
{"ts": {"$gt": {"$timestamp": {"t": 1666619908,"i": 1}}, "$lte": {"$timestamp": {"t": 1666619998,"i": 1}}}}
TIME
Full backup(n)
Full backup(m)

PITR
Full backup : 1pm
Incremental backups: every 1 hour
4:30pm - Delete operation on users.meta namespace
TIME
Disaster
RTO
Full backup(n)
DOWNTIME
RPO

PITR
Identify the erroneous operation start time
bsondump --quiet /backup/mongo_20230523/oplog_4/local/oplog.rs.bson | grep '"op":"d","ns":"users.meta"' {"op": "d","ns": "users.meta","ui":
{"$binary": {"base64": "ZlkpNQpySoG98UFLQvYZnw==","subType": "04"}},"o": {"_id": {"$oid": "636a7cd663089cc37ae7c96c"}},"ts": {"$timestamp":
{"t": 1684859539,"i": 1}},"t": {"$numberLong": "6"},"v": {"$numberLong": "2"},"wall":{"$date": {"$numberLong": "1667917939452"}}}
Erroneous operation : "ts": {"$timestamp": {"t": 1684859539,"i": 1}}
TIME
RTO
Full backup(n)
DOWNTIME
RPO
Disaster

PITR
Restore the Full backup(n)
Restore the incremental backups using mongorestore --oplogReplay
cp /backup/mongo_20230523/oplog_1/local/oplog.rs.bson /backup/oplogR1/oplog.bson
mongorestore --port 57017 –u < username > -p < password > --authenticationDatabase=admin --oplogReplay /backup/oplogR1
TIME
RTO
Full backup(n)
DOWNTIME
RPO
Disaster

PITR
Erroneous operation : "ts": {"$timestamp": {"t": 1684859539,"i": 1}}
Restore the ﬁnal incremental backup using mongorestore --oplogReplay --oplogLimit
cp /backup/mongo_20230523/oplog_4/local/oplog.rs.bson /backup/oplog4/oplog.bson
mongorestore --port 57017 -u < username > -p < password > --authenticationDatabase=admin --oplogReplay --oplogLimit 1684859539, 1 /backup/oplog4
TIME
RTO >= 0
Full backup(n)
DOWNTIME
RPO=0
Disaster

Summary
• Run backups as part of your organization’s DR strategy
• Logical backups work well with smaller DB size, for larger DBs use physical
• Backups are only good as long as they can be restored
• For Replica sets run backups on a MongoDB Secondary node (hidden or
delayed member)
• For Sharded clusters you need to backup each shard and the conﬁg servers
metadata while the balancer is stopped
• Cloud manager/Ops manager backup are enterprise licenced
• Full physical backup and incremental oplog backups will allow you PITR
• RPO ~ 0 means you need to take backups more frequently
• RTO ~ 0 means you need to invest in automating the restore

ChatGPT - Explain best practices for MongoDB backups and PITR
MongoDB backups and point-in-time recovery (PITR) are critical aspects of ensuring data availability and recovery in the event of data loss or system failure.
Here are some best practices to consider when performing backups and PITR in MongoDB:
1. Choose the right backup strategy: MongoDB offers several backup strategies, including file system backups, logical backups, and
continuous backups. Choose the right backup strategy based on your business requirements, data size, and recovery objectives.
2. Set backup schedules and retention periods: Create a backup schedule that fits your business needs and set a retention period that
aligns with your recovery objectives. Ensure that you have enough backup storage to retain backups for the desired duration.
3. Use replica sets: MongoDB replica sets provide high availability and data redundancy. By configuring a replica set, you can create
multiple copies of your data across different nodes. This ensures that you always have a copy of your data available for recovery.
4. Enable journaling: Journaling is a feature that ensures data consistency and durability. By enabling journaling, MongoDB can recover
data from incomplete write operations in the event of a system failure.
5. Monitor backup performance: Monitor backup performance to ensure that backups are completed within the scheduled window. Use
tools such as MongoDB Management Service (MMS) to monitor backup status and performance.
6. Test backups regularly: Regularly test your backups to ensure that you can restore your data when needed. Perform a full restore
from a backup to a separate environment to test backup integrity and recovery procedures.
7. Implement PITR: Point-in-time recovery (PITR) allows you to restore your database to a specific point in time. This can be useful
when you need to recover from a data corruption or user error. Ensure that you have enabled oplog (operation log) to enable PITR.
8. Use automation: Use automation tools to simplify backup and PITR operations. Tools such as MongoDB Ops Manager and MongoDB
Atlas provide automated backup and recovery processes.
By following these best practices, you can ensure that your MongoDB backups and PITR processes are reliable and effective in maintaining data
availability and recovery.

References
How to backup MongoDB database using lvm snapshots - Part 1
MongoDB incremental backups using the oplog - Part 2
MongoDB Backups and Point In Time Recovery - Part 3
Percona Backup for MongoDB Documentation
FAQ: Backup and Restore — MongoDB Cloud Manager
FAQ: Backup and Restore — MongoDB Ops Manager
MongoDB Backup and Restore - Best Practices and Techniques

MongoDB Backups and PITR

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to MongoDB Backups and PITR

Similar to MongoDB Backups and PITR (20)

More from Igor Donchovski

More from Igor Donchovski (8)

Recently uploaded

Recently uploaded (20)

MongoDB Backups and PITR