Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DevOps Fest 2020. Николай Маржан. Consistent backups of multi-shard MongoDB

11 views

Published on

It only sounds like a simple thing to make a consistent backup of the database which builds around eventual consistency ideas.
Let's review the current types of backups and their limitations when it comes to sharding. From there we will move into why you can't be consistent with a single node, and how you can take sharded or unsharded consistent backups.

Published in: Economy & Finance
  • Be the first to comment

  • Be the first to like this

DevOps Fest 2020. Николай Маржан. Consistent backups of multi-shard MongoDB

  1. 1. Percona Backup for MongoDB Solution for consistent backups of MongoDB sharded cluster
  2. 2. The Main Problem Consistency If there are write operations during the dump operation, the dump will not reflect a single moment in time. Changes made to the database during the backup process can affect the output of the backup. Source: https://docs.mongodb.com/manual/reference/program/mongodump/ © 2019 Percona 2
  3. 3. Single-Shard Backups Use “mongodump --oplog” option Creates a oplog file as part of the mongodump output. Which contains oplog entries that occur during the mongodump operation. This file provides an consistent snapshot of the state of a mongod instance. Source: https://docs.mongodb.com/manual/reference/program/mongodump/ © 2019 Percona 3
  4. 4. Single-Shard Backups 4 Start Consistencyattheend mongodump oplog
  5. 5. Multi-Shard Backups Backups created with mongodump do not maintain the atomicity guarantees of transactions across shards. Source: https://docs.mongodb.com/manual/reference/program/mongodump/ © 2019 Percona 5
  6. 6. Multi-Shard Backups 6 Start NOConsistency ShardB mongodump ShardB oplog ShardC mongodump ShardA oplog ShardC oplog ShardA mongodump
  7. 7. Why? With mongodump it is basically impossible to stop oplog exactly on the same transaction on all shards simultaneously without stop writes. © 2019 Percona 7
  8. 8. Multi-Shard Backups 8 Start Consistencyattheend ShardB mongodump ShardB oplog ShardC mongodump ShardA oplog ShardC oplog ShardA mongodump
  9. 9. Percona Backup for MongoDB Solution for consistent backups of MongoDB sharded cluster
  10. 10. Agent ● Agent must be attached on localhost of each mongod instance (including secondaries and config server instances). ● Agent connects and works only with/via MongoDB. No need to handle additional security aspects: open ports, TLS/SSL, authorization etc. ● Agent detects if it is a good candidate to do the backup or restore operations and coordinates with the other instances to accomplish the requested actions. © 2019 Percona10
  11. 11. CLI ● Administrators observe and control the backups or restores with a pbm CLI command that they can run from any host with the access to the MongoDB cluster. ● CLI connects and works only with/via MongoDB. Not needed to handle agent addresses, TLS/SSL, authorization etc. © 2019 Percona11
  12. 12. How to start using it? ● Create user If authentication is enabled ● Run agent locally for each mongod instance ● Configure location of remote storage ● Run “pbm backup” command © 2019 Percona12
  13. 13. Current state and Roadmap
  14. 14. Current state ● MongoDB 3.6, 4.0, 4.2 versions are supported. ● Backup and restore via S3-protocol. Officially supported object stores: AWS S3, GCP GCS (interoperability), MinIO. ● Backup and restore from locally mounted NAS/SAN. ● Take backup from Secondary or hidden nodes if possible. ● Klaus Post S2 compression (extension of Snappy) enabled by default for oplogs and logical backups. Gzip and LZ4 are supported also. © 2019 Percona14 Percona Solutions
  15. 15. Roadmap ● Recovery to any point-in-time (PITR) ● Centralize logs ● Encryption support ● Enhanced selection of backup source © 2019 Percona15 Percona Solutions
  16. 16. We are Hiring! Speak to me ;) © 2019 Percona16
  17. 17. Q & A Mykola Marzhan Director of Server Engineering

×