The presentation discussed MongoDB's approach to modern data backup and recovery from on-premises to the public cloud. It introduced WiredTiger checkpoints that take checkpoints of data and move them to long-term storage, eliminating the need for initial syncs and reducing infrastructure requirements. It also discussed Atlas' multi-AZ disaster recovery, point-in-time data recovery with queryable backups, and continuous backup regions using cloud provider snapshots to satisfy data sovereignty at a lower cost. Future improvements may allow point-in-time restores with customizable windows and building new clusters from backups.
MongoDB World 2019: Modern Data Backup and Recovery from On-premises to the Public Cloud
1. Benjamin Cefalo, Lead Product Manager, MongoDB
Modern Data Backup and Recovery from On Premises to the
Public Cloud
@bencefalo
2. Safe Harbor Statement
This presentation contains “forward-looking statements” within the meaning of Section 27A of the Securities Act of 1933,
as amended, and Section 21E of the Securities Exchange Act of 1934, as amended. Such forward-looking statements are
subject to a number of risks, uncertainties, assumptions and other factors that could cause actual results and the timing of
certain events to differ materially from future results expressed or implied by the forward-looking statements. Factors that
could cause or contribute to such differences include, but are not limited to, those identified our filings with the Securities
and Exchange Commission. You should not rely upon forward-looking statements as predictions of future events.
Furthermore, such forward-looking statements speak only as of the date of this presentation.
In particular, the development, release, and timing of any features or functionality described for MongoDB products
remains at MongoDB’s sole discretion. This information is merely intended to outline our general product direction and it
should not be relied on in making a purchasing decision nor is this a commitment, promise or legal obligation to deliver any
material, code, or functionality. Except as required by law, we undertake no obligation to update any forward-looking
statements to reflect events or circumstances after the date of such statements.
3. Quick level set — We’re not going to talk about
disaster recovery
19. What is it?
● Enhanced WiredTiger to take checkpoints of itself
● Checkpoints are then moved to long term storage
● HeadDB’s are completely eliminated — no more initial syncs!
○ Reduced storage requirements
○ Reduces architectural complexities
○ Reduces infrastructure required
● Consolidated agents, from three agents to one
20. ...
...
...
...
Oplog04
Oplog02
Blockstore06
Blockstore04
Blockstore02
Oplog06
On-Premises Backup Architecture Today
Ops Manager
OM01
OM02
OM03
OM04
OM05
OM06
Backup Agent
Monitoring Agent
Backup Agent
Monitoring Agent
Backup Agent
Monitoring Agent
Backup Agent
Monitoring Agent
Backup Agent
Monitoring Agent
Backup Agent
Monitoring Agent
OM Group
BackupLoadBalancer
App Server for
Backup
App Server for
Backup
App Server for
Backup
Oplog01
Oplog03
Oplog05
Backup
Daemon
HeadDB
Backup
Daemon
HeadDB
Backup
Daemon
HeadDB
Blockstore01
Blockstore03
Blockstore05
21. Oplog02
Blockstore
New Backup Architecture
Ops Manager
OM01
OM02
OM03
OM04
OM05
OM06
MongoDB Agent
MongoDB Agent
MongoDB Agent
MongoDB Agent
MongoDB Agent
MongoDB Agent
OM Group
BackupLoadBalancer
App
Server/Backup
Daemon
App
Server/Backup
Daemon
Oplog01
S3 Snapshot Store
22. Backup Node Selection Order
1. Hidden Secondaries
2. A secondary we have already taken a snapshot from
3. Secondary closet to the time of the time of the snapshot
4. Any available secondary
5. Primary
23. Phase 1 Details
• Backup agent required to exist on every node
• OM 4.2 required for backing up 4.2
• Backwards compatible for 4.0 and below
• Daemons are used for background tasks
• Replica sets only, sharded clusters coming later
• Advanced features coming later
24. S3 Oplog
Blockstore
New Backup Architecture — Future
Ops Manager
OM01
OM02
OM03
OM04
OM05
OM06
MongoDB Agent
MongoDB Agent
MongoDB Agent
MongoDB Agent
MongoDB Agent
MongoDB Agent
OM Group
BackupLoadBalancer
App Server for
Backup
App Server for
Backup
Oplog01
S3 Snapshot Store
28. What this means for you...
● Less people, time, and energy you have to spend to deliver a best-in-class data
recovery solution for your on-premises and self-managed MongoDB
deployments
30. Private DBaaS: On-Prem Public DBaaS: Fully Managed
Built on the Same Code Base, Same API, Same Management UI
Atlas: Same Features as Ops Manager & Cloud Manager
Hybrid DBaaS
31. The Latest MongoDB Features
• MongoDB Atlas comes out-of-the-box with
MongoDB 3.4, 3.6, 4.0
• Transactions (4.0)
• Change Streams (3.6)
• JSON Schema (3.6)
• Expressive nested array updates (3.6)
• Expressive joins: $lookup (3.6)
• Graph queries (3.4)
• Facets & expressive aggregations (3.4)
• Minor updates and major upgrades without
downtime
● Data Explorer
○ Inspect schema & index utilization
● Real-Time Performance Panel
○ Live telemetry: in-flight operations & resource
consumption
● Performance Advisor
○ Always-on index recommendations
33. ...with Optional Multi-Region Fault Tolerance
AZ IIAZ I AZ III
Preferred Region
AZ I AZ II AZ I AZ II
Failover Region Failover Region
34. Atlas Backups: Data Recovery
Roll back the clock when you run
into issues triggered by user or
application errors that are
replicated from the primary to
the rest of your cluster.
35. Point-in-Time Data Recovery
• Lets you select a restore time based on your PIT window
• Restores the closest snapshot and rolls ahead
• Reduces the possibility of data loss
36. What About Small Disasters?
• The application is working fine
• But there is data missing or has
been altered
• No time to do a full restore
37. Queryable Backups
• Ability to query your snapshots and
restore data at the document level in
minutes.
• Reduces the operational overhead
associated with:
• Identifying whether data of interest has
been altered
• Pinpointing the best point in time to
restore a database
38. db = source.locations
db2 = destination.locations
zips = db.zipcodes
zips2 = db2.zipcodes
def restore():
print "Finding Missing Data"
query = {'state': ‘NY'}
try:
cursor = zips.find(query)
except Exception as e:
print "Unexpected error:", type(e), e
for doc in cursor:
zips2.insert(doc)
Sample Queryable Script
41. Cloud Provider Snapshots (CPS)
• Utilizes each providers native snapshot capabilities
• Granular backup region selection
• Satisfy data sovereignty requirements
• Supports replica sets and sharded clusters
• Pricing is based on snapshot size, not data size
• Less expensive, starting at $0.08 per GB of snapshot
size (varies per provider and region)
• Now available on all cloud providers!
• AWS and GCP snapshots are incremental
Incremental snapshots for Atlas
customers deploying on AWS
10 GB 15 GB (5 new) 20 GB (5 new)
42. Cloud Provider Snapshots (CPS) Updates — Continued
• Bring your own keys
• AWS KMS
• Azure Key Vault
• GCP KMS
• Backup Policies
• On-demand snapshots (quicksave)
• Takes a snapshot immediately if there’s not already one in progress
• API for pipeline integrations
• M2/M5 Backups (Coming Soon!)
47. Cloud Provider Snapshots - Future
● Point in Time Restores with customizable window
● Build a new cluster from a backup
● Queryable Backup with Selective restore
48. Pick the fully managed backup solution that’s right for you
Within Atlas is a modern data recovery platform that enables you to choose how you want to backup your
data, depending on your requirements.
Continuous
● Point-in-time restore
○ All but eliminate the possibility of data
loss
● Queryable snapshots
○ Rapidly restore granular data
Cloud Provider Snapshots
● Localized backup
○ Satisfy data sovereignty requirements
● Bring your own keys
● Take snapshots on-demand
● The cost effective option