"MongoDB isn't just software we release; it's software we rely on every day as part of the MongoDB Management Service (MMS). We use MongoDB to store monitoring data and backups for the databases of over 50,000 customers. Get the inside scoop on running an overwhelmingly write-heavy application as MMS engineers Steve Briskin and John Morales explain the tweaks and optimizations they use to get most out of every machine, including: - RRD schema design to balance write throughput and read latency - Warming indexes before inserts - Backup snapshot storage design optimized for insert-only workloads They will also discuss the process of migrating to 3.0 and how it has simplified their lives."

1. Evolution and Scaling of MongoDB
Management Service Running on MongoDB
Steve Briskin, Lead Engineer, MMS Backup
John Morales, Senior Engineer, MMS Monitoring

2. 2
Agenda
● What is MongoDB Management Service
● MMS Backup
o Schema evolution and optimizations
o How we scaled
● MMS Monitoring
o Read-optimized time series schema
o Write-optimized time series schema
o Benchmarks

3. 3
MongoDB Management Service (MMS)
mms.mongodb.com
Automation and Provisioning
Single-click provisioning, scaling &
upgrades, administrative tasks
Monitoring
Charts, dashboards, and alerts on 100+
metrics
Backup
Backup and restore, with point-in-time
recovery, and support for sharded clusters

4. 4
MMS Backup
● Cloud Backup service
● Takes periodic snapshots
● Manages storage
● Premium features
o Point in time recovery
o Consistent snapshots of sharded clusters

5. 5
MMS Backup Architecture
Backup
Agent
MMS Backup
Ingestion
MMS Backup
Backup Process
Customer’s
MongoDB
Oplog
Store
Snapshot
Store

6. 6
Oplog Store
● “Circular Buffer” of operations
o Many concurrent inserts
o Time-bound (e.g. 24 hours)
o Lifecycle: insert, read once, delete
● Concerns
o Lock contention
o Data purging
o Freelist fragmentation
MMS Backup
Ingestion
Oplog
Store

7. 7
Oplog Store
● Developed with MongoDB 2.2
● Lock Contention
o DB per customer
● Data Purging
o Use TTL Index
● Freelist fragmentation
o Use Power Of 2 Allocation
MMS Backup
Ingestion
Oplog
Store

8. 8
Oplog Store on MongoDB 3.0
• MongoDB 3.0
o More granular locking
o Freelist management improvements
Upgrade
Upgrade
30% Faster

9. 9
Snapshot Storage (Blockstore)
● Backup Snapshot Storage
o File storage in MongoDB
● Design
o Block: 64KB - 15MB of binary data
 SHA256 hash as a unique identifier
o File: List of blocks
o Insert only schema
o De-duplication and compression
MMS Backup
Backup Process
Snapshot
Store

10. 10
Blockstore
● Insert Only + Power Of 2 Allocation = Wasted Space
o Example: 9k document will use 16k
o Worst case: Need 2x disk space
● Writes are sporadic
o Indexes are cold and need to be paged in
o Can be slow and I/O-expensive

11. 11
Blockstore
● Disable Power Of 2 Allocation
o MongoDB 2.2 - 2.6:
db.runCommand({collMod : “collection”,
usePowerOf2Sizes : false})
o MongoDB 3.0:
db.runCommand({collMod : “collection”,
noPadding : true})
● Warm indexes before bulk insertions
db.runCommand({touch : “collection”,
index : true,
data : false})

12. 12
Scaling - Replica Sets
• Started with a single replica set
• Split into purpose-based replica sets
Blockstore (Large HDDs)
Primary
Secondary Secondary
Backup Metadata (Small SSDs)
Primary
Secondary Secondary Secondary Arbiter
Oplog Store (Small HDDs)
Primary
Secondary Secondary Secondary Arbiter

13. 13
Scaling - Application Sharding
• Application sharding for horizontal scaling
• Each customer is assigned to one replica set
Application
Customer A
Customer B
Customer C
Blockstore_1
Primary
Secondary Secondary
Blockstore_2
Primary
Secondary Secondary
Blockstore_0
Primary
Secondary Secondary

14. 14
Scaling - MongoDB Sharding
• MongoDB Sharding
Application
Blockstore_shard_1
Primary
Secondary Secondary
Blockstore_shard_2
Primary
Secondary Secondary
Blockstore_shard_0
Primary
Secondary Secondary
Mongos Mongos

15. MMS Monitoring Schema Evolution

16. 16
Introduction to MMS Monitoring
Design Objectives circa 2012
•Fast chart load times
•Chart ~80 metrics per host
•Minute-level resolution
Inherent Advantages
•Control our own rate of samples
Browser Users
Monitoring Agent
Metric Data
Sharded ClusterCustomer
Deployment
mms.mongodb.com

17. 17
Circa 2012: Read-Optimized Schema
{
hid: “id”, // Host ID
cid: ObjectId(“...”), // Group ID
g: “network”, // Metric group
i: “bytesOut”, // Specific metric
mn: { // hour worth of points stored
together
“00”: {
n: NumberLong(“...”), // value
t: 1430918626 // time
},
“01”: {
...
},
...,
“59”: { ... }
}
}
● Store points for same metric together

18. 18
Circa 2012: Scaling up Writes
● Write Performance when Read-Optimized
○ Updates $set the time and value sub-doc
○ Documents grow, move on disk
○ I/O mostly random
● Mitigate
○ Ensure updates always in-place (MMAPv1-only)

19. 19
Circa 2012 to Today: Performance
Hooray
●Average chart load time: 15ms
●Today MMS actively monitoring 60k+ hosts
●Storing average of 128 metrics per host

20. 20
2015: What’s Next
Upcoming MMS Monitoring features
●High resolution monitoring
●Charting more metrics
Whoops
●Read-optimized schema inflexible
●Each new metric means new write

21. Coming Soon: Write-Optimized
{
"_id" : “...”,
"n" : { // network
"bi" : NumberLong(123), // bytesIn
"bo" : NumberLong(234), // bytesOut
"r" : NumberLong(34), // requests
},
"e" : { // page faults
"pf" : NumberLong(3564),
},
"g" : { // queues
"cr" : NumberLong(3564),
},
...,
// sample time for all these points
"t" : ISODate("2015-06-02T15:35:43.189Z")
}
● Store points across metrics together
● Insert-only versus random updates

22. 22
Benchmarking and Tradeoffs
Writes: time (millis) to ingest 4500 hosts Read Latency: millis to read 24-hour chart
200x more write
throughput
~18ms latency tradeoff

23. Wrap Up and Q & A
● Tailoring configuration for workload
● Schema design and managing tradeoffs
● IOPS often the limiting resource