Keeping MongoDB Data Safe

8,783 views

Published on

Deck presented at 2012 MongoSF on how you can keep your MongoDB data safe

Published in: Technology
0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
8,783
On SlideShare
0
From Embeds
0
Number of Embeds
2,516
Actions
Shares
0
Downloads
38
Comments
0
Likes
7
Embeds 0
No embeds

No notes for slide

Keeping MongoDB Data Safe

  1. Keeping your MongoDB Data Safe Tony Tam @fehguy
  2. Backups
  3. You care because…• Your data matters• You run experiments on prod data• Your devs have sudo on production• Youve seen this before
  4. BackgroundWho am I?• MongoDB user • Migrated Wordnik to MongoDB in 2009• MongoDB admin • Had to keep it runningWho is Wordnik?• Data-driven technology company• MongoDB is our primary data store
  5. Strategies• Its a function of your data size, state of your business
  6. Whats in the Standard Toolbox• Dump files via mongodump• Exports via mongoexport• Binary data files• Redundancy• Oplog• 3 party or community-developed OSS rd• Hosted MongoDB
  7. The Lazy Developer• One server (Youve been there)• Small data, small usage, small problems• Mongodump is great! • Small(ish) files (gzip will help you) • FAST to create • (typically) FAST to restore via mongorestore
  8. Tradeoffs with dump/restore• Can be done with no downtime. But…• Potentially inconsistent snapshot • Why? One collection at a time • Non-blocking (will yield to writes)• All or nothing• Remove then restore• Restore *might* take time • Indexes!
  9. Tradeoffs with dump/restore• Can be done with no downtime. But…• Potentially inconsistent snapshot • Why? One collection at a time This can • Non-blocking (will yield to writes) take• All or nothing DAYS• Remove then restore• Restore *might* take time • Indexes! Cost of being lazy
  10. Replication• Right! Replica sets• HA by redundancy• Auto fail-over• Maintenance without downtime• You can STILL use mongodump There is NO excuse
  11. Replica Sets• Lost a server? Add a new one • Sync from nearby master • Announce to clients when ready• Time depends on data size • And… oh yea, index size• Gah! WTF!?
  12. Replica Sets• Lost a server? Add a new one • Sync from nearby master • Announce to clients when ready• Time depends on data size • And… oh yea, index size You need• Gah! WTF!? MORE RAM to rebuild indexes!
  13. Replica Sets• They are Awesome! Really! But… Test the process before you need it!
  14. Tradeoffs with Replica Sets• Need multiple servers• Fat finger?• Malicious access?• Software bug?• You still need backups
  15. Options with Replica Sets• Slave Delay • Keep one slave behind by X seconds • *Read* is delayed, not *write*
  16. Options with Replica Sets• Slave Delay • Keep one slave behind by X seconds Fat finger • *Read* is delayed, not *write* problem solved? No! Shut them all down! Hurry!
  17. Alternative to Mongodump?• Snapshot the data files • Stop server, back them up • Its consistent! Snapshot time is well known• Restoring is easy • Copy the files, start a server, add to replica set • NO index rebuilding delays
  18. In action• Stop server• Snapshot data• Archive• Restart• Repeat Daily? Hourly?
  19. Repeat often or lose data!• Data copy time (EC2 => 20mbps if lucky) • 1GB => 1 min • 100GB => 1.5 hours • 1TB => 14 hours• Cant write to data files while copying!
  20. Repeat often or lose data!• Data copy time (EC2 => 20mbps if lucky) • 1GB => 1 min • 100GB => 1.5 hours • 1TB => 14 hours• Cant write to data files while copying! Multiple backup Fancy servers? storage device?
  21. Plain-old copying might not cut it• Many alternatives • EBS Snapshots • Logical Volume Manager (LVM) • RYOR (Roll your own RAID) • Other IT Black Magic
  22. But what about Snapshot Gaps?• The gaps can be real (and painful)• Your DRP might need more • OH, and we still have the fat finger issue • Retention? • "Rollback everything but one operation"?• You can do incremental backups • (with a little help)• Easy to add to your automated snapshots
  23. More about the OpLog• All participating members have one• Capped collection of all write ops t3 time t0 t1 t2 primary replica replica
  24. OpLog for incremental BU• SAME mechanism used by slaves (its rock solid) • Just write operations to disk! Its just BSON• How? (write some code) cursor = oplog.find(); cursor.addOption(Bytes.QUERYOPTION_TAILABLE); cursor.addOption(Bytes.QUERYOPTION_AWAITDATA) ; while(cursor.hasNext) { DBObject x = cursor.next(); outputStream.write(BSON.encode(x)); ... }
  25. OpLog for incremental BU• Already done for youhttps://github.com/wordnik/wordnik-oss• For the lazy:• Get com.wordnik.mongo-admin-utils-distribution from sonatype/maven central./bin/run.shcom.wordnik.system.mongodb.IncrementalBackupUtil -?
  26. Using Wordnik Admin Tools• Start the IncrementalBackupUtil• Write to rotating files, last timestamp• Kill at will• Restart, picks up from last query• Restore using RestoreUtil, mongorestore
  27. How does it work?• Easy, of course
  28. In Summary• Technique depends on your deployment• Lots of tools available• Fine grained control is available Test before you need it!
  29. Questions?

×