Managing a MongoDB Deployment

Managing a MongoDB deployment
Tony Tam @fehguy

Areas of discussion
Many drivers to go non-relational
Maybe you already are?
SOA?
“Yes we should!” Now what?
Implementation details, best practices
Depends on what system you choose
Operational Architecture
Making your deployment “maintainable”
Maximizing performance
Software + Hardware tuning

Wordnik Says Yes to MongoDB
And we’re alive to talk about it
Our stats
14 M REST API calls yesterday
7 ms avg. response
3 TB of MongoDB data
Mostly all “un-cached” responses

Software Architecture
Java/Scala/Jetty application servers
Share (almost) nothing
All roads in must speak REST

Deployment architecture
Wordnik uses physical servers

Operational Constraints
Fundamental question to answer
Can we support this?
You need to consider
If you want it to scale
If you have a SLA to support
If you carry the pager
Also important if
You have customers
You like your job

Operational Constraints
Core items for Wordnik
Redundancy with master/slave
Multi-datacenter deployment possible?
Design for read-only mode
Control at the driver level
Blocking
When is it OK?
Steps around with MongoDB
Consistency
When does it matter?
Scale-out
What limits our speed/size?
Other?
Some features => opportunity

Redundancy
MongoDB supports MS and Replica Sets
RS is like MS with auto-failover
Driver intelligence
Still one master, elected by peers
Master
Slave
Master
Master
Slave
Slave
Slave
Slave
Slave
Slave
Slave

Redundancy
Master/Slave fine for us but…
Not fine for multi operational datacenters
resync?
Your data drives the appropriate solution
Custom tools to support this
https://github.com/wordnik/wordnik-oss
Primary Datacenter
Hot Datacenter
Incremental Backup Files
Master
Master
Replay Util
SCP

Redundancy
Multi-datacenter deployment
Some data need not be consistent
Some does
Replication tricks for master-master
db.documents
documents.src == 1
Master 1
Master 2
db.documents
documents.src == 2

Controlling blocking operations
Important to know what blocks
Some situations can create global locks
Know what happens with:
Adding data
Deleting data
Modifying data
…Then toss in 100 reads/sec
Only access your DB “the right way”
Enforce by good software design
The right way == “right way for your db”

Controlling blocking operations
Wordnik Solution
MongoDB access via service layer
Only allow queries you want run
All others => non-production server, Hadoop
Smart driver manager
Slave bias
$inc:{page_views:1} => $set{page_views:10038}
Read-only mode
Hot-remove server from pool

Handling eventual consistency
EC makes things fast
May create timing issues
Sometimes need current data
val db:DB = getReadWriteConnection()
Avoid this a much as possible
But if you must…
Write then call “getLastError”
Forces commit to DB
RS allows “replicate to n nodes”
Forces commit to master, n slaves

Scale-out
What are you scaling?
Size of data
Size of indexes
Operations
Add slave servers
Won’t help if size > index/server capacity
Sharding if done right
Autoshard solution
Roll your own

Scale-out
Selecting a shard key
Can be tricky
Depends on what you need

Keeping it running
Backups
Export with “mongodump”
No incremental backup solution included
Wordnik tools
https://github.com/wordnik/wordnik-oss

Wordnik OSS tools
Take our tools-They work!!!
SnapshotUtil
Selectively snapshot in BSON
Index info too!
IncrementalBackupUtil
Tail the oplog, stream to disk
Only the collections you want!
Compress & rotate
RestoreUtil
Recover your snapshots
Apply indexes yourself
ReplayUtil
Apply your Incremental backups

What if Scenarios
One collection gets corrupt?
Restore it
Apply all operations to it
“My top developer dropped a collection!”
Restore just that one
Apply operations to it until that POT
“We got hacked!”
Restore it all
Apply operations until that POT

Wrap up: We like MongoDB
See more @ http://developer.wordnik.com
Questions?

Managing a MongoDB Deployment

by Tony Tam on Feb 17, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Managing a MongoDB Deployment — Presentation Transcript