Managing a MongoDB deployment<br />Tony Tam @fehguy<br />
Areas of discussion<br />Many drivers to go non-relational<br />Maybe you already are?<br />SOA?<br />“Yes we should!”  No...
Wordnik Says Yes to MongoDB<br />And we’re alive to talk about it<br />Our stats<br />14 M REST API calls yesterday<br />7...
Software Architecture<br />Java/Scala/Jetty application servers<br />Share (almost) nothing<br />All roads in must speak R...
Deployment architecture<br />Wordnik uses physical servers<br />
Operational Constraints<br />Fundamental question to answer<br />Can we support this?<br />You need to consider<br />If yo...
Operational Constraints<br />Core items for Wordnik<br />Redundancy with master/slave<br />Multi-datacenter deployment pos...
Redundancy<br />MongoDB supports MS and Replica Sets<br />RS is like MS with auto-failover<br />Driver intelligence<br />S...
Redundancy<br />Master/Slave fine for us but…<br />Not fine for multi operational datacenters<br />resync?<br />Your data ...
Redundancy<br />Multi-datacenter deployment<br />Some data need not be consistent<br />Some does<br />Replication tricks f...
Controlling blocking operations<br />Important to know what blocks<br />Some situations can create global locks<br />Know ...
Controlling blocking operations<br />Wordnik Solution<br />MongoDB access via service layer<br />Only allow queries you wa...
Handling eventual consistency<br />EC makes things fast<br />May create timing issues<br />Sometimes need current data<br ...
Scale-out<br />What are you scaling?<br />Size of data<br />Size of indexes<br />Operations<br />Add slave servers<br />Wo...
Scale-out<br />Selecting a shard key<br />Can be tricky<br />Depends on what you need<br />
Keeping it running<br />Backups<br />Export with “mongodump”<br />No incremental backup solution included<br />Wordnik too...
Wordnik OSS tools<br />Take our tools-They work!!!<br />SnapshotUtil<br />Selectively snapshot in BSON<br />Index info too...
What if Scenarios<br />One collection gets corrupt?<br />Restore it<br />Apply all operations to it<br />“My top developer...
Wrap up: We like MongoDB<br />See more @ http://developer.wordnik.com<br />Questions?<br />
Upcoming SlideShare
Loading in …5
×

Managing a MongoDB Deployment

6,306 views

Published on

Talk by Wordnik's Tony Tam on managing a large scale MongoDB deployment

Published in: Technology
  • Be the first to comment

Managing a MongoDB Deployment

  1. 1. Managing a MongoDB deployment<br />Tony Tam @fehguy<br />
  2. 2. Areas of discussion<br />Many drivers to go non-relational<br />Maybe you already are?<br />SOA?<br />“Yes we should!” Now what?<br />Implementation details, best practices<br />Depends on what system you choose<br />Operational Architecture<br />Making your deployment “maintainable”<br />Maximizing performance<br />Software + Hardware tuning<br />
  3. 3. Wordnik Says Yes to MongoDB<br />And we’re alive to talk about it<br />Our stats<br />14 M REST API calls yesterday<br />7 ms avg. response<br />3 TB of MongoDB data<br />Mostly all “un-cached” responses<br />
  4. 4. Software Architecture<br />Java/Scala/Jetty application servers<br />Share (almost) nothing<br />All roads in must speak REST<br />
  5. 5. Deployment architecture<br />Wordnik uses physical servers<br />
  6. 6. Operational Constraints<br />Fundamental question to answer<br />Can we support this?<br />You need to consider<br />If you want it to scale<br />If you have a SLA to support<br />If you carry the pager<br />Also important if<br />You have customers<br />You like your job<br />
  7. 7. Operational Constraints<br />Core items for Wordnik<br />Redundancy with master/slave<br />Multi-datacenter deployment possible?<br />Design for read-only mode<br />Control at the driver level<br />Blocking<br />When is it OK?<br />Steps around with MongoDB<br />Consistency<br />When does it matter?<br />Scale-out<br />What limits our speed/size?<br />Other?<br />Some features => opportunity<br />
  8. 8. Redundancy<br />MongoDB supports MS and Replica Sets<br />RS is like MS with auto-failover<br />Driver intelligence<br />Still one master, elected by peers<br />Master<br />Slave<br />Master<br />Master<br />Slave<br />Slave<br />Slave<br />Slave<br />Slave<br />Slave<br />Slave<br />
  9. 9. Redundancy<br />Master/Slave fine for us but…<br />Not fine for multi operational datacenters<br />resync?<br />Your data drives the appropriate solution<br />Custom tools to support this<br />https://github.com/wordnik/wordnik-oss<br />Primary Datacenter<br />Hot Datacenter<br />Incremental Backup Files<br />Master<br />Master<br />Replay Util<br />SCP<br />
  10. 10. Redundancy<br />Multi-datacenter deployment<br />Some data need not be consistent<br />Some does<br />Replication tricks for master-master<br />db.documents<br />documents.src == 1<br />Master 1<br />Master 2<br />db.documents<br />documents.src == 2<br />
  11. 11. Controlling blocking operations<br />Important to know what blocks<br />Some situations can create global locks<br />Know what happens with:<br />Adding data<br />Deleting data<br />Modifying data<br />…Then toss in 100 reads/sec<br />Only access your DB “the right way”<br />Enforce by good software design<br />The right way == “right way for your db”<br />
  12. 12. Controlling blocking operations<br />Wordnik Solution<br />MongoDB access via service layer<br />Only allow queries you want run<br />All others => non-production server, Hadoop<br />Smart driver manager <br />Slave bias<br /> $inc:{page_views:1} => $set{page_views:10038}<br />Read-only mode<br />Hot-remove server from pool<br />
  13. 13. Handling eventual consistency<br />EC makes things fast<br />May create timing issues<br />Sometimes need current data<br />val db:DB = getReadWriteConnection()<br />Avoid this a much as possible<br />But if you must…<br />Write then call “getLastError”<br />Forces commit to DB<br />RS allows “replicate to n nodes”<br />Forces commit to master, n slaves<br />
  14. 14. Scale-out<br />What are you scaling?<br />Size of data<br />Size of indexes<br />Operations<br />Add slave servers<br />Won’t help if size > index/server capacity<br />Sharding if done right<br />Autoshard solution<br />Roll your own<br />
  15. 15. Scale-out<br />Selecting a shard key<br />Can be tricky<br />Depends on what you need<br />
  16. 16. Keeping it running<br />Backups<br />Export with “mongodump”<br />No incremental backup solution included<br />Wordnik tools<br /> https://github.com/wordnik/wordnik-oss<br />
  17. 17. Wordnik OSS tools<br />Take our tools-They work!!!<br />SnapshotUtil<br />Selectively snapshot in BSON<br />Index info too!<br />IncrementalBackupUtil<br />Tail the oplog, stream to disk<br />Only the collections you want!<br />Compress & rotate<br />RestoreUtil<br />Recover your snapshots<br />Apply indexes yourself<br />ReplayUtil<br />Apply your Incremental backups<br />
  18. 18. What if Scenarios<br />One collection gets corrupt?<br />Restore it<br />Apply all operations to it<br />“My top developer dropped a collection!”<br />Restore just that one<br />Apply operations to it until that POT<br />“We got hacked!”<br />Restore it all<br />Apply operations until that POT<br />
  19. 19. Wrap up: We like MongoDB<br />See more @ http://developer.wordnik.com<br />Questions?<br />

×