Managing a Maturing MongoDB Ecosystem


Published on

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Managing a Maturing MongoDB Ecosystem

  1. 1. Charity Majors@mipsytipsyThursday, June 20, 13
  2. 2. Managing a maturing MongoDBecosystemThursday, June 20, 13
  3. 3. automating with chefperformance tuningdisaster recoveryThursday, June 20, 13
  4. 4. chef.Thursday, June 20, 13
  5. 5. Basic replica setThursday, June 20, 13
  6. 6. How do I chef that?... grab the AWS and mongodb cookbooks,create a site wrapper cookbookThursday, June 20, 13
  7. 7. make a role for your cluster,launch some nodes,Thursday, June 20, 13
  8. 8. initiate the replica set,... and you’re done.Thursday, June 20, 13
  9. 9. Adding snapshotsThursday, June 20, 13
  10. 10. adding RAID for EBS volumesThursday, June 20, 13
  11. 11. this will bootstrap a new node for the cluster from snapshotswith this role ...Thursday, June 20, 13
  12. 12. multiple clustersdistinct cluster name, backup host, backup volumesThursday, June 20, 13
  13. 13. shardingThursday, June 20, 13
  14. 14. assign a shard name per cluster, per roletreat them like ordinary replica setsThursday, June 20, 13
  15. 15. Arbiters• Mongod processes that do nothing but vote• Highly reliable• To provision an arbiter, use the LWRP• Easy to run multiple arbiters on a single hostThursday, June 20, 13
  16. 16. arbiter LWRPThursday, June 20, 13
  17. 17. replica set with arbitersThursday, June 20, 13
  18. 18. run multiple arbiters on a single host:Thursday, June 20, 13
  19. 19. Managing votes with arbitersThursday, June 20, 13
  20. 20. tuning and performance.Thursday, June 20, 13
  21. 21. resources and provisioningtuning your filesystemsnapshotting and warmupsfragmentationThursday, June 20, 13
  22. 22. Provisioning tips• Memory is your primary scaling constraint• Your working set must fit in to memory• in 2.4, estimate with:• Page faults? Your working set may not fitThursday, June 20, 13
  23. 23. Disk options• If you’re on Amazon:• EBS• Dedicated SSD• Provisioned IOPS• Ephemeral• If not:• use SSDs!Thursday, June 20, 13
  24. 24. EBS classicEBS withPIOPS:... just say no to EBSThursday, June 20, 13
  25. 25. SSD(hi1.4xlarge)• 8 cores• 60 gigs RAM• 2 1-TB SSD drives• 120k random reads/sec• 85k random writes/sec• expensive! $2300/mo on demandThursday, June 20, 13
  26. 26. PIOPS• Up to 2000 IOPS/volume• Up to 1024 GB/volume• Variability of < 0.1%• Costs double regular EBS• Supports snapshots• RAID together multiple volumesfor more storage/performanceThursday, June 20, 13
  27. 27. • multiply that by 2-3x depending on your spikinessEstimating PIOPS• estimate how many IOPS to provision with the “tps”column of sar -d 1Thursday, June 20, 13
  28. 28. EphemeralStorage• Cheap• Fast• No network latency• No snapshot capability• Data is lost forever if you stop orresize the instanceThursday, June 20, 13
  29. 29. Filesystem and limits• Raise file descriptor limits• Raise connection limits• Mount with noatime and nodiratime• Consider putting the journal on a separate volumeThursday, June 20, 13
  30. 30. Blockdev• Your default blockdev is probably wrong• Too large? you will underuse memory• Too small? you will hit the disk too much• Experiment.Thursday, June 20, 13
  31. 31. Snapshot best practices• Set priority = 0• Set hidden = 1• Consider setting votes = 0• Lock mongo or stop mongod before snapshot• Consider running continuous compaction onsnapshot nodeThursday, June 20, 13
  32. 32. Restoring from snapshot• EBS snapshot will lazily-load blocks from S3• run “dd” on each of the data files to pull blocks down• Always warm up a secondary before promoting• warm up both indexes and data•• in mongodb 2.2 and above you can use the touch command:Thursday, June 20, 13
  33. 33. Fragmentation• Your RAM gets fragmented too!• Leads to underuse of memory• Deletes are not the only source of fragmentation• Repair, compact, or resync regularlyThursday, June 20, 13
  34. 34. 3 ways to fix fragmentation:• Re-sync a secondary from scratch• hard on your primary; rs.syncFrom() a secondary• Repair a secondary• can cause small discrepancies in your data• Run continuous compaction on your snapshotnode• won’t reset padding factors• not appropriate if you do lots of deletesThursday, June 20, 13
  35. 35. Fragmentation is terribleThursday, June 20, 13
  36. 36. Upgrade!mongo is getting faster. :)Thursday, June 20, 13
  37. 37. disasters and recovery.Thursday, June 20, 13
  38. 38. Finding bad queries• db.currentOp()• mongodb.log• profiling collectionThursday, June 20, 13
  39. 39. db.currentOp()• Check the queue size• Any indexes building?• Sort by num_seconds• Sort by num_yields, locktype• Consider adding comments to your queries• Run explain() on queries that are long-runningThursday, June 20, 13
  40. 40. mongodb.log• Configure output with --slowms• Look for high execution time, nscanned, ntoreturn• See which queries are holding long locks• Match connection ids to IPsThursday, June 20, 13
  41. 41. system.profile collection• Enable profiling with db.setProfiling()• Does not persist through restarts• Like mongodb.log, but queryable• Writes to this collection incur some cost• Use db.system.profile.find() to get slow queries fora certain collection, time range, execution time, etcThursday, June 20, 13
  42. 42. • Know what your tipping point looks like• Don’t switch your primary or restart• Do kill queries before the tipping point• Write your kill script before you need it• Don’t kill internal mongo operations, only queries.... when queries pile up ...Thursday, June 20, 13
  43. 43. can’t elect a master?• Never run with an even number of votes (max 7)• You need > 50% of votes to elect a primary• Set your priority levels explicitly if you needwarmup• Consider delegating voting to arbiters• Set snapshot nodes to be nonvoting if possible.• Check your mongo log. Is something vetoing? Dothey have an inconsistent view of the cluster state?Thursday, June 20, 13
  44. 44. secondaries crashing?• Some rare mongo bugs will cause all secondariesto crash unrecoverably• Never kill oplog tailers or other internal databaseoperations, this can also trash secondaries• Arbiters are more stable than secondaries,consider using them to form a quorum with yourprimaryThursday, June 20, 13
  45. 45. replication stops?• Other rare bugs will stop replication or causesecondaries to exit without a corrupt op• The correct way to fix this is to re-snapshot offthe primary and rebuild your secondaries.• However, you can sometimes *dangerously* repaira secondary:1. stop mongo2. bring it back up in standalone mode3. repair the offending collection4. restart mongo again as part of the replica setThursday, June 20, 13
  46. 46. • Everything is getting vaguely slower?• check your padding factor, try compaction• You rs.remove() a node and get weird drivererrors?• always shut down mongod after removing from replica set• Huge background flush spike?• probably an EBS or disk problem• You run out of connection limits?• possibly a driver bug• hard-coded to 80% of soft ulimit until 20k is reached.Thursday, June 20, 13
  47. 47. • It looks like all I/O stops for a while?• check your mongodb.log for large newExtent warnings• also make sure you aren’t reaching PIOPS limits• You get weird driver errors after adding/removing/re-electing?• some drivers have problems with this, you may have to restartThursday, June 20, 13
  48. 48. Glossary of resources• Opscode AWS cookbook•• edelight MongoDB cookbook•• Parse MongoDB cookbook fork•• Parse compaction scripts and warmup scripts••, June 20, 13
  49. 49. Charity Majors@mipsytipsyThursday, June 20, 13