Monitoring MongoDB (MongoUK)
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Monitoring MongoDB (MongoUK)

on

  • 1,150 views

Presentation by David Mytton about monitoring MongoDB at the MongoUK conference 21st Mar 2011....

Presentation by David Mytton about monitoring MongoDB at the MongoUK conference 21st Mar 2011.

A full blog series covering everything in this presentation is at http://blog.boxedice.com/mongodb-monitoring/

Statistics

Views

Total Views
1,150
Views on SlideShare
1,016
Embed Views
134

Actions

Likes
2
Downloads
24
Comments
1

2 Embeds 134

http://www.nosqldatabases.com 132
http://lanyrd.com 2

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • A full blog series covering everything in this presentation is at http://blog.boxedice.com/mongodb-monitoring/
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Monitoring MongoDB (MongoUK) Presentation Transcript

  • 1. MongoDB Health Tips
  • 2. It’s a little different,but not entirely new.
  • 3. Keep it in RAM. Obviously.www.flickr.com/photos/comedynose/4388430444/
  • 4. How do you know? > db.stats() { ! "collections" : 3, ! "objects" : 379970142, ! "avgObjSize" : 146.4554114991488, ! "dataSize" : 55648683504, 51GB ! "storageSize" : 61795435008, ! "numExtents" : 64, ! "indexes" : 1, ! "indexSize" : 21354514128, 19GB ! "fileSize" : 100816388096, ! "ok" : 1 }http://www.flickr.com/photos/comedynose/4388430444/
  • 5. Where should it go? Should it be in What? memory? Indexes Always Data If you canhttp://www.flickr.com/photos/comedynose/4388430444/
  • 6. How you’ll know1) Slow queries Thu Oct 14 17:01:11 [conn7410] update sd.apiLog query: { c: "android/setDeviceToken", a: 1466, u: "blah", ua: "Server Density Android" } 51926mswww.flickr.com/photos/tonivc/2283676770/
  • 7. How you’ll know2) Timeouts cursor timed out (20000 ms)
  • 8. How you’ll know3) Disk i/o spikeswww.flickr.com/photos/daddo83/3406962115/
  • 9. Watch your storage1) Pre-alloc
  • 10. Watch your storage2) Sharding maxSize
  • 11. Watch your storage3) Logging --quiet db.runCommand("logRotate"); killall -SIGUSR1 mongod
  • 12. Watch your storage4) Journaling david@rs2b ~: ls -alh /mongodbdata/journal/ total 538M drwxrwxr-x 2 david david 29 Mar 20 16:50 . drwx------ 4 david david 4.0K Mar 13 09:50 .. -rw------- 1 david david 538M Mar 20 17:00 j._862 -rw------- 1 david david 88 Mar 20 17:00 lsn
  • 13. db.serverStatus()
  • 14. db.serverStatus()1) Used connectionswww.flickr.com/photos/armchaircaver/2061231069/
  • 15. db.serverStatus()2) Available connections
  • 16. Fri Nov 19 17:24:32 [mongosMain] Listener: accept() returns -1 errno:24 Too many open filesFri Nov 19 17:24:32 [mongosMain] Listener: accept() returns -1 errno:24 Too many open filesFri Nov 19 17:24:32 [mongosMain] Listener: accept() returns -1 errno:24 Too many open filesFri Nov 19 17:24:32 [mongosMain] Listener: accept() returns -1 errno:24 Too many open filesFri Nov 19 17:24:32 [mongosMain] Listener: accept() returns -1 errno:24 Too many open filesFri Nov 19 17:24:32 [mongosMain] Listener: accept() returns -1 errno:24 Too many open filesFri Nov 19 17:24:32 [mongosMain] Listener: accept() returns -1 errno:24 Too many open filesFri Nov 19 17:24:32 [mongosMain] Listener: accept() returns -1 errno:24 Too many open filesFri Nov 19 17:24:32 [mongosMain] Listener: accept() returns -1 errno:24 Too many open filesFri Nov 19 17:24:32 [mongosMain] Listener: accept() returns -1 errno:24 Too many open filesFri Nov 19 17:24:32 [mongosMain] Listener: accept() returns -1 errno:24 Too many open filesFri Nov 19 17:24:32 [mongosMain] Listener: accept() returns -1 errno:24 Too many open filesFri Nov 19 17:24:32 [mongosMain] Listener: accept() returns -1 errno:24 Too many open filesFri Nov 19 17:24:32 [conn2335] getaddrinfo("rs1b") failed: No address associated with hostnameFri Nov 19 17:24:32 [conn2335] getaddrinfo("rs1d") failed: No address associated with hostnameFri Nov 19 17:24:32 [conn2335] getaddrinfo("rs1c") failed: No address associated with hostnameFri Nov 19 17:24:32 [conn2335] getaddrinfo("rs2b") failed: No address associated with hostnameFri Nov 19 17:24:32 [conn2335] getaddrinfo("rs2d") failed: No address associated with hostnameFri Nov 19 17:24:32 [conn2335] getaddrinfo("rs2c") failed: No address associated with hostnameFri Nov 19 17:24:32 [conn2335] getaddrinfo("rs2a") failed: No address associated with hostnameFri Nov 19 17:24:32 [conn2268] checkmaster: rs2b:27018 { setName: "set2", ismaster: false, secondary: true, hosts: [ "rs2b:27018", "rs2d:27018", "rs2c:27018", "rs2a:27018" ], arbiters:[ "rs2arbiter:27018" ], primary: "rs2a:27018", maxBsonObjectSize: 8388608, ok: 1.0 }MessagingPort say send() errno:9 Bad file descriptor (NONE)Fri Nov 19 17:24:32 [conn2268] checkmaster: caught exception rs2d:27018 socket exceptionFri Nov 19 17:24:32 [conn2268] MessagingPort say send() errno:9 Bad file descriptor (NONE)Fri Nov 19 17:24:32 [conn2268] checkmaster: caught exception rs2c:27018 socket exceptionFri Nov 19 17:24:32 [conn2268] MessagingPort say send() errno:9 Bad file descriptor (NONE)Fri Nov 19 17:24:32 [conn2268] checkmaster: caught exception rs2a:27018 socket exceptionFri Nov 19 17:24:33 [conn2330] getaddrinfo("rs1a") failed: No address associated with hostnameFri Nov 19 17:24:33 [conn2330] getaddrinfo("rs1b") failed: No address associated with hostnameFri Nov 19 17:24:33 [conn2330] getaddrinfo("rs1d") failed: No address associated with hostnameFri Nov 19 17:24:33 [conn2330] getaddrinfo("rs1c") failed: No address associated with hostnameFri Nov 19 17:24:33 [conn2327] getaddrinfo("rs2b") failed: No address associated with hostnameFri Nov 19 17:24:33 [conn2327] getaddrinfo("rs2d") failed: No address associated with hostnameFri Nov 19 17:24:33 [conn2327] getaddrinfo("rs2c") failed: No address associated with hostnameFri Nov 19 17:24:33 [conn2327] getaddrinfo("rs2a") failed: No address associated with hostnameFri Nov 19 17:24:33 [conn2126] getaddrinfo("rs2b") failed: No address associated with hostnameFri Nov 19 17:24:33 [conn2126] getaddrinfo("rs2d") failed: No address associated with hostnameFri Nov 19 17:24:33 [conn2126] getaddrinfo("rs2c") failed: No address associated with hostnameFri Nov 19 17:24:33 [conn2126] getaddrinfo("rs2a") failed: No address associated with hostnameFri Nov 19 17:24:33 [conn2343] getaddrinfo("rs1b") failed: No address associated with hostnameFri Nov 19 17:24:33 [conn2343] getaddrinfo("rs1d") failed: No address associated with hostnameFri Nov 19 17:24:33 [conn2343] getaddrinfo("rs1c") failed: No address associated with hostnameFri Nov 19 17:24:34 [conn2332] getaddrinfo("rs1b") failed: No address associated with hostnameFri Nov 19 17:24:34 [conn2332] getaddrinfo("rs1d") failed: No address associated with hostnameFri Nov 19 17:24:34 [conn2332] getaddrinfo("rs1c") failed: No address associated with hostnameFri Nov 19 17:24:34 [conn2332] getaddrinfo("rs2b") failed: No address associated with hostnameFri Nov 19 17:24:34 [conn2332] getaddrinfo("rs2d") failed: No address associated with hostnameFri Nov 19 17:24:34 [conn2332] getaddrinfo("rs2c") failed: No address associated with hostnameFri Nov 19 17:24:34 [conn2332] getaddrinfo("rs2a") failed: No address associated with hostnameFri Nov 19 17:24:34 [conn2343] getaddrinfo("rs2d") failed: No address associated with hostnameFri Nov 19 17:24:34 [conn2343] getaddrinfo("rs2c") failed: No address associated with hostnameFri Nov 19 17:24:34 [conn2343] getaddrinfo("rs2a") failed: No address associated with hostnameFri Nov 19 17:24:34 [conn2343] trying reconnect to rs2d:27018
  • 17. connPoolStats> db.runCommand("connPoolStats"){! "hosts" : {! ! "config1:27019" : {! ! ! "available" : 2,! ! ! "created" : 6! ! },! ! "set1/rs1a:27018,rs1b:27018" : {! ! ! "available" : 1,! ! ! "created" : 249! ! }, ...! },! "totalAvailable" : 5,! "totalCreated" : 1002,! "numDBClientConnection" : 3490,! "numAScopedConnection" : 3,}
  • 18. db.serverStatus()3) Index counters "indexCounters" : { ! ! "btree" : { ! ! ! "accesses" : 15180175, ! ! ! "hits" : 15178725, ! ! ! "misses" : 1450, ! ! ! "resets" : 0, ! ! ! "missRatio" : 0.00009551932 ! ! } ! },
  • 19. db.serverStatus()4) Op counterswww.flickr.com/photos/cosmic_bandita/2395369614/
  • 20. db.serverStatus()5) Background flushingPicture is unrelated! Mmm, ice cream.
  • 21. db.serverStatus()6) Dur
  • 22. rs.status() { ! "_id" : 1, ! "name" : "rs3b:27018", ! "health" : 1, ! "state" : 2, ! "stateStr" : "SECONDARY", ! "uptime" : 1886098, ! "optime" : { ! ! "t" : 1291252178000, ! ! "i" : 13 ! }, ! "optimeDate" : ISODate("2010-12-02T01:09:38Z"), "lastHeartbeat" : ISODate("2010-12-02T01:09:38Z") },www.ex-astris-scientia.org/inconsistencies/ent_vs_tng.htm (yes it’s a replicator from Star Trek)
  • 23. rs.status()1) myState Value Meaning 0 Starting up (phase 1) 1 Primary 2 Secondary 3 Recovering 4 Fatal error 5 Starting up (phase 2) 6 Unknown state 7 Arbiter 8 Downen.wikipedia.org/wiki/State_of_matter
  • 24. rs.status()2) Optime "optimeDate" : ISODate("2010-12-02T01:09:38Z")www.flickr.com/photos/robbie73/4244846566/
  • 25. rs.status()3) Heartbeat "lastHeartbeat" : ISODate("2010-12-02T01:09:38Z")www.flickr.com/photos/drawblindfaith/3400981091/
  • 26. mongostat
  • 27. mongostat 1) faultsPicture is unrelated! Snowmobile in Norway.
  • 28. mongostat2) lockedwww.flickr.com/photos/bbusschots/4541573665/
  • 29. mongostat3) index misswww.flickr.com/photos/gareandkitty/276471187/
  • 30. mongostat4) queues
  • 31. mongostat5) Diagnostics
  • 32. Current operations db.currentOp(); { ! ! ! "opid" : "shard1:299939199", ! ! ! "active" : true, ! ! ! "lockType" : "write", ! ! ! "waitingForLock" : false, ! ! ! "secs_running" : 15419, ! ! ! "op" : "remove", ! ! ! "ns" : "sd.metrics", ! ! ! "query" : { ! ! ! ! "accId" : 1391, ! ! ! ! "tA" : { ! ! ! ! ! "$lte" : ISODate("2010-11-24T19:53:00Z") ! ! ! ! } ! ! ! }, ! ! ! "client" : "10.121.12.228:44426", ! ! ! "desc" : "conn" ! ! },www.flickr.com/photos/jeffhester/2784666811/
  • 33. Monitoring toolsRun yourself Ganglia
  • 34. Monitoring toolsServer Density
  • 35. Monitoring toolswww.mongomonitor.com
  • 36. Recap
  • 37. RecapKeep it in RAM
  • 38. RecapKeep it in RAMWatch your storage
  • 39. RecapKeep it in RAMWatch your storagedb.serverStatus()rs.status()
  • 40. David Mytton @davidmyttondavid@boxedice.comwww.mongomonitor.comWoop Japan!