1404 app dev series - session 8 - monitoring & performance tuning

727 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
727
On SlideShare
0
From Embeds
0
Number of Embeds
159
Actions
Shares
0
Downloads
19
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

1404 app dev series - session 8 - monitoring & performance tuning

  1. 1. Application Development Series Back to Basics Monitoring & Performance Tuning Daniel Roberts @dmroberts #MongoDBBasics
  2. 2. 2 • Recap from last session • Monitoring Tools – MongoDB command line and shell • Key Metrics in MongoDB • Logs – Log levels – mtools • Disk saturation Agenda
  3. 3. 3 • Virtual Genius Bar – Use the chat to post questions – EMEA Solution Architecture / Support team are on hand – Make use of them during the sessions!!! Q & A
  4. 4. Recap from last time….
  5. 5. 5 • mongodump & mongorestore • File system copy • Files system snapshot • Don’t use mongoimport & mongoexport! Backup Tools & Approaches //mongodump Example server> mongodump -h myhost -d cms -c articles
  6. 6. 6 • Creates a .bson file – (and a json metadata file) – Over network or direct from file system. Mongodump >mongodump –h myhost -d cms -c articles connected to: myhost 2014-04-16T12:54:56.758+0100 DATABASE: cms to dump/cms 2014-04-16T12:54:56.759+0100 cms.articles to dump/cms/articles.bson 2014-04-16T12:54:56.816+0100 7 documents 2014-04-16T12:54:56.817+0100 Metadata for cms.articles to dump/cms/articles.metadata.json
  7. 7. 7 • Use secondary for backup – Or hidden secondary • Fsync+Lock • Balance off – In sharded cluster Backup Architecture mongodump
  8. 8. Monitoring Tools
  9. 9. 9 • Key MongoDB tools – Mongostat – Mongo shell – Mongo Management Service mms • Mtools – Logs • OS – iostat Tools
  10. 10. 10 MMS Architecture
  11. 11. 11 MMS Alerts
  12. 12. 12 Metrics Collection and Reporting
  13. 13. 13 Logs and Profile data
  14. 14. 14 • Access to key statistics – Connect to mongod or mongos for sharded cluster stats – Shows, operations per second, memory usage, page faults, queues and more Mongostat >mongostat -h localhost --port 27017 connected to: localhost:27017 insert query update delete getmore command flushes mapped vsize res faults locked db idx miss % qr|qw ar|aw netIn netOut conn time *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:24 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:25 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:26 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:27 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:28 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:29 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:30 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:31 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.1% 0 0|0 0|0 62b 5k 2 09:08:32 *0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:33
  15. 15. 15 • Commands for access metrics – db.serverStatus() – workingSet analyzer Mongo shell //to access a subset of serverStatus output specify the sub document >db.serverStatus().mem { "bits" : 64, "resident" : 266, "virtual" : 5920, "supported" : true, "mapped" : 1712, "mappedWithJournal" : 3424 }
  16. 16. 16 • Estimates the ‘working set’ of the database – Number of ‘pagesInMemory’ accessed in RAM over ‘overSeconds’ number of seconds. – Default Page is 4k – overSecond – delta between oldest and newest WorkingSet analyzer >db.serverStatus({ workingSet : 1 }) { "note" : "thisIsAnEstimate", "pagesInMemory" : 723, "computationTimeMicros" : 4601, "overSeconds" : 2311 } overSeconds decreasing or small, workingSet could be larger than RAM
  17. 17. MongoDB Metrics
  18. 18. 18 • Queued readers | writers • Page faults • OpCounters • Background flush process • Memory usage • Lock % • Btree misses • connections Key Metrics Available from : • MMS • db.serverStatus() • mongostat
  19. 19. 19 Queued Reader | Writers • Number of operations waiting for: • Read Lock • Write Lock
  20. 20. 20 Page Faults • How often we are going to disk to retrieve data? • Can be a problem if disk IO is saturated • Combine with iostat
  21. 21. 21 Background Flush Process • Length of time to flush data back to disk • Again can point to disk saturation.
  22. 22. 22 Memory • Virtual • Mapped • Resident
  23. 23. 23 Lock % • Percentage of time in the global lock
  24. 24. 24 btree • Accesses • Hits • Misses • Bad! • Points to Indexes not in RAM
  25. 25. 25 connections • Connections to the database • Growing over time? • Check pool logic in your application code.
  26. 26. Logs
  27. 27. 27 • Increase the log level – See further insight to operation performance – Index efficiency – Document moves – Operation time Set log level //Increase log level verbosity from the shell > db.adminCommand( { setParameter:1, logLevel:1 } ) { "was" : 0, "ok" : 1 } > -v [ --verbose ] be more verbose (include multiple times for more verbosity e.g. -vvvvv)
  28. 28. 28 • Pinpoint problems – In this example we can see that there has been a document move. Log output example 2014-05-02T13:55:02.047+0100 [conn7] update cms.articles query: { _id: ObjectId('532198379fb5ba99a6bd4063') } update: { $inc: { comment_count: 1 }, $push: { comments: { $each: [ { date: new Date(1399035302013), text: "Data locality provides an amazing performance boost over relational" } ], $slice: -10, $sort: { date: 1 } } } } nscanned:1 nscannedObjects:1 nmoved:1 nMatched:1 nModified:1 keyUpdates:0 numYields:0 locks(micros) w:33529 33ms
  29. 29. 29 • Log analysis (example syntax) – Show me queries that took more than 1000 ms from 6 am to 6 pm: • Now, graph those queries: mtools prompt> mlogfilter mongodb.log --from 06:00 --to 18:00 -- slow 1000 > mongodb-filtered.log prompt> mplotqueries --logscale mongodb-filtered.log
  30. 30. 30 mtools output
  31. 31. Disk saturation
  32. 32. 32 • iostat – Disk IO usage – Use to review % utilisation of disk – High % • If it’s sustained, you may need to increase disk IO. – Sharding – RAID 1+0 – Partitioning on different disk – Provisioned IOPS OS Tools and Paramenters prompt> iostat –xmt 1
  33. 33. Summary
  34. 34. 34 • Monitor regularly – Set alerts for changes • Quick check with command line and shell • View trends with MMS graphs over time • High levels of utilisation – Increase RAM, disk IO – Scale out – (Having first checked Indexes etc!) • Database profiler (covered in Indexing session) Summary

×