1. Application Development Series
Back to Basics
Monitoring & Performance Tuning
Daniel Roberts
@dmroberts
#MongoDBBasics

2. 2
• Recap from last session
• Monitoring Tools
– MongoDB command line and shell
• Key Metrics in MongoDB
• Logs
– Log levels
– mtools
• Disk saturation
Agenda

3. 3
• Virtual Genius Bar
– Use the chat to post
questions
– EMEA Solution
Architecture / Support
team are on hand
– Make use of them
during the sessions!!!
Q & A

4. Recap from last time….

5. 5
• mongodump & mongorestore
• File system copy
• Files system snapshot
• Don’t use mongoimport & mongoexport!
Backup Tools & Approaches
//mongodump Example
server> mongodump -h myhost -d cms -c articles

6. 6
• Creates a .bson file
– (and a json metadata
file)
– Over network or direct
from file system.
Mongodump
>mongodump –h myhost -d cms -c articles
connected to: myhost
2014-04-16T12:54:56.758+0100 DATABASE: cms to dump/cms
2014-04-16T12:54:56.759+0100 cms.articles to
dump/cms/articles.bson
2014-04-16T12:54:56.816+0100 7 documents
2014-04-16T12:54:56.817+0100 Metadata for cms.articles to
dump/cms/articles.metadata.json

7. 7
• Use secondary for backup
– Or hidden secondary
• Fsync+Lock
• Balance off
– In sharded cluster
Backup Architecture
mongodump

8. Monitoring Tools

9. 9
• Key MongoDB tools
– Mongostat
– Mongo shell
– Mongo Management Service mms
• Mtools
– Logs
• OS
– iostat
Tools

10. 10
MMS Architecture

11. 11
MMS Alerts

12. 12
Metrics Collection and Reporting

13. 13
Logs and Profile data

14. 14
• Access to key statistics
– Connect to mongod or mongos for sharded cluster stats
– Shows, operations per second, memory usage, page faults, queues and more
Mongostat
>mongostat -h localhost --port 27017
connected to: localhost:27017
insert query update delete getmore command flushes mapped vsize res faults locked db idx miss % qr|qw ar|aw
netIn netOut conn time
*0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:24
*0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:25
*0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:26
*0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:27
*0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:28
*0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:29
*0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:30
*0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:31
*0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.1% 0 0|0 0|0 62b 5k 2 09:08:32
*0 *0 *0 *0 0 1|0 0 1.67g 5.78g 29m 0 .:0.0% 0 0|0 0|0 62b 5k 2 09:08:33

15. 15
• Commands for access metrics
– db.serverStatus()
– workingSet analyzer
Mongo shell
//to access a subset of serverStatus output specify the sub document
>db.serverStatus().mem
{
"bits" : 64,
"resident" : 266,
"virtual" : 5920,
"supported" : true,
"mapped" : 1712,
"mappedWithJournal" : 3424
}

16. 16
• Estimates the ‘working set’ of the database
– Number of ‘pagesInMemory’ accessed in RAM over
‘overSeconds’ number of seconds.
– Default Page is 4k
– overSecond – delta between oldest and newest
WorkingSet analyzer
>db.serverStatus({ workingSet : 1 })
{
"note" : "thisIsAnEstimate",
"pagesInMemory" : 723,
"computationTimeMicros" : 4601,
"overSeconds" : 2311
}
overSeconds
decreasing or small,
workingSet could be
larger than RAM

17. MongoDB Metrics

18. 18
• Queued readers | writers
• Page faults
• OpCounters
• Background flush process
• Memory usage
• Lock %
• Btree misses
• connections
Key Metrics
Available from :
• MMS
• db.serverStatus()
• mongostat

19. 19
Queued Reader | Writers
• Number of
operations waiting
for:
• Read Lock
• Write Lock

20. 20
Page Faults
• How often we are
going to disk to
retrieve data?
• Can be a problem if disk IO
is saturated
• Combine with iostat

21. 21
Background Flush Process
• Length of time to
flush data back to
disk
• Again can point to
disk saturation.

22. 22
Memory
• Virtual
• Mapped
• Resident

23. 23
Lock %
• Percentage of time
in the global lock

24. 24
btree
• Accesses
• Hits
• Misses
• Bad!
• Points to Indexes
not in RAM

25. 25
connections
• Connections to the
database
• Growing over time?
• Check pool logic
in your
application code.

26. Logs

27. 27
• Increase the log level
– See further insight to operation performance
– Index efficiency
– Document moves
– Operation time
Set log level
//Increase log level verbosity from the shell
> db.adminCommand( { setParameter:1, logLevel:1 } )
{ "was" : 0, "ok" : 1 }
>
-v [ --verbose ] be more verbose (include multiple times for more
verbosity e.g. -vvvvv)

28. 28
• Pinpoint problems
– In this example we can see that there has been a document move.
Log output example
2014-05-02T13:55:02.047+0100 [conn7] update cms.articles query: {
_id: ObjectId('532198379fb5ba99a6bd4063') } update: { $inc: {
comment_count: 1 }, $push: { comments: { $each: [ { date: new
Date(1399035302013), text: "Data locality provides an amazing
performance boost over relational" } ], $slice: -10, $sort: { date: 1 } } } }
nscanned:1 nscannedObjects:1 nmoved:1 nMatched:1 nModified:1
keyUpdates:0 numYields:0 locks(micros) w:33529 33ms

29. 29
• Log analysis (example syntax)
– Show me queries that took more than 1000 ms from 6
am to 6 pm:
• Now, graph those queries:
mtools
prompt> mlogfilter mongodb.log --from 06:00 --to 18:00 --
slow 1000 > mongodb-filtered.log
prompt> mplotqueries --logscale mongodb-filtered.log

30. 30
mtools output

31. Disk saturation

32. 32
• iostat
– Disk IO usage
– Use to review % utilisation of disk
– High %
• If it’s sustained, you may need to increase disk IO.
– Sharding
– RAID 1+0
– Partitioning on different disk
– Provisioned IOPS
OS Tools and Paramenters
prompt> iostat –xmt 1

33. Summary

34. 34
• Monitor regularly
– Set alerts for changes
• Quick check with command line and shell
• View trends with MMS graphs over time
• High levels of utilisation
– Increase RAM, disk IO
– Scale out
– (Having first checked Indexes etc!)
• Database profiler (covered in Indexing session)
Summary