• Like
MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation


MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation

MongoDB performance tuning and load testing, NOSQL Now! 2013 Conference presentation

Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On SlideShare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Enteros, Inc. MongoDB Performance Tuning Load Testing NoSQL Now! 2013 Conference San Jose ron@enteros.com Ron Warshawsky August 23, 2013
  • 2. 2 MongoDB Overview Agenda Overview performance tuning Overview load testing
  • 3. 3 MongoDB Areas Of Performance Tuning OS tuning Storage tuning Database tuning Load Testing
  • 4. 4 MongoDB Areas Of Performance Tuning • OS Tuning • Follow ulimitsper mongo manual • http://docs.mongodb.org/manual/reference/ulimit/ • Follow production notes per mongo manual • http://docs.mongodb.org/manual/administration/production-notes/ • No atime on fs • No hugepages • Use NTT to sync time between the nodes • Try “tuned” • Best when used in staging/load testing environment under realisticload • CPU, Network and Storage tuning
  • 5. 5 MongoDB Areas Of Performance Tuning • Storage Tuning • RAID 10 • Ext4 or XFS • Delaysharding with better I/O • SSD + FlashCache – shutterflytested • https://github.com/facebook/flashcache/ • Provisioned IOPS on AWS
  • 6. 6 MongoDB Areas Of Performance Tuning • Storage Tuning • Ext4 + external journal on SSD – good I/O results • Make sure to test • journal_async_commit is used • http://www.raid6.com.au/posts/fs_ext4_external_journal/
  • 7. 7 MongoDB Areas Of Performance Tuning Softlayerstudy/ benchmarking “MongoDB Performance Analysis: Bare Metal v. Virtual” http://blog.softlayer.com/2012/mongodb-performance-analysis-bare-metal-v- virtual/ Highlights: • When a working data set is smaller than available memory, query performance increases. • The number of clients performing queries has an impact on query performance because more data is being actively cached at a rapid rate. • The addition of a separate Journal Mount volume significantly improves performance. Because the Small (SM) engineered server does not include a secondary mount for Journals, whenever MongoDB began to journal, the disk I/O associated with journaling was disruptive to the query and update operations performed on the Data Mount. • The best deployments in terms of operations per second,stability and control were the configurations with a RAID10 SSD Data Mount and a RAID1 SSD Journal Mount. These configurations are available in both Medium and Large offerings,and I’d highly recommend them.
  • 8. 8 MongoDB Areas Of Performance Tuning OS Performancemonitoring and analysis tools: • A) Linux utilities: iostat, vmstat, mpstat, sar, free -tm • B) Open source monitoring / performance analysissystems: Nagios, Cacti • http://tag1consulting.com/blog/mongodb-cacti-graphs • https://github.com/mzupan/nagios-plugin-mongodb C) Oracle OS Watcher– Free tool, performance collection and chart generator, easy to operate, very reliable, covers CPU, processes, top processes, I/O, network, etc OSWatcher Black Box Analyzer User Guide (Doc ID 461053.1)
  • 9. 9 MongoDB Areas Of Performance Tuning Database Tuning • Database configuration • Preallocatespace (noprealloc = false) • journalCommitInterval – range of 2 to 300, default 100. Can be adjusted to for better performance • Option to disable services: 1. HTTP Interface using nohttpinterface = true. 2. ScriptingEngine using noscripting = true (server-side java script). 3. RESTservice using rest = false.
  • 10. 10 MongoDB Areas Of Performance Tuning Database Tuning • Database monitoring tools • Mongostat– based on “eBay” “freecon” utility. • Use “–discover” option for membersof replicaSet / shared cluster • Displays critical performance metrics: • Flushes/Faults/locked db/idxmiss/qr/qw • Collect 24x7 and save for analysis
  • 11. 11 MongoDB Areas Of Performance Tuning Database Tuning • Database monitoring tools • Mongotop • Per collection I/O time tracking • Per database lock tracking (--lock option)
  • 12. 12 MongoDB Areas Of Performance Tuning Database networktraffic analysis • Database networkmonitoringtools • Mongosniff– sniffing MongoDb requests • Can capture invalid BSON requests from a network (--objcheck option) • Wireshark– support for MongoDB protocol (not BSON), great options for network performanceand latencyanalysis
  • 13. 13 MongoDB Areas Of Performance Tuning Database Tuning • Collections/ Fragmentation Collections in a MongoDB database can become fragmented. This can be a particularly serious problem if data usage patterns are relatively unstructured. In the long run, this will result in databases taking up more space on disk and in RAM to hold the same amount of data, it will make many database operations noticeably slower, and it can reduce overall query capacity significantly. compactcollection: db.runCommand ( { compact: '<collection>', paddingBytes: 100 } ) Automate compaction: http://blog.parse.com/2013/03/26/always-be-compacting/ ./mongo_compact.rb -d userdata1,userdata2,userdata3
  • 14. 14 MongoDB Areas Of Performance Tuning Database Tuning Profiler db.setProfilingLevel(level, slowms) db.setProfilingLevel(1)– collects slow requests Output: system.profile.op – operation system.profile.ns – namespace system.profile.ntoreturn – documentsreturned system.profile.nscanned – documentsscanned … http://docs.mongodb.org/manual/reference/database-profiler/ Profilerdeep dive https://speakerdeck.com/kgorman/mongodb-profiler-deep-dive
  • 15. 15 MongoDB Areas Of Performance Tuning Database Tuning • Indexes • Explain db.collection.find({…}).explain() Watch for scans • Indexes + Profiler= DEX - suggest what indexesare needed https://github.com/jwilder/mongodb-tools http://architects.dzone.com/articles/mongodb-performance-tuning- dex
  • 16. 16 MongoDB Areas Of Performance Tuning Database Monitoring and performance analysisUI a) MongoDB monitoring service (MMS) b) Open Source monitoringsoftware http://docs.mongodb.org/manual/administration/monitoring/ c) Enteros High Load Capture for MongoDB
  • 17. 17 MongoDB Areas Of Performance Tuning • Replication/ Replica set tuning Watch for replicationlag Problem: a) Weaksecondary b) Write bursts c) Index build d) Secondarylocked for backup e) Secondaryoffline Measure lag: db.printSlaveReplicationInfo() db.printReplicationInfo() Get time difference MMS provides replication lag chart and alerts
  • 18. 18 MongoDB Areas Of Performance Tuning • Sharding Tuning • Shard size • Migration of chunks • Failed migrationof chunks
  • 19. 19 MongoDB Load Testing Load testing 1) Whyload testing? • Validate upgrades • Validate fixes • Validate hardware and platform changes (to / from cloud, etc) • Validate multiplesof production load
  • 20. 20 MongoDB Load Testing 2) Options for Load testing A) BenchmarkLoad Testing • Mongo-Perf - Create standard TPCC-like load on MongoDB • https://github.com/mongodb/mongo-perf B) Disk I/O load testing • Mongoperf • Iozone http://www.iozone.org/
  • 21. 21 MongoDB Load Testing C) Real traffic Load Testing Parse PCAP files and generate production-like load http://docs.mongodb.org/meta-driver/latest/legacy/mongodb-wire- protocol/ Create custom tool Enteros Load2Test (pre-GA)
  • 22. Understandbusiness requirements and it’s impact on production system design Designfor performance, scalability and availability Implement monitoring and performance analysis across layers Implement reliable production-like load testing MongoDB Summary
  • 23. Enteros, Inc http://www.enteros.com/ Enteros is an innovative software company specializing in Performance Management and Load Testing Software for Production Databases - RDBMS and NOSQL/Big Data Enteros solutions enable IT professionals to identify and remediate performance problems in business-critical databases with unprecedented speed, accuracy and scope. Ron Warshawsky; ron@enteros.com 408-207-8408 MongoDB Summary