Mongo scaling

SCALING MONGODB IN THE CLOUD Simon Maynard - Bugsnag CTO @snmaynard

WHERE HAVE I USED MONGODB?

HEYZAP• Largest mobile gaming social network• MongoDB the main datastore • Also MySQL & Redis• High number of reads, fewer writes

BUGSNAG bugsnag.com• Exception tracking service for mobile and web• MongoDB only persistent datastore • Redis caching• Lots of writes, fewer reads

WHAT ARE THE PROS & CONS OF MONGODB?

MONGODB PROS & CONS Pros• Schemaless• Fire & Forget• Scalable writes / reads• Fast!

MONGODB PROS & CONS Pros Cons• Schemaless • Schemaless• Fire & Forget • Fire & Forget• Scalable writes / reads • No joins• Fast! • No transactions • Database level locking

WHEN SHOULD YOU THINK ABOUT SCALING?

• From the start!• Monitor• Anticipate• React early

WHAT ARE THE KEY RESOURCES?

RAM• Heavily reliant on available RAM• “Working set” should ﬁt in RAM• Indexes & documents

RAM in RAM not in RAM

I/O• When data is not in RAM, MongoDB hits the disk• Ensure this happens infrequently• When it does, it should be fast• EBS throughput sucks

HOW TO KEEP I/O FAST• Fast ﬁlesystem - 10gen recommends xfs• Use RAID - e.g. RAID 10 (stripe of mirrors)• Increase ﬁle descriptor limits• Turn off atime and diratime• Tweak read-ahead settings• http://www.mongodb.org/display/DOCS/Production+Notes

HOW CAN YOU ARCHITECT MONGODB TO SCALE?

VERTICAL SCALING• Buy more resources on single machine • RAM • I/O

HORIZONTAL SCALING • Buy more machines • Replica sets • Sharding

REPLICA SETS• Scales reads well• One primary, many secondaries• Read from all members• Write to primary only• Inconsistent reads from secondaries

SHARDING• Many primaries, many secondaries• Scales writes and reads• Harder to set up well

WHAT OTHER TECHNIQUES TO SCALE?

STANDARD RULES• Standard DB scaling rules apply to MongoDB • Use skip() and limit() • Return subsets of ﬁelds • Index all your queries • Run explain() on new/slow queries

SCHEMA DESIGN• De-normalize { "_id" : ObjectId("505bd6a6c6b6b99254000003"), "author" : "Simon Maynard", "post" : "Hey everyone!", "comments" : [ { "author" : "anonymous", "text" : "Hey!", },{ "author" : "James Smith", "text" : "Hey Simon!", } }

SCHEMA DESIGN• Indexes should be minimized in size and number { { "name" : "Angry Birds", "name" : "Angry Birds", "android" : true, "platform" : 3 "iphone" : true } }

SCHEMA DESIGN• Minimize key lengths on small documents• Can reduce storage requirements and performance increase{ "_id":"AHAHSPGPGSAVKLPAPHSVGKSALR", "game_id":"8122", "user_id":"1854", "session_start":"51067007", "session_end":"51067085"} 92 bytes

SCHEMA DESIGN• Minimize key lengths on small documents• Can reduce storage requirements and performance increase{ { "_id":"AHAHSPGPGSAVKLPAPHSVGKSALR", "_id":"AHAHSPGPGSAVKLPAPHSVGKSALR", "game_id":"8122", "g":"8122", "user_id":"1854", "u":"1854", "session_start":"51067007", "s":"51067007", "session_end":"51067085" "e":"51067085"} } 92 bytes 58 bytes About 1/3 memory saved!

PROFILER• MongoDB has a built in profiler• Use the profiler all the time• db.setProfilingLevel(1, 100)• ‘show profile’ shows recent profiles• Stored in db.system.profile

PROFILER OUTPUT"ts" : ISODate("2012-09-24T23:24:28.908Z"), "nscanned" : 1,"op" : "query", "scanAndOrder" : true,"ns" : "bugsnag.errors", "numYield" : 0,"query" : { "lockStats" : { "query" : { "timeLockedMicros" : { }, "errorHash":"2ff33b4f86543972577cdee34f60e4b2", "timeAcquiringMicros" : { "project_id" :"4ff24b7e2511bb1a70000004" "r" : NumberLong(2), }, "w" : NumberLong(3) "orderby" : { } "_id" : 1 }, } "nreturned" : 1,}, "responseLength" : 5240,"ntoreturn" : 1, "millis" : 0,"ntoskip" : 0,

WHAT SHOULD I MONITOR?

MONITORS• Chart the index size• Chart the number of current ops• Monitor index misses• Monitor replication lag• Monitor I/O performance (iostat)• Monitor disk space

HOW CAN I MONITOR MONGODB?

db.currentOp(){ "opid" : 783608, "active" : true, "secs_running" : 149, "op" : "query", "ns" : "bugsnag.accounts", "query" : { "_id" : ObjectId("505bd6a6c6b6b99254000003"), }, "waitingForLock" : false, "numYields" : 349,}

db.serverStatus()"locks" : { ! "misses" : 0,! "bugsnag" : { ! "resets" : 0, "timeLockedMicros" : { ! "missRatio" : 0 ! "r" : NumberLong(1639187950), } ! "w" : NumberLong(1313312267) }, }, "opcounters" : { "timeAcquiringMicros" : { ! "insert" : 13674147, "r" : NumberLong(1041368094), ! "query" : 5261723, "w" : NumberLong(630905947) ! "update" : 2576757, } ! "delete" : 22324, }, ! "getmore" : 4459,}, ! "command" : 4382007"indexCounters" : { },! "btree" : { "accesses" : 610645909, ! "hits" : 610645909,

db.stats(){! "db" : "bugsnag",! "collections" : 14,! "objects" : 68081951,! "avgObjSize" : 10147.85350585104,! "dataSize" : 690885147618,! "storageSize" : 1290028235245,! "numExtents" : 67,! "indexes" : 28,! "indexSize" : 21240430449,! "fileSize" : 1925185536051,! "nsSizeMB" : 16,! "ok" : 1}

MONGOTOP ns total read write bugsnag.events 80ms 12ms 68ms bugsnag.projects 2ms 2ms 0ms bugsnag.users 1ms 1ms 0msbugsnag.system.indexes 4ms 4ms 0ms

MONGOSTAT insert query update delete getmore command ﬂushes faults locked db idx miss %localhost 147 210 51 13 4 215 0 0 14% 0

MONGO MONITORING SERVICE• MMS is 10gen hosted Mongo monitoring• Available as web app (https://mms.10gen.com)• Android client also available from Google Play

KIBANA & LOGSTASH• Logstash is open-source log parser - http://logstash.net/• Kibana is an alternative UI for Logstash - http://kibana.org/• Cool trend analysis for mongo logs

• Questions?• Check out www.bugsnag.com• Follow me on twitter @snmaynard

Mongo scaling

by Simon Maynard on Sep 27, 2012

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Mongo scaling Presentation Transcript