• Like
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
829
On Slideshare
0
From Embeds
0
Number of Embeds
5

Actions

Shares
Downloads
20
Comments
0
Likes
4

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. MongoDB Debugging Performance Problems @ConradIrwin
  • 2. 20M crashes/day 2TB data / 50GB index 11 nodes / 3* availability zones
  • 3. I don't care if MongoDB is slow I care if my app is slow
  • 4. How to make app fast agai it's slow bec of the way I'm
  • 5. What does slow mean?
  • 6. Performance over time
  • 7. Performance over time
  • 8. It's slow :(
  • 9. Solution 1 Denormalize
  • 10. It's fast :)
  • 11. It's slow :(
  • 12. db.errors. find({project_id: x}). sort({ _id: -1}). limit(30)
  • 13. db.errors. find({project_id: x}). sort({ _id: -1}). limit(30). explain()
  • 14. { "cursor":"BtreeCursor_id_reverse", "isMultiKey":false, "n":0, "nscannedObjects":227756, "nscanned":227756, "nscannedObjectsAllPlans":227756, "nscannedAllPlans":227756, "scanAndOrder":false, "indexOnly":false, "nYields":1779, "nChunkSkips":0, "millis":461, "indexBounds":{ "_id":[ [ { "$maxElement":1 }, { "$minElement":1 } ] ] }, "server":"Jaroussky.local:27017", "filterSet":false }
  • 15. { "cursor" : "BtreeCursor _id_ reverse", "nscanned" : 227756, "indexBounds" : {"_id" : [ [{"$maxElement" : 1}, {"$minElement" : 1}] ] } }
  • 16. { "cursor" : "BtreeCursor _id_ reverse", "nscanned" : 227756, "indexBounds" : {"_id" : [ [{"$maxElement" : 1}, {"$minElement" : 1}] ] } }
  • 17. Solution 2 Index
  • 18. db.errors.ensureIndex( {project_id: 1, id: -1} )
  • 19. {"cursor" : "BtreeCursor project_id_1__id_1", "nscanned" : 6, "indexBounds" : { "project_id" : [[ ObjectId(x),ObjectId(x) ]], "_id" : [[ {"$minElement" : 1}, {"$maxElement" : 1} ]] }}
  • 20. It's fast :)
  • 21. It's slow :(
  • 22. Only 1 query... Indexed properly...
  • 23. mongostat
  • 24. insert queryupdatedeletegetmorecommandflushesmapped 5 97 57 *0 95 127|0 0 320g 5 98 61 *0 113 146|0 0 320g 8 94 61 *0 95 137|0 0 320g vsize resfaults lockeddbidxmiss% qr|qw ar|aw 641g 8.65g 7bugsnag:47.7% 0 0|0 0|0 641g 8.66g 2bugsnag:21.0% 0 0|0 0|0 641g 8.64g 3bugsnag:23.4% 0 0|0 0|0 netInnetOut conn setrepl time 47k 73k 145bugsnag1 PRI 06:34:05 63k 99k 146bugsnag1 PRI 06:34:06 98k 124k 146bugsnag1 PRI 06:34:07
  • 25. command locked db time 127|0 bugsnag:47.7% 06:34:05 146|0 bugsnag:21.0% 06:34:06 137|0 bugsnag:23.4% 06:34:07
  • 26. Solution 3 Shard
  • 27. sh.shardCollection("errors" { project_id: 1, _id: -1 })
  • 28. command locked db time 83|0 bugsnag:3.1% 06:46:12 69|0 bugsnag:4.1% 06:46:13 73|0 bugsnag:2.4% 06:46:14
  • 29. It's fast :)
  • 30. It's slow :(
  • 31. Only 1 query... Indexed properly... Lock % ok...
  • 32. iostat
  • 33. Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/savgrq-sz xvdi 0.00 2.00 15.00 743.00 436.00 7905.50 22.01 xvdi 0.00 2.00 0.00 663.00 0.00 6706.00 20.23 xvdi 0.00 1.00 0.00 726.00 0.00 5593.50 15.41 avgqu-sz awaitr_awaitw_await svctm %util 11.02 14.54 5.33 14.73 0.60 45.60 8.04 12.12 0.00 12.12 0.51 33.60 13.01 17.92 0.00 17.92 0.50 36.40
  • 34. rkB/s wkB/s await %util 7660 138 14.73 83.60 2248 2043 12.12 65.20 7905 436 17.92 45.60
  • 35. db.stats()['indexSize'] = 3852071824 = 3932614656 free -b(m3.medium)
  • 36. Solution 4 Scale
  • 37. db.stats()['indexSize'] = 3852071824 = 15775363072 free -b(r3.large)
  • 38. It's fast :)
  • 39. It's slow :(
  • 40. Main query seems fine... "Quick" queries sometimes slow...
  • 41. mongotop
  • 42. ns total read write 2 014-06-22T19:11:35 bugsnag.events 35ms 0ms 35ms bugsnag.errors 30ms 6ms 24ms bugsnag.system.namespaces 26ms 26ms 0ms bugsnag.projects 12ms 5ms 7ms bugsnag.users 15ms 2ms 13ms bugsnag.error_aggregates 4ms 0ms 4ms bugsnag.deploys 4ms 3ms 1ms bugsnag.event_tallies 3ms 0ms 3ms
  • 43. ns tot r w bugsnag.events 35 0 35 bugsnag.errors 30 6 24 bugsnag.projects 12 5 7 bugsnag.users 15 7 8
  • 44. Solution 5 Tag shards
  • 45. It's fast :)
  • 46. Solution 1 Denormalize
  • 47. Solution 2 Index
  • 48. Solution 3 Shard
  • 49. Solution 4 Scale
  • 50. Solution 5 Tag shards
  • 51. Solution 6 ...
  • 52. NewRelic / Skylight explain() mongostat iostat ...
  • 53. It will be slow
  • 54. You can speed it up
  • 55. Thanks!@ConradIrwin