Running a MongoDB cluster is usually smooth sailing, but as your load increases you may notice things start to slow down. This talk will run through a few of the options you have to notice problems, and the ways to fix them. We’ll be focussing mainly on running a cluster inside EC2, as the challenges are slightly different, but you should learn something regardless of where you’re hosted.

1. MongoDB
Debugging Performance Problems
@ConradIrwin

3. 50M crashes/day
4TB data / 100GB index
15 nodes / 3* availability zones

4. I don't care if MongoDB is slow
I care if my app is slow

5. How to make app fast again it's slow because
of the way I'm

6. What does slow mean?

8. Performance over time

9. Performance over time

11. It's slow :(

14. Solution 1
Denormalize

16. It's fast :)

17. It's slow :(

18. db.errors.
find({project_id: x}).
sort({ _id: -1}).
limit(30)

19. db.errors.
find({project_id: x}).
sort({ _id: -1}).
limit(30).
explain()

20. {
"cursor" : "BtreeCursor _id_ reverse",
"isMultiKey" : false,
"n" : 0,
"nscannedObjects" : 227756,
"nscanned" : 227756,
"nscannedObjectsAllPlans" : 227756,
"nscannedAllPlans" : 227756,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 1779,
"nChunkSkips" : 0,
"millis" : 461,
"indexBounds" : {
"_id" : [
[
{
"$maxElement" : 1
},
{
"$minElement" : 1
}
]
]
},
"server" : "Jaroussky.local:27017",
"filterSet" : false
}

21. { "cursor" :
"BtreeCursor _id_ reverse",
"nscanned" : 227756,
"indexBounds" : {"_id" : [
[{"$maxElement" : 1},
{"$minElement" : 1}]
]
}
}

22. Solution 2
Index

23. db.errors.ensureIndex(
{project_id: 1, _id: -1}
)

24. {"cursor" : "BtreeCursor
project_id_1__id_1",
"nscanned" : 6,
"indexBounds" : {
"project_id" : [[
ObjectId(x),ObjectId(x)
]],
"_id" : [[
{"$minElement" : 1},
{"$maxElement" : 1}
]]
}}

25. It's fast :)

26. It's slow :(

27. Only 1 query...
Indexed properly...

28. mongostat

29. insert query update delete getmore command flushes mapped
5 97 57 *0 95 127|0 0 320g
5 98 61 *0 113 146|0 0 320g
8 94 61 *0 95 137|0 0 320g
vsize res faults locked db idx miss % qr|qw ar|aw
641g 8.65g 7 bugsnag:47.7% 0 0|0 0|0
641g 8.66g 2 bugsnag:21.0% 0 0|0 0|0
641g 8.64g 3 bugsnag:23.4% 0 0|0 0|0
netIn netOut conn set repl time
47k 73k 145 bugsnag1 PRI 06:34:05
63k 99k 146 bugsnag1 PRI 06:34:06
98k 124k 146 bugsnag1 PRI 06:34:07

30. command locked db time
127|0 bugsnag:47.7% 06:34:05
146|0 bugsnag:21.0% 06:34:06
137|0 bugsnag:23.4% 06:34:07

32. Solution 3
Shard

34. sh.shardCollection("errors"
{
project_id: 1,
_id: -1
})

35. command locked db time
83|0 bugsnag:3.1% 06:46:12
69|0 bugsnag:4.1% 06:46:13
73|0 bugsnag:2.4% 06:46:14

36. It's fast :)

37. It's slow :(

38. Only 1 query...
Indexed properly...
Lock % ok...

39. iostat

40. Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
xvdi 0.00 2.00 15.00 743.00 436.00 7905.50 22.01
xvdi 0.00 2.00 0.00 663.00 0.00 6706.00 20.23
xvdi 0.00 1.00 0.00 726.00 0.00 5593.50 15.41
avgqu-sz await r_await w_await svctm %util
11.02 14.54 5.33 14.73 0.60 45.60
8.04 12.12 0.00 12.12 0.51 33.60
13.01 17.92 0.00 17.92 0.50 36.40

41. rkB/s wkB/s await %util
7660 138 14.73 83.60
2248 2043 12.12 65.20
7905 436 17.92 45.60

42. db.stats()['indexSize']
= 3852071824
= 3932614656
free -b (m3.medium)

43. Solution 4
Scale

44. db.stats()['indexSize']
= 3852071824
= 15775363072
free -b (r3.large)

45. It's fast :)

46. It's slow :(

47. Main query seems fine...
"Quick" queries sometimes slow...

48. mongotop

49. ns total read write 20
14-06-22T19:11:35
bugsnag.events 95ms 0ms 95ms
bugsnag.errors 80ms 6ms 74ms
bugsnag.system.namespaces 26ms 26ms 0ms
bugsnag.projects 12ms 5ms 7ms
bugsnag.users 15ms 2ms 13ms
bugsnag.error_aggregates 4ms 0ms 4ms
bugsnag.deploys 4ms 3ms 1ms
bugsnag.event_tallies 3ms 0ms 3ms

50. ns tot r w
bugsnag.events 95 0 95
bugsnag.errors 80 6 74
bugsnag.projects 12 5 7
bugsnag.users 15 7 8

52. Solution 5
Tag shards

54. It's fast :)

55. Solution 1
Denormalize

56. Solution 2
Index

57. Solution 3
Shard

58. Solution 4
Scale

59. Solution 5
Tag shards

60. Solution 6
...

61. NewRelic / Skylight
explain()
mongostat
iostat
mongotop
...

62. It will
be slow

63. You can
speed it up

64. Thanks!
@ConradIrwin