0
MongoDB
Debugging Performance Problems
@ConradIrwin
20M crashes/day
2TB data / 50GB index
11 nodes / 3* availability zones
I don't care if MongoDB is slow
I care if my app is slow
How to make
app fast agai
it's slow bec
of the way I'm
What does slow mean?
Performance over time
Performance over time
It's slow :(
Solution 1
Denormalize
It's fast :)
It's slow :(
db.errors.
find({project_id: x}).
sort({ _id: -1}).
limit(30)
db.errors.
find({project_id: x}).
sort({ _id: -1}).
limit(30).
explain()
{
"cursor":"BtreeCursor_id_reverse",
"isMultiKey":false,
"n":0,
"nscannedObjects":227756,
"nscanned":227756,
"nscannedObje...
{
"cursor" :
"BtreeCursor _id_ reverse",
"nscanned" : 227756,
"indexBounds" : {"_id" : [
[{"$maxElement" : 1},
{"$minEleme...
{
"cursor" :
"BtreeCursor _id_ reverse",
"nscanned" : 227756,
"indexBounds" : {"_id" : [
[{"$maxElement" : 1},
{"$minEleme...
Solution 2
Index
db.errors.ensureIndex(
{project_id: 1, id: -1}
)
{"cursor" : "BtreeCursor
project_id_1__id_1",
"nscanned" : 6,
"indexBounds" : {
"project_id" : [[
ObjectId(x),ObjectId(x)
...
It's fast :)
It's slow :(
Only 1 query...
Indexed properly...
mongostat
insert queryupdatedeletegetmorecommandflushesmapped
5 97 57 *0 95 127|0 0 320g
5 98 61 *0 113 146|0 0 320g
8 94 61 *0 95 1...
command locked db time
127|0 bugsnag:47.7% 06:34:05
146|0 bugsnag:21.0% 06:34:06
137|0 bugsnag:23.4% 06:34:07
Solution 3
Shard
sh.shardCollection("errors"
{
project_id: 1,
_id: -1
})
command locked db time
83|0 bugsnag:3.1% 06:46:12
69|0 bugsnag:4.1% 06:46:13
73|0 bugsnag:2.4% 06:46:14
It's fast :)
It's slow :(
Only 1 query...
Indexed properly...
Lock % ok...
iostat
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/savgrq-sz
xvdi 0.00 2.00 15.00 743.00 436.00 7905.50 22.01
xvdi 0.00 2.00 0.00 663...
rkB/s wkB/s await %util
7660 138 14.73 83.60
2248 2043 12.12 65.20
7905 436 17.92 45.60
db.stats()['indexSize']
= 3852071824
= 3932614656
free -b(m3.medium)
Solution 4
Scale
db.stats()['indexSize']
= 3852071824
= 15775363072
free -b(r3.large)
It's fast :)
It's slow :(
Main query seems fine...
"Quick" queries sometimes slow...
mongotop
ns total read write 2
014-06-22T19:11:35
bugsnag.events 35ms 0ms 35ms
bugsnag.errors 30ms 6ms 24ms
bugsnag.system.namespac...
ns tot r w
bugsnag.events 35 0 35
bugsnag.errors 30 6 24
bugsnag.projects 12 5 7
bugsnag.users 15 7 8
Solution 5
Tag shards
It's fast :)
Solution 1
Denormalize
Solution 2
Index
Solution 3
Shard
Solution 4
Scale
Solution 5
Tag shards
Solution 6
...
NewRelic / Skylight
explain()
mongostat
iostat
...
It will
be slow
You can
speed it up
Thanks!@ConradIrwin
MongoDB Performance Debugging
MongoDB Performance Debugging
MongoDB Performance Debugging
MongoDB Performance Debugging
MongoDB Performance Debugging
MongoDB Performance Debugging
MongoDB Performance Debugging
MongoDB Performance Debugging
Upcoming SlideShare
Loading in...5
×

MongoDB Performance Debugging

1,474

Published on

Published in: Technology, Business

Transcript of "MongoDB Performance Debugging"

  1. 1. MongoDB Debugging Performance Problems @ConradIrwin
  2. 2. 20M crashes/day 2TB data / 50GB index 11 nodes / 3* availability zones
  3. 3. I don't care if MongoDB is slow I care if my app is slow
  4. 4. How to make app fast agai it's slow bec of the way I'm
  5. 5. What does slow mean?
  6. 6. Performance over time
  7. 7. Performance over time
  8. 8. It's slow :(
  9. 9. Solution 1 Denormalize
  10. 10. It's fast :)
  11. 11. It's slow :(
  12. 12. db.errors. find({project_id: x}). sort({ _id: -1}). limit(30)
  13. 13. db.errors. find({project_id: x}). sort({ _id: -1}). limit(30). explain()
  14. 14. { "cursor":"BtreeCursor_id_reverse", "isMultiKey":false, "n":0, "nscannedObjects":227756, "nscanned":227756, "nscannedObjectsAllPlans":227756, "nscannedAllPlans":227756, "scanAndOrder":false, "indexOnly":false, "nYields":1779, "nChunkSkips":0, "millis":461, "indexBounds":{ "_id":[ [ { "$maxElement":1 }, { "$minElement":1 } ] ] }, "server":"Jaroussky.local:27017", "filterSet":false }
  15. 15. { "cursor" : "BtreeCursor _id_ reverse", "nscanned" : 227756, "indexBounds" : {"_id" : [ [{"$maxElement" : 1}, {"$minElement" : 1}] ] } }
  16. 16. { "cursor" : "BtreeCursor _id_ reverse", "nscanned" : 227756, "indexBounds" : {"_id" : [ [{"$maxElement" : 1}, {"$minElement" : 1}] ] } }
  17. 17. Solution 2 Index
  18. 18. db.errors.ensureIndex( {project_id: 1, id: -1} )
  19. 19. {"cursor" : "BtreeCursor project_id_1__id_1", "nscanned" : 6, "indexBounds" : { "project_id" : [[ ObjectId(x),ObjectId(x) ]], "_id" : [[ {"$minElement" : 1}, {"$maxElement" : 1} ]] }}
  20. 20. It's fast :)
  21. 21. It's slow :(
  22. 22. Only 1 query... Indexed properly...
  23. 23. mongostat
  24. 24. insert queryupdatedeletegetmorecommandflushesmapped 5 97 57 *0 95 127|0 0 320g 5 98 61 *0 113 146|0 0 320g 8 94 61 *0 95 137|0 0 320g vsize resfaults lockeddbidxmiss% qr|qw ar|aw 641g 8.65g 7bugsnag:47.7% 0 0|0 0|0 641g 8.66g 2bugsnag:21.0% 0 0|0 0|0 641g 8.64g 3bugsnag:23.4% 0 0|0 0|0 netInnetOut conn setrepl time 47k 73k 145bugsnag1 PRI 06:34:05 63k 99k 146bugsnag1 PRI 06:34:06 98k 124k 146bugsnag1 PRI 06:34:07
  25. 25. command locked db time 127|0 bugsnag:47.7% 06:34:05 146|0 bugsnag:21.0% 06:34:06 137|0 bugsnag:23.4% 06:34:07
  26. 26. Solution 3 Shard
  27. 27. sh.shardCollection("errors" { project_id: 1, _id: -1 })
  28. 28. command locked db time 83|0 bugsnag:3.1% 06:46:12 69|0 bugsnag:4.1% 06:46:13 73|0 bugsnag:2.4% 06:46:14
  29. 29. It's fast :)
  30. 30. It's slow :(
  31. 31. Only 1 query... Indexed properly... Lock % ok...
  32. 32. iostat
  33. 33. Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/savgrq-sz xvdi 0.00 2.00 15.00 743.00 436.00 7905.50 22.01 xvdi 0.00 2.00 0.00 663.00 0.00 6706.00 20.23 xvdi 0.00 1.00 0.00 726.00 0.00 5593.50 15.41 avgqu-sz awaitr_awaitw_await svctm %util 11.02 14.54 5.33 14.73 0.60 45.60 8.04 12.12 0.00 12.12 0.51 33.60 13.01 17.92 0.00 17.92 0.50 36.40
  34. 34. rkB/s wkB/s await %util 7660 138 14.73 83.60 2248 2043 12.12 65.20 7905 436 17.92 45.60
  35. 35. db.stats()['indexSize'] = 3852071824 = 3932614656 free -b(m3.medium)
  36. 36. Solution 4 Scale
  37. 37. db.stats()['indexSize'] = 3852071824 = 15775363072 free -b(r3.large)
  38. 38. It's fast :)
  39. 39. It's slow :(
  40. 40. Main query seems fine... "Quick" queries sometimes slow...
  41. 41. mongotop
  42. 42. ns total read write 2 014-06-22T19:11:35 bugsnag.events 35ms 0ms 35ms bugsnag.errors 30ms 6ms 24ms bugsnag.system.namespaces 26ms 26ms 0ms bugsnag.projects 12ms 5ms 7ms bugsnag.users 15ms 2ms 13ms bugsnag.error_aggregates 4ms 0ms 4ms bugsnag.deploys 4ms 3ms 1ms bugsnag.event_tallies 3ms 0ms 3ms
  43. 43. ns tot r w bugsnag.events 35 0 35 bugsnag.errors 30 6 24 bugsnag.projects 12 5 7 bugsnag.users 15 7 8
  44. 44. Solution 5 Tag shards
  45. 45. It's fast :)
  46. 46. Solution 1 Denormalize
  47. 47. Solution 2 Index
  48. 48. Solution 3 Shard
  49. 49. Solution 4 Scale
  50. 50. Solution 5 Tag shards
  51. 51. Solution 6 ...
  52. 52. NewRelic / Skylight explain() mongostat iostat ...
  53. 53. It will be slow
  54. 54. You can speed it up
  55. 55. Thanks!@ConradIrwin
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×