Webinar: Ensuring High Availabilityfor Real-time Analytics
~3333 write ops/s   0.07 - 0.05 ms response
David MyttonWoop Japan!
MongoDB at Server Density
MongoDB at Server Density•27 nodes
MongoDB at Server Density•27 nodes• June 2009 - +3yrs
MongoDB at Server Density•27 nodes• June 2009 - +3yrs•MySQL -> MongoDB
MongoDB at Server Density•27 nodes• June 2009 - +3yrs•MySQL -> MongoDB•15TB data per month
Why?
Why?• Replication
Why?• Replication• Official drivers
Why?• Replication• Official drivers• Easy deployment
Why?• Replication• Official drivers• Easy deployment• Fast out of the box
Fast out of the box?Photo: dannychoo.com
~3333 write ops/s   0.07 - 0.05 ms response
Fast out of the box?• Softlayer cloud (1 core, 8GB)Photo: dannychoo.com
Fast out of the box?• Softlayer cloud (1 core, 8GB)• Local instance storagePhoto: dannychoo.com
Fast out of the box?• Softlayer cloud (1 core, 8GB)• Local instance storage• Ubuntu 10.04 LTSPhoto: dannychoo.com
Fast out of the box?• JournalingPhoto: dannychoo.com
Fast out of the box?• Journaling• ReplicationPhoto: dannychoo.com
Fast out of the box?• Journaling• Replication• Unsafe writesPhoto: dannychoo.com
Fast out of the box?Picture is unrelated! Mmm, ice cream.
Fast out of the box?• Fast networkPicture is unrelated! Mmm, ice cream.
Fast out of the box?• Fast network• Working set in RAMPicture is unrelated! Mmm, ice cream.
mongos> db.metrics_20120508_15_1m.stats(){    "sharded" : true,    "flags" : 1,    "ns" : "metrics.metrics_20120508_15_1m"...
"size" : 746837640,    "totalIndexSize" : 517581680Indexes = 493MB      Data = 712MB
"size" : 746837640,    "totalIndexSize" : 517581680Indexes = 493MB      Data = 712MB
"size" : 746837640,    "totalIndexSize" : 517581680Indexes = 493MB        Data = 712MB            Total = 1205MB
Where should it go?                                                     Should it be in                            What?  ...
How you’ll know1) Slow queries                 Thu Oct 14 17:01:11 [conn7410] update sd.apiLog                query: { c: ...
How you’ll know2) Timeouts    cursor timed out (20000 ms)
How you’ll know3) Disk i/o spikeswww.flickr.com/photos/daddo83/3406962115/
Fast out of the box?• Fast network• Working set in RAMPicture is unrelated! Mmm, ice cream.
Fast out of the box?• Fast network• Working set in RAM• Fast disks (optional)Picture is unrelated! Mmm, ice cream.
Fast out of the box?• Fast network• Working set in RAM• Fast disks (optional)Picture is unrelated! Mmm, ice cream.
Fast out of the box?• Fast network• Working set in RAM• Fast disks (optional)• Sharding (optional)Picture is unrelated! Mm...
Scaling writes
Scaling writes• Global DB lock
Scaling writes• Global DB lock• Concurrency
http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
http://bit.ly/mongolock
Scaling writes• Global DB lock• Concurrency• Sharding
Scaling writes
Failover•Replica sets
Failover•Replica sets                •Master/slave
Failover•Replica sets                •Master/slave                •Min 3 nodes
Failover•Replica sets                •Master/slave                •Min 3 nodes            •Automatic failover
rs.status()       {       !     "_id" : 1,       !     "name" : "rs3b:27018",       !     "health" : 1,       !     "state...
rs.status()1) myState       Value           Meaning         0   Starting up (phase 1)         1   Primary         2   Seco...
rs.status()2) Optime         "optimeDate" : ISODate("2010-12-02T01:09:38Z")www.flickr.com/photos/robbie73/4244846566/
rs.status()3) Heartbeat         "lastHeartbeat" : ISODate("2010-12-02T01:09:38Z")www.flickr.com/photos/drawblindfaith/34009...
Scaling reads
Scaling reads•Replica slaves
Scaling reads•Replica slaves•Consistency
Scaling reads•Replica slaves•Consistency•w flag / tags
Unsafe writes
Unsafe writes• Fast by default
Unsafe writes• Fast by default•J
Unsafe writes• Fast by default•J• fsync
Unsafe writes• Fast by default•J• fsync•w
Tags{    _id : "someSet",    members : [        {_id : 0, host : "A", tags : {"dc":   "ny"}},        {_id : 1, host : "B",...
Tags{    _id : "someSet",                               (A or B) + (C or D) + E    members : [        {_id : 0, host : "A"...
Tags{    _id : "someSet",                                 (A + C) or (D + E) ...    members : [        {_id : 0, host : "A...
Bottlenecks•EC2
Bottlenecks•EC2       •Local storage
Bottlenecks•EC2             •Local storage   •EBS: RAID10 4-8 volumes
Bottlenecks•EC2               •Local storage    •EBS: RAID10 4-8 volumes•i/o: rand but not sequential
http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs
http://bit.ly/ec2mongodb
Bottlenecks•CPU       •Index building
mongostat
mongostatLocks/Queues
mongostatDiagnostics
Current operations    db.currentOp();    {    ! ! ! "opid" : "shard1:299939199",    ! ! ! "active" : true,    ! ! ! "lockT...
Monitoring toolsRun yourself    Ganglia
Monitoring toolsServer Density
Fast out of the box?• Fast network• Working set in RAMPicture is unrelated! Mmm, ice cream.
Fast out of the box?• Fast network• Working set in RAM• bit.ly/benchrunPicture is unrelated! Mmm, ice cream.
David Mytton @davidmyttondavid@serverdensity.comwww.serverdensity.comWoop Japan!
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Server Density Webinar
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Server Density Webinar
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Server Density Webinar
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Server Density Webinar
Upcoming SlideShare
Loading in …5
×

Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Server Density Webinar

1,081 views

Published on

This will cover what to consider for high write throughput performance from hardware configuration through to the use of replica sets, multi-data centre deployments, monitoring and sharding to ensure your database is fast and stays online.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,081
On SlideShare
0
From Embeds
0
Number of Embeds
423
Actions
Shares
0
Downloads
11
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Server Density Webinar

  1. 1. Webinar: Ensuring High Availabilityfor Real-time Analytics
  2. 2. ~3333 write ops/s 0.07 - 0.05 ms response
  3. 3. David MyttonWoop Japan!
  4. 4. MongoDB at Server Density
  5. 5. MongoDB at Server Density•27 nodes
  6. 6. MongoDB at Server Density•27 nodes• June 2009 - +3yrs
  7. 7. MongoDB at Server Density•27 nodes• June 2009 - +3yrs•MySQL -> MongoDB
  8. 8. MongoDB at Server Density•27 nodes• June 2009 - +3yrs•MySQL -> MongoDB•15TB data per month
  9. 9. Why?
  10. 10. Why?• Replication
  11. 11. Why?• Replication• Official drivers
  12. 12. Why?• Replication• Official drivers• Easy deployment
  13. 13. Why?• Replication• Official drivers• Easy deployment• Fast out of the box
  14. 14. Fast out of the box?Photo: dannychoo.com
  15. 15. ~3333 write ops/s 0.07 - 0.05 ms response
  16. 16. Fast out of the box?• Softlayer cloud (1 core, 8GB)Photo: dannychoo.com
  17. 17. Fast out of the box?• Softlayer cloud (1 core, 8GB)• Local instance storagePhoto: dannychoo.com
  18. 18. Fast out of the box?• Softlayer cloud (1 core, 8GB)• Local instance storage• Ubuntu 10.04 LTSPhoto: dannychoo.com
  19. 19. Fast out of the box?• JournalingPhoto: dannychoo.com
  20. 20. Fast out of the box?• Journaling• ReplicationPhoto: dannychoo.com
  21. 21. Fast out of the box?• Journaling• Replication• Unsafe writesPhoto: dannychoo.com
  22. 22. Fast out of the box?Picture is unrelated! Mmm, ice cream.
  23. 23. Fast out of the box?• Fast networkPicture is unrelated! Mmm, ice cream.
  24. 24. Fast out of the box?• Fast network• Working set in RAMPicture is unrelated! Mmm, ice cream.
  25. 25. mongos> db.metrics_20120508_15_1m.stats(){ "sharded" : true, "flags" : 1, "ns" : "metrics.metrics_20120508_15_1m", "count" : 2752934, "numExtents" : 46, "size" : 746837640, "storageSize" : 823717888, "totalIndexSize" : 517581680, "indexSizes" : { "_id_" : 130358144, "a_1_i_1" : 155711920, "a_1_i_1_m_1_t_1" : 231511616 }, "avgObjSize" : 271.2878841265355, "nindexes" : 3, "nchunks" : 61,
  26. 26. "size" : 746837640, "totalIndexSize" : 517581680Indexes = 493MB Data = 712MB
  27. 27. "size" : 746837640, "totalIndexSize" : 517581680Indexes = 493MB Data = 712MB
  28. 28. "size" : 746837640, "totalIndexSize" : 517581680Indexes = 493MB Data = 712MB Total = 1205MB
  29. 29. Where should it go? Should it be in What? memory? Indexes Always Data If you canhttp://www.flickr.com/photos/comedynose/4388430444/
  30. 30. How you’ll know1) Slow queries Thu Oct 14 17:01:11 [conn7410] update sd.apiLog query: { c: "android/setDeviceToken", a: 1466, u: "blah", ua: "Server Density Android" } 51926mswww.flickr.com/photos/tonivc/2283676770/
  31. 31. How you’ll know2) Timeouts cursor timed out (20000 ms)
  32. 32. How you’ll know3) Disk i/o spikeswww.flickr.com/photos/daddo83/3406962115/
  33. 33. Fast out of the box?• Fast network• Working set in RAMPicture is unrelated! Mmm, ice cream.
  34. 34. Fast out of the box?• Fast network• Working set in RAM• Fast disks (optional)Picture is unrelated! Mmm, ice cream.
  35. 35. Fast out of the box?• Fast network• Working set in RAM• Fast disks (optional)Picture is unrelated! Mmm, ice cream.
  36. 36. Fast out of the box?• Fast network• Working set in RAM• Fast disks (optional)• Sharding (optional)Picture is unrelated! Mmm, ice cream.
  37. 37. Scaling writes
  38. 38. Scaling writes• Global DB lock
  39. 39. Scaling writes• Global DB lock• Concurrency
  40. 40. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
  41. 41. http://blog.pythonisito.com/2011/12/mongodbs-write-lock.html
  42. 42. http://bit.ly/mongolock
  43. 43. Scaling writes• Global DB lock• Concurrency• Sharding
  44. 44. Scaling writes
  45. 45. Failover•Replica sets
  46. 46. Failover•Replica sets •Master/slave
  47. 47. Failover•Replica sets •Master/slave •Min 3 nodes
  48. 48. Failover•Replica sets •Master/slave •Min 3 nodes •Automatic failover
  49. 49. rs.status() { ! "_id" : 1, ! "name" : "rs3b:27018", ! "health" : 1, ! "state" : 2, ! "stateStr" : "SECONDARY", ! "uptime" : 1886098, ! "optime" : { ! ! "t" : 1291252178000, ! ! "i" : 13 ! }, ! "optimeDate" : ISODate("2010-12-02T01:09:38Z"), "lastHeartbeat" : ISODate("2010-12-02T01:09:38Z") },www.ex-astris-scientia.org/inconsistencies/ent_vs_tng.htm (yes it’s a replicator from Star Trek)
  50. 50. rs.status()1) myState Value Meaning 0 Starting up (phase 1) 1 Primary 2 Secondary 3 Recovering 4 Fatal error 5 Starting up (phase 2) 6 Unknown state 7 Arbiter 8 Down
  51. 51. rs.status()2) Optime "optimeDate" : ISODate("2010-12-02T01:09:38Z")www.flickr.com/photos/robbie73/4244846566/
  52. 52. rs.status()3) Heartbeat "lastHeartbeat" : ISODate("2010-12-02T01:09:38Z")www.flickr.com/photos/drawblindfaith/3400981091/
  53. 53. Scaling reads
  54. 54. Scaling reads•Replica slaves
  55. 55. Scaling reads•Replica slaves•Consistency
  56. 56. Scaling reads•Replica slaves•Consistency•w flag / tags
  57. 57. Unsafe writes
  58. 58. Unsafe writes• Fast by default
  59. 59. Unsafe writes• Fast by default•J
  60. 60. Unsafe writes• Fast by default•J• fsync
  61. 61. Unsafe writes• Fast by default•J• fsync•w
  62. 62. Tags{ _id : "someSet", members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } }}> db.foo.insert({x:1})> db.runCommand({getLastError : 1, w : "veryImportant"})
  63. 63. Tags{ _id : "someSet", (A or B) + (C or D) + E members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } }}> db.foo.insert({x:1})> db.runCommand({getLastError : 1, w : "veryImportant"})
  64. 64. Tags{ _id : "someSet", (A + C) or (D + E) ... members : [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}} ] settings : { getLastErrorModes : { veryImportant : {"dc" : 3}, sortOfImportant : {"dc" : 2} } }}> db.foo.insert({x:1})> db.runCommand({getLastError : 1, w : "sortOfImportant"})
  65. 65. Bottlenecks•EC2
  66. 66. Bottlenecks•EC2 •Local storage
  67. 67. Bottlenecks•EC2 •Local storage •EBS: RAID10 4-8 volumes
  68. 68. Bottlenecks•EC2 •Local storage •EBS: RAID10 4-8 volumes•i/o: rand but not sequential
  69. 69. http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs
  70. 70. http://bit.ly/ec2mongodb
  71. 71. Bottlenecks•CPU •Index building
  72. 72. mongostat
  73. 73. mongostatLocks/Queues
  74. 74. mongostatDiagnostics
  75. 75. Current operations db.currentOp(); { ! ! ! "opid" : "shard1:299939199", ! ! ! "active" : true, ! ! ! "lockType" : "write", ! ! ! "waitingForLock" : false, ! ! ! "secs_running" : 15419, ! ! ! "op" : "remove", ! ! ! "ns" : "sd.metrics", ! ! ! "query" : { ! ! ! ! "accId" : 1391, ! ! ! ! "tA" : { ! ! ! ! ! "$lte" : ISODate("2010-11-24T19:53:00Z") ! ! ! ! } ! ! ! }, ! ! ! "client" : "10.121.12.228:44426", ! ! ! "desc" : "conn" ! ! },www.flickr.com/photos/jeffhester/2784666811/
  76. 76. Monitoring toolsRun yourself Ganglia
  77. 77. Monitoring toolsServer Density
  78. 78. Fast out of the box?• Fast network• Working set in RAMPicture is unrelated! Mmm, ice cream.
  79. 79. Fast out of the box?• Fast network• Working set in RAM• bit.ly/benchrunPicture is unrelated! Mmm, ice cream.
  80. 80. David Mytton @davidmyttondavid@serverdensity.comwww.serverdensity.comWoop Japan!

×