Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Scaling to 30,000 Requests Per Second
and Beyond
with MongoDB
Mike Chesnut
Director of Operations Engineering
Crittercism
Scaling to 30,000 Requests Per Second
and Beyond
with MongoDB
Mike Chesnut
Director of Operations Engineering
Crittercism
How a Startup Gets Started
● Pick something and go with it
How a Startup Gets Started
● Pick something and go with it
● Make mistakes along the way
How a Startup Gets Started
● Pick something and go with it
● Make mistakes along the way
● Correct the mistakes you can
How a Startup Gets Started
● Pick something and go with it
● Make mistakes along the way
● Correct the mistakes you can
● Work around the ones you ca...
What I’ll Talk About
What I’ll Talk About
● Crittercism - Background and Architecture
What I’ll Talk About
● Crittercism - Background and Architecture
● Router (mongos) Architecture
What I’ll Talk About
● Crittercism - Background and Architecture
● Router (mongos) Architecture
● Sharding Considerations
What I’ll Talk About
● Crittercism - Background and Architecture
● Router (mongos) Architecture
● Sharding Considerations
...
What I’ll Talk About
● Crittercism - Background and Architecture
● Router (mongos) Architecture
● Sharding Considerations
...
Critter-What?
Critter-What?
A Brief History...
Critter-What?
Our Founders
(Rob, Andrew, Jeeyun)
Critter-What?
Our Founders
(Rob, Andrew, Jeeyun)
Let’s make a mobile app!
It’ll be awesome!
Critter-What?
(Unnamed Dating App)
Critter-What?
Critter-What?
Critter-What?
Our Founders
(Rob, Andrew, Jeeyun)
Our app isn’t so awesome
after all...
Critter-What?
Critter-What?
Critter-What?
Critter-What?
Critter-What?
Critter-What?
Critter-What?
Critter-What?
Architecture
Architecture
Architecture
API
Architecture
APIFeedback
Architecture
APIFeedback
Crashes
Architecture
APIFeedback
App Loads
Crashes
Architecture
APIFeedback
App Loads
Crashes
Handled
Exceptions
Architecture
APIFeedback
App Loads
Crashes
Handled
Exceptions
Architecture
API
App Loads
Crashes
Handled
Exceptions
Architecture
API
App Loads
Crashes
Handled
Exceptions
Architecture
API
Crashes
Handled
Exceptions
App Loads
batch
Architecture
API
Crashes
Handled
Exceptions
Metadata
App Loads
batch
Architecture
DynamoDB
API
Crashes
Handled
Exceptions
Metadata
App Loads
batch
Architecture
DynamoDB
API
Crashes
Handled
Exceptions
Metadata
App Loads
batch
Architecture
DynamoDB
API
API
Crashes
Handled
Exceptions
Metadata
Performance
Data
Geo Data
App Loads
batch
Architecture
DynamoDB
API
API
Crashes
Handled
Exceptions
Metadata
Performance
Data
Geo Data
40,000 req/s
App Loads
batch
Growth
Router Architecture
Router Architecture
mongod
server
mongod
server
mongod
server
replica set
mongod
server
mongod
server
mongod
server
replic...
Router Architecture
mongod
server
mongod
server
mongod
server
replica set
mongod
server
mongod
server
mongod
server
replic...
Router Architecture
mongod
server
mongod
server
mongod
server
replica set
mongod
server
mongod
server
mongod
server
replic...
Router Architecture
mongod
server
mongod
server
mongod
server
replica set
mongod
server
mongod
server
mongod
server
replic...
Router Architecture
mongod
server
mongod
server
mongod
server
replica set
mongod
server
mongod
server
mongod
server
replic...
Router Architecture
mongod
server
mongod
server
mongod
server
replica set
mongod
server
mongod
server
mongod
server
replic...
Router Architecture
mongod
server
mongod
server
mongod
server
replica set
mongod
server
mongod
server
mongod
server
replic...
Router Architecture
RS
RS
RS
conf
ms
app
ms
app
Router Architecture
RS
RS
RS
conf
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
appms
app
ms
ap...
Router Architecture
RS
RS
RS
conf
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
appms
app
ms
ap...
Single mongos per client problems we encountered:
Router Architecture
Router Architecture
Single mongos per client problems we encountered:
● thousands of connections to config servers
Router Architecture
Single mongos per client problems we encountered:
● thousands of connections to config servers
● confi...
Router Architecture
Single mongos per client problems we encountered:
● thousands of connections to config servers
● confi...
Router Architecture
RS
RS
RS
conf
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
appms
app
ms
ap...
Router Architecture
RS
RS
RS
conf
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
appms
app
ms
ap...
Router Architecture
mongod
server
mongod
server
mongod
server
replica set
mongod
server
mongod
server
mongod
server
replic...
Router Architecture
mongod
server
mongod
server
mongod
server
replica set
mongod
server
mongod
server
mongod
server
replic...
Router Architecture
Separate mongos tier advantages:
Router Architecture
Separate mongos tier advantages:
● greatly reduced number of connections to each mongod
Router Architecture
Separate mongos tier advantages:
● greatly reduced number of connections to each mongod
● far fewer ho...
Router Architecture
Separate mongos tier advantages:
● greatly reduced number of connections to each mongod
● far fewer ho...
Router Architecture
Separate mongos tier advantages:
● greatly reduced number of connections to each mongod
● far fewer ho...
Router Architecture
Separate mongos tier advantages:
● greatly reduced number of connections to each mongod
● far fewer ho...
Router Architecture
Separate mongos tier advantages:
● greatly reduced number of connections to each mongod
● far fewer ho...
Router Architecture
RS
RS
RS
conf
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
appms
app
ms
ap...
Router Architecture
RS
RS
RS
conf
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
appms
app
ms
ap...
Router Architecture
RS
RS
RS
conf
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
appms
app
ms
ap...
Router Architecture
RS
RS
RS
conf
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
appms
app
ms
ap...
Router Architecture
RS
RS
RS
conf
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
appms
app
ms
ap...
Router Architecture
RS
RS
RS
conf
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
appms
app
ms
ap...
Router Architecture
RS
RS
RS
conf
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
appms
app
ms
ap...
Router Architecture
RS
RS
RS
conf
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
app
ms
appms
app
ms
ap...
Router Architecture - Evolve!
Router Architecture - Evolve!
mongod
server
mongod
server
mongod
server
replica set
mongod
server
mongod
server
mongod
ser...
Maybe at first,
doing the
mongos-per-host
architecture
is fine.
And it will probably
remain fine
for quite a while.
Router...
Router Architecture - Evolve!
mongod
server
mongod
server
mongod
server
replica set
mongod
server
mongod
server
mongod
ser...
Sharding Considerations
Pick something you want to live with.
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
What could we have done differently?
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
Sharding Considerations
The Balancing Act
The Balancing Act
Why wouldn’t you run the balancer in the first place?
The Balancing Act
Why wouldn’t you run the balancer in the first place?
● great question
The Balancing Act
Why wouldn’t you run the balancer in the first place?
● great question
● for us, it’s because we deleted...
The Balancing Act
Why wouldn’t you run the balancer in the first place?
● great question
● for us, it’s because we deleted...
The Balancing Act
Why wouldn’t you run the balancer in the first place?
● great question
● for us, it’s because we deleted...
The Balancing Act
Why wouldn’t you run the balancer in the first place?
● great question
● for us, it’s because we deleted...
The Balancing Act
Why wouldn’t you run the balancer in the first place?
● great question
● for us, it’s because we deleted...
The Balancing Act
Why wouldn’t you run the balancer in the first place?
● great question
● for us, it’s because we deleted...
The Balancing Act
Fresh, new, empty cluster… But no balancer running.
The Balancing Act
The Balancing Act
The Balancing Act
The Balancing Act
The Balancing Act
The Balancing Act
The Balancing Act
The Balancing Act
The Balancing Act
The Balancing Act
The Balancing Act
The Balancing Act
The Balancing Act
The Balancing Act
The Balancing Act
The Balancing Act
The Balancing Act
The Balancing Act
The Balancing Act
The Balancing Act
Now we’re pretty full, so let’s add another shard...
The Balancing Act
The Balancing Act
And keep inserting...
The Balancing Act
The Balancing Act
The Balancing Act
The Balancing Act
The Balancing Act
Suddenly we find ourselves with a very unbalanced cluster.
The Balancing Act
But if we enable the balancer, it will DoS the 5th shard!
The Balancing Act
The approximate effect looks something like this:
The Balancing Act
The approximate effect looks something like this:
The Balancing Act
The approximate effect looks something like this:
The Balancing Act
The approximate effect looks something like this:
The Balancing Act
The approximate effect looks something like this:
The Balancing Act
The approximate effect looks something like this:
The Balancing Act
The approximate effect looks something like this:
So what can we do?
The Balancing Act
So what can we do?
1. add IOPS
The Balancing Act
So what can we do?
1. add IOPS
2. make sure your config servers have plenty of CPU (and IOPS)
The Balancing Act
So what can we do?
1. add IOPS
2. make sure your config servers have plenty of CPU (and IOPS)
3. slowly move chunks manual...
So what can we do?
1. add IOPS
2. make sure your config servers have plenty of CPU (and IOPS)
3. slowly move chunks manual...
So what can we do?
1. add IOPS
2. make sure your config servers have plenty of CPU (and IOPS)
3. slowly move chunks manual...
So what can we do?
1. add IOPS
2. make sure your config servers have plenty of CPU (and IOPS)
3. slowly move chunks manual...
How to manually balance:
1. determine a chunk on a hot shard
2. monitor effects on both the source and target shards
3. mo...
Conclusion here:
Run the balancer!
The Balancing Act
● Design ahead of time
o “NoSQL” lets you play it by ear
o but some of these decisions will bite you later
● Be willing to...
References
● MongoDB Blog post (details on shard
migration):http://blog.mongodb.org/post/77278906988/crittercism-scaling-t...
Q&A
Thank You!
Upcoming SlideShare
Loading in …5
×

Scaling Crittercism to 30,000 Requests Per Second and Beyond with MongoDB

3,574 views

Published on

  • Be the first to comment

Scaling Crittercism to 30,000 Requests Per Second and Beyond with MongoDB

  1. 1. Scaling to 30,000 Requests Per Second and Beyond with MongoDB Mike Chesnut Director of Operations Engineering Crittercism
  2. 2. Scaling to 30,000 Requests Per Second and Beyond with MongoDB Mike Chesnut Director of Operations Engineering Crittercism
  3. 3. How a Startup Gets Started
  4. 4. ● Pick something and go with it How a Startup Gets Started
  5. 5. ● Pick something and go with it ● Make mistakes along the way How a Startup Gets Started
  6. 6. ● Pick something and go with it ● Make mistakes along the way ● Correct the mistakes you can How a Startup Gets Started
  7. 7. ● Pick something and go with it ● Make mistakes along the way ● Correct the mistakes you can ● Work around the ones you can’t How a Startup Gets Started
  8. 8. What I’ll Talk About
  9. 9. What I’ll Talk About ● Crittercism - Background and Architecture
  10. 10. What I’ll Talk About ● Crittercism - Background and Architecture ● Router (mongos) Architecture
  11. 11. What I’ll Talk About ● Crittercism - Background and Architecture ● Router (mongos) Architecture ● Sharding Considerations
  12. 12. What I’ll Talk About ● Crittercism - Background and Architecture ● Router (mongos) Architecture ● Sharding Considerations ● The Balancing Act
  13. 13. What I’ll Talk About ● Crittercism - Background and Architecture ● Router (mongos) Architecture ● Sharding Considerations ● The Balancing Act ● Q&A
  14. 14. Critter-What?
  15. 15. Critter-What? A Brief History...
  16. 16. Critter-What? Our Founders (Rob, Andrew, Jeeyun)
  17. 17. Critter-What? Our Founders (Rob, Andrew, Jeeyun) Let’s make a mobile app! It’ll be awesome!
  18. 18. Critter-What? (Unnamed Dating App)
  19. 19. Critter-What?
  20. 20. Critter-What?
  21. 21. Critter-What? Our Founders (Rob, Andrew, Jeeyun) Our app isn’t so awesome after all...
  22. 22. Critter-What?
  23. 23. Critter-What?
  24. 24. Critter-What?
  25. 25. Critter-What?
  26. 26. Critter-What?
  27. 27. Critter-What?
  28. 28. Critter-What?
  29. 29. Critter-What?
  30. 30. Architecture
  31. 31. Architecture
  32. 32. Architecture API
  33. 33. Architecture APIFeedback
  34. 34. Architecture APIFeedback Crashes
  35. 35. Architecture APIFeedback App Loads Crashes
  36. 36. Architecture APIFeedback App Loads Crashes Handled Exceptions
  37. 37. Architecture APIFeedback App Loads Crashes Handled Exceptions
  38. 38. Architecture API App Loads Crashes Handled Exceptions
  39. 39. Architecture API App Loads Crashes Handled Exceptions
  40. 40. Architecture API Crashes Handled Exceptions App Loads batch
  41. 41. Architecture API Crashes Handled Exceptions Metadata App Loads batch
  42. 42. Architecture DynamoDB API Crashes Handled Exceptions Metadata App Loads batch
  43. 43. Architecture DynamoDB API Crashes Handled Exceptions Metadata App Loads batch
  44. 44. Architecture DynamoDB API API Crashes Handled Exceptions Metadata Performance Data Geo Data App Loads batch
  45. 45. Architecture DynamoDB API API Crashes Handled Exceptions Metadata Performance Data Geo Data 40,000 req/s App Loads batch
  46. 46. Growth
  47. 47. Router Architecture
  48. 48. Router Architecture mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set MongoDB Cluster
  49. 49. Router Architecture mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set client process application server client process application server Client Application(s) MongoDB Cluster
  50. 50. Router Architecture mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongos client process application server mongos client process application server Client Application(s) MongoDB Cluster
  51. 51. Router Architecture mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongos client process application server mongos client process application server Client Application(s) MongoDB Cluster
  52. 52. Router Architecture mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongos client process application server mongos client process application server Client Application(s) MongoDB Cluster mongod server mongod server config server config servers
  53. 53. Router Architecture mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongos client process application server mongos client process application server Client Application(s) MongoDB Cluster mongod server mongod server config server config servers
  54. 54. Router Architecture mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongos client process application server mongos client process application server Client Application(s) MongoDB Cluster mongod server mongod server config server config servers
  55. 55. Router Architecture RS RS RS conf ms app ms app
  56. 56. Router Architecture RS RS RS conf ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms appms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app
  57. 57. Router Architecture RS RS RS conf ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms appms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app . . .
  58. 58. Single mongos per client problems we encountered: Router Architecture
  59. 59. Router Architecture Single mongos per client problems we encountered: ● thousands of connections to config servers
  60. 60. Router Architecture Single mongos per client problems we encountered: ● thousands of connections to config servers ● config server CPU load
  61. 61. Router Architecture Single mongos per client problems we encountered: ● thousands of connections to config servers ● config server CPU load ● configdb propagation delays
  62. 62. Router Architecture RS RS RS conf ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms appms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app . . . We went from this...
  63. 63. Router Architecture RS RS RS conf ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms appms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app . . . . . . To this.
  64. 64. Router Architecture mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongos client process application server mongos client process application server Client Application(s) MongoDB Cluster
  65. 65. Router Architecture mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongos client process application server mongos client process application server Client Application(s) MongoDB ClusterRouter Tier
  66. 66. Router Architecture Separate mongos tier advantages:
  67. 67. Router Architecture Separate mongos tier advantages: ● greatly reduced number of connections to each mongod
  68. 68. Router Architecture Separate mongos tier advantages: ● greatly reduced number of connections to each mongod ● far fewer hosts talking to the config servers
  69. 69. Router Architecture Separate mongos tier advantages: ● greatly reduced number of connections to each mongod ● far fewer hosts talking to the config servers ● much faster configdb propagation
  70. 70. Router Architecture Separate mongos tier advantages: ● greatly reduced number of connections to each mongod ● far fewer hosts talking to the config servers ● much faster configdb propagation Disadvantages:
  71. 71. Router Architecture Separate mongos tier advantages: ● greatly reduced number of connections to each mongod ● far fewer hosts talking to the config servers ● much faster configdb propagation Disadvantages: ● additional network hop
  72. 72. Router Architecture Separate mongos tier advantages: ● greatly reduced number of connections to each mongod ● far fewer hosts talking to the config servers ● much faster configdb propagation Disadvantages: ● additional network hop ● host failure has a larger effect
  73. 73. Router Architecture RS RS RS conf ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms appms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app . . . mongos-per-host failure:
  74. 74. Router Architecture RS RS RS conf ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms appms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app . . . mongos-per-host failure:
  75. 75. Router Architecture RS RS RS conf ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms appms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app . . . mongos-per-host failure:
  76. 76. Router Architecture RS RS RS conf ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms appms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app . . . . . . Separate mongos tier failure:
  77. 77. Router Architecture RS RS RS conf ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms appms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app . . . . . . Separate mongos tier failure:
  78. 78. Router Architecture RS RS RS conf ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms appms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app . . . . . . Separate mongos tier failure:
  79. 79. Router Architecture RS RS RS conf ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms appms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app . . . . . . So increase the number of mongos routers:
  80. 80. Router Architecture RS RS RS conf ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms appms app ms app ms app ms app ms app ms app ms app ms app ms app ms app ms app . . . . . . ms ms So increase the number of mongos routers:
  81. 81. Router Architecture - Evolve!
  82. 82. Router Architecture - Evolve! mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongos client process application server mongos client process application server Client Application(s) MongoDB ClusterMaybe at first, doing the mongos-per-host architecture is fine.
  83. 83. Maybe at first, doing the mongos-per-host architecture is fine. And it will probably remain fine for quite a while. Router Architecture - Evolve! mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongos client process application server mongos client process application server Client Application(s) MongoDB Cluster
  84. 84. Router Architecture - Evolve! mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongod server mongod server mongod server replica set mongos client process application server mongos client process application server Client Application(s) MongoDB ClusterRouter TierThis is an area where you can and should be willing to adapt as you go (and as needed).
  85. 85. Sharding Considerations
  86. 86. Pick something you want to live with. Sharding Considerations
  87. 87. Sharding Considerations
  88. 88. Sharding Considerations
  89. 89. Sharding Considerations
  90. 90. Sharding Considerations
  91. 91. Sharding Considerations
  92. 92. Sharding Considerations
  93. 93. Sharding Considerations
  94. 94. Sharding Considerations
  95. 95. Sharding Considerations
  96. 96. Sharding Considerations
  97. 97. Sharding Considerations
  98. 98. Sharding Considerations
  99. 99. Sharding Considerations
  100. 100. Sharding Considerations
  101. 101. Sharding Considerations
  102. 102. Sharding Considerations
  103. 103. Sharding Considerations
  104. 104. Sharding Considerations
  105. 105. Sharding Considerations
  106. 106. Sharding Considerations
  107. 107. Sharding Considerations
  108. 108. Sharding Considerations
  109. 109. Sharding Considerations
  110. 110. Sharding Considerations
  111. 111. Sharding Considerations
  112. 112. Sharding Considerations
  113. 113. Sharding Considerations
  114. 114. Sharding Considerations
  115. 115. Sharding Considerations
  116. 116. Sharding Considerations
  117. 117. Sharding Considerations
  118. 118. What could we have done differently? Sharding Considerations
  119. 119. Sharding Considerations
  120. 120. Sharding Considerations
  121. 121. Sharding Considerations
  122. 122. Sharding Considerations
  123. 123. Sharding Considerations
  124. 124. Sharding Considerations
  125. 125. Sharding Considerations
  126. 126. Sharding Considerations
  127. 127. Sharding Considerations
  128. 128. Sharding Considerations
  129. 129. Sharding Considerations
  130. 130. Sharding Considerations
  131. 131. Sharding Considerations
  132. 132. Sharding Considerations
  133. 133. Sharding Considerations
  134. 134. Sharding Considerations
  135. 135. Sharding Considerations
  136. 136. Sharding Considerations
  137. 137. Sharding Considerations
  138. 138. Sharding Considerations
  139. 139. The Balancing Act
  140. 140. The Balancing Act Why wouldn’t you run the balancer in the first place?
  141. 141. The Balancing Act Why wouldn’t you run the balancer in the first place? ● great question
  142. 142. The Balancing Act Why wouldn’t you run the balancer in the first place? ● great question ● for us, it’s because we deleted some old data at one point, and left a bunch of holes
  143. 143. The Balancing Act Why wouldn’t you run the balancer in the first place? ● great question ● for us, it’s because we deleted some old data at one point, and left a bunch of holes ○ we turned it off while deleting this data
  144. 144. The Balancing Act Why wouldn’t you run the balancer in the first place? ● great question ● for us, it’s because we deleted some old data at one point, and left a bunch of holes ○ we turned it off while deleting this data ○ and then were unable to turn it back on
  145. 145. The Balancing Act Why wouldn’t you run the balancer in the first place? ● great question ● for us, it’s because we deleted some old data at one point, and left a bunch of holes ○ we turned it off while deleting this data ○ and then were unable to turn it back on ● but maybe you start without it
  146. 146. The Balancing Act Why wouldn’t you run the balancer in the first place? ● great question ● for us, it’s because we deleted some old data at one point, and left a bunch of holes ○ we turned it off while deleting this data ○ and then were unable to turn it back on ● but maybe you start without it ● or maybe you need to turn it off for maintenance and forget to turn it back on
  147. 147. The Balancing Act Why wouldn’t you run the balancer in the first place? ● great question ● for us, it’s because we deleted some old data at one point, and left a bunch of holes ○ we turned it off while deleting this data ○ and then were unable to turn it back on ● but maybe you start without it ● or maybe you need to turn it off for maintenance and forget to turn it back on Obviously, don’t do this. But if you do, here’s what happens...
  148. 148. The Balancing Act Fresh, new, empty cluster… But no balancer running.
  149. 149. The Balancing Act
  150. 150. The Balancing Act
  151. 151. The Balancing Act
  152. 152. The Balancing Act
  153. 153. The Balancing Act
  154. 154. The Balancing Act
  155. 155. The Balancing Act
  156. 156. The Balancing Act
  157. 157. The Balancing Act
  158. 158. The Balancing Act
  159. 159. The Balancing Act
  160. 160. The Balancing Act
  161. 161. The Balancing Act
  162. 162. The Balancing Act
  163. 163. The Balancing Act
  164. 164. The Balancing Act
  165. 165. The Balancing Act
  166. 166. The Balancing Act
  167. 167. The Balancing Act
  168. 168. The Balancing Act Now we’re pretty full, so let’s add another shard...
  169. 169. The Balancing Act
  170. 170. The Balancing Act And keep inserting...
  171. 171. The Balancing Act
  172. 172. The Balancing Act
  173. 173. The Balancing Act
  174. 174. The Balancing Act
  175. 175. The Balancing Act Suddenly we find ourselves with a very unbalanced cluster.
  176. 176. The Balancing Act But if we enable the balancer, it will DoS the 5th shard!
  177. 177. The Balancing Act The approximate effect looks something like this:
  178. 178. The Balancing Act The approximate effect looks something like this:
  179. 179. The Balancing Act The approximate effect looks something like this:
  180. 180. The Balancing Act The approximate effect looks something like this:
  181. 181. The Balancing Act The approximate effect looks something like this:
  182. 182. The Balancing Act The approximate effect looks something like this:
  183. 183. The Balancing Act The approximate effect looks something like this:
  184. 184. So what can we do? The Balancing Act
  185. 185. So what can we do? 1. add IOPS The Balancing Act
  186. 186. So what can we do? 1. add IOPS 2. make sure your config servers have plenty of CPU (and IOPS) The Balancing Act
  187. 187. So what can we do? 1. add IOPS 2. make sure your config servers have plenty of CPU (and IOPS) 3. slowly move chunks manually The Balancing Act
  188. 188. So what can we do? 1. add IOPS 2. make sure your config servers have plenty of CPU (and IOPS) 3. slowly move chunks manually 4. approach a balanced state The Balancing Act
  189. 189. So what can we do? 1. add IOPS 2. make sure your config servers have plenty of CPU (and IOPS) 3. slowly move chunks manually 4. approach a balanced state 5. hold your breath The Balancing Act
  190. 190. So what can we do? 1. add IOPS 2. make sure your config servers have plenty of CPU (and IOPS) 3. slowly move chunks manually 4. approach a balanced state 5. hold your breath 6. try re-enabling the balancer The Balancing Act
  191. 191. How to manually balance: 1. determine a chunk on a hot shard 2. monitor effects on both the source and target shards 3. move the chunk 4. allow the system to settle 5. repeat The Balancing Act
  192. 192. Conclusion here: Run the balancer! The Balancing Act
  193. 193. ● Design ahead of time o “NoSQL” lets you play it by ear o but some of these decisions will bite you later ● Be willing to correct past mistakes o dedicate time and resources to adapting o learn how to live with the mistakes you can’t correct Summary
  194. 194. References ● MongoDB Blog post (details on shard migration):http://blog.mongodb.org/post/77278906988/crittercism-scaling-to-billions-of- requests-per-day-on ● MongoDB Webinar (details on manual chunk migrations):http://www.mongodb.com/presentations/webinar-back-basics-3-scaling-30000-requests- second-mongodb ● Documentation on mongos routers:http://docs.mongodb.org/master/core/sharded- cluster-query-routing/ ● Documentation on the balancer:http://docs.mongodb.org/manual/tutorial/manage- sharded-cluster-balancer/ ● Documentation on shard keys:http://docs.mongodb.org/manual/core/sharding-shard- key/ Crittercism: http://www.crittercism.com/ to learn more, and http://www.crittercism.com/careers/ if you want to help us!
  195. 195. Q&A
  196. 196. Thank You!

×