Learn with this webinar how to plan your MongoDB scaling strategy up front and how to prevent ending up with unusable secondary nodes and shards. Finally, we’ve showed you how to leverage ClusterControl’s MongoDB scaling capabilities and have ClusterControl manage your shards. AGENDA What are the differences in read and write scaling with MongoDB Read scaling considerations with MongoDB MongoDB read preference explained How sharding works in MongoDB Adding new shards and balance data How to scale and shard MongoDB using ClusterControl Live Demo SPEAKER Art van Scheppingen is a Senior Support Engineer at Severalnines. He’s a pragmatic MySQL and Database expert with over 15 years experience in web development. He previously worked at Spil Games as Head of Database Engineering, where he kept a broad vision upon the whole database environment: from MySQL to Couchbase, Vertica to Hadoop and from Sphinx Search to SOLR. He regularly presents his work and projects at various conferences (Percona Live, FOSDEM) and related meetups.

1. Your host & some logistics
I'm Jean-Jérôme from the Severalnines Team and I'm
your host for today's webinar!
Feel free to ask any questions in the Questions section of
this application or via the Chat box.
You can also contact me directly via the chat box or via
email: jj@severalnines.com during or after the webinar.

2. About Severalnines and ClusterControl

3. What we do
Manage Scale
MonitorDeploy

4. ClusterControl Automation & Management
☐ Provisioning
☐ Deploy a cluster in minutes
☐ On-premises or in the cloud
(AWS)
☐ Monitoring
☐ Systems view
☐ 1sec resolution
☐ DB / OS stats & performance
advisors
☐ Configurable dashboards
☐ Query Analyzer
☐ Real-time / historical
☐ Management
☐ Multi cluster/data-center
☐ Automate repair/recovery
☐ Database upgrades
☐ Backups
☐ Configuration management
☐ Cloning
☐ One-click scaling

5. Supported Databases

6. Customers

7. Become a MongoDB DBA
Scaling and Sharding
Art van Scheppingen, Senior Support Engineer

8. Logistics
☐ Webinar is recorded
☐ Replay available soon
☐ Feel free to ask questions at any time
☐ Use your control panel to contact us
☐ Or email us as well: info@severalnines.com

9. Agenda
☐ When scalability is needed
☐ Scaling out reads with MongoDB
☐ Sharding with MongoDB
☐ Maintaining shards with MongoDB
☐ ClusterControl and MongoDB scaling
☐ NinesControl and MongoDB
☐ Live Demo

10. When scalability is needed
MongoDB scaling and sharding

11. Scaling MongoDB
☐ Why do we need to scale?
☐ Workload too large for the current system
☐ Peak loads are saturating the system
☐ Used disk space is nearing the capacity
☐ How do we scale?
☐ Vertical
☐ Horizontal

12. Vertical scaling
☐ Scale by adding more power
☐ Adding faster CPU
☐ More memory
☐ More/faster storage disks
☐ You may outpace Moore’s law
☐ Larger growth than new developments
☐ Costs may be very high
☐ Faster CPUs and disks are more expensive

13. Horizontal scaling
☐ Scale by adding more machines
☐ Split workload over multiple nodes
☐ Many hands make light work
☐ Complexity
☐ Horizontal scaling adds complexity
☐ Orchestration systems are needed

14. Horizontal database scaling
☐ Read scaling
☐ Offloading read operations to slave / secondary nodes
☐ Caching layers
☐ Write scaling
☐ Offloading write operations to another master / primary node
☐ This is often referred to as sharding

15. Sharding types
☐ Functional sharding
☐ Place the database / collection on a single node (or replicaSet)
☐ Uneven reads / writes
☐ Can’t join data between databases / collections
☐ Only vertical scaling
☐ Partitioned sharding
☐ Spread the data of a database / collection over multiple nodes
☐ Even reads / writes
☐ Horizontal scaling

16. Scaling out reads with MongoDB

17. Typical MongoDB ReplicaSet

18. Read scaling considerations
☐ By default both read and write requests go to primary
☐ MongoDB replication is asynchronous
☐ Inconsistent data may be returned from a secondary
☐ Overloading secondaries may be dangerous for failover
☐ Sharded environments
☐ Data might have been migrated from one shard to another

19. Confidential
Asynchronous replication
☐ Replication happens, like MySQL replication, asynchronously
☐ Eventual consistency: nodes will eventually contain the data
☐ Not all nodes may be in the same state
☐ Writeconcern to enforce data is secure
☐ Writeconcern
☐ HA: wait for confirmation from secondary nodes
☐ numeric, majority or <tag>
☐ Durability: Wait for write to journal

20. Confidential
Durability

21. Confidential
Durability

22. Confidential
Durability

23. Confidential
Eventual consistency

24. Confidential
Overloading secondaries
☐ Secondaries are used for HA
☐ Contain a copy of the data
☐ Take part in the (majority) voting process for a new primary
☐ Using more capacity than (number of secondaries - 1)
☐ If one secondary fails, the load is distributed over remaining
nodes
☐ Another node fails, due to the increased load
☐ No majority of secondaries remain
☐ Primary gets demoted to a secondary
☐ No more writes are allowed

25. Confidential
Read scaling considerations checklist

26. Offloading reads to secondaries

27. Setting the read preference
primary Always read from the primary (default)
primaryPreferred Always read from the primary, read from secondary if the primary is unavailable
secondary Always read from a secondary
secondaryPreferred Always read from a secondary, read from the primary if no secondary is available
nearest Always read from the node with the lowest network latency

28. Read preference primary (default)

29. Read preference primary (default)

30. Read preference primaryPreferred

31. Read preference primaryPreferred

32. Read preference secondary

33. Read preference secondary

34. Read preference secondaryPreferred

35. Read preference secondaryPreferred

36. Read preference nearest

37. Read preference nearest

38. Filtering nodes with tags
☐ Nodes in a replicaSet can be tagged
☐ You can use these tags to filter nodes for read requests
db.getMongo().setReadPref(‘nearest’, [ { "dc": "2" } ] )

39. {
"_id" : "myrs",
"version" : 2,
"members" : [
{
"_id" : 0,
"host" : "host1:27017",
"tags" : {
"dc": "1",
"rack": "e3"
}
}, {
"_id" : 1,
"host" : "host2:27017",
"tags" : {
"dc": "1",
"rack": "b2"
}
}, {
"_id" : 0,
"host" : "host3:27017",
"tags" : {
"dc": "2",
"rack": "q1"
}
}
]
}

40. Filtering nodes with tags

41. Filtering nodes with tags

42. Scaling out limitations
☐ Adding more secondaries
☐ Maximum of 50 nodes in a replicaSet
☐ Maximum of 7 voting nodes in a replicaSet
☐ Arbiter nodes also count in the 50 node limit
☐ No need for intermediate masters
☐ MongoDB secondaries can replicate from secondaries

43. Sharding with MongoDB

44. MongoDB sharding components
☐ Shard router
☐ Small router/proxy connecting to the configserver
☐ Configserver
☐ A special replicaSet containing the shard administration
☐ Shards
☐ ReplicaSet containing a piece of the data
☐ Shards share nothing with other shards

45. Typical MongoDB sharding environment

46. Shard router: transparent sharding

47. Sharding meta data
☐ Sharding Meta data
☐ Data gets partitioned by slicing it into ranges (chunks)
☐ Chunks are defined by the shard key
☐ Chunks are distributed evenly over all shards
☐ MongoDB Balancer will balance chunks over shards
☐ If a chunk is too large, a new chunk is created
☐ Shard router creates chunks on the fly
☐ New chunks assigned to the shard with most space
☐ Sharding meta data is stored in the Configserver

48. Chunks and shards

49. Balancing shards

50. Shard key
☐ Placed on an indexed field or indexed compound field
☐ Shard keys can’t be altered
☐ Only the _id field and shard key can be unique
☐ The shard key is the most important factor in sharding
☐ It determines the data distribution
☐ Influences the effectiveness of the shards

51. Shard key distribution

52. Shard key considerations: writing data
☐ Sequential writes
☐ E.g. sequential identifiers, timestamps
☐ Only one shard at the time will be written to
☐ Use a hash shard key to randomly distribute
☐ Random writes
☐ E.g. Username, UUID, date of birth
☐ Will write over all shards

53. Shard key without hash function

54. Shard key with hash function

55. Shard key considerations: reading data
☐ Range queries covered by the shard key
☐ Shard router will know which shard(s) to query
☐ Very efficient if only one shard needs to be queried
☐ Performance degrades the more shards need to be queried
☐ Range queries not covered by the shard key
☐ Shard router will not know which shard(s) to query
☐ Will require all shards to be queried
☐ Fields not covered by an index will perform a table scan on all
shards

56. Maintaining shards with MongoDB

57. Backups
☐ Create a backup of a replicaSet: backup any node
☐ Create a backup of a sharded cluster:
☐ Backup the Configserver replicaSet (meta data)
☐ Backup each shard (one node per shard, pref. primary)
☐ Backups started at the same time, may not end at the same
time
☐ Percona MongoDB Consistent backup tool
☐ Backup orchestration tool
☐ Will stream the oplog till all shards have been backed up
☐ https://www.percona.com/blog/2016/07/25/mongodb-consistent-backups/

58. Monitoring a MongoDB sharded cluster
☐ Write capacity has increased largely
☐ Look for the next bottleneck
☐ Most obvious one:
☐ Range queries may touch every shard
☐ Connections will increase
☐ Watch out for non-sharded collections
☐ Will use up all available space, if written to extensively

59. Capacity planning
☐ Plan for scaling out
☐ Monitor the number of chunks per shard
☐ Watch the disk space per (node in the) shard
☐ Adding a new shard will start balancing process
☐ Increase in IO
☐ Balancing large clusters may take forever
☐ Watch for new database / collections

60. Scaling out
☐ Adding new shards is easy
☐ Deploy a new (empty) replicaSet
☐ Add to cluster
mongo --eval 'sh.addShard("sh1/node1.domain.com:27018,node2.domain.com:27018,node3.domain.com:27018");'

61. Balancing shards
☐ MongoDB shard balancer
☐ Runs in the background
☐ Disable when making backups
☐ Balancer can also be scheduled:
db.settings.update(
{ _id: "balancer" },
{ $set: { activeWindow : { start : "02:00", stop : "04:00" } } },
{ upsert: true }
)

62. Disabling the balancer on a collection
☐ Sometimes keeps balancing chunks back and forth
☐ Some collections (archive) data needs to remain on its initial
shard
☐ Disable the balancer per collection:
sh.disableBalancing("mydata.hugearchive")

63. Consistency
☐ Don’t read from secondaries
☐ Balancer may be migrating chunks between shards
☐ Inconsistent data between primary and secondary
☐ Don’t connect to shards directly
☐ Data on a shard may actually reside on another shard
☐ Shard router will hide this

64. ClusterControl and MongoDB scaling

65. MongoDB replicaSet
☐ Deploy a MongoDB replicaSet
☐ Scale the replicaSet by adding secondaries
☐ Add arbiters for voting
☐ Make reliable backups

66. MongoDB sharded cluster
☐ Deploy a fully sharded cluster
☐ Deploy shard routers (mongos)
☐ Deploy Configservers
☐ Deploy shards
☐ Scale by adding shards
☐ Convert a replicaSet into a sharded cluster (1.3.3)
☐ Make cluster wide consistent backups
☐ Percona Consistent MongoDB backup

67. NinesControl and MongoDB

68. NinesControl

69. NinesControl deployment

70. NinesControl deployment

71. NinesControl deployment

72. NinesControl deployment

73. NinesControl monitoring

74. NinesControl monitoring

75. ClusterControl: live demo

76. Demo

77. Q & A

78. Thank you!
☐ NinesControl
☐ https://ninescontrol.com/
☐ Severalnines Blog
☐ www.severalnines.com/blog-categories/mongodb
☐ ClusterControl
☐ www.severalnines.com/product/clustercontrol
☐ Contact: info@severalnines.com