Aaron Staple's presentation at Mongo Seattle

1. Scaling with MongoDB Aaron Staple aaron@10gen.com Mongo Seattle July 27, 2010

2. MongoDB 1.6 Comes out next week!

3. Differences from Typical RDBMS Memory mapped data All data in memory (if it fits), synced to disk periodically No joins Reads have greater data locality No joins between servers No transactions Improves performance of various operations No transactions between servers

4. Topics Single server read scaling Single server write scaling Scaling reads with a master/slave cluster Scaling reads with replica sets Scaling reads and writes with sharding

5. Denormalize { userid: 100, books: [ { title: ‘James and the Giant Peach’, author: ‘Roald Dahl’ }, { title: ‘Charlotte’s Web’, author: ‘E B White’ }, { title: ‘A Wrinkle in Time’, author: ‘Madeleine L’Engle’ } ] }

6. Use Indices Find by value db.users.find( { userid: 100 } ) Find by range of values db.users.find( { age: { $gte: 20, $lte: 40 } } ) db.users.find( { hobbies: { $in: [ ‘biking’, ‘running’, ‘swimming’ ] } ) Find with a sort spec db.users.find().sort( { signup_ts: -1 } ) db.users.find( { hobbies: ‘snorkeling’ } ).sort( { signup_ts: -1 } ) Index on { hobbies: 1, signup_ts: -1 }

7. Use Indices Writes with a query component db.users.remove( { userid: 100 } ) Other operations count distinct group map/reduce anything with a query spec

8. Use Indices Look for slow operations Mongod log Profiling Examine how your indexes are used db.users.find( { age: 90, hobbies: ‘snowboarding’ } ).explain() { age: 1 } { hobbies: 1 } Index numbers rather than strings

9. Leverage RAM Indexes perform best when they fit in RAM db.users.stats() Index sizes db.serverStatus() Index hit rate in RAM Check paging vmstat

10. Restrict Fields db.users.find( { userid: 100 }, { hobbies: 1 } ) Just returns hobbies No less work for mongo, but less network traffic and less work for the app server to parse result

11. Topics Single server read scaling Single server write scaling Scaling reads with a master/slave cluster Scaling reads with replica sets Scaling reads and writes with sharding

12. Use Modifiers Update in place db.users.update( { userid: 100 }, { $inc: { views: 1 } } ) db.users.update( { userid: 100 }, { $set: { pet: ‘dog’ } } ) performs pretty well too For very complex modifiers, consider cost of performing operation on database versus app server (generally easier to add an app server) Balance against atomicity requirements Even without modifiers, consistency in object size can help

13. Drop Indices Avoid redundant indices { userid: 1 } { userid: -1 } { userid: 1, signup_ts: -1 } db.users.update( { userid: 100 }, { $inc: { views: 1 } } ) don’t index views db.user15555.drop() not db.user15555.remove( {} )

14. Fire and forget Unsafe “asynchronous” writes No confirmation from mongo that write succeeded Reduce latency at app server Writes queued in mongod server’s network buffer

15. Use Capped Collections Fixed size collection When space runs out, new documents replace the oldest documents Simple allocation model means writes are fast No _id index by default db.createCollection( ‘log’, {capped:true, size:30000} );

16. Wordnik Configuration 1000 requests of various types / second 5 billion documents (1.2TB) Single 2x4 core server 32gb ram, FC SAN non virtualized NOTE: Virtualized storage tends to perform poorly, for example if you are on EC2 you should run several EBS volumes striped

17. Topics Single server read scaling Single server write scaling Scaling reads with a master/slave cluster Scaling reads with replica sets Scaling reads and writes with sharding

18. Master/Slave Easy to set up mongod --master mongod --slave --source <host> App server maintains two connections Writes go to master Reads come from slave Slave will generally be a bit behind master Can sync writes to slave(s) using getlasterror ‘w’ parameter

19. Master/Slave MASTER SLAVE 1 SLAVE 2 APP SERVER 1 APP SERVER 2

20. Monotonic Read Consistency MASTER SLAVE 1 SLAVE 2 APP SERVER 1 APP SERVER 2 Sourceforge uses this configuration, with 5 read slaves, to power most content for all projects

21. Master/Slave A master experiences some additional read load per additional read slave A slave experiences the same write load as the master Consider --only option to reduce write load on slave Delayed slave Diagnostics use local; db.printReplicationInfo() use local; db.printSlaveReplicationInfo()

22. Topics Single server read scaling Single server write scaling Scaling reads with a master/slave cluster Scaling reads with replica sets Scaling reads and writes with sharding

23. Replica Sets Cluster of N servers Only one node is ‘primary’ at a time This is equivalent to master The node where writes go Primary is elected by concensus Automatic failover Automatic recovery of failed nodes

24. Replica Sets - Writes A write is only ‘committed’ once it has been replicated to a majority of nodes in the set Before this happens, reads to the set may or may not see the write On failover, data which is not ‘committed’ may be dropped (but not necessarily) If dropped, it will be rolled back from all servers which wrote it For improved durability, use getLastError/w Other criteria – block writes when nodes go down or slaves get too far behind Or, to reduce latency, reduce getLastError/w

25. Replica Sets - Nodes Nodes monitor each other’s heartbeats If primary can’t see a majority of nodes, it relinquishes primary status If a majority of nodes notice there is no primary, they elect a primary using criteria Node priority Node data’s freshness

26. Replica Sets - Nodes Member 1 Member 2 Member 3

27. Replica Sets - Nodes {a:1} Member 1 SECONDARY {a:1} {b:2} Member 2 SECONDARY {a:1} {b:2} {c:3} Member 3 PRIMARY

28. Replica Sets - Nodes {a:1} Member 1 SECONDARY {a:1} {b:2} Member 2 PRIMARY {a:1} {b:2} {c:3} Member 3 DOWN

29. Replica Sets - Nodes {a:1} {b:2} Member 1 SECONDARY {a:1} {b:2} Member 2 PRIMARY {a:1} {b:2} {c:3} Member 3 RECOVERING

30. Replica Sets - Nodes {a:1} {b:2} Member 1 SECONDARY {a:1} {b:2} Member 2 PRIMARY {a:1} {b:2} Member 3 SECONDARY

31. Replica Sets – Node Types Standard – can be primary or secondary Passive – will be secondary but never primary Arbiter – will vote on primary, but won’t replicate data

32. SlaveOk db.getMongo().setSlaveOk(); Syntax varies by driver Writes to master, reads to slave Slave will be picked arbitrarily

33. Topics Single server read scaling Single server write scaling Scaling reads with a master/slave cluster Scaling reads with replica sets Scaling reads and writes with sharding

34. Sharding Architecture

35. Shard A master/slave cluster Or a replica set Manages a well defined range of shard keys

36. Shard Distribute data across machines Reduce data per machine Better able to fit in RAM Distribute write load across shards Distribute read load across shards, and across nodes within shards

37. Shard Key { user_id: 1 } { lastname: 1, firstname: 1 } { tag: 1, timestamp: -1 } { _id: 1 } This is the default

38. Mongos Routes data to/from shards db.users.find( { user_id: 5000 } ) db.users.find( { user_id: { $gt: 4000, $lt: 6000 } } ) db.users.find( { hometown: ‘Seattle’ } ) db.users.find( { hometown: ‘Seattle’ } ).sort( { user_id: 1 } )

39. Secondary Index db.users.find( { hometown: ‘Seattle’ } ).sort( { lastname: 1 } )

40. SlaveOk Works for a replica set acting as a shard the same as for a standard replica set

41. Writes work similarly db.users.save( { user_id: 5000, … } ) Shard key must be supplied db.users.update( { user_id: 5000 }, { $inc: { views: 1 } } ) db.users.remove( { user_id: { $lt: 1000 } } ) db.users.remove( { signup_ts: { $lt: oneYearAgo } }

42. Writes across shards Asynchronous writes (fire and forget) Writes sent to all shards sequentially, executed per shard in parallel Synchronous writes (confirmation) Send writes sequentially, as above Call getLastError on shards sequentially Mongos limits shards which must be touched Data partitioning limits data each node must touch (for example, it may be more likely to fit in RAM)

43. Increasing Shard Key What if I keep inserting data with increasing values for the shard key? All new data will go to last shard initially We have special purpose code to handle this case, but it can still be less performant than a more uniformally distributed key Example: auto generated mongo ObjectId

44. Adding a Shard Monitor your performance If you need more disk bandwidth for writes, add a shard Monitor your RAM usage – vmstat If you are paging too much, add a shard

45. Balancing Mongo automatically adjusts the key ranges per shard to balance data size between shards Other metrics will be possible in future – disk ops, cpu Currently move just one “chunk” at a time Keeps overhead of balancing slow

46. Sharding models Database not sharded Collections within database are sharded Documents within collection are sharded If remove a shard, any unsharded data on it must be migrated manually (for now).

47. Give it a Try! Download from mongodb.org Sharding and replica sets production ready in 1.6, which is scheduled for release next week For now use 1.5 (unstable) to try sharding and replica sets