Deploying for the Win with MongoDB 3.0
•Download as PPTX, PDF•
2 likes•924 views
"Deploying MongoDB for the Win" presented by Chris Biow, SVP US Public Sector, Basis Technology at MongoDB Evenings DC on October 13, 2015 at UberOffices Tysons.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Deploying for the Win with MongoDB 3.0
Similar to Deploying for the Win with MongoDB 3.0 (20)
More from MongoDB
More from MongoDB (20)
Recently uploaded
Recently uploaded (20)
Deploying for the Win with MongoDB 3.0
- 2. Deploying for the Win with MongoDB 3.0 Chris Biow Basis Technology @chris_biow
- 4. Agenda General Production Considerations Durability, Scalability, Availability Deployment Architectures & Operations Take home code: three servers in two data centers
- 6. 6 Expectations 5-9’s / High Availability with replication • No scheduled downtime • Zero-downtime maintenance Linear scale-out for read and write • Commodity hardware • Cloud – Public – Private – Hybrid
- 7. 7 Infrastructure Priorities 1. Storage. It’s all about the IOPS! RAID 10 or 0. 2. RAM. Working set (only) in cache for web-scale reads. 3. CPU. Web-scale writes with WiredTiger storage engine. 4. Network.
- 8. 8 Optimal Power to Price Ratio Commodity server or virtual instance • Dual-CPU Intel, 128GB+ • Locally mounted block storage – Spinning disk – SSD – Enterprise storage with guaranteed IOPS
- 9. Production in 16 grams
- 11. Standalone
- 12. Replica Sets Replica Set – 2 to 50 copies Self-healing shard Data Center Aware Addresses availability considerations: High Availability Disaster Recovery Maintenance Workload Isolation: operational & analytics
- 13. Write Concern for Durability
- 15. Automatic Sharding Three types: hash-based, range-based, location-aware Increase or decrease capacity as you go Automatic balancing
- 16. Query Routing Multiple query optimization models Each sharding option appropriate for different apps
- 17. Deployment Architectures & Operations
- 18. Development Architecture Laptop Application mongod SSD 127.0.0.1 / wire protocol SATA / MMAPv1 or Wired Tiger Driver
- 19. Single Data Center, 3 Racks Automated failover Tolerates server failures Tolerates rack failures Number of replicas defines failure tolerance
- 20. DMZDMZ Ideal: 3 Full Data Centers App Server Application Driver mongos DC1 Primary Storage DC2 Secondary Storage DC3 Secondary Storage
- 21. Knobs and Levers, Oh My! replset:PRIMARY> rs.conf().members // (output filtered) [ { "_id" : 0, "host" : "mbp-cbiow-4921.local:27017", "priority" : 1, "votes" : 1 }, { "_id" : 1, "host" : "mbp-cbiow-4921.local:27018", "priority" : 1, "votes" : 1 }, { "_id" : 2, "host" : "mbp-cbiow-4921.local:27019", "priority" : 1, "votes" : 1 } ]
- 22. DMZDMZ Ideal: 3 Full Data Centers App Server Application Driver mongos DC1 Down Storage DC2 Primary Storage DC3 Secondary Storage✗
- 23. DMZDMZ Hybrid Cloud App Server Application Driver mongos DC1 Primary Storage DC2 Secondary Storage The Cloud Secondary Storage
- 24. 3 Data Centers (or servers, or racks…) You can have it all • Durable commits (w:majority) • Automatic failover and recovery • Lose any server • Lose any data center
- 25. DMZDMZ 2.1 Data Centers App Server Application Driver mongos DC1 Primary [Anywhere] Arbiter DC2 Secondary
- 26. DMZDMZ 2.1 Data Centers App Server Application Driver mongos DC1 Primary [Anywhere] Arbiter DC2 Primary ✗
- 27. Active/Active Data Center Tolerates server, rack, data center failures, network partitions
- 28. DMZDMZ Only 2 Data Centers App Server Application Driver mongos DC1 Primary [Nowhere] Nothing DC2 Secondary
- 29. DMZDMZ Only 2 Data Centers App Server Application Driver mongos DC1 Primary DC2 Secondary
- 30. 2 Data Centers (or 2 servers, 2 racks…) Can’t have it all with two data centers • Durable commits (w:majority) • Automatic failover and recovery • Lose any server …OK so far… • Lose either data center
- 31. 2 Data Centers (or 2 servers, 2 racks…) Can’t have it all with two data centers • Durable commits (w:majority) • Automatic failover and recovery • Lose any server • Lose either data center
- 32. DMZDMZ Only 2 Data Centers App Server Application Driver mongos DC1 Primary DC2 Secondary
- 33. DMZDMZ Only 2 Data Centers App Server Application Driver mongos DC1 Primary DC2 Secondary Secondary
- 34. DMZDMZ Only 2 Data Centers App Server Application Driver mongos DC1 Primary DC2 Secondary Secondary ✗
- 35. DMZDMZ Only 2 Data Centers App Server Application Driver mongos DC1 Down DC2 Secondary Primary ✗
- 36. DMZDMZ Only 2 Data Centers App Server Application Driver mongos DC1 Primary DC2 Secondary Secondary✗
- 37. 2 Data Centers Mutually exclusive • Durable commits (w:majority) • Automatic failover and recovery • Lose either data center
- 38. 2 Data Centers Mutually exclusive • Durable commits (w:majority) • Automatic failover and recovery • Lose either data center We need an out-of-band actor
- 39. Some Servers Are Less Equal replset:PRIMARY> rs.conf().members // (output filtered) [ { "_id" : 0, "host" : "mbp-cbiow-4921.local:27017", "priority" : 1, "votes" : 1 }, { "_id" : 1, "host" : "mbp-cbiow-4921.local:27018", "priority" : 1, "votes" : 1 }, { "_id" : 2, "host" : "mbp-cbiow-4921.local:27019", "priority" : 0.5, "votes" : 1 } ]
- 40. DMZDMZ Only 2 Data Centers App Server Application Driver mongos DC1 Primary DC2 Secondary priority:0.5 Secondary
- 42. DMZDMZ Only 2 Data Centers App Server Application Driver mongos DC2 Secondary priority:0.5 DC1 Down Primary ✗
- 44. DMZDMZ At least one DC1 Server Must Stay Down! App Server Application Driver mongos DC1 Down DC2 Secondary priority:0.5 Powered Off ✗
- 45. Dictatorship: Remove Voting Levers replset:PRIMARY> rs.conf().members // (output filtered) [ { "_id" : 0, "host" : "mbp-cbiow-4921.local:27017", "priority" : 1, "votes" : 0 }, { "_id" : 1, "host" : "mbp-cbiow-4921.local:27018", "priority" : 1, "votes" : 0 }, { "_id" : 2, "host" : "mbp-cbiow-4921.local:27019", "priority" : 0.5, "votes" : 1 } ]
- 46. DMZDMZ Remove DC1 votes (3.0 required) App Server Application Driver mongos DC1 votes:0 DC2 Secondary votes:0 ✗
- 47. DMZDMZ Majority of one! App Server Application Driver mongos DC1 votes:0 DC2 Primary votes:0 ✗
- 48. DMZDMZ This must not happen: split brain! App Server Application Driver mongos DC1 Primary DC2 Primary Secondary
- 49. DMZDMZ Recovering DC1 App Server Application Driver mongos DC1 Secondary DC2 Primary Powered off
- 50. DMZDMZ Recovering DC1 App Server Application Driver mongos DC1 Primary DC2 Secondary Powered Off
- 51. DMZDMZ Recovery Complete App Server Application Driver mongos DC1 Primary DC2 Secondary Secondary
- 52. One Server, One Vote replset:PRIMARY> rs.conf().members // (output filtered) [ { "_id" : 0, "host" : "mbp-cbiow-4921.local:27017", "priority" : 1, "votes" : 1 }, { "_id" : 1, "host" : "mbp-cbiow-4921.local:27018", "priority" : 1, "votes" : 1 }, { "_id" : 2, "host" : "mbp-cbiow-4921.local:27019", "priority" : 0.5, "votes" : 1 } ]
- 53. 2 Data Centers Maximum capability with two data centers • Durable commits (w:majority) –possible rollback, if DC1 goes down • Automatic failover and recovery –unless DC1 lost • Lose any server • Lose either data center
- 55. Install and Configure two DCs brew install mongodb pip install mtools pymongo [[ -d ~/data/replset ]] && rm -rf ~/data/replset mlaunch init --nodes 3 --replicaset mongo localhost:27017 #DC1 //rs.init() rs.status() //rs.add("cbiow.local:27018") //DC1 //rs.status() //rs.add("cbiow.local:27019") //simulating DC2 //rs.status() //rs.status()
- 56. Reconfigure and Fail Over in DC1 db.mycoll.insert({a:1},{writeConcern: {w: "majority", wtimeout: 5000}}) db.mycoll.find() r = rs.config() r.members[2].priority = 0.5 rs.reconfig(r) pkill -f 27017 mongo localhost:27018 rs.status() db.mycoll.insert({a:2},{writeConcern: {w: "majority", wtimeout: 5000}}) db.mycoll.find() db.mycoll.count()
- 57. DC1 Down and Recover in DC2 pkill –f 27018 mongo localhost:27019 rs.status() db.mycoll.insert({a:3},{writeConcern: {w: "majority", wtimeout: 5000}}) r = rs.config() r.members[0].votes = 0 r.members[1].votes = 0 rs.reconfig(r, { force: true }) rs.status() db.mycoll.insert({a:4},{writeConcern: {w: "majority", wtimeout: 5000}}) db.mycoll.find()
- 58. DC1 Recovery and Restore mlaunch start 27017 mongo localhost:27017 rs.status() db.mycoll.insert({a:5},{writeConcern: {w: "majority", wtimeout: 5000}}) db.mycoll.find() mlaunch start 27018 mongo localhost:27017 rs.status() r = rs.config() r.members[0].votes = 1 r.members[1].votes = 1 rs.reconfig(r)
Editor's Notes
- High Availability – Ensure application availability during many types of failures Disaster Recovery – Address the RTO and RPO goals for business continuity Maintenance – Perform upgrades and other maintenance operations with no application downtime Secondaries can be used for a variety of applications – failover, hot backup, rolling upgrades, data locality and privacy and workload isolation
- High Availability – Ensure application availability during many types of failures Disaster Recovery – Address the RTO and RPO goals for business continuity Maintenance – Perform upgrades and other maintenance operations with no application downtime Secondaries can be used for a variety of applications – failover, hot backup, rolling upgrades, data locality and privacy and workload isolation
- High Availability – Ensure application availability during many types of failures Disaster Recovery – Address the RTO and RPO goals for business continuity Maintenance – Perform upgrades and other maintenance operations with no application downtime Secondaries can be used for a variety of applications – failover, hot backup, rolling upgrades, data locality and privacy and workload isolation
- High Availability – Ensure application availability during many types of failures Disaster Recovery – Address the RTO and RPO goals for business continuity Maintenance – Perform upgrades and other maintenance operations with no application downtime Secondaries can be used for a variety of applications – failover, hot backup, rolling upgrades, data locality and privacy and workload isolation
- MongoDB provides horizontal scale-out for databases using a technique called sharding, which is trans- parent to applications. Sharding distributes data across multiple physical partitions called shards. Sharding allows MongoDB deployments to address the hardware limitations of a single server, such as bottlenecks in RAM or disk I/O, without adding complexity to the application. MongoDB supports three types of sharding: • Range-based Sharding. Documents are partitioned across shards according to the shard key value. Documents with shard key values “close” to one another are likely to be co-located on the same shard. This approach is well suited for applications that need to optimize range- based queries. • Hash-based Sharding. Documents are uniformly distributed according to an MD5 hash of the shard key value. Documents with shard key values “close” to one another are unlikely to be co-located on the same shard. This approach guarantees a uniform distribution of writes across shards, but is less optimal for range-based queries. • Tag-aware Sharding. Documents are partitioned according to a user-specified configuration that associates shard key ranges with shards. Users can optimize the physical location of documents for application requirements such as locating data in specific data centers. MongoDB automatically balances the data in the cluster as the data grows or the size of the cluster increases or decreases.
- Sharding is transparent to applications; whether there is one or one hundred shards, the application code for querying MongoDB is the same. Applications issue queries to a query router that dispatches the query to the appropriate shards. For key-value queries that are based on the shard key, the query router will dispatch the query to the shard that manages the document with the requested key. When using range-based sharding, queries that specify ranges on the shard key are only dispatched to shards that contain documents with values within the range. For queries that don’t use the shard key, the query router will dispatch the query to all shards and aggregate and sort the results as appropriate. Multiple query routers can be used with a MongoDB system, and the appropriate number is determined based on performance and availability requirements of the application.
- MongoDB provides horizontal scale-out for databases using a technique called sharding, which is trans- parent to applications. Sharding distributes data across multiple physical partitions called shards. Sharding allows MongoDB deployments to address the hardware limitations of a single server, such as bottlenecks in RAM or disk I/O, without adding complexity to the application. MongoDB supports three types of sharding: • Range-based Sharding. Documents are partitioned across shards according to the shard key value. Documents with shard key values “close” to one another are likely to be co-located on the same shard. This approach is well suited for applications that need to optimize range- based queries. • Hash-based Sharding. Documents are uniformly distributed according to an MD5 hash of the shard key value. Documents with shard key values “close” to one another are unlikely to be co-located on the same shard. This approach guarantees a uniform distribution of writes across shards, but is less optimal for range-based queries. • Tag-aware Sharding. Documents are partitioned according to a user-specified configuration that associates shard key ranges with shards. Users can optimize the physical location of documents for application requirements such as locating data in specific data centers. MongoDB automatically balances the data in the cluster as the data grows or the size of the cluster increases or decreases.