Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MongoDB World 2018: Active-Active Application Architectures: Become a MongoDB Multi-Data Center Master

1,345 views

Published on

Speaker: Jay Runkel, Master Solutions Architect, MongoDB

Published in: Technology
  • Hi there! I just wanted to share a list of sites that helped me a lot during my studies: .................................................................................................................................... www.EssayWrite.best - Write an essay .................................................................................................................................... www.LitReview.xyz - Summary of books .................................................................................................................................... www.Coursework.best - Online coursework .................................................................................................................................... www.Dissertations.me - proquest dissertations .................................................................................................................................... www.ReMovie.club - Movies reviews .................................................................................................................................... www.WebSlides.vip - Best powerpoint presentations .................................................................................................................................... www.WritePaper.info - Write a research paper .................................................................................................................................... www.EddyHelp.com - Homework help online .................................................................................................................................... www.MyResumeHelp.net - Professional resume writing service .................................................................................................................................. www.HelpWriting.net - Help with writing any papers ......................................................................................................................................... Save so as not to lose
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

MongoDB World 2018: Active-Active Application Architectures: Become a MongoDB Multi-Data Center Master

  1. 1. Active-Active Application Architectures Become a MongoDB Multi-Data Center Master
  2. 2. Jay Runkel Master Solutions Architect, MongoDB jay.runkel@mongodb.com @jayrunkel
  3. 3. Today’s Challenge
  4. 4. Organizations Want Active Active
  5. 5. Replica Sets Appear to be a Mismatch?
  6. 6. Can MongoDB used for Active Active Applications? Hmmm…. What are all the requirements?
  7. 7. Active Active Requirements Performance/Latency – read and write locally. Millisecond performance. Consistency – reads see results of previous writes, readers in various regions get same results, no conflict resolution Availability – operation continues through loss of nodes, networks, data centers, etc. Durability – data replicated among nodes. Data is not lost when failures occur
  8. 8. Requirement 1: Latency Read ReadWrite Write
  9. 9. Active DR solution many not meet latency requirements 10ms 2 ms 45 ms 35 ms 75 ms 105 ms 85 ms
  10. 10. Active Active provides consistent low latency 10ms 2 ms 10 ms 4 ms 2ms 10 ms 4 ms 2ms
  11. 11. Requirement 2: Consistency Write {x:1} Write {x:2} {x:0} {x:0} {x:0} {x:0}
  12. 12. Requirement 2: Consistency Write {x:1} Write {x:2} {x:0} {x:1} {x:0} {x:2} Reads return the results of previous writes? {x:0}
  13. 13. Requirement 2: Consistency Write {x:1} Write {x:2} {x:0} {x:1} {x:0} {x:2} Reads in different regions return the same results? {x:1} {x:2}
  14. 14. Requirement 2: Consistency Write {x:1} Write {x:2} {x:1} {x:1} {x:2} {x:2} No conflict resolution? {x:1} {x:2}
  15. 15. Requirement 2: Consistency Write {x:1} Write {x:2} {x:1} {x:1} {x:1} {x:1} No Data Loss? {x:1} {x:2}
  16. 16. Requirement 3: Availability Resilience from loss of a node
  17. 17. Requirement 3: Availability Resilience from loss of a region
  18. 18. Requirement 3: Availability Resilience from loss of network
  19. 19. Requirement 4: Durability Write {x:1} {x:0} {x:1} {x:0} {x:0} Writes survive loss of nodes, data center, network?
  20. 20. “You can't always get what you want But if you try sometimes you might find You get what you need” - Rolling Stones
  21. 21. Trade Offs Performance 0 ms 1 sec Durability 1 node 2 nodes 3 nodes n nodes Consistency eventual causal linearizable Read Availability primary (with secondary reads)~100% Write Availability primary ~100%retryable strong readConcern: majority
  22. 22. Agenda For Remainder of This Talk? 1. MongoDB Architecture Review a. MongoDB Replica Sets b. Trading Off: • Consistency • Availability • Durability • Performance 2. Three Multi-Data Center Deployment Patterns
  23. 23. MongoDB Architecture
  24. 24. Tuning MongoDB to meet Active Active Requirements Active Active Requirements MongoDB Capability Performance Zone Sharding Availability Secondary Reads Retryable Writes Consistency Read Preference Read Concern Durability Write Concern
  25. 25. Replica Sets Replica Set – 2 to 50 copies Self-healing shard Data Center Aware Addresses availability considerations: High Availability Disaster Recovery Maintenance Workload Isolation: operational & analytics
  26. 26. Sharding: Scaling MongoDB
  27. 27. Tuning Read and Write Availability Read Preference Retryable Writes
  28. 28. Node 1 (Primary) Node 2 (Secondary) Node 3 (Secondary) Application Driver Read Read Preference: Primary Default All reads from Primary
  29. 29. Node 1 Node 2 (Secondary) Node 3 (Secondary) Application Driver Read Preference: Primary {x: 100} Node 1 (Primary) Read Can’t read, if no primary
  30. 30. Node 1 Node 2 (Secondary) Node 3 (Secondary) Application Driver Read Preference: PrimaryPreferred {x: 100} Node 1 (Primary) Read Read from secondary, if no primary 100% read availability SecondaryPreferred also Read Read
  31. 31. Node 1 Node 2 (Secondary) Node 3 (Secondary) Application Driver Write Availability → Retryable Writes {x: 100} Node 1 (Primary) Write Can’t write, if no primary Retryable writes - retries failed writes
  32. 32. Tuning Consistency Primary vs. Secondary Reads Read Concern
  33. 33. Node 1 (Primary) Node 2 (Secondary) Node 3 (Secondary) Application Driver Read Primary Reads → Strong Consistency Reads return all previous writes
  34. 34. Node 1 (Primary) Node 2 (Secondary) Node 3 (Secondary) Application Driver Secondary Reads → Eventually Consistent Read Read Reads may reflect a past state of the data
  35. 35. Challenge #1 - Reading Rolled Back Data Node 1 Node 2 Node 3 Application Driver {x: 101} {x: 100} {x : 100}{x: 100} write {x: 101} Application Driver {x: 101} Node 1 (Primary)
  36. 36. Challenge #1 - Reading Rolled Back Data Node 2 Node 3 Application Driver {x: 101} {x : 100}{x: 100} Application Driver {x: 101} {x: 100} {x: 100}
  37. 37. Node 1 Node 2 Node 3 Application Driver {x: 101} {x: 101} {x : 100}{x: 100} write Application Driver {x: 100} ReadConcern: Majority
  38. 38. Challenge #2 - Secondary Reads Don’t Reflect My Writes Node 1 Node 2 Node 3 Application Driver {x: 101} {x: 100} {x : 100}{x: 100} write {x: 101} {x : 100} read
  39. 39. Causal Consistency Node 1 Node 2 Node 3 Application Driver {x: 101} {x: 100} {x : 100}{x: 100} write {x: 101} read {x: 101}
  40. 40. Tuning Durability Write Concern
  41. 41. Node 1 Node 2 Node 3 Application Driver Replica Set Write Durability {x: 100} {x: 100} {x : 100}{x: 100} writeack
  42. 42. Node 1 Node 2 Node 3 Application Driver Replica Set Write Durability {x: 100} {x: 100} write Node 1 (Primary)
  43. 43. Node 1 Node 2 Node 3 Application Driver Replica Set Write Durability {x: 100} {x: 100} write Node 1 (Primary)
  44. 44. Node 1 (Primary) Node 2 (Primary) Node 3 (Secondary) Application Driver Write Durability {x : 100} Node 1 (Primary) {x : 100}
  45. 45. Write Durability is Configured via Write Concern • Intelligent write receipt/confirmation – Specifieds the the number of nodes that must have written the write to disk – Default number is 1 • NOT a distributed transaction db.test.insert({x:100},{writeConcern:{w:2}})
  46. 46. Node 1 Node 2 Node 3 Application Driver Write Concern Majority {x: 100} {x: 100} {x : 100}{x: 100} writeack db.test.insert({x:100}, {writeConcern:{w:”majority”}})
  47. 47. MongoDB Retryable Writes Write failure handling moved from the app to the database for transient network errors or primary elections – Driver automatically retries failed write – With a unique transaction identifier, server enforces exactly-once processing semantics • Properties – Supports idempotent & non-idempotent operations, and errors caused by time-outs • Delivers always-on, global availability of write operations – Overcomes the complexity imposed by multi-master, eventually consistent systems
  48. 48. Zone Sharding
  49. 49. Configuring MongoDB Performance 0 ms 1 sec Durability 1 node 2 nodes 3 nodes n nodes Consistency eventual causal linearizable Read Availability primary (with secondary reads)~100% Write Availability primary ~100%retryable strong readConcern: majority
  50. 50. Performance Optimized Performance 0 ms 1 sec Durability 1 node 2 nodes 3 nodes n nodes Consistency eventual causal linearizable Read Availability primary (with secondary reads)~100% Write Availability primary ~100%retryable strong readConcern: majority
  51. 51. Durability and Consistency Optimized Performance 0 ms 1 sec Durability 1 node 2 nodes 3 nodes n nodes Consistency eventual causal linearizable Read Availability primary (with secondary reads)~100% Write Availability primary ~100%retryable strong readConcern: majority
  52. 52. MongoDB Multi-Data Center Deployments
  53. 53. Writes: Multi-Data Center 3 Alternatives: Two-phase commit across data centers • Consistent • Slow Multi master • Each data item can be written to any data center • Inconsistent - Conflict resolution and data loss • Fast Sharded Database • Multiple masters; each document/partition has a single master • Multiple masters can be deployed in various data centers • Consistent • Fast
  54. 54. Writes: Multi-Data Center 3 Alternatives: Two-phase commit across data centers • Consistent • Slow Multi master • Each data item can be written to any data center • Inconsistent - Conflict resolution and data loss • Fast Sharded Database • Multiple masters; each document/partition has a single master • Multiple masters can be deployed in various data centers • Consistent • Fast
  55. 55. MongoDB Multi Data Center Architecture Options Active DR Partitioned (Sharded) Database “Multi-master”
  56. 56. Option 1: Active DR
  57. 57. Active DR
  58. 58. Multi Region
  59. 59. Multi Region Tune ReadPreference, ReadConcern, WriteConcern, Causal Consistency
  60. 60. Option 2: Sharded Database
  61. 61. Routing Requests To Nearest Region Request Router
  62. 62. Sharded Database (zone sharding) Writes •Each data center owns a partition of the data •External routing process routes requests to the data center owning the request’s data •Application writes to local primary •Occasional cross data writes may occur if routing process isn’t perfect Reads •Read from local primary for consistency •Global data access using “nearest" read preference. Eventually consistent.
  63. 63. Zone Sharding Request Router
  64. 64. Zone Sharding Request Router
  65. 65. Zone Sharding Request Router
  66. 66. Zone Sharding Request Router
  67. 67. Location-Aware Data Distribution Shard Key: {regionCode} {userId} Zone Start End West West, MinKey West, MaxKey East East, MinKey East, MaxKey ∞… East Zone … … ∞ ∞… … ∞ West Zone { _id : ObjectID(“abc…”), regionCode: “West”, userId: 12345, … }
  68. 68. Read Global/Write Local
  69. 69. Option 3: Multi-Master Database
  70. 70. How to implement Multi-Master 1. Insert only • Updates → Inserts • Aggregate on read 2. Always write to local primary 3. All reads are scatter gather
  71. 71. Updates db.carts.find({shopCartId: 1234}) db.carts.update( {shopCartId: 1234}, {$push: {items: “socks12”}}) { shopCartId: 1234, items: [“shoe32”, “coat43”] } ShoppingCart Collection
  72. 72. Insert only db.carts.aggregate( [{$match: {shopCartId: 1234, op: “add”}}, {$group: {_id: “$shopCardId”, items: {$push: “$items”}}}] db.carts.insert( {shopCartId: 1234, op: “add”, items: “socks12”} ) { shopCartId: 1234, op: “add”, items: “shoe32” } { shopCartId: 1234, op: “add”, items: “coat32” } CartOperations Collection
  73. 73. Multi-Master Writes (inserts) Request Router App Server assigned data center field
  74. 74. Mult-master zone sharding configuration Shard Key: {dc} {shopCartId} Zone Start End Virginia Virginia, MinKey Virginia, MaxKey Oregon Oregon, MinKey Oregon, MaxKey ∞… Virginia Zone … … ∞ ∞… … ∞ Oregon Zone { shopCartId: 1234, dc: “Virginia”, op: “add”, items: “shoe32” }
  75. 75. Insert only db.carts.aggregate( [{$match: {shopCartId: 1234, op: “add”}}, {$group: {_id: “$shopCartId”, items: {$push : “$items”}}}] db.carts.insert( {shopCartId: 1234, dc: “Oregon”, op: “add”, items : “socks12”} ) { shopCartId: 1234, dc: “Oregon”, op: “add”, items: “shoe32” } { shopCartId: 1234, dc: “Virginia” op: “add”, items: “coat32” } CartOperations Collection
  76. 76. Multi-Master Reads (scatter gather)
  77. 77. Multi-Data Center Approach Summary • • •
  78. 78. Tune to application requirements Performance 0 ms 1 sec Durability 1 node 2 nodes 3 nodes n nodes Consistency eventual causal linearizable Read Availability primary (with secondary reads)~100% Write Availability primary ~100%retryable strong readConcern: majority

×