Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building a Fault Tolerant Distributed Architecture

431 views

Published on

This talk will highlight some of the challenges to building a fault tolerant distributed architecture, and how MemSQL's architecture tackles these challenges.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Building a Fault Tolerant Distributed Architecture

  1. 1. Building a Fault Tolerant Distributed Architecture Rodrigo Toste Gomes Software Engineer
  2. 2. A MemSQL Cluster 2 - Master Aggregator - Leaves Leaves store shards of data. Master Aggregator is the source of truth for cluster state: - Location of data shards;
  3. 3. Fault Tolerance and High Availability 3 Maintain replicas of shards in different nodes. System can tolerate one node failure. Master Aggregator is also responsible for: - Location of data shard replicas; - Replication states; - Moving shards around.
  4. 4. 4 Cluster State: Node State: - Leaf A is online - Leaf B is online Databases: - data Shards: - data:0 - primary on A - data:0 - replica on B - data:1 - primary on B - data:1 - replica on A Example Cluster Nodes: Master Aggregator Leaf A Leaf B
  5. 5. Maintaining the Cluster State 5 Cluster State: - Leaf A is online Leaf A: Offline MA BA Heartbeats Cluster State: - Leaf A is offline Leaf A: Offline
  6. 6. Maintaining the Cluster State 6 Cluster State: - Leaf A is online - data:0 - primary on A - data:0 - replica on B Leaf A: Offline Leaf B: Online; replica of tweets:0 Cluster State: - Leaf A is offline - data:0 - primary on B Leaf A: Offline Leaf B: Online; replica of data:0
  7. 7. Maintaining the Cluster State 7 Cluster State: - Leaf A is online - data:0 - primary on A - data:0 - replica on B Leaf A: Offline Leaf B: Online; replica of tweets:0 Cluster State: - Leaf A is offline - data:0 - primary on B Leaf A: Offline Leaf B: Online; replica of data:0 Leaf B: Online; primary of data:0
  8. 8. MemSQL Distributed Architecture 8 Master Aggregator: - Maintains primary copy of cluster state; - Monitors node state via heartbeats. Leaves: - Monitor for changes in local and cluster state; - Reconcile local state with cluster state. What happens when a node is unable to reconcile its local state with the cluster state?
  9. 9. Shard Can’t Replicate 9 Cluster State: - data:0 primary on A - data:0 replica on B Leaf A: Primary data:0 Replicating data:0 to B Leaf B: Replica of data:0 MA BA Heartbeats data:0
  10. 10. Shard Can’t Replicate 10 Cluster State: - data:0 primary on A - data:0 replica on B Leaf A: Primary data:0 Leaf B: Replica of data:0 Network Partition A | B MA BA Heartbeats data:0
  11. 11. Strategy 1: Don’t block writes 11 MA BA Heartbeats data:0 Cluster State: - data:0 primary on A - data:0 replica on B Leaf A: Primary data:0 Leaf B: Out of Date Replica of data:0 Network Partition A | B
  12. 12. Strategy 1: Don’t block writes (Leaf A crashes) 12 MA BA Heartbeats data:0 Cluster State: - data:0 primary on A - data:0 replica on B Leaf A: Offline Leaf B: Out of Date Replica of data:0 Network Partition A | B Cluster State: - data:0 primary on B Leaf B: Primary of data:0 Not a MemSQL approach given potential for data loss
  13. 13. Strategy 2: Do nothing - block writes indefinitely 13 MA BA Heartbeats data:0 Cluster State: - data:0 primary on A - data:0 replica on B Leaf A: Primary data:0 Leaf B: Replica of data:0 Network Partition A | B Write workload stalls
  14. 14. Strategy 3: Leaf Notify 14 MA BA Heartbeats data:0 Cluster State: - data:0 primary on A - data:0 synchronous replica on B Leaf A: Primary data:0 Leaf B: Replica of data:0 Network Partition A | B Write workload is stalled
  15. 15. Strategy 3: Leaf Notify 15 MA BA Heartbeats data:0 Cluster State: - data:0 primary on A - data:0 synchronous replica on B Leaf A: Primary data:0 Leaf Notifies MA - no replication Leaf B: Replica of data:0 Network Partition A | B
  16. 16. Strategy 3: Leaf Notify 16 MA BA Heartbeats data:0 Cluster State: - data:0 primary on A - data:0 asynchronous replica on B Leaf A: Primary data:0 Leaf B: Out of Date Replica of data:0 Network Partition A | B
  17. 17. MemSQL Distributed Architecture 17 Master Aggregator: - Maintains primary copy of cluster state; - Monitors node state via heartbeats. Leaves: - Monitor for changes in local and cluster state; - Reconcile local state with cluster state; - Notify MA to change cluster state when reconciling is impossible.
  18. 18. Questions?
  19. 19. Thank you memsql.com

×