Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Corpus collapsum 
Partition tolerance of Galera in a noisy high load 
environment 
Highload++ 2014 
Raghavendra Prabhu 
 ...
The Title?
Our Cluster
Split brain
Seed quotes.. 
“ ’Network is reliable’ - a fallacy of the distributed system. ” - Peter 
Deutsch 
“ A distributed system i...
Seed quotes.. 
“ ’Network is reliable’ - a fallacy of the distributed system. ” - Peter 
Deutsch 
“ A distributed system i...
Seed quotes.. 
“ ’Network is reliable’ - a fallacy of the distributed system. ” - Peter 
Deutsch 
“ A distributed system i...
Seed quotes.. 
“ ’Network is reliable’ - a fallacy of the distributed system. ” - Peter 
Deutsch 
“ A distributed system i...
The fallacies 
▶ The network is reliable 
▶ Latency is zero 
▶ Bandwidth is infinite 
▶ The network is secure 
9 / 92
The fallacies 
▶ Topology doesn’t change 
▶ There is one administrator 
▶ Transport cost is zero 
▶ The network is homogen...
20000 feet view
Actors 
▶ Database - WSREP/PXC 
▶ Plugin - Galera 
▶ Traffic control 
♦ Traffic Control - tc 
♦ NetEm 
12 / 92
Actors 
▶ Database - WSREP/PXC 
▶ Plugin - Galera 
▶ Traffic control 
♦ Traffic Control - tc 
♦ NetEm 
13 / 92
Actors 
▶ Database - WSREP/PXC 
▶ Plugin - Galera 
▶ Traffic control 
♦ Traffic Control - tc 
♦ NetEm 
14 / 92
Actors 
▶ Containers - Docker 
▶ Load 
♦ Generators - Sysbench, RQG 
▶ Network 
♦ Dnsmasq 
♦ nsenter 
15 / 92
Actors 
▶ Containers - Docker 
▶ Load 
♦ Generators - Sysbench, RQG 
▶ Network 
♦ Dnsmasq 
♦ nsenter 
16 / 92
Actors 
▶ Jenkins 
♦ Build flow and CI 
▶ Storage 
♦ Why 
17 / 92
But why 
▶ The ‘P’ in CAP 
▶ WAN scalability 
▶ Real Reason - fun! 
▶ Tolerance to latency variance 
18 / 92
But why 
▶ The ‘P’ in CAP 
▶ WAN scalability 
▶ Real Reason - fun! 
▶ Tolerance to latency variance 
19 / 92
But why 
▶ The ‘P’ in CAP 
▶ WAN scalability 
▶ Real Reason - fun! 
▶ Tolerance to latency variance 
20 / 92
But why 
▶ The ‘P’ in CAP 
▶ WAN scalability 
▶ Real Reason - fun! 
▶ Tolerance to latency variance 
21 / 92
But why 
▶ Failures in warehouses. 
▶ Not quorum, but consensus. 
▶ Real world networks and synchronous replication 
- Del...
Galera
Galera 
▶ Data-centric approach 
▶ EVS 
▶ Causality and Synchronous 
▶ Latency 
24 / 92
Where did it start
Where did it start 
▶ Bug! https://bugs.launchpad.net/galera/+bug/1274192 
▶ Loss of PC 
▶ Crash 
▶ HA goal 
29 / 92
One can bring the whole down
The Flow
Basic Flow 
Jenkins Build image(s?) Start Dnsmasq Bootstrap 
nsenter/netem Pre-sanity SST/Others Load/Sysbench 
32 / 92
Basic Flow 
Jenkins Build image(s?) Start Dnsmasq Bootstrap 
nsenter/netem Pre-sanity SST/Others Load/Sysbench 
33 / 92
Basic Flow 
Jenkins Build image(s?) Start Dnsmasq Bootstrap 
nsenter/netem Pre-sanity SST/Others Load/Sysbench 
34 / 92
Basic Flow 
Jenkins Build image(s?) Start Dnsmasq Bootstrap 
nsenter/netem Pre-sanity SST/Others Load/Sysbench 
35 / 92
Basic Flow 
Jenkins Build image(s?) Start Dnsmasq Bootstrap 
nsenter/netem Pre-sanity SST/Others Load/Sysbench 
36 / 92
Basic Flow 
Jenkins Build image(s?) Start Dnsmasq Bootstrap 
nsenter/netem Pre-sanity SST/Others Load/Sysbench 
37 / 92
Basic Flow 
Jenkins Build image(s?) Start Dnsmasq Bootstrap 
nsenter/netem Pre-sanity SST/Others Load/Sysbench 
38 / 92
Basic Flow 
Jenkins Build image(s?) Start Dnsmasq Bootstrap 
nsenter/netem Pre-sanity SST/Others Load/Sysbench 
39 / 92
Basic Flow 
RR sysbench 
Detach/Keep 
Post sanity Core trace 
Sanity check Reconciliation 
Examine! 
Cleanup Collect logs ...
Basic Flow 
RR sysbench 
Detach/Keep 
Post sanity Core trace 
Sanity check Reconciliation 
Examine! 
Cleanup Collect logs ...
Basic Flow 
RR sysbench 
Detach/Keep 
Post sanity Core trace 
Sanity check Reconciliation 
Examine! 
Cleanup Collect logs ...
Basic Flow 
RR sysbench 
Detach/Keep 
Post sanity Core trace 
Sanity check Reconciliation 
Examine! 
Cleanup Collect logs ...
Basic Flow 
RR sysbench 
Detach/Keep 
Post sanity Core trace 
Sanity check Reconciliation 
Examine! 
Cleanup Collect logs ...
Basic Flow 
RR sysbench 
Detach/Keep 
Post sanity Core trace 
Sanity check Reconciliation 
Examine! 
Cleanup Collect logs ...
Basic Flow 
RR sysbench 
Detach/Keep 
Post sanity Core trace 
Sanity check Reconciliation 
Examine! 
Cleanup Collect logs ...
Basic Flow 
RR sysbench 
Detach/Keep 
Post sanity Core trace 
Sanity check Reconciliation 
Examine! 
Cleanup Collect logs ...
Cluster Resilience 
48 / 92
Parameters 
▶ Sysbench 
▶ Segments 
▶ Reconciliation period 
▶ Number of nodes with loss 
49 / 92
Parameters 
▶ Sysbench 
▶ Segments 
▶ Reconciliation period 
▶ Number of nodes with loss 
50 / 92
Parameters 
▶ Sysbench 
▶ Segments 
▶ Reconciliation period 
▶ Number of nodes with loss 
51 / 92
Parameters 
▶ Sysbench 
▶ Segments 
▶ Reconciliation period 
▶ Number of nodes with loss 
52 / 92
Parameters 
▶ NetEm 
▶ Detach qdisc or keep it? 
▶ Fsync 
▶ Shutdown 
53 / 92
Parameters 
▶ NetEm 
▶ Detach qdisc or keep it? 
▶ Fsync 
▶ Shutdown 
54 / 92
Parameters 
▶ NetEm 
▶ Detach qdisc or keep it? 
▶ Fsync 
▶ Shutdown 
55 / 92
Parameters 
▶ NetEm 
▶ Detach qdisc or keep it? 
▶ Fsync 
▶ Shutdown 
56 / 92
Parameters
Containers!
Docker 
▶ Why not virtualize 
♦ Occam 
♦ Namespaces 
▶ Simplicity 
♦ Network 
♦ One application per node 
59 / 92
Docker 
▶ Portability 
- See same qualitative behavior that I do. 
▶ Reproducibility 
- Makes it determinstic 
▶ Configura...
Docker 
▶ QEMU and Docker 
▶ Scalability 
♦ Performance 
♦ Feature 
▶ Abstraction of channels 
61 / 92
Container Networking 
▶ Linking didn’t help 
▶ Dnsmasq to rescue! 
♦ Hosts file and volumes 
♦ SIGHUP and refresh 
▶ More ...
Noise 
▶ Initial setup 
- Bridge 
- Egress only 
- IFB 
▶ Present state 
▶ NetEm 
- tc qdisc buckets 
- packet loss, delay...
Testing methods
Method I 
▶ Qdisc is detached after load 
▶ Objective 
- Time to recover of full cluster 
▶ Done with a larger subset 
65 ...
Method II 
▶ Qdisc is kept till the end 
▶ Objective 
- Formation of primary component 
▶ Comparatively smaller subset 
66...
Observations 
▶ Post sanity types 
- Why 
- Whom 
▶ Which method is more pertinent 
▶ State transfer issues 
- Beginning 
...
Observations 
▶ Direct load to affected nodes 
▶ Logs 
- journalctl 
- Streaming? 
68 / 92
Other noises 
▶ Aim 
▶ Fsync 
- libeatmydata 
- Variance 
▶ Correlation with network 
▶ How with Docker 
- LD_PRELOAD 
69 ...
System Load
Load generation 
▶ Sysbench 
- Generation 
- Reconnect on partition 
▶ Sockets chosen 
- Load on affected nodes 
▶ Distrib...
Load generation 
▶ Nature of data/load 
- DDL 
▶ RQG in future 
- Fuzz testing 
72 / 92
The Fix
Strike Out!
Eviction 
▶ STONITH 
▶ Permanent eviction 
▶ ’N’ strikes & out! 
- Timers - evs parameters 
- wsrep_evs_delayed and wsrep_...
Eviction 
▶ Aim 
▶ Quorum required 
- Keep majority of the group live or to avoid eviction. 
- Why? - Not shoot each other...
Eviction 
▶ Aim 
▶ Quorum required 
- Keep majority of the group live or to avoid eviction. 
- Why? - Not shoot each other...
Eviction 
▶ EVS version and upgrade 
▶ TODO! 
- Ingress only 
- Follow here. 
▶ Credits to Teemu Ollakka, Yan Zhang and Al...
Coredumps with Docker 
▶ Breakdown of abstraction 
▶ Lack of isolation 
▶ What was done 
- Volumes 
- core_pattern & sysct...
WAN Segments 
▶ How they work 
▶ Random allocation 
▶ Joiner starvation 
▶ Simulates data center 
▶ Donor selection 
80 / ...
The code 
▶ Github: https://github.com/percona/pxc-docker 
▶ Jenkins: http://jenkins.percona.com/job/PXC-5.6-netem/ 
▶ Con...
Code: todo 
▶ Docker automated builds 
▶ Orchestration 
- Kubernetes, Mesos et.al. 
▶ Docker 
♦ Injection 
♦ Signal proxyi...
Code: todo 
▶ Poll and exit during reconciliation 
- Instrument v/s number of nodes 
▶ Use actual channels 
▶ Run it bare ...
Future work
Future work 
▶ Fault injection 
♦ Memory 
- Poisoned memory/madvise 
♦ Disk 
- libeatmydata 
- Opposite 
- ENOSPC 
85 / 92
Fault injection 
▶ CPU 
- NUMA? 
- Hotplug 
▶ More network 
- corruption, duplication, reordering, rate-limit 
- Better di...
More Chaos
Future work 
▶ Disturb cluster more! 
- Membership changes 
* Manual eviction 
* Pull the cord! 
- Corrupt nodes 
▶ Consis...
Further Reading 
▶ Jepsen 
▶ Byzantine fault tolerance 
- Reaching agreement in presence of faults 
▶ The Network is Relia...
About 
▶ /me: Raghavendra Prabhu, Product Lead, Percona XtraDB Cluster, Percona. 
▶ Slides will be at slideshare. 
▶ About...
Image Credits 
▶ http://galeracluster.com/documentation-webpages/ 
▶ http://www.thelastdragontribute.com/40th-anniversary-...
Spaseeba & Da sveedaneeya!
Corpus collapsum: Partition tolerance of Galera in a noisy high load environment
Corpus collapsum: Partition tolerance of Galera in a noisy high load environment
Corpus collapsum: Partition tolerance of Galera in a noisy high load environment
Upcoming SlideShare
Loading in …5
×

Corpus collapsum: Partition tolerance of Galera in a noisy high load environment

520 views

Published on

This is the talk given at Highload++ 2014 in Moscow, Russia. The topic was partition tolerance testing of Galera in a noisy high load environment with NetEm and Docker.

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Corpus collapsum: Partition tolerance of Galera in a noisy high load environment

  1. 1. Corpus collapsum Partition tolerance of Galera in a noisy high load environment Highload++ 2014 Raghavendra Prabhu  raghavendra.d.prabhu@gmail.com Percona  raghavendra.prabhu@percona.com  randomsurfer  wnohang.net  rdprabhu  ronin13
  2. 2. The Title?
  3. 3. Our Cluster
  4. 4. Split brain
  5. 5. Seed quotes.. “ ’Network is reliable’ - a fallacy of the distributed system. ” - Peter Deutsch “ A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable. ” - Leslie Lamport “ Never attribute to malice that which is adequately explained by stupidity. ” - Hanlon’s Razor “ Never attribute to Byzantine failure which can be explained by an ill node(s) ” - Me 5 / 92
  6. 6. Seed quotes.. “ ’Network is reliable’ - a fallacy of the distributed system. ” - Peter Deutsch “ A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable. ” - Leslie Lamport “ Never attribute to malice that which is adequately explained by stupidity. ” - Hanlon’s Razor “ Never attribute to Byzantine failure which can be explained by an ill node(s) ” - Me 6 / 92
  7. 7. Seed quotes.. “ ’Network is reliable’ - a fallacy of the distributed system. ” - Peter Deutsch “ A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable. ” - Leslie Lamport “ Never attribute to malice that which is adequately explained by stupidity. ” - Hanlon’s Razor “ Never attribute to Byzantine failure which can be explained by an ill node(s) ” - Me 7 / 92
  8. 8. Seed quotes.. “ ’Network is reliable’ - a fallacy of the distributed system. ” - Peter Deutsch “ A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable. ” - Leslie Lamport “ Never attribute to malice that which is adequately explained by stupidity. ” - Hanlon’s Razor “ Never attribute to Byzantine failure which can be explained by an ill node(s) ” - Me 8 / 92
  9. 9. The fallacies ▶ The network is reliable ▶ Latency is zero ▶ Bandwidth is infinite ▶ The network is secure 9 / 92
  10. 10. The fallacies ▶ Topology doesn’t change ▶ There is one administrator ▶ Transport cost is zero ▶ The network is homogeneous 10 / 92
  11. 11. 20000 feet view
  12. 12. Actors ▶ Database - WSREP/PXC ▶ Plugin - Galera ▶ Traffic control ♦ Traffic Control - tc ♦ NetEm 12 / 92
  13. 13. Actors ▶ Database - WSREP/PXC ▶ Plugin - Galera ▶ Traffic control ♦ Traffic Control - tc ♦ NetEm 13 / 92
  14. 14. Actors ▶ Database - WSREP/PXC ▶ Plugin - Galera ▶ Traffic control ♦ Traffic Control - tc ♦ NetEm 14 / 92
  15. 15. Actors ▶ Containers - Docker ▶ Load ♦ Generators - Sysbench, RQG ▶ Network ♦ Dnsmasq ♦ nsenter 15 / 92
  16. 16. Actors ▶ Containers - Docker ▶ Load ♦ Generators - Sysbench, RQG ▶ Network ♦ Dnsmasq ♦ nsenter 16 / 92
  17. 17. Actors ▶ Jenkins ♦ Build flow and CI ▶ Storage ♦ Why 17 / 92
  18. 18. But why ▶ The ‘P’ in CAP ▶ WAN scalability ▶ Real Reason - fun! ▶ Tolerance to latency variance 18 / 92
  19. 19. But why ▶ The ‘P’ in CAP ▶ WAN scalability ▶ Real Reason - fun! ▶ Tolerance to latency variance 19 / 92
  20. 20. But why ▶ The ‘P’ in CAP ▶ WAN scalability ▶ Real Reason - fun! ▶ Tolerance to latency variance 20 / 92
  21. 21. But why ▶ The ‘P’ in CAP ▶ WAN scalability ▶ Real Reason - fun! ▶ Tolerance to latency variance 21 / 92
  22. 22. But why ▶ Failures in warehouses. ▶ Not quorum, but consensus. ▶ Real world networks and synchronous replication - Delay - Partition 22 / 92
  23. 23. Galera
  24. 24. Galera ▶ Data-centric approach ▶ EVS ▶ Causality and Synchronous ▶ Latency 24 / 92
  25. 25. Where did it start
  26. 26. Where did it start ▶ Bug! https://bugs.launchpad.net/galera/+bug/1274192 ▶ Loss of PC ▶ Crash ▶ HA goal 29 / 92
  27. 27. One can bring the whole down
  28. 28. The Flow
  29. 29. Basic Flow Jenkins Build image(s?) Start Dnsmasq Bootstrap nsenter/netem Pre-sanity SST/Others Load/Sysbench 32 / 92
  30. 30. Basic Flow Jenkins Build image(s?) Start Dnsmasq Bootstrap nsenter/netem Pre-sanity SST/Others Load/Sysbench 33 / 92
  31. 31. Basic Flow Jenkins Build image(s?) Start Dnsmasq Bootstrap nsenter/netem Pre-sanity SST/Others Load/Sysbench 34 / 92
  32. 32. Basic Flow Jenkins Build image(s?) Start Dnsmasq Bootstrap nsenter/netem Pre-sanity SST/Others Load/Sysbench 35 / 92
  33. 33. Basic Flow Jenkins Build image(s?) Start Dnsmasq Bootstrap nsenter/netem Pre-sanity SST/Others Load/Sysbench 36 / 92
  34. 34. Basic Flow Jenkins Build image(s?) Start Dnsmasq Bootstrap nsenter/netem Pre-sanity SST/Others Load/Sysbench 37 / 92
  35. 35. Basic Flow Jenkins Build image(s?) Start Dnsmasq Bootstrap nsenter/netem Pre-sanity SST/Others Load/Sysbench 38 / 92
  36. 36. Basic Flow Jenkins Build image(s?) Start Dnsmasq Bootstrap nsenter/netem Pre-sanity SST/Others Load/Sysbench 39 / 92
  37. 37. Basic Flow RR sysbench Detach/Keep Post sanity Core trace Sanity check Reconciliation Examine! Cleanup Collect logs 40 / 92
  38. 38. Basic Flow RR sysbench Detach/Keep Post sanity Core trace Sanity check Reconciliation Examine! Cleanup Collect logs 41 / 92
  39. 39. Basic Flow RR sysbench Detach/Keep Post sanity Core trace Sanity check Reconciliation Examine! Cleanup Collect logs 42 / 92
  40. 40. Basic Flow RR sysbench Detach/Keep Post sanity Core trace Sanity check Reconciliation Examine! Cleanup Collect logs 43 / 92
  41. 41. Basic Flow RR sysbench Detach/Keep Post sanity Core trace Sanity check Reconciliation Examine! Cleanup Collect logs 44 / 92
  42. 42. Basic Flow RR sysbench Detach/Keep Post sanity Core trace Sanity check Reconciliation Examine! Cleanup Collect logs 45 / 92
  43. 43. Basic Flow RR sysbench Detach/Keep Post sanity Core trace Sanity check Reconciliation Examine! Cleanup Collect logs 46 / 92
  44. 44. Basic Flow RR sysbench Detach/Keep Post sanity Core trace Sanity check Reconciliation Examine! Cleanup Collect logs 47 / 92
  45. 45. Cluster Resilience 48 / 92
  46. 46. Parameters ▶ Sysbench ▶ Segments ▶ Reconciliation period ▶ Number of nodes with loss 49 / 92
  47. 47. Parameters ▶ Sysbench ▶ Segments ▶ Reconciliation period ▶ Number of nodes with loss 50 / 92
  48. 48. Parameters ▶ Sysbench ▶ Segments ▶ Reconciliation period ▶ Number of nodes with loss 51 / 92
  49. 49. Parameters ▶ Sysbench ▶ Segments ▶ Reconciliation period ▶ Number of nodes with loss 52 / 92
  50. 50. Parameters ▶ NetEm ▶ Detach qdisc or keep it? ▶ Fsync ▶ Shutdown 53 / 92
  51. 51. Parameters ▶ NetEm ▶ Detach qdisc or keep it? ▶ Fsync ▶ Shutdown 54 / 92
  52. 52. Parameters ▶ NetEm ▶ Detach qdisc or keep it? ▶ Fsync ▶ Shutdown 55 / 92
  53. 53. Parameters ▶ NetEm ▶ Detach qdisc or keep it? ▶ Fsync ▶ Shutdown 56 / 92
  54. 54. Parameters
  55. 55. Containers!
  56. 56. Docker ▶ Why not virtualize ♦ Occam ♦ Namespaces ▶ Simplicity ♦ Network ♦ One application per node 59 / 92
  57. 57. Docker ▶ Portability - See same qualitative behavior that I do. ▶ Reproducibility - Makes it determinstic ▶ Configurable and CI - Byproducts 60 / 92
  58. 58. Docker ▶ QEMU and Docker ▶ Scalability ♦ Performance ♦ Feature ▶ Abstraction of channels 61 / 92
  59. 59. Container Networking ▶ Linking didn’t help ▶ Dnsmasq to rescue! ♦ Hosts file and volumes ♦ SIGHUP and refresh ▶ More elegant methods - Swarm 62 / 92
  60. 60. Noise ▶ Initial setup - Bridge - Egress only - IFB ▶ Present state ▶ NetEm - tc qdisc buckets - packet loss, delay, corruption, duplication, reordering - nsenter ▶ Future - Docker exec 63 / 92
  61. 61. Testing methods
  62. 62. Method I ▶ Qdisc is detached after load ▶ Objective - Time to recover of full cluster ▶ Done with a larger subset 65 / 92
  63. 63. Method II ▶ Qdisc is kept till the end ▶ Objective - Formation of primary component ▶ Comparatively smaller subset 66 / 92
  64. 64. Observations ▶ Post sanity types - Why - Whom ▶ Which method is more pertinent ▶ State transfer issues - Beginning - During re-emergence 67 / 92
  65. 65. Observations ▶ Direct load to affected nodes ▶ Logs - journalctl - Streaming? 68 / 92
  66. 66. Other noises ▶ Aim ▶ Fsync - libeatmydata - Variance ▶ Correlation with network ▶ How with Docker - LD_PRELOAD 69 / 92
  67. 67. System Load
  68. 68. Load generation ▶ Sysbench - Generation - Reconnect on partition ▶ Sockets chosen - Load on affected nodes ▶ Distribution of Load - RR with socat - Native sysbench support - HAProxy? 71 / 92
  69. 69. Load generation ▶ Nature of data/load - DDL ▶ RQG in future - Fuzz testing 72 / 92
  70. 70. The Fix
  71. 71. Strike Out!
  72. 72. Eviction ▶ STONITH ▶ Permanent eviction ▶ ’N’ strikes & out! - Timers - evs parameters - wsrep_evs_delayed and wsrep_evs_evict_list 75 / 92
  73. 73. Eviction ▶ Aim ▶ Quorum required - Keep majority of the group live or to avoid eviction. - Why? - Not shoot each other - Non-PC nodes also. 76 / 92
  74. 74. Eviction ▶ Aim ▶ Quorum required - Keep majority of the group live or to avoid eviction. - Why? - Not shoot each other - Non-PC nodes also. 77 / 92
  75. 75. Eviction ▶ EVS version and upgrade ▶ TODO! - Ingress only - Follow here. ▶ Credits to Teemu Ollakka, Yan Zhang and Alex Yurchenko from codership. 78 / 92
  76. 76. Coredumps with Docker ▶ Breakdown of abstraction ▶ Lack of isolation ▶ What was done - Volumes - core_pattern & sysctl - suid and ulimit 79 / 92
  77. 77. WAN Segments ▶ How they work ▶ Random allocation ▶ Joiner starvation ▶ Simulates data center ▶ Donor selection 80 / 92
  78. 78. The code ▶ Github: https://github.com/percona/pxc-docker ▶ Jenkins: http://jenkins.percona.com/job/PXC-5.6-netem/ ▶ Contributions/testing welcome! ▶ Dependencies - Sysbench 81 / 92
  79. 79. Code: todo ▶ Docker automated builds ▶ Orchestration - Kubernetes, Mesos et.al. ▶ Docker ♦ Injection ♦ Signal proxying 82 / 92
  80. 80. Code: todo ▶ Poll and exit during reconciliation - Instrument v/s number of nodes ▶ Use actual channels ▶ Run it bare - CoreOS etc. ▶ Overlay with etcd/fleet/libswarm 83 / 92
  81. 81. Future work
  82. 82. Future work ▶ Fault injection ♦ Memory - Poisoned memory/madvise ♦ Disk - libeatmydata - Opposite - ENOSPC 85 / 92
  83. 83. Fault injection ▶ CPU - NUMA? - Hotplug ▶ More network - corruption, duplication, reordering, rate-limit - Better distribution - Other shaping 86 / 92
  84. 84. More Chaos
  85. 85. Future work ▶ Disturb cluster more! - Membership changes * Manual eviction * Pull the cord! - Corrupt nodes ▶ Consistency voting 88 / 92
  86. 86. Further Reading ▶ Jepsen ▶ Byzantine fault tolerance - Reaching agreement in presence of faults ▶ The Network is Reliable ▶ NetEm ▶ Latency: The New Web Performance Bottleneck ▶ Galera Cluster Documentation ▶ Auto eviction code ▶ Don’t Settle for Eventual Consistency ▶ Extended Virtual Synchrony 89 / 92
  87. 87. About ▶ /me: Raghavendra Prabhu, Product Lead, Percona XtraDB Cluster, Percona. ▶ Slides will be at slideshare. ▶ About.me: raghavendra.prabhu ▶ Keybase.io: rdprabhu ▶ Presentation under CC BY-SA 4.0 90 / 92
  88. 88. Image Credits ▶ http://galeracluster.com/documentation-webpages/ ▶ http://www.thelastdragontribute.com/40th-anniversary-death-of-bruce-lee/ ▶ https://upload.wikimedia.org/wikipedia/commons/6/60/Corpus_callosum.png ▶ http://www.thebarrow.org/Neurological_Services/Epilepsy/204354 ▶ https://flic.kr/p/9J6GNu ▶ https://secure.flickr.com/photos/brewbooks/7780990192 ▶ https://www.flickr.com/photos/kwerfeldein/2649294869 ▶ https://secure.flickr.com/photos/mindmob/51951632 ▶ https://secure.flickr.com/photos/arenamontanus/2227769907 ▶ https://www.flickr.com/photos/markop/477199204 ▶ http://galeracluster.com/wp-content/uploads/2013/10/galera_replication1.png ▶ https://www.flickr.com/photos/gcwest/281385801 ▶ https://www.flickr.com/photos/opethdamna/360934079 ▶ http://digital-amphetamine.deviantart.com/art/Sky-82555664 ▶ http://highload.co/i/logo.png ▶ https://flic.kr/p/xTT8n ▶ https://www.flickr.com/photos/29233640@N07/13466208953 ▶ https://www.flickr.com/photos/bob_in_thailand/9782777742/ 91 / 92
  89. 89. Spaseeba & Da sveedaneeya!

×