Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MySQL and the CAP theorem: relevance & misconceptions

165 views

Published on

The CAP theorem is often used to describe the tradeoffs of available versus consistent systems. But the CAP theorem is often misunderstood and misrepresented. In this session we investigate the CAP properties of various MySQL replication setups, and show that it is impossible to claim a setup is AP or that it is CP. We will see how the properties can be non binary, and change across time, geography and workload. We will also observe where CAP's terms can be made irrelevant. This session takes the practical engineer's perspective and shows how we can gain the upper hand with reasonable tradeoffs.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

MySQL and the CAP theorem: relevance & misconceptions

  1. 1. MySQL and the CAP theorem: relevance & misconceptions Dissecting,affirmingandrefutingCAPassumptionsin realproductionsystems. ShlomiNoach GitHub FOSDEM2019
  2. 2. About me @github/database-infrastructure Authoroforchestrator,gh-ost,freno,ccql andothers. Blogathttp://openark.org 
 github.com/shlomi-noach
 @ShlomiNoach
  3. 3. GitHub
 Built for developers Largestopensourcehosting 100M+repositories,31M+developers,
 2.1M+organizations SupplierofoctocatT-Shirtsandstickers
  4. 4. Incentive
  5. 5. Incentive PeoplefrequentlyciteCAPtheorembywayof: • Modellingasystem(asAP/CP). • Projectingthatitisimpossibletoachievesomedesired setup.
  6. 6. CAP Conjecture SuggestedbyEricBrewer,1998-1999 Terms: • [atomic]Consistency • [high]Availability • Partitiontolerance
  7. 7. CAP Theorem Mathematicallyproved(hence“theorem”)bySethGilbert& NancyLynch,MIT. UsingdifferenttermsthanthosecoinedbyBrewer. InMath,atheoremonlyholdsaslongasitsterms&conditions aremet.
  8. 8. CAP Theorem Terms,accordingtoGilbert&Lynch: -Consistency -Availability -Partitiontolerance
  9. 9. CAP Theorem: Consistency Onceawriteissuccessfulonanode,anyreadonanynodemust reflectthatwriteoranylaterwrite. akaAtomicConsistency,akaLinearconsistency,aka Linearizability.
  10. 10. CAP Theorem: Availability Everynon-crashednodemustrespondtorequestsinafinite amountoftime. Thereisnoconstraintontheactualamountoftime. Thoughitisnotspecified,itisimpliedthatresponsemustbe valid,non-error. ContrastwithBrewer’sdefinitonofHighavailability: consumer alwayshasaccesstodataviaatleastonereplica.
  11. 11. CAP Theorem: Partition Tolerance Thesystemisabletooperateonnetworkpartitioning. Partitiontoleranceisconsideredasagivencondition,since networkpartitioningcananddoestakeplaceregardlessofa system’sdesign.
  12. 12. CAP Theorem Adistributeddatastore[webservice]cannotprovidemorethan twooutofthethreeproperties.Betterillustratedas: • Ifthenetworkisgood,youmayachievebothAvailibilityand Consistency. • Ifthenetworkispartitioned,youmustchoosebetween AvailabilityandConsistency.
  13. 13. Proof of CAP theorem Simplified,notbymuch. Giventwonodesreplicatingfromeachother ! ! n1 n2
  14. 14. Proof of CAP theorem Wepartitionthenetworkbetweenthetwonodestoaninfinite amountoftime. ! ! n1 n2
  15. 15. Proof of CAP theorem Wewritedatatoonenode.IfthesystemisAvailable,thewrite compeltesinafiniteamountoftime. ! ! n1 n2
  16. 16. Proof of CAP theorem Wereaddatafromtheothernode.IfthesystemisAvailable thatreadcompletesinafinitemountoftime. ! ! n1 n2
  17. 17. Proof of CAP theorem Duringthatfinitetimethenetworkwaspartitioned,henceour readcouldnotreflectchangesmadebythewrite. QED ! ! n1 n2
  18. 18. CAP is mathematical Asengineerswehavedifferentproductionobjectives.
  19. 19. CAP is binary Thetheoremhasnothingtosayof99.95%Availabilitywith 99.99%Consistency Tradeoffsandpartialavailability&consistencyarediscussedin bothBrewer’sConjecturepaperandGilbert&Lynch’sTheorem paper,buthavenopartintheproof.
  20. 20. CAP does not discuss DisasterRecovery Failover Latency(seePACELC) Durability,noderecovery SLA,uptime,nines Scalability,capacity
  21. 21. CAP & MySQL in production
  22. 22. Standard MySQL replication Singlemaster,multiplereplicas,asynchronousreplication ! ! ! ! ! !
  23. 23. Standard MySQL replication:
 CAP properties NotConsistent:asynchronousreplication,replicationlag. NotAvailable: • Replicanodesareunabletoservewrites(returnwitherror). • Ifmastercrashes,noservertobeabletotakewrites. • CAPdoesnotdiscussfailovers/SLA. ! ! ! ! ! !
  24. 24. Standard MySQL replication:
 consistency in practice Readfrommaster? Scalability/capacity ! ! ! ! ! !
  25. 25. Standard MySQL replication:
 consistency in practice ProxySQLGTIDcasualreads: • Appliestosameconnectiononly. • Ifmasterwithinhostgroup,canfailbacktomaster • NotCAPConsistent.Aproxyisn’taknowncomponentinCAP Theorem. 
 https://proxysql.com/blog/proxysql-gtid-causal-reads ! ! ! ! ! ! "
  26. 26. Standard MySQL replication:
 consistency in practice ! ! ! ! ! ! frenotomonitorreplicationlag: • Suggestswhenreplicasareuptodate • HAProxyroutesqueriestoreplicas,offloadingtrafficfrom master • NotCAPConsistent.NotpartofCAP. 
 https://githubengineering.com/mitigating-replication-lag-and-reducing-read-load-with-freno/ "
  27. 27. Semi-sync replication Singlemaster,multiplereplicas,semi-synctoall/subsetof replicas. ! ! ! ! ! !
  28. 28. Semi-sync replication:
 CAP properties StillnotConsistent. • Thoughwegetdurability. StillnotAvailable. • semi-syncwriteAvailabilitydependsonconfiguration;on identitiesofsemi-syncreplicas. ! ! ! ! ! !
  29. 29. Hybrid cluster IsitCA?IsitCP? Dependsongeographiclocation,time,workload. ! ! ! ! ! !
  30. 30. Vitess SQLlayes:vttablet,proxytobackend MySQLstandardreplicationbackendshards Suggestedconfiguration: • semi-sync • onlysendtraffictomaster " " " ! ! ! ! ! !
  31. 31. Vitess: CAP properties Givensuggestedconfiguration: • NotConsistent • NotAvailable " " " ! ! ! ! ! !
  32. 32. But! Withfailover,wecanhavethesystemhighlyavailable. Withreads-on-masteroranotherproxysolution,wecanmake thesystemhighlyconsistent. " " " ! ! ! ! ! !
  33. 33. CAP != E=MC²
  34. 34. InnoDB Cluster, single writer Singlewriternode,paxosconsensus ! ! ! < 8.0.14
  35. 35. InnoDB Cluster, single writer: CAP properties NotAvailable:ifnetworkpartitionedseverely,quorumcanbe lostandnoleadertoacceptwrites. NotConsistent;changesare: • propagatedtomajorityofnodes,notall. • madedurable,notapplied. ! ! ! < 8.0.14
  36. 36. InnoDB Cluster, single writer: availability, consistency in practice MySQLRouterasProxy,routetraffictoleader. ProxySQLforper-connectionconsistentreads.! ! ! " < 8.0.14
  37. 37. InnoDB Cluster, multi writer Multiwriternodes,paxosconsensus ! ! ! < 8.0.14
  38. 38. InnoDB Cluster, multi writer: CAP properties NotAvailable,asbefore. • Debatable:arewriteconflictscompatiblewithAvailability? Notconsistent,asbefore. ! ! ! < 8.0.14
  39. 39. Galera: CAP properties wsrep_sync_wait=0 • NotAvailable,notConsistent wsrep_sync_wait != 0 (e.g.=3) • NotAvailable • Consistentuptotimeoutonisolatednodes:i.e.not Consistent. ! ! !
  40. 40. InnoDB Cluster, >= 8.0.14 group_replication_consistency= BEFORE_AND_AFTER Supportsconsistentreads,uptotimeoutonisolatednode. NotAvailable,notConsistent. STONITHforisolatednodes. ! ! !
  41. 41. Is there any AP setup?
  42. 42. AP setups Disjointmasters ! !! ! !! ! ! ! ! ! Writablemaster-master,asyncreplication Writablecircular[n>2]mastertopology,asyncreplication
  43. 43. Takeaways MathandEngineeringarenotrequiredtoagree.Practical engneeringmakestradeoffs. CAPTheoremdoesnotrepresentcomplexproductionsystems, nordoesitmeetproductionrequirements. CAPTheoremmaynotevenapplytoyoursystem.Itmightnot bethemodelweshouldbelookingat. EvenasCAPsaysyouhavetochooseAoverCincaseofP,it doesnotmeanyoucannotachievehighavailabilityandhigh consistency.
  44. 44. PACELC AnextentiontoCAP: -ifPartitioned,tradeAvailabilityforConsistency. -Else,tradeLatencyforConsistency Morerealistic,stilldoesnotmeetproductionneeds.
  45. 45. Harvest, Yield, and ScalableTolerant Systems,ArmandoFox,EricA.Brewer
 https://pdfs.semanticscholar.org/5015/8bc1a8a67295ab7bce0550886a9859000dc2.pdf Brewer’s Conjecture and the Feasibility of Consistent, Available, Partition-TolerantWeb Services,SethGilbert,NancyLynch
 https://www.glassbeam.com/sites/all/themes/glassbeam/images/blog/10.1.1.67.6951.pdf A Critique of the CAPTheorem,MartinKleppmann
 https://arxiv.org/pdf/1509.05393.pdf CAPTwelve Years Later: How the "Rules" Have Changed,EricBrewer
 https://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed Please stop calling databases CP or AP,MartinKleppmann
 https://martin.kleppmann.com/2015/05/11/please-stop-calling-databases-cp-or-ap.html Further reading
  46. 46. NewSQL database systems are failing to guarantee consistency, and I blame Spanner,Daniel Abadi
 http://dbmsmusings.blogspot.com/2018/09/newsql-database-systems-are-failing-to.html Problems with CAP, andYahoo’s little known NoSQL system,DanielAbadi
 http://dbmsmusings.blogspot.com/2010/04/problems-with-cap-and-yahoos-little.html PACELC theorem,Wikipedia
 https://en.wikipedia.org/wiki/PACELC_theorem Further reading
  47. 47. CAP theorem is mostly misunderstood,SuguSougoumarane
 http://ssougou.blogspot.com/2017/09/cap-theorem-is-mostly-misunderstood.html "A Critique of the CAPTheorem”,JuliaEvans
 https://jvns.ca/blog/2016/11/19/a-critique-of-the-cap-theorem/ CAPTheorem,FoundationDB
 https://apple.github.io/foundationdb/cap-theorem.html Further reading
  48. 48. Questions? github.com/shlomi-noach @ShlomiNoach Thank you!

×