MySQL HA An overview Kris Buytaert
Kris Buytaert <ul><li>Senior Linux and Open Source Consultant @inuits.be
„ Infrastructure Architect“
I don't remember when I started using MySQL :)
Specializing in Automated , Large Scale Deployments , Highly Available infrastructures, since 2008 also known as “the Cloud”
Surviving the 10 th  floor test
DevOp </li></ul>
What is HA Clustering ? <ul><li>One service goes down  </li><ul><li>=> others take over its work </li></ul><li>IP address ...
Not  designed for high-performance
Not designed for high troughput (load balancing) </li></ul>
Does it Matter ? <ul><li>Downtime is expensive
You mis out on $$$
Your boss complains
New users don't return </li></ul>
Lies, Damn Lies, and Statistics Counting nines (slide by Alan R)
The Rules of HA <ul><li>Keep it Simple
Keep it Simple
Prepare for Failure
Complexity is the enemy of reliability </li></ul>
You care about ? <ul><li>Your data ? </li><ul><li>Consistent
Realitime
Eventual Consistent  </li></ul><li>Your Connection </li><ul><li>Always
Most of the time </li></ul></ul>
Eliminating the SPOF <ul><li>Find out what Will Fail  </li><ul><li>Disks
Fans
Power (Supplies) </li></ul><li>Find out what Can Fail </li><ul><li>Network
Going Out Of Memory  </li></ul></ul>
Split Brain <ul><li>Communications failures can lead to separated partitions of the cluster
If those partitions each try and take control of the cluster, then it's called a split-brain condition
If this happens, then bad things will happen </li><ul><li>http://linux-ha.org/BadThingsWillHappen </li></ul></ul>
Historical MySQL HA <ul><li>Replication  </li><ul><li>1 read write node
Upcoming SlideShare
Loading in...5
×

MySQL HA Alternatives 2010

6,897

Published on

The presentation I gave in the MySQL Devroom at Fosdem 2010 , summarizing the different alternatives for MySQL HA

Published in: Technology
0 Comments
15 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
6,897
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
15
Embeds 0
No embeds

No notes for slide

Transcript of "MySQL HA Alternatives 2010"

  1. 1. MySQL HA An overview Kris Buytaert
  2. 2. Kris Buytaert <ul><li>Senior Linux and Open Source Consultant @inuits.be
  3. 3. „ Infrastructure Architect“
  4. 4. I don't remember when I started using MySQL :)
  5. 5. Specializing in Automated , Large Scale Deployments , Highly Available infrastructures, since 2008 also known as “the Cloud”
  6. 6. Surviving the 10 th floor test
  7. 7. DevOp </li></ul>
  8. 8. What is HA Clustering ? <ul><li>One service goes down </li><ul><li>=> others take over its work </li></ul><li>IP address takeover, service takeover,
  9. 9. Not designed for high-performance
  10. 10. Not designed for high troughput (load balancing) </li></ul>
  11. 11. Does it Matter ? <ul><li>Downtime is expensive
  12. 12. You mis out on $$$
  13. 13. Your boss complains
  14. 14. New users don't return </li></ul>
  15. 15. Lies, Damn Lies, and Statistics Counting nines (slide by Alan R)
  16. 16. The Rules of HA <ul><li>Keep it Simple
  17. 17. Keep it Simple
  18. 18. Prepare for Failure
  19. 19. Complexity is the enemy of reliability </li></ul>
  20. 20. You care about ? <ul><li>Your data ? </li><ul><li>Consistent
  21. 21. Realitime
  22. 22. Eventual Consistent </li></ul><li>Your Connection </li><ul><li>Always
  23. 23. Most of the time </li></ul></ul>
  24. 24. Eliminating the SPOF <ul><li>Find out what Will Fail </li><ul><li>Disks
  25. 25. Fans
  26. 26. Power (Supplies) </li></ul><li>Find out what Can Fail </li><ul><li>Network
  27. 27. Going Out Of Memory </li></ul></ul>
  28. 28. Split Brain <ul><li>Communications failures can lead to separated partitions of the cluster
  29. 29. If those partitions each try and take control of the cluster, then it's called a split-brain condition
  30. 30. If this happens, then bad things will happen </li><ul><li>http://linux-ha.org/BadThingsWillHappen </li></ul></ul>
  31. 31. Historical MySQL HA <ul><li>Replication </li><ul><li>1 read write node
  32. 32. Multiple read only nodes
  33. 33. Application needed to be modified </li></ul></ul>
  34. 34. Solutions <ul><li>BYO
  35. 35. DRBD
  36. 36. MySQL Cluster NDBD
  37. 37. Multi Master Replication
  38. 38. MySQL Proxy
  39. 39. MMM
  40. 40. Flipper </li></ul>
  41. 41. Data vs Connection <ul><li>DATA : </li><ul><li>Replication
  42. 42. DRBD </li></ul><li>Connection </li><ul><li>LVS
  43. 43. Proxy
  44. 44. Heartbeat / Pacemaker </li></ul></ul>
  45. 45. Shared Storage <ul><li>1 MySQL instance
  46. 46. Monitor MySQL node
  47. 47. Stonith
  48. 48. $$$ 1+1 <> 2
  49. 49. Storage = SPOF
  50. 50. Split Brain :( </li></ul>
  51. 51. DRBD <ul><li>Distributed Replicated Block Device
  52. 52. In the Linux Kernel (as of very recent)
  53. 53. Usually only 1 mount </li><ul><li>Multi mount as of 8.X </li><ul><li>Requires GFS / OCFS2 </li></ul></ul><li>Regular FS ext3 ...
  54. 54. Only 1 MySQL instance Active accessing data
  55. 55. Upon Failover MySQL needs to be started on other node </li></ul>
  56. 56. DRBD(2) <ul><li>What happens when you pull the plug of a Physical machine ? </li><ul><li>Minimal Timeout
  57. 57. Why did the crash happen ?
  58. 58. Is my data still correct ?
  59. 59. Innodb Consistency Checks ? </li><ul><li>Lengthy ?
  60. 60. Check your BinLog size </li></ul></ul></ul>
  61. 61. MySQL Cluster NDBD <ul><li>Shared-nothing architecture
  62. 62. Automatic partitioning
  63. 63. Synchronous replication
  64. 64. Fast automatic fail-over of data nodes
  65. 65. In-memory indexes
  66. 66. Not suitable for all query patterns (multi-table JOINs, range scans) </li></ul>
  67. 67. Title <ul><ul><li>Data </li></ul></ul>
  68. 68. MySQL Cluster NDBD <ul><li>All indexed data needs to be in memory
  69. 69. Good and bad experiences </li><ul><li>Better experiences when using the API
  70. 70. Bad when using the MySQL Server </li></ul><li>Test before you deploy
  71. 71. Does not fit for all apps </li></ul>
  72. 72. Multi Master Replication <ul><li>Replicating the same table data both ways can lead to race conditions </li><ul><li>Auto_increment, unique keys, etc.. could cause problems If you write them 2x </li></ul><li>Both nodes are master
  73. 73. Both nodes are slave
  74. 74. Write in 1 get updates on the other </li></ul>M|S M|S
  75. 75. MySQL Proxy <ul><li>Man in the middle
  76. 76. Decides where to connect to </li><ul><li>LUA </li></ul><li>Write rules to </li><ul><li>Redirect traffic </li></ul></ul>
  77. 77. Master Slave & Proxy <ul><li>Split Read and Write Actions
  78. 78. No Application change required
  79. 79. Sends specific queries to a specific node
  80. 80. Based on </li><ul><li>Customer
  81. 81. User
  82. 82. Table
  83. 83. Availability </li></ul></ul>
  84. 84. MySQL Proxy <ul><li>Your new SPOF
  85. 85. Make your Proxy HA too ! </li><ul><li>Heartbeat OCF Resource </li></ul></ul>
  86. 86. Breaking Replication <ul><li>If the master and slave gets out of sync
  87. 87. Updates on slave with identical index id </li><ul><li>Check error log for disconnections and issues with replication </li></ul></ul>
  88. 88. Monitor your Setup <ul><li>Not just connectivity
  89. 89. Also functional </li><ul><li>Query data
  90. 90. Check resultset is correct </li></ul><li>Check replication </li><ul><li>MaatKit
  91. 91. OpenARK </li></ul></ul>
  92. 92. Pulling Traffic <ul><li>Eg. for Cluster, MultiMaster setups </li><ul><li>DNS
  93. 93. Advanced Routing
  94. 94. LVS
  95. 95. Or the upcoming slides </li></ul></ul>
  96. 96. MMM <ul><li>Multi-Master Replication Manager for MySQL </li><ul><li>Perl scripts to perform monitoring/failover and management of MySQL master-master replication configurations </li></ul><li>Balance master / slave configs based on replication state </li><ul><li>Map Virtual IP to the Best Node </li></ul><li>http://mysql-mmm.org/ </li></ul>
  97. 97. Flipper <ul><li>Flipper is a Perl tool for managing read and write access pairs of MySQL servers
  98. 98. master-master MySQL Servers
  99. 99. Clients machines do not connect &quot;directly&quot; to either node instead,
  100. 100. One IP for read,
  101. 101. One IP for write.
  102. 102. Flipper allows you to move these IP addresses between the nodes in a safe and controlled manner.
  103. 103. http://provenscaling.com/software/flipper/ </li></ul>
  104. 104. Linux-HA PaceMaker <ul><li>Plays well with others
  105. 105. Manages more than MySQL
  106. 106. ...v3 .. don't even think about the rest anymore
  107. 107. http://clusterlabs.org/ </li></ul>
  108. 109. Conclusion <ul><li>Plenty of Alternatives
  109. 110. Think about your Data
  110. 111. Think about getting Queries to that Data
  111. 112. Complexity is the enemy of reliability
  112. 113. Keep it Simple
  113. 114. Monitor inside the DB </li></ul>
  114. 115. ` Kris Buytaert < [email_address] > Further Reading http://www.krisbuytaert.be/blog/ http://www.inuits.be/ http://www.virtualization.com/ http://www.oreillygmt.com/ ? !

×