0
MySQL High Availability Solutions


   Lenz Grimmer <lenz@grimmer.com>
                  <
    http://lenzg.net/ | Twitter...
Agenda
●   High Availability: Concepts &
    Considerations
●   MySQL Replication
●   DRBD / Heartbeat / Pacemaker
●   MyS...
What Is HA Clustering?
●   Redundancy
●   One system or service goes down → others
    take over
●   IP address takeover, ...
Replication vs. Shared Storage
●   Shared storage resource can become SPOF
●   Split brain situations can lead to mayhem
 ...
MySQL Replication
●   One-way, statement- or row-based
●   One Master, many Slaves
●   Asynchronous – Slaves can lag
●   M...
Statement-based replication
●   Pro
      ✔   Proven (around since MySQL 3.23)
      ✔   Smaller log files
      ✔   Audit...
Row-based replication
●   Pro
      ✔   All changes can be replicated
      ✔   Similar technology used by other RDBMSes
 ...
MySQL Replication Overview
        Read & Write


            Web/App
             Server
                                ...
Replication Topologies

Master > Slave                          Master > Slaves




Master > Slave > Slaves            Mas...
Master-Master Replication
●   Useful for easier failover
●   Not suitable for load-balancing
●   Don't write to both maste...
MySQL Replication as an HA Solution

●   What happens if the Master fails?
●   What happens if the Slave fails?
●   This d...
Replication & HA
●   Combined with Heartbeat
●   Virtual IP takeover
●   Slave gets promoted to Master
●   Side benefits: ...
Linux-HA / Heartbeat / Pacemaker
●   Supports 2 or more nodes
●   Resource monitoring
●   Active fencing mechanism: STONIT...
Applications




Master   Virtual IP          HA Slave   Scale-out Slave




               Replication

                 ...
DRBD
●   Distributed Replicated Block Device
●   “RAID-1 over network”
●   Synchronous or asynchronous
●   Automatic resyn...
MySQL on DRBD
●   MySQL runs on the Active node as usual
●   MySQL is dormant on the Passive node



                     ...
MySQL Cluster
●   Shared-nothing architecture
●   Automatic partitioning
●   Distributed Fragments
●   Synchronous replica...
MySQL Cluster
●   In-memory indexes
●   Not suitable for all query patterns (multi-table
    JOINs, range scans)
●   No su...
MySQL Cluster & Replication
●   MySQL Cluster
      ●   Easy failover from one master to another
      ●   Scaling writes ...
http://johanandersson.blogspot.com/2009/05/ha-mysql-write-scaling-using-cluster-to.html
MySQL Replicat ion &   MySQL, Hear t beat &     MySQL
                    Requir em ent s      MySQL Replicat ion       He...
MMM
●   MySQL Master-Master Replication Manager
●   Collection of Perl scripts to perform
    monitoring/failover and mana...
Flipper
●   Perl Script that manages pairs of MySQL servers
    replicating to each other.
●   Automatic switchover, trigg...
Red Hat Cluster Suite
●   HA and load balancing
●   Supports up to 128 nodes
●   Shared storage
●   Monitors services, fil...
Solaris Cluster / OpenHA Cluster
●   Provides failover and scalability services
●   Solaris / OpenSolaris
●   Kernel-level...
Related Tools / Links
●   Linux Heartbeat
    http://linux-ha.org/
●   Red Hat Cluster Suite
    http://www.redhat.com/clu...
More Tools / Links
●   MySQL Multi-Master Replication Manager
    http://code.google.com/p/mysql-master-master/
●   Maatki...
Q&A




Questions, Comments?
Thank you!




Lenz Grimmer <lenz@grimmer.com>
         Twitter: @lenzgr
         http://lenzg.net/
Eliminating the SPOF
●   Identify what will fail
    ●   Disks
    ●   Fans
●   Find out what can fail
    ●   Network cab...
HA Components
●   Heartbeat
       ●   Checks that services that are being failed over,
           are alive.
       ●   C...
Rules of High Availability
●   Prepare for failure
●   Aim to ensure that no important data is lost
●   Keep it simple, st...
Split-Brain
●   Communications failures can lead to
    separated partitions of the cluster
●   If those partitions each t...
MySQL High Availability Solutions
MySQL High Availability Solutions
MySQL High Availability Solutions
MySQL High Availability Solutions
MySQL High Availability Solutions
MySQL High Availability Solutions
MySQL High Availability Solutions
MySQL High Availability Solutions
Upcoming SlideShare
Loading in...5
×

MySQL High Availability Solutions

13,927

Published on

A presentation about how to make MySQL highly available, presented at the San Francisco MySQL Meetup (http://www.sfmysql.org/events/15760472/) on January 26th, 2011.

A video recording of this presentation is available from Ustream: http://ustre.am/fyLk

Published in: Technology
1 Comment
30 Likes
Statistics
Notes
No Downloads
Views
Total Views
13,927
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
662
Comments
1
Likes
30
Embeds 0
No embeds

No notes for slide

Transcript of "MySQL High Availability Solutions"

  1. 1. MySQL High Availability Solutions Lenz Grimmer <lenz@grimmer.com> < http://lenzg.net/ | Twitter: @lenzgr 2009-12-03 SAPO CodeBits | Lisbon | Portugal
  2. 2. Agenda ● High Availability: Concepts & Considerations ● MySQL Replication ● DRBD / Heartbeat / Pacemaker ● MySQL Cluster ● Other HA tools/applications ● Links
  3. 3. What Is HA Clustering? ● Redundancy ● One system or service goes down → others take over ● IP address takeover, service takeover ● Ensuring data availability & integrity ● Not designed for high-performance
  4. 4. Replication vs. Shared Storage ● Shared storage resource can become SPOF ● Split brain situations can lead to mayhem (e.g. mounting file systems concurrently) ● SAN/NAS read I/O overhead ● Consistency of replicated data ● Synchronous vs. asynchronous replication
  5. 5. MySQL Replication ● One-way, statement- or row-based ● One Master, many Slaves ● Asynchronous – Slaves can lag ● Master maintains binary logs & index ✔ Easy to use and set up ✔ Built into MySQL ✗ Replication is single-threaded ✗ No automated fail-over ✗ No heartbeat, no monitoring
  6. 6. Statement-based replication ● Pro ✔ Proven (around since MySQL 3.23) ✔ Smaller log files ✔ Auditing of actual SQL statements ✔ No primary key requirement for replicated tables ● Con ✗ Non-deterministic functions and UDFs ✗ LOAD_FILE(), UUID(), CURRENT_USER(), FOUND_ROWS() (but RAND() and NOW() work)
  7. 7. Row-based replication ● Pro ✔ All changes can be replicated ✔ Similar technology used by other RDBMSes ✔ Fewer locks required for some INSERT, UPDATE or DELETE statements ● Con ✗ More data to be logged ✗ Log file size increases (backup/restore implications) ✗ Replicated tables require explicit primary keys ✗ Possible different result sets on bulk INSERTs
  8. 8. MySQL Replication Overview Read & Write Web/App Server Write Relay Log mysqld I/O SQL Thread Thread Index & Binlogs Data Replication Binlog Data mysqld MySQL Master MySQL Slave
  9. 9. Replication Topologies Master > Slave Master > Slaves Master > Slave > Slaves Masters > Slave (Multi-Source) Ring (Multi-Master) Master < > Master (Multi-Master)
  10. 10. Master-Master Replication ● Useful for easier failover ● Not suitable for load-balancing ● Don't write to both masters! ● Use Sharding or Partitioning instead (e.g. MySQL Proxy)
  11. 11. MySQL Replication as an HA Solution ● What happens if the Master fails? ● What happens if the Slave fails? ● This doesn’t sound like High Availability!
  12. 12. Replication & HA ● Combined with Heartbeat ● Virtual IP takeover ● Slave gets promoted to Master ● Side benefits: load balancing & backup ● Can be tricky to fail back ● No automatic conflict resolution ● Proper failover needs to be scripted
  13. 13. Linux-HA / Heartbeat / Pacemaker ● Supports 2 or more nodes ● Resource monitoring ● Active fencing mechanism: STONITH ● Policy-based resource management, dependencies & constraints ● Time-based rules ● Includes support for many applications ● GUI support ● Low dependencies / requirements ● Subsecond node failure detection
  14. 14. Applications Master Virtual IP HA Slave Scale-out Slave Replication Replication
  15. 15. DRBD ● Distributed Replicated Block Device ● “RAID-1 over network” ● Synchronous or asynchronous ● Automatic resync on recovery ● Can mask local I/O errors ● Active/passive configuration by default ● Dual-primary mode (requires a cluster filesystem like GFS or OCFS2) ● http://drbd.org/
  16. 16. MySQL on DRBD ● MySQL runs on the Active node as usual ● MySQL is dormant on the Passive node Applications Active Node Virtual IP Passive Node DRBD
  17. 17. MySQL Cluster ● Shared-nothing architecture ● Automatic partitioning ● Distributed Fragments ● Synchronous replication ● Fast automatic fail-over of data nodes ● Automatic resynchronization ● Transparent to Application ● Supports Transactions
  18. 18. MySQL Cluster ● In-memory indexes ● Not suitable for all query patterns (multi-table JOINs, range scans) ● No support for foreign key constraints ● Not suitable for large datasets/transactions ● Latency matters ● Can be combined with MySQL Replication (RBR)
  19. 19. MySQL Cluster & Replication ● MySQL Cluster ● Easy failover from one master to another ● Scaling writes to multiple masters ● Asynchronous replication from Cluster to multiple slaves ● Reads are distributed to slave servers (running InnoDB/MyISAM) ● Quick setup of new slaves (using Cluster Online Backup) ● Easy failover and quick recovery
  20. 20. http://johanandersson.blogspot.com/2009/05/ha-mysql-write-scaling-using-cluster-to.html
  21. 21. MySQL Replicat ion & MySQL, Hear t beat & MySQL Requir em ent s MySQL Replicat ion Hear t beat DRBD Clust er Aut om at ed IP Failover No Yes Yes No Aut om at ed DB Availabilit y Failover No No Yes Yes Typical Failover t im e Var ies Var ies < 30s < 3s Aut o r esync of dat a No No Yes Yes Geogr aphic MySQL r edundancy Yes Yes MySQL Replicat ion Replicat ion Built -in load Scalabilit y balancing MySQL Replicat ion MySQL Replicat ion MySQL Replicat ion Yes Read-int ensive Yes Yes MySQL Replicat ion Yes Wr it e-int ensive No No Possible Yes # Nodes/Clust er Mast er /Slave(s) Mast er /Slave(s) Act ive/Passive 255
  22. 22. MMM ● MySQL Master-Master Replication Manager ● Collection of Perl scripts to perform monitoring/failover and management ● Master-Master replication configurations (one writable node) ● Failover by moving virtual IP ● http://mysql-mmm.org/
  23. 23. Flipper ● Perl Script that manages pairs of MySQL servers replicating to each other. ● Automatic switchover, triggered manually ● Enables one half of the pair to be taken offline for maintenance work while the other half carries on dealing with queries from clients ● Moves IP addresses based on a role ("read-only", "writable") between two nodes in the master pair, to ensure that each role is available ● http://provenscaling.com/software/flipper/
  24. 24. Red Hat Cluster Suite ● HA and load balancing ● Supports up to 128 nodes ● Shared storage ● Monitors services, file systems and network status ● Fencing (STONITH) or distributed lock manager ● Configuration synchronization ● http://www.redhat.com/cluster_suite/
  25. 25. Solaris Cluster / OpenHA Cluster ● Provides failover and scalability services ● Solaris / OpenSolaris ● Kernel-level components plus userland ● Agents monitor applications ● Geo Edition to provide Disaster Recovery using Replication ● Open Source since June 2007 ● See Thorsten's FrOSCon talk for more details! (Sunday, 11:15, HS4) ● http://www.opensolaris.org/os/community/ha-clusters/
  26. 26. Related Tools / Links ● Linux Heartbeat http://linux-ha.org/ ● Red Hat Cluster Suite http://www.redhat.com/cluster_suite/ ● Sun Open High Availability Cluster http://opensolaris.org/os/project/ha-mysql/ ● Linux Cluster Information Center http://www.lcic.org/ha.html
  27. 27. More Tools / Links ● MySQL Multi-Master Replication Manager http://code.google.com/p/mysql-master-master/ ● Maatkit http://maatkit.sourceforge.net/ ● Mon – scheduler and alert management http://www.kernel.org/software/mon/ ● Continuent Tungsten Replicator https://community.continuent.com/community/tungsten-replicator
  28. 28. Q&A Questions, Comments?
  29. 29. Thank you! Lenz Grimmer <lenz@grimmer.com> Twitter: @lenzgr http://lenzg.net/
  30. 30. Eliminating the SPOF ● Identify what will fail ● Disks ● Fans ● Find out what can fail ● Network cables ● Power supplies ● CPUs ● NICs ● OOM
  31. 31. HA Components ● Heartbeat ● Checks that services that are being failed over, are alive. ● Can check individual servers, software services, networking etc. ● HA Monitor ● Configuration of the services ● Ensures proper shutdown and startup ● Allows manual control ● Shared storage / Replication
  32. 32. Rules of High Availability ● Prepare for failure ● Aim to ensure that no important data is lost ● Keep it simple, stupid (KISS) ● Complexity is the enemy of reliability ● Automate it ● Test your setup frequently!
  33. 33. Split-Brain ● Communications failures can lead to separated partitions of the cluster ● If those partitions each try and take control of the cluster, then it's called a split-brain condition ● If this happens, then bad things will happen http://linux-ha.org/BadThingsWillHappen ● Use Fencing or Moderation/Arbitration to avoid it
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×