MySQL High Availability - Managing Farms of Distributed Servers

MySQL High Availability:
Managing Farms of Distributed Servers
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.
|
Narayanan Venkateswaran
Alfranio Correia
Mats Kindahl

Safe Harbor Statement
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Jan
2010
Sept
2014
2x Engineering Staf
3x QA Staf
2x Support Staf
The following is intended to outline our general product direction. It
is intended for information purposes only, and may not be
incorporated into any contract. It is not a commitment to deliver any
material, code, or functionality, and should not be relied upon in
making purchasing decision. The development, release, and timing
of any features or functionality described for Oracle’s products
remains at the sole discretion of Oracle.

Agenda
Jan
2010
Sept
2014
➢ Building Reliable Systems
➢ MySQL Fabric Overview
➢ Groups
➢ Failure Detection
➢ Integration with other HA Solutions
➢ High Availability @ Fabric Node
➢ Future Work

Causes of Downtime
● System Failures
Jan
2010
Sept
2014
2x Engineering Staf
● Server faults
3x ● Software QA bugs Staf
or crashes
2x Support Staf
● Physical Disasters
● Scheduled Maintenance
● User Errors

2x Engineering Staf
3x QA Staf
2x Support Staf High Availability is an integral part of designing a reliable system
Jan
2010
Sept
2014

High Availability Solutions
● E.g. MySQL Replication
● Shared Nothing Clusters
● Primary – Secondary
Jan
2010
Sept
2014
2x Engineering Staf
3x ● MySQL QA Cluster
Staf
2x Support Staf
● Tightly Coupled Clusters
● DRBD
● WSFC
● Solaris Clustering
● Oracle VM High Availability

An extensible and easy-to-use
framework for managing a farm
of MySQL servers supporting
high-availability and sharding.
MySQL Fabric
Jan
2010
Sept
2014

What does all that mean?
● Management System
– Manages a MySQL Farm
– Distributed Framework
Copyright © 2014, Oracle and/or its affiliates. 2014-10-02 All rights reserved. |
● Framework
– Procedure execution
– State store
– Transaction Routing
● Extensible
– High-availability groups
– “Semi-automatic” sharding
● Written in Python
● Latest Release 1.5.2
GA
● Open Source
– You can participate
– Suggest features
– Report bugs
– Contribute patches
● MySQL 5.6 is focus

Birds-eye View
Application
SQL
Connector
Connector
Connector
High Availability
Groups
XML-RPC
MySQL-RPC
MySQL
Fabric
Node
SQL
Operator
Database
Servers

High-Level Components
● Fabric-aware
Connectors
– Python, PHP, Java, .NET,
C
– Enhanced Connector API
● MySQL Fabric Node
– Manage information about
farm
– Provide status information
– Execute procedures
● MySQL Servers
– Organized in high-availability
groups
– Handling application data
Application Connector
High Availability
Group
Connector
Connector
MySQL
Fabric
Node

MySQL Fabric Node Architecture
Connector
Connector ?
MySQL
MySQL Fabric
Framework
Executor State Store
(Persister)
Sh
HA
AMQP MySQL-RPC XML-RPC
Connector
Protocols
Extensions
Backing
Store

Groups
Jan
2010
Sept
2014
● Group of Servers
– Hardware Redundancy
– Data Redundancy
● Generic Concept
– Default Master-Slave
configuration
– Supports generic High
Availability methods
● DRBD
● MySQL Cluster
● MySQL Replication
● Etc.
DRBD
ndbd ndbd
ndbd ndbd

Creating a MySQL Fabric Group
Jan
2010
Sept
2014
Creating a Group
Command:
mysqlfabric group create <group_id> [--description=NONE]
E.g.
mysqlfabric group create group-1 –description=”Example Group”
Creates a logical entity named group-1 that represents a farm of
MySQL Servers in a HA configuration

MySQL Fabric Group – Add Servers
Jan
2010
Sept
2014
Adding Servers to a Group
Command:
mysqlfabric group add <group_id> <address>
E.g.
mysqlfabric group add group-1 server1.example.com
mysqlfabric group add group-1 server2.example.com
Adds two servers to group-1

MySQL Fabric Group – Create Master / Slave Setup
Command:
mysqlfabric group promote group_id
[ --slave_id=NONE]
E.g.
mysqlfabric group promote group-1
--slave_id=”server1.example.com”
Promote server1 as the master
Jan
2010
Sept
2014

Built-In Failure Detector
Jan
2010
Sept
2014
● Group level failure detector
● Monitor servers within groups
● On master failure
– Mark master as faulty
– Trigger fail-over
● On slave failure
– Mark slave as faulty
● Solution is only for servers
managed by Fabric

Built-in Failure detector configuration
[failure_tracking]
detections = 3
detection_interval = 6
detection_timeout = 1
detections
Number of times a server must fail an alive check before being
marked as FAULTY.
detection_interval
Interval at which a server must be checked to be alive.
detection_timeout
Elapsed time after which a server alive ping must timeout.
Jan
2010
Sept
2014
2x Engineering Staf
3x QA Staf
2x Support Staf

Distributed Failure Detection
Jan
2010
Sept
2014
● Connector reports error to
Fabric
● Error used to update backing
store
● Trigger Failover
– Error count reaches
threshold
● The report API can be used by
any generic process to report
failures

Distributed Failure Detection – MySQL Fabric
Configuration
[failure_tracking]
notifications = 300
notification_clients = 50
notification_interval = 60
failover_interval = 0
prune_time = 3600
notifications
The number of error notifications required to mark a server
FAULTY.
notification_clients
The number of clients that need to report an error to mark a
server FAULTY.
Jan
2010
Sept
2014
2x Engineering Staf
3x QA Staf
2x Support Staf

Distributed Failure Detection – MySQL Fabric
Configuration
notification_interval
The interval of interest in which we want to verify error
notifications.
failover_interval
The interval at which we can safely do a failover in a group
without causing system instability.
prune_time
Interval at which we need to prune the error logs.
Jan
2010
Sept
2014
2x Engineering Staf
3x QA Staf
2x Support Staf

Distributed Failure Detection – Connector – Enabling
Error Reporting
● The option to report errors is part of the Fabric configuration
● “report_errors” can be turned on as follows
'host': ..,
'report_errors': True,
}
cnx = mysql.connector.connect(fabric=fabric_config)
fabric_config = {
Jan
2010
Sept
2014
2x Engineering Staf
3x QA Staf
2x Support Staf
● Causes an error during a connection to a server to be reported

Distributed Failure Detection – Connector – Error
Reporting
● The following errors are reported by default
REPORT_ERRORS = (
errorcode.CR_SERVER_LOST,
errorcode.CR_SERVER_GONE_ERROR,
errorcode.CR_CONN_HOST_ERROR,
errorcode.CR_CONNECTION_ERROR,
errorcode.CR_IPSOCK_ERROR,
)
Jan
2010
Sept
2014
2x Engineering Staf
3x QA Staf
2x Support Staf
● The above can be extended by setting extra_failure_report
from mysql.connector.fabric import extra_failure_report
extra_failure_report([error_code_0,
error_code_1, ...])

Distributed Failure Detection – Connector – Error
Reporting
● The following errors are reported by default
REPORT_ERRORS = (
errorcode.CR_SERVER_GONE_ERROR,
errorcode.CR_CONN_HOST_ERROR,
errorcode.CR_CONNECTION_ERROR,
errorcode.CR_IPSOCK_ERROR,
)
Jan
2010
Sept
2014
2x Engineering Staf
3x QA Staf
2x Support Staf
● The above can be extended by setting extra_failure_report
from mysql.connector.fabric import
extra_failure_report
extra_failure_report([error_code_0, error_code_1,
...])

Distributed Failure Detection – Connector – Cache
Invalidation
● Define which errors cause invalidation of the connector cache
● The following errors cause a cache invalidation by default
RESET_CACHE_ON_ERROR = (
errorcode.ER_OPTION_PREVENTS_STATEMENT,)
Jan
2010
Sept
2014
2x Engineering Staf
3x QA Staf
2x Support Staf
• The above can be extended by
from mysql.connector.fabric import
RESET_CACHE_ON_ERROR
RESET_CACHE_ON_ERROR.append(error_code_0)

Distributed Failure Detection – Avoiding thundering
herds
Jan
2010
Sept
2014
Application
1. Error
MySQL
Fabric Node
Application
Application
Application
2. Error
3. Error
4.Error
● When to report errors ?
● Reporting errors at the session
level can cause flooding
● E.g. Orders Page
– Orders Page writes customer
orders to the database
– Every failing order will cause
a report

Distributed Failure Detection – Connector – Role
● MySQL User role for accessing to Fabric dump and reporting
● Adding users done using
mysqlfabric user add <username> --role
3 connector Role for MySQL Connectors
+ Access to dump commands
+ Reporting to Fabric
Jan
2010
Sept
2014
2x Engineering Staf
3x QA Staf
2x Support Staf
● connector role
● Causes an error during a connection to a server to be reported

Update Only
Jan
2010
Sept
2014
Scenario
● Using Fabric as a lookup server
● Servers managed outside Fabric
– For E.g. using DRBD
Requirement
● Update Fabric without affecting
servers

Group Based and Virtual IP Based solutions
● Group Based Solutions
Jan
2010
Sept
2014
2x Engineering Staf
3x ● Offer QA a notion Staf
of Group Membership
2x Support Staf
● Virtual IP Based Solutions
● Only one virtual server identified by a virtual IP Address
● E.g. WFSC, DRBD

Group Based System
Jan
2010
Sept
2014
● Notion of membership
● Dead servers automatically
excluded from the group
● Use update_only to store
topology information in Fabric

Group Based System
● Creating a Group
● Adding Servers to a Group
● Promote
Creating a Group
mysqlfabric group create "group-1"
mysqlfabric group add group-1 server1.example.com --update_only
mysqlfabric group add group-1 server2.example.com -- update_only
Promote
mysqlfabric group promote group-1 --slave_id=7bcb0804-
41bb-11e4-b4d2-f4ce2963772b --update_only
Jan
2010
Sept
2014

Group Based System – Updating Failures
● Updating the state when all the servers are dead
● Updating the state when the system is brought back
online
When all the servers are dead
mysqlfabric threat report_error 7bcb0804-41bb-11e4-b4d2-
f4ce2963772b --error=FAULTY --update_only
When the system is brought back online
Jan
2010
Sept
2014

Virtual IP System – Integrating DRBD
● Creating a Group
● Adding Virtual IP to Group
Creating a Group
mysqlfabric group create "group-1"
mysqlfabric group add group-1 IPAddress:Portno --update_only
● Add the virtual IP of the DRBD setup to the group definition.
● The DRBD setup handles the maintanence of the cluster
Jan
2010
Sept
2014

Virtual IP System – Updating Failures
● Updating the state when all the servers are dead
● Updating the state when the system is brought back
online
When all the servers are dead
mysqlfabric threat report_error 7bcb0804-41bb-11e4-b4d2-
f4ce2963772b --error=FAULTY --update_only
When the system is brought back online
Jan
2010
Sept
2014

HA @ The MySQL Fabric Node
Jan
2010
Sept
2014
● Fabric Node Fails
● Crash-Safe Procedures
● State Store Fails
● Safeguarding meta-data
● Both Fabric and State Store Fails

Crash Safe Procedures
Jan
2010
Sept
2014
● Each Fabric Operation is a procedure
● Procedure consists of a set of steps/jobs
● Steps/Jobs
● Can be Rolled Back
● Are Idempotent

State Store failure
Jan
2010
Sept
2014
● Storing Data in MySQL Cluster
● Storing Data in replicated Storage

Future Work
Jan
2010
Sept
2014
● Ideas for the future

Detecting and Restarting when Fabric Fails
Jan
2010
Sept
2014
● Detecting Fabric Failure
● Using Pacemaker
● Avoiding split-brain
● Re-Starting Fabric
● Providing script for restart

Handling Failure of MySQL Fabric and State Store
Jan
2010
Sept
2014
● Building a replicated state machine
● Multiple Fabric instance that can take over from any of
the other instances
● Uses Paxos/Raft-like implementation

Handling Connector Failover when Fabric goes down
Jan
2010
Sept
2014
● Alternate Fabric Addresses
– Connector has a list of potential addresses
– Try these addresses when a Fabric node fails
● Connector notices server failure
– PRIMARY failure
– SECONDARY failure
– All servers fail

Error reporting when Fabric goes down
Jan
2010
Sept
2014
● PRIMARY failure
– Report error to application
– Remove from cache
– Set group to read only
● SECONDARY failure
– Remove from cache
– Pick alternate server
● All server failure
– Report Error to Application

Reading for the Interested
● MySQL Forum: Fabric, Sharding, HA,
Utilities
http://forums.mysql.com/list.php?144
● MySQL Fabric Documentation
http://dev.mysql.com/doc/mysql-utilities/1.4/en/fabric.htm
l
● Migrating From an Unsharded to a Sharded
Setup
http://vnwrites.blogspot.com/2013/09/mysqlfabric-shardin
g-migration.html
● Configuring and running MySQL Fabric
http://alfranio-distributed.blogspot.com/2014/03/mysqlfabric-installation.html

Want to contribute?
● Check it
– … and send us use-case and feature suggestions
● Test it
– … and send comments to the forum
● Break it
– … and send in bugs to http://bugs.mysql.com

Keeping in Touch
Mats Kindahl
Twitter: @mkindahl
http://mysqlmusings.blogspot
.com
Alfranio Correia
Twitter: @alfranio
http://alfranio-distributed.blo
gspot.com
Narayanan
Venkateswaran
Twitter: @vn_tweets
http://vnwrites.blogspot
.com
Geert Vanderkelen
Twitter: @geertjanvdk
http://geert.vanderkelen.
org

Thank You !
Jan
2010
Sept
2014
2x Engineering Staf
3x QA Staf
2x Support Staf

MySQL High Availability - Managing Farms of Distributed Servers

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (8)

Similar to MySQL High Availability - Managing Farms of Distributed Servers

Similar to MySQL High Availability - Managing Farms of Distributed Servers (20)

Recently uploaded

Recently uploaded (20)

MySQL High Availability - Managing Farms of Distributed Servers