High-Availability using MySQL Fabric

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
High Availability using MySQL Fabric:
Managing Farms of Servers
Mats Kindahl
Copyright © 2015, Oracle and/or its affiliates. All rights reserved.

Copyright © 2015, Oracle and/or its affiliates. All rights reserved.2
The following is intended to outline our general product direction. It is intended
for information purposes only, and may not be incorporated into any contract.
It is not a commitment to deliver any material, code, or functionality, and
should not be relied upon in making purchasing decision. The development,
release, and timing of any features or functionality described for Oracle’s
products remains at the sole discretion of Oracle.
Safe Harbor Statement

Program Agenda
 Building reliable systems
 MySQL Fabric overview
 Managing redundancy
 Procedure automation and the Executor
 Failure detection and failure handling

Program Agenda
 Using Fabric with existing high-availability setups
 Making Fabric highly available
 Thoughts for the future
 Closing remarks

Insert Picture Here
Building Reliable Systems
Insert Picture Here

Copyright © 20135 Oracle and/or its affiliates. All rights reserved.6
Insert Picture Here
High-availability is an
integral part of designing a
reliable system
Building for reliability

What causes downtime?
●
System failures
●
Hardware faults
●
Software bugs
●
Disasters
●
Maintenance
●
User errors

High-availability concepts
●
Redundancy
●
Duplicate critical components
●
Monitoring
●
Detecting failing components
●
Monitor load
●
Procedures
●
Activate replacements
●
Distribute load

High-availability solutions
●
Primary-seconday approach
●
MySQL Replication
●
Shared-nothing clusters
●
MySQL Cluster
●
MySQL Group Replication (not
GA)
●
Tightly coupled clusters
●
DRBD
●
WSFC
●
Solaris Clustering
●
Oracle Clusterware
●
Oracle VM High Availability

Insert Picture Here
MySQL Fabric Overview

What is MySQL Fabric?
An extensible and easy-to-use
framework for managing a farm of
MySQL servers supporting high-
availability and sharding.

What does it mean?
●
Management system
●
Manages a MySQL Farm
●
Distributed framework
●
Framework
●
Procedure execution
●
State store
●
Transaction Routing
●
Extensible
●
High-availability groups
●
Sharding
●
Cloud support
●
Written in Python
●
MySQL 5.6 (or later)
●
Open Source
●
You can participate

Birds-eye view
MySQL
Fabric Node
Application
Operator
High-Availability Groups
(Shards)

MySQL Fabric Components
●
Fabric-aware connectors
●
Enhanced Connector API
●
Python, PHP, Java, .NET, C
●
MySQL Fabric controller
●
Manage farm meta-data
●
Provide status information
●
Execute Procedures
●
MySQL servers
●
Organized in high-availability groups
●
Handle application data
MySQL Fabric
controller node
High-availability
group
Application with
Fabric-aware connectors

MySQL Fabric Controller Architecture
XML-RPC
MySQL-RPC
AMQP
Protocol Server
XML-RPC
MySQL-RPC
AMQP
Protocol Server
Sharding
Master-Slave
Providers
Extensions
State
Store
XML-RPC
MySQL-RPC
AMQP
Protocol Server
Executor
Model
Persistance
Fabric Core
Requests
Events
Results
Eample only!
Eample only!

Insert Picture Here
Managing Redudancy

High-Availability Group Concept
● Group of servers
● Hardware redundancy
● Data redundancy
● Generic Concept
● Implementation-independent
● Self-managed or externally managed
● Different Types
● Primary-Backup (Master-Slave)
● Shared or Replicated Storage
● MySQL Cluster
DRBD
ndbdndbd
ndbd ndbd
Default
Eamples Only
Not Implemented

●
Create a logical group for the servers
●
Empty initially
mysqlfabric group create my_group
--description='My Group'
Creating a high-availability group
Create an empty group

●
Add servers to group
●
Group will have no master
●
All servers are secondaries (!)
Adding servers to the group
mysqlfabric group add my_group server1.example.com

●
Promote one secondary to primary
●
Selects secondary at random
●
Specific secondary can be selected
Promote a primary
mysqlfabric group promote my_group
mysqlfabric group promote my_group
--slave_id='server1.example.com'

●
Enable built-in failure detector
●
Monitor servers in group
Enable failure detector
mysqlfabric group activate my_group

●
On primary failure
●
Mark primary as faulty
●
Trigger fail-over
●
On secondary failure
●
Mark secondary as faulty

Insert Picture Here
Procedure Automation and
the Executor

●
Management Procedures
●
Fail-over
●
Slave promotion
●
Shard split
●
Triggered on events
●
Crashing server
●
Administrative decision
●
Increasing load
●
Resilient execution
●
Controller node can crash
●
Recover partially executed procedure
Automating management of a farm
Find
Candidate
Check
Candidate
Disable
Read-only
Process
Backlog
Re-direct
Slaves
SLAVE_PROMOTED
SERVER_LOST

MySQL Fabric executor
●
Event driven executor
●
Events will trigger execution of procedures
●
Procedures can trigger events themselves
●
Each step of a procedure is called a job
●
Procedures
●
Written in Python
●
Interacts with servers
●
Write state changes into backing store
●
Lock manager for conflict resolution
– Conservative 2PL
– Avoid deadlocks
Queue
Backing
Store
Events

Example: keep high-availability profile

●
Register job for event
●
@on_event decorator
●
Register job with event
●
Fetch group of lost server
●
Fetch new server from provider
●
Add server to group
@on_event(SERVER_LOST)
def _add_server(group_id, server_uuid):
group = Group.fetch(group_id)
machines = PROV.create_machines(
parameters
)
server = MySQLServer(
server_uuid, address
)
MySQLServer.add(server)
group.add(server)
_configure_as_slave(server)
Automating adding a server

●
Before starting a job:
●
Aquire the necessary locks
●
Checkpoint execution state in backing store
●
Start a transaction on the backing store
●
When executing job:
●
Updates to backing store inside transaction
●
Interact with servers
●
After executing a job:
●
Mark job completed in internal log
●
Commit transaction on backing store
What about crashes?
Queue
Backing
Store
Events
MySQL Fabric execution flow

●
Two types of jobs:
●
Idempotent: Restart the job
●
Not idempotent: Execute compensation
●
Recovery procedure
●
Start the executor
●
Collect unfinished checkpoints
●
Execute compensation actions
… if there are any
●
Re-schedule each job in checkpoint
Queue
Backing
Store
Events
MySQL Fabric executor recovery

Insert Picture Here
Failure detection and
failure handling

●
Group level detection
●
Fabric node ping servers in group
●
Servers need to be managed by Fabric
●
On primary failure
●
Mark primary as faulty
●
Trigger fail-over of connectors and slaves
●
On secondary failure
●
Mark secondary as faulty
Built-in failure detector

Built-in failure detector
Configuration
● Detections
● Number of failed pings before
marked as faulty
● Detection Interval
● Interval between server ping,
in seconds
● Detection Timeout
● Timeout for ping, in seconds
[failure_tracking]
detections = 3
detection_interval = 6
detection_timeout = 1

●
External failure detectors
●
Connectors
●
Custom failure detectors
●
Reporting API
●
Error: suspected server failure
●
Failure: server is known to have failed
●
Reporting server error
●
Trigger fail-over if threshold is exceeded
●
Reporting server failure
●
Trigger immediate fail-over
?
!
MySQL Fabric
controller node

Configuration
● Notifications
● Error threshold
● Notification clients
● Threshold for number of
unique clients
● Notification interval
● Notification window
[failure_tracking]
notifications = 300
notification_clients = 50
notification_interval = 60
failover_interval = 0
prune_time = 3600

Configuration
● Failover interval
● Minimum interval between
failovers
● Used to prevent flapping
● Prune time
● Size of error log (in seconds)
to keep
[failure_tracking]
notifications = 300
notification_clients = 50
notification_interval = 60
failover_interval = 0
prune_time = 3600

Connector as external failure detector
● Error reporting from connector
● Depends on connector support
● Report suspected failures
● Enabling error reporting
● Error reporting off by default
● Avoid a thundering herd
● Do not enable error reporting
for all connectors!
● Failing server will cause all
connectors to report failure
fabric_config = {
…
'report_errors': True,
…
}
cnx = connect(
…
fabric=fabric_config
…
)

Error reporting
● Default errors reported
● Extra errors can be added
● extra_failure_report
CR_SERVER_LOST
CR_SERVER_GONE_ERROR
CR_CONN_HOST_ERROR
CR_CONNECTION_ERROR
CR_IPSOCK_ERROR
from mysql.connector.fabric import extra_failure_report
extra_failure_report([error1, error2, …, errorn])

Cache invalidation
● Cache invalidation by default on
● Server Lost (CR_SERVER_LOST)
● Server read-only (ER_OPTION_PREVENTS_STATEMENT)
from mysql.connector.fabric import RESET_CACHE_ON_ERROR
RESET_CACHE_ON_ERROR.append(error)

Insert Picture Here
Using Fabric with Existing
High-availability Setups

Using Fabric with Existing Solution
● Servers already managed
● Group Based Solutions
● Virtual IP-based solutions
● Fabric as lookup server
● Connectors can route transactions
● Application can retrieve information from Fabric
● Update state store only

Example: An existing setup
● DRBD for redundancy
● Disk replicated
● Pacemaker for fail-over
● Heartbeat detect failure
● Resource agent handle
fail-over
● Fabric as lookup server
● Fabric for routing
transactions
Secondary
Node
Primary
Node
DRBD Replication
Pacemaker Pacemaker

Create a group
● Create a group
● Add server to group
● Fabric should only update state store
● “Promote” the DRBD primary to be primary in group
mysqlfabric group create my_group
mysqlfabric group add my_group server1.example.com --update_only
mysqlfabric group promote my_group --update_only
--slave_id=...

Update resource agent
● Change resource agent script
● On Ubuntu: /usr/lib/ocf/resource.d/heartbeat/mysql
● Update resource agent actions to inform Fabric
● Remove old server
● Only update the state store
mysqlfabric group demote --update_only
--slave_id=7bcb0804-...
mysqlfabric group remove --update_only
7bcb0804-...

Update resource agent
● Change resource agent script
● On Ubuntu: /usr/lib/ocf/resource.d/heartbeat/mysql
● Update resource agent actions to inform Fabric
● Add standby server
● Only update the state store
mysqlfabric group add --update_only standby.example.com
mysqlfabric group promote --update_only
8308b0c4-...

Insert Picture Here
Making Fabric highly
available

Making Fabric highly available
● Standard deployment
● Fabric node and state store on
same machine
● Need to use TCP
– Socket connection not
available yet (Bug#71946)
● Three things can fail:
● State store
● Fabric node
● Machine

Handling state store failure
● If state store connection is lost:
● Fabric retry until state store
becomes available
● Ongoing transactions fail
● Fabric report error if
connection not recovered
“quickly enough”
● Solution: restart state store
● MySQL handle recovery
● Fabric re-connect
automatically

Handling state store failure
● Connection timeout
● Timeout (in seconds) for
connection attempt
● Connection attempts
● Number of attempts before
reporting state store failed
● Connection interval
● Delay (in seconds) between
connection attempts
[storage]
connection_timeout = 6
connection_attempts = 6
connection_interval = 1

Handling Fabric controller node failure
● If Fabric node is lost:
● Ongoing jobs fail
● Execution state checkpointed
● On Fabric node restart:
● Execution state recovered
● Solution: restart Fabric node
● Detect failure
– Local ping script
● Restart Fabric node
– init.d script
● Neither distributed with Fabric

Making Fabric Highly Available
Handling machine failures
● If the machine fails:
● State store is lost
● Fabric node is lost
● Catastrophic failures can
prevent machine recovery
● Solution:
● Replicate meta-data
● Detect machine failure
● Activate duplicate deployment

Replicate meta-data
● Replicate state store
● DRBD
● MySQL Cluster
● MySQL Replication
● Configure DRBD
● Version 8.3 or later
● Replicate block device
● Configure MySQL Servers
● Data directory on replicated
device

Replicate meta-data
● Active node
● MySQL Fabric
● MySQL Server
● DRBD primary
● Passive node
● DRBD secondary
● Server and Fabric started on
fail-over

Detect machine failure & activate replacement
● Detecting machine failure
● Corosync
● Activate Replacement
● Pacemaker

● Configure MySQL Fabric
● State store in DRBD volume
● Configure Corosync
● Set no-quorum-policy to
'ignore'
– Prevent remaining node to
shut down
● Turn off STONITH
– Node will commit suicide

● Configure Pacemaker
● Add MySQL Fabric resource
agent
● Colocate Fabric, DRBD, and
MySQL and order them
● Avoiding split-brain
● Reliably detect network
partition
● Ping reliable resource
– Example: Router

Insert Picture Here
Closing Remarks &
Ideas for the Future

Multi-Node Fabric
Replicated State Machine
● Multiple Fabric Nodes
● Built-in support
● Fail-over
● Local read instance
● Distributed execution
● Replicated State Machine
● Coordinate procedure execution
● Automatic fail-over
● Paxos or Raft-like implementation

More Flexibility
● Server Providers
● Amazon AWS
● Kubernetes?
● Built-in high-availability group types
● DRBD
● MySQL Cluster
● Amazon RDS?

MySQL Fabric Resources
Useful links
● Download and try
● http://dev.mysql.com/downloads/utilities/
● MySQL Fabric Documentation
● http://dev.mysql.com/doc/mysql-utilities/1.5/en/fabric.html
● Forum (MySQL Fabric, Sharding, HA, Utilities)
● http://forums.mysql.com/list.php?144

MySQL Fabric Resources
Blogs
● MySQL High-Availability
● http://mysqlhighavailability.com
● Mats Kindahl
● http://mysqlmusings.blogspot.
com
● Alfranio Correia
● http://alfranio-distributed.blog
spot.com
● Narayanan Venkateswaran
● http://vnwrites.blogspot.com

Thank You!

High-Availability using MySQL Fabric

More Related Content

What's hot

Similar to High-Availability using MySQL Fabric

More from Mats Kindahl

Recently uploaded

High-Availability using MySQL Fabric