MySQL Fabric: Easy Management of MySQL Servers

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1
MySQL Fabric:
Easy Management of MySQL Servers
杜修文 Ivan.Tu@Oracle.Com
MySQL Principal Sales Consultant
Oracle LLC
July 17, 2014

2
Insert Picture Here
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |
An extensible and
easy-to-use framework
for managing a farm of
MySQL server supporting
high-availability and
sharding
MySQL Fabric

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |3
MySQL Fabric: Goals & Features
 Connector API Extensions
 Support Transactions
 Support full SQL
 Decision logic in connector
 Reducing network load
 Load Balancing
 Read-Write Split
 Distribute transactions
 Global Updates
 Global tables
 Schema updates
 Shard Multiple Tables
 Using same key
 Sharding Functions
 Range
 (Consistent) Hash
 Shard Operations
 Using built-in executor
 Shard move
 Shard split

Transaction Handling

 Routing Transactions
 Pre-declare properties of transactions
 Detecting transaction boundaries
 Push as much as possible to server
 Managing Session State
 Move session state between servers
− Easy to use
− Expensive and error prone
 Reset state after each transaction
− Transactions start with default session state
Transaction Handling

Routing Transactions
Shard #2
Shard #1
Shard #3
Switch
State
Store Executor
QUERY
KEY
KEY
QUERY
Contain decision logic
for routing queries
Contain information
about location of shards
SHARD#

Birds-eye View
High Availability Groups (Shards)
MySQL Fabric
Node
Application
XML-RPC
SQL
Connector
Connector
Connector
Operator

MySQL Fabric: Prerequisites
 MySQL Servers (version 5.6.10 or later)
 Server for meta-data backing store
 Servers being managed
 Python 2.6 or 2.7
 No support for 3.x yet
 MySQL Utilities 1.4.0
 Available at http://labs.mysql.com/

MySQL Fabric: Configuration
 Backing Store
 MySQL server
 Persistent storage for state
 Storage engine-agnostic
 Protocol
 Address where node will be
 Currently only XML-RPC
 Logging
 Chatty: INFO (default)
 Moderate: WARNING
 URL for rotating log
[storage]
address = localhost:3306
user = fabric
password =
database = fabric
connection_timeout = 6
[protocol.xmlrpc]
threads = 5
[logging]
level = INFO
url =
file:///var/log/fabric.log

10
Insert Picture Here
Connecting to a
MySQL Fabric Farm

Fabric-aware Connector API
 Load Balancing
 Read-Write Split
 Distribute transactions
 Fabric-aware Connectors
 Connector/J
 Connector/Python
 Connector/PHP
 Fabric-aware Frameworks
 Doctrine
 Hibernate
 Focus on Connector/Python

12
Insert Picture Here
Architecture for
Sharding

Benefits of Sharding
 Write scalability
 Can handle more writes
 Large data set
 Database too large
 Does not fit on single server
 Improved performance
 Smaller index size
 Smaller working set
 Improve performance
UID 10000-20000 UID 20001-40000

MySQL Fabric: Sharding Goals & Features
 Shard Multiple Tables
 Using same key
 Global Updates
 Global tables
 Schema updates
 Sharding Functions
 Range
 (Consistent) Hash
 Shard Operations
 Using built-in executor
 Shard move
 Shard split

Mapping the Sharding Key
 What is a sharding key?
 Single column
 Multi column
− Same table?
− Different tables?
 How is the key transformed?
 Hash
 Range
 User-defined
Compute
Shard#
KeyShard#
(X)
(X,Y,...)
RANGE
HASH
Something else

Sharded Tables
In desperate need
of sharding!
Foreign keys

Multi-table Query with Sharded Tables
SELECT first_name, last_name, salary
FROM salaries JOIN employees USING (emp_no)
WHERE emp_no = 21012
AND CURRENT_DATE BETWEEN from_date AND to_date;
 Referential Integrity Constraint
 Example query joining salaries and employees
 Same key, same shard: co-locate rows for same user
 JOIN normally based on equality
 Using non-equality defeats purpose of foreign key

Global Tables
Does not need
to be sharded

Multi-table Query with Global Tables
SELECT first_name, last_name, GROUP_CONCAT(dept_name)
FROM employees JOIN dept_emp USING (emp_no)
JOIN departments USING (dept_no)
WHERE emp_no = 21012 GROUP BY emp_no;
 JOIN with departments table
 Has no employee number, hence no sharding key
 Table need to be present on all shards
 How do we update global tables?

Sharding Architecture
Shards
MySQL Fabric Node
Application
Global
Group
Global Updates
Shard
Updates
Replication
Support global update
for off-line shards

Demo

Demo Server Side – Startup Instances
3306
3606
Fabric
Repository
3506
Fabric Node
32274
3308 3314 3408 3406

Demo Server Side – Configuration of Fabric
/etc/mysql/fabric.cfg
[DEFAULT]
prefix =
sysconfdir = /etc
logdir = /var/log
[storage]
user = fabric
password = welcome1
database = fabric
auth_plugin = mysql_native_password
connection_timeout = 6
connection_attempts = 6
connection_delay = 1
[servers]
user = fabric
password = welcome1
[protocol.xmlrpc]
threads = 5
user = admin
password =welcome1
disable_authentication = no
realm = MySQL Fabric
ssl_ca =
ssl_cert =
ssl_key =
[executor]
executors = 5
[logging]
level = INFO
url = file:///var/log/fabric.log

Startup Fabric
•Fabric help command
mysqlfabric help commands
•Instantiate Fabric
mysqlfabric manage setup
•Bring up Fabric
mysqlfabric manage start --deamonize

Demo – Configure by Fabric
GP1
3306
GP-Global
3606
GP2
Fabric
Repository
3506
Fabric Node
32274
3308 3314 3408 3406

Fabric Group Operation
 Create 1 global group, 2 shard group
mysqlfabric group create gp-global
mysqlfabric group create gp1
mysqlfabric group create gp2
 Assign Servers to groups
mysqlfabric group add gp-global 127.0.0.1:3606
mysqlfabric group add gp1 127.0.0.1:3306
 Pick the master server by Fabric
mysqlfabric group promote gp1
mysqlfabric group promote gp2

Fabric Group Operations
• Let Fabric manage auto-failover for you
mysqlfabric group activate gp-global
mysqlfabric group activate gp1
mysqlfabric group activate gp2
• Define Range shard on to global group
mysqlfabric sharding create_definition RANGE gp-global
• Add shard table – employees to the shard created (shard map
id = 1) shard by id column
mysqlfabric sharding add_table 1 employees.employees id
• Assign rows with id value 1~999 to group gp1, 1000 above to
group gp2
mysqlfabric sharding add_shard 1 gp1/1,gp2/1000 --state=ENABLED

Demo – Databases is Ready
GP1
3306
GP-Global
3606
GP2
Fabric
Repository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406

Demo Application Side
– Need Connector/J 5.1.30 or later
GP1
3306
GP-Global
3606
GP2
Fabric
Repository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406
JavaConnector/J

Demo Application Side
– Cache Fabric Configuration on creating DB Connection
GP1
3306
GP-Global
3606
GP2
Fabric
Repository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406
JavaConnector/J
Connection
Object

Java Application
- Global tables and DDLs are accessed by property fabricServerGroup
import com.mysql.fabric.jdbc.FabricMySQLConnection;
…
rawConnection = DriverManager.getConnection( baseUrl +
database + "?fabricServerGroup=gp-global“,user, password);
statement = rawConnection.createStatement();
…
statement.executeUpdate("create table employees.employees (emp_no int
not null,first_name varchar(50),last_name varchar(50), department_id
int,primary key (emp_no))");
…
statement.executeUpdate("insert into employees.departments values
(1,'Sales'),(2,'Financial')");

Demo Environment
– Dispatch Request Based On Cached Configuration with Connection
GP1
3306
GP-Global
3606
GP2
Fabric
Repository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406
JavaConnector/J
Connection
Object –
Shard Table
Write Sharded Table
Write Global Table

Demo Environment
GP1
3306
GP-Global
3606
GP2
Fabric
Repository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406
JavaConnector/J
Connection
Object –
Shard Table
Write Sharded Table
Write Global Table

Shared tables are accessed by property
fabricShardTable and fabericShardKey
rawConnection =
DriverManager.getConnection( "jdbc:mysql:fabric://127.0.
0.1:32274/mysql?fabricShardTable=employees.employees“, user,
password);
…
for (int i = 0; i < 4; ++i) {
connection.setShardKey(ids[i].toString());
ps.setInt(1, ids[i]);
ps.setString(2, firstNames[i]);
ps.setString(3, lastNames[i]);
ps.setString(4, departmentIds[i]);
ps.executeUpdate();
}

Demo Environment
– Read/Write Split within Shard
GP1
3306
GP-Global
3606
GP2
Fabric
Repository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406
JavaConnector/J
Connection
Object –
Shard Table
Read Only Operations
Read Global Table

Read Only Operations
FabricMySQLConnection newConn = (FabricMySQLConnection)
rawConnection;
newConn.setReadOnly(true);
ps = newConn.prepareStatement(
"select emp_no, first_name, last_name, d.name, i.variable_value
from employees.employees e, information_schema.global_variables i,
employees.departments d where e.department_id = d.department_id
and i.variable_name='port' and emp_no = ?");

Demo Environment – Failure Detected
GP1
3306
GP-Global
3606
GP2
Fabric
Repository
3506
Fabric Node
32274
3308 3314-M 3408-M 3406
JavaConnector/J
Connection
Object
Write Global Table
Change Master to 3606
Change Master to 3308
X

Demo Environment – Failover within Shard
GP1
3306
GP-Global
3606
GP2
Fabric
Repository
3506
Fabric Node
32274
3308-M 3314-M 3408-M 3406
JavaConnector/J
Connection
Object
Write Shard Table
Write Global Table
X

40
Insert Picture Here
Closing Remarks

Thoughts for the Future
 Connector multi-cast
 Scatter-gather
 UNION of result sets
 More complex operations?
 Internal interfaces
 Improve extension support
 Improve procedures
support
 Command-line interface
 Improving usability
 Focus on ease-of-use
 More protocols
 MySQL-RPC Protocol?
 AMQP?
 More frameworks?
 More HA group types
 DRBD
 MySQL Cluster
 Fabric-unaware connectors?

MySQL Fabric: Easy Management of MySQL Servers

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to MySQL Fabric: Easy Management of MySQL Servers

Similar to MySQL Fabric: Easy Management of MySQL Servers (20)

More from Ivan Tu

More from Ivan Tu (13)

MySQL Fabric: Easy Management of MySQL Servers