Your SlideShare is downloading. ×
0
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.1
Sharding and Scale-out
using MySQL Fabric
Mats Kinda...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |2
The following is int...
3 Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |
Presentation Outlin...
4
Insert Picture Here
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |5
What does all that m...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |6
MySQL Fabric: Goals ...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |7
A Brief History of M...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |8
Birds-eye View
High ...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |9
High-Level Component...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |10
MySQL Fabric Node A...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |11
MySQL Fabric: Prere...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |12
MySQL Fabric: Confi...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |13
MySQL Fabric: Basic...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |14
Setting up and Tear...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |15
Starting and Stoppi...
16
Insert Picture Here
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 ...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |17
High-Availability C...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |18
High-Availability G...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |19
High-Availability G...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |20
Create Groups and a...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |21
Activate High-Avail...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |22
Distributed Failure...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |23
Update Only Operati...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |24
Credentials in MySQ...
25
Insert Picture Here
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 ...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |26
● Fabric-aware Conn...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |27
Routing Transaction...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |28
Fabric-aware Connec...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |29
Enable Connector/Py...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |30
Connector API: Exec...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |31
START TRANSACTION;
...
33
Insert Picture Here
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 ...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |34
Benefits of Shardin...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |35
MySQL Fabric: Shard...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |36
Sharded Tables
Tabl...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |37
Mapping the Shardin...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |38
Sharded Tables: Mul...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |39
Digression: Computi...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |40
Multi-table Query w...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |41
Global Tables
Table...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |42
Multi-table Query w...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |43
Sharding Architectu...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |44
MySQL Fabric: Shard...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |45
MySQL Fabric: Set u...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |46
MySQL Fabric: Add S...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |47
MySQL Fabric: Movin...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |48
Connector API: Shar...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |49
Connector API: Shar...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |50
Connector API: Glob...
51
Insert Picture Here
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 ...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |52
What do we have now...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |53
Thoughts for the Fu...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |54
Thoughts for the Fu...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |55
Reading for the Int...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |56
Want to contribute?...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |57
Mats Kindahl
Twitte...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.58
Graphic Section Divider
MySQL Connect @ Oracle
Open...
Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |59
Thank you!
Upcoming SlideShare
Loading in...5
×

Sharding and Scale-out using MySQL Fabric

3,546

Published on

MySQL Fabric is an open-source solution released by the MySQL Engineering team at Oracle. It make management of farms of MySQL servers easy and available for both applications with small and large number of servers.

This is the presentation from Percona Live Santa Clara.

Published in: Data & Analytics, Technology
1 Comment
6 Likes
Statistics
Notes
  • I create 3 groups, global-group, group_id-1 and group_id-2, then built a shard. Till now, everything is OK. But when i split the shard with a new group named group_id-3, there is an error occurred. The message is as follows:
    # mysqlfabric sharding split_shard 2 group_id-3 –split_value=100000
    Password for admin:
    Procedure :
    { uuid = a0eaff93-f6fe-4a7d-9c44-e7e53882fbfe,
    finished = True,
    success = False,
    return = BackupError: (‘Error while restoring the backup using the mysql client\n, %s’, “ERROR 1840 (HY000) at line 24 in file: ‘MySQL_132.228.239.19_3316.sql’: @@GLOBAL.GTID_PURGED can only be set when @@GLOBAL.GTID_EXECUTED is empty.\n”),
    activities =
    }
    I run the “reset master” command first, but the error is still there.what is this error?

    Regards
    Shen
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
3,546
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
112
Comments
1
Likes
6
Embeds 0
No embeds

No notes for slide

Transcript of "Sharding and Scale-out using MySQL Fabric"

  1. 1. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.1 Sharding and Scale-out using MySQL Fabric Mats Kindahl (mats.kindahl@oracle.com) Principal Senior Software Developer
  2. 2. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |2 The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decision. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. Safe Harbor Statement
  3. 3. 3 Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 | Presentation Outline  Introducing MySQL Fabric  Architecture for High-Availability  Connecting to a MySQL Fabric Farm  Architecture for Sharding  Summary and Closing Remarks
  4. 4. 4 Insert Picture Here Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 | An extensible and easy-to-use framework for managing a farm of MySQL servers supporting high-availability and sharding MySQL Fabric
  5. 5. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |5 What does all that mean? ● “Farm” Management System ● Farm: Collection of components ● Distributed Framework ● Framework ● Procedure execution ● State store ● Transaction Routing ● Extensible ● Extensions are first-class ● High-Availability Groups ● “Semi-Automatic” Sharding ● Written in Python ● Latest Release 1.4.2 ● Release Candidate ● Open Source ● You can participate ● Suggest features ● Report bugs ● Contribute patches ● MySQL 5.6 is focus
  6. 6. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |6 MySQL Fabric: Goals & Features ● Decision logic in connector ● Eliminate one network hop ● Reducing network load ● Connector API Extensions ● Support Transactions ● Support full SQL ● Load Balancing ● Read-Write Split ● Round-robin ● Multi-Table Sharding ● Sharding Functions ● Range ● (Consistent) Hash ● Shard Operations ● Shard move ● Shard split ● Global Updates ● Global tables ● Schema updates
  7. 7. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |7 A Brief History of MySQL Fabric ● MySQL Fabric 1.4.0 ● September, 2013 ● First public release ● High-Availability Groups ● Slave Promotion ● Range and Hash Sharding ● Shard move and split ● Connector/Python Support ● Connector/J Support ● Connector/PHP Support ● MySQL Fabric 1.4.1 ● December, 2013 ● Alpha release ● Sharding refactorings ● MySQL Fabric 1.4.2 ● April, 2014 ● Release Candidate ● Distributed Failure Detection ● Credentials ● Weighted Round-Robin
  8. 8. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |8 Birds-eye View High Availability Groups Application XML-RPC SQL SQL Connector Connector Connector Operator MySQL Fabric Node Database Servers
  9. 9. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |9 High-Level Components ● Fabric-aware Connectors ● Python, PHP, and Java ● Enhanced Connector API ● MySQL Fabric Node ● Manage information about farm ● Provide status information ● Execute procedures ● MySQL Servers ● Organized in High-Availability Groups ● Handling application data High Availability Group Application Connector Connector Connector MySQL Fabric Node
  10. 10. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |10 MySQL Fabric Node Architecture MySQL MySQL Fabric Framework Executor State Store (Persister) Sh ?HA MySQL-RPCAMQP XML-RPC ?? Connector Connector Connector Protocols Extensions Backing Store
  11. 11. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |11 MySQL Fabric: Prerequisites ● MySQL Servers (version 5.6.10 or later) ● Backing store database server ● Application database servers ● Python 2.6 or 2.7 ● No support for 3.x yet ● MySQL Utilities 1.4 ● Available at https://dev.mysql.com/downloads/tools/utilities ● “Development release” tab
  12. 12. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |12 MySQL Fabric: Configuration ● Backing Store ● MySQL server ● Persistent storage for state ● Storage engine-agnostic ● Protocol ● Address where node will be ● Currently only XML-RPC ● Logging ● Chatty: INFO (default) ● Moderate: WARNING ● URL for rotating log [storage] address = localhost:3306 user = fabric password =  database = fabric [servers] user = fabric password =  [protocol.xmlrpc] address = localhost:32274 threads = 5 disable_authentication = yes [logging] level = INFO url = file:///var/log/fabric.log
  13. 13. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |13 MySQL Fabric: Basic Commands and Help ● Command Structure mysqlfabric group command ... ● Getting help mysqlfabric help mysqlfabric help commands mysqlfabric help manage mysqlfabric help manage setup ● MySQL Utilities Documentation: ● http://dev.mysql.com/doc/mysql-utilities/1.4/en/index.html ● MySQL Fabric Documentation: ● http://dev.mysql.com/doc/mysql-utilities/1.4/en/fabric.html
  14. 14. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |14 Setting up and Tearing down MySQL Fabric ● Create and populate the necessary tables in backing store mysqlfabric manage setup ● Remove the tables from backing store mysqlfabric manage teardown ● Connects to the database server in “storage” section ● Ensure that you have the necessary users and privileges
  15. 15. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |15 Starting and Stopping MySQL Fabric ● Start MySQL Fabric node in foreground – print log to terminal mysqlfabric manage start ● Start MySQL Fabric node in background – print log to file mysqlfabric manage start ­­daemonize  ● Stop MySQL Fabric node mysqlfabric manage stop
  16. 16. 16 Insert Picture Here Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 | Architecture for High-Availability
  17. 17. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |17 High-Availability Concepts ● Redundancy ● Duplicate critical components ● Monitoring ● Detecting failing components ● Monitor load ● Procedures ● Activate replacements ● Distribute load
  18. 18. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |18 High-Availability Group Concept ● Group of servers ● Hardware redundancy ● Data redundancy ● Generic Concept ● Implementation-independent ● Self-managed or externally managed ● Different Types ● Primary-Backup (Master-Slave) ● Shared or Replicated Storage ● MySQL Cluster Done! Examples Only Not Implemented
  19. 19. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |19 High-Availability Group Concept ● Abstract Concept ● Set of servers ● Server attributes ● Connector Attributes ● Connection information ● Mode: read-only, read-write, ... ● Weight: distribute load ● Management Attributes ● Status: state/role of the server Status: Primary Mode: Read-Write Host: server- 1.example.com
  20. 20. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |20 Create Groups and add Servers ● Define a group mysqlfabric group create my_group ● Add servers to group mysqlfabric group add my_group server1.example.com mysqlfabric group add my_group server2.example.com
  21. 21. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |21 Activate High-Availability Group ● Promote one server to be primary mysqlfabric group promote my_group ● Tell built-in failure detector to monitor group mysqlfabric group activate my_group
  22. 22. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |22 Distributed Failure Detector New in MySQL Fabric 1.4.2 ● Connectors report errors ● Report that an error was noticed ● Failover based on statistics ● report_error(server, source, error) ● Report failure ● A server is known to have failed ● Failover occurs immediately ● report_fault(server, source, error) Connector report_error report_fault
  23. 23. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |23 Update Only Operations New in MySQL Fabric 1.4.2 ● Situation: ● Server promotion is done elsewhere ● Real situation does not match content of state-store ● Problem: ● Need to update state store to match real situation ● Should not touch application servers ● Solution: Use update-only option mysqlfabric group promote my_group ­­update_only   ­­slave_uuid=29bf3b2d­b5ac­11e3­a383­58946b051f64
  24. 24. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |24 Credentials in MySQL Fabric New in MySQL Fabric 1.4.2 ● Credentials ● Digest Authentication ● SSL connection ● Digest Authentication ● RFC 2617 ● Server: Realm ● Client: User + Password ● SSL connection ● Setup Like MySQL Server [storage] address = localhost:3306 user = fabric password = xyzzy database = fabric [protocol.xmlrpc] address = localhost:32274 threads = 5 disable_authentication = no realm = MySQL Fabric user = admin password = xyzzy ssl_ca = /etc/mysql/fabric_ca.pem ssl_key = /etc/mysql/fabric_key.pem ssl_cert = /etc/mysql/fabric_cert.pem
  25. 25. 25 Insert Picture Here Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 | Connecting to a MySQL Fabric Farm
  26. 26. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |26 ● Fabric-aware Connectors ● Connector/J ● Connector/Python ● Connector/PHP ● Fabric-aware Frameworks ● Doctrine ● Hibernate ● In this presentation: ● Connector/Python Fabric-aware Connector ● Connector API Extensions ● Support Transactions ● Support full SQL ● Decision logic in connector ● Reducing network load ● Load Balancing ● Read-Write Split ● Distribute transactions
  27. 27. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |27 Routing Transactions prop1=val1 prop2=val2 execute(query) srv1.example.com:3306 srv2.example.com:3306 srv3.example.com:3307 MySQL Fabric Node Application prop1=val1 prop2=val2 execute(query) Connector Cache srv1.example.com srv2.example.com srv3.example.com
  28. 28. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |28 Fabric-aware Connector API import mysql.connector conn = mysql.connector.connect(     fabric={"host": "fabric.example.com"},     user='mats', password='xyzzy', database="employees" ) ● Establish a “virtual” connection ● Real server connection established lazily ● Provide connection information for the Fabric node ● Connector will fetch information about servers
  29. 29. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |29 Enable Connector/Python Error Reporting New in Connector/Python 1.2.1 import mysql.connector conn = mysql.connector.connect(     fabric={"host": "fabric.example.com"},     user='mats', password='xyzzy', database="employees", report_error=True, ) ● Connectors can report errors to Fabric node ● Enable using report_error ● Defaults to False ● Require MySQL Fabric 1.4.2
  30. 30. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |30 Connector API: Executing a Transaction ● Provide group name ● Property: group ● Fabric will compute candidate servers conn.set_property(group='my_group', mode=MODE_READWRITE) cur = conn.cursor() cur.execute("INSERT INTO employees VALUES (%s,%s,%s)",             (emp_no, first_name, last_name)) cur.execute("INSERT INTO titles(emp_no,title,from_date)"             “ VALUES (%s,%s,CURDATE())",             (emp_no, 'Intern')); conn.commit() ● Provide transaction mode ● Property: mode ● Fabric will pick server in right mode Same as before
  31. 31. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |31 START TRANSACTION; SELECT salary INTO @s FROM salaries WHERE emp_no = 20101; SET @s = 1.1 * @s; INSERT INTO salaries VALUES (20101, @s); COMMIT; BEGIN; CALL update_salary(20202, @s);  COMMIT; Where's the sharding key? Ah, there it is! Session state? Hmm... looks like a read transaction Oops.. it was a write transaction! Transaction done! Clear session state? New transaction! Different connection? What about the session state? Executing a Transaction What about connection pools? Application error? What does this procedure update?
  32. 32. 33 Insert Picture Here Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 | Architecture for Sharding
  33. 33. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |34 Benefits of Sharding ● Write scalability ● Can handle more writes ● Large data set ● Database too large ● Does not fit on single server ● Improved performance ● Smaller index size ● Smaller working set UID 10000-20000 UID 20001-40000
  34. 34. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |35 MySQL Fabric: Sharding Goals & Features ● Connector API Extensions ● Support Transactions ● Support full SQL ● Decision logic in connector ● Reducing network load ● Shard Multiple Tables ● Using same key ● Global Updates ● Global tables ● Schema updates ● Sharding Functions ● Range ● (Consistent) Hash ● Shard Operations ● Using built-in executor ● Shard move ● Shard split
  35. 35. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |36 Sharded Tables Table Rows salaries 284 404 700 titles 44 330 800 employees 30 002 400 dept_emp 33 160 300 dept_manager 2 400 departments 900 In desperate need of sharding! Foreign keys
  36. 36. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |37 Mapping the Sharding Key ● What is a sharding key? ● Single column ● Multi column – Same table? – Different tables? ● How is the key transformed? ● Hash ● Range ● User-defined Compute Shard# KeyShard# (X) (X,Y,...) RANGE HASH Something else
  37. 37. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |38 Sharded Tables: Multiple Mappings Library Schema Employee Schema
  38. 38. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |39 Digression: Computing Shards ● Multiple Mappings ● Which mapping to use? ● Application don't care … but know tables in transaction ● Currently only one mapping ● Computing shard requires ● Tables + sharding key ● Map# + sharding key ● Enhanced Connector API Compute Shard# Key Shard#Map# Compute Map# Tables
  39. 39. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |40 Multi-table Query with Sharded Tables SELECT first_name, last_name, salary FROM salaries JOIN employees USING (emp_no) WHERE emp_no = 21012 AND CURRENT_DATE BETWEEN from_date AND to_date; ● Referential Integrity Constraint ● Example query joining salaries and employees ● Same key, same shard: co-locate rows for same user ● JOIN normally based on equality ● Using non-equality defeats purpose of foreign key
  40. 40. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |41 Global Tables Table Rows salaries 284 404 700 titles 44 330 800 employees 30 002 400 dept_emp 33 160 300 dept_manager 2 400 departments 900 Do not really need to be sharded Global tables exist on all shards!
  41. 41. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |42 Multi-table Query with Global Tables SELECT first_name, last_name, GROUP_CONCAT(dept_name) FROM employees JOIN dept_emp USING (emp_no) JOIN departments USING (dept_no) WHERE emp_no = 21012 GROUP BY emp_no; ● JOIN with departments table ● Has no employee number, hence no sharding key ● Table need to be present on all shards ● But... how do we update global tables?
  42. 42. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |43 Sharding Architecture Shards MySQL Fabric Node Application Connector Connector Connector Global Group Global Updates Shard Updates Replication Support global update for off-line shards
  43. 43. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |44 MySQL Fabric: Sharding Setup ● Set up some groups ● my_global – for global updates ● my_group.* – for the shards ● Add servers to the groups ● Create a shard mapping ● A “distributed database” ● Give information on what tables are sharded ● Add shards ● Mapping keys to shards
  44. 44. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |45 MySQL Fabric: Set up Shard Mapping ● Define shard mapping mysqlfabric sharding    create_definition hash my_global ● Add tables that should be sharded mysqlfabric sharding add_table 1    employees.employees emp_no mysqlfabric sharding add_table 1    employees.salaries emp_no ● Tables not added are considered global Shard map identifier Will return a shard map identifier
  45. 45. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |46 MySQL Fabric: Add Shards ● Add shards to shard mapping mysqlfabric sharding add_shard 1    “my_group.1,...,my_group.N” ­­state=ENABLED Shard map identifier
  46. 46. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |47 MySQL Fabric: Moving and Splitting Shards ● Moving a shard from one group to another mysqlfabric sharding move 5 my_group.5 ● Splitting a shard into two pieces (hash) mysqlfabric sharding split 5 my_group.6 Shard ID
  47. 47. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |48 Connector API: Shard Specific Query ● Provide tables in query ● Property: tables ● Fabric will compute map conn.set_property(tables=['employees.employees', 'employees.titles'],                   key=emp_no) cur = conn.cursor() cur.execute("INSERT INTO employees VALUES (%s,%s,%s)",             (emp_no, first_name, last_name)) cur.execute("INSERT INTO titles(emp_no, title, from_date)"             " VALUES (%s, %s, CURDATE())",             (emp_no, 'Intern')); conn.commit() ● Provide sharding key ● Property: key ● Fabric will compute shard
  48. 48. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |49 Connector API: Shard Specific Query ● Provide tables in query ● Property: tables ● Fabric will compute map conn.set_property(tables=['employees.employees', 'employees.titles'],                   key=emp_no) cur = conn.cursor() cur.execute(     "SELECT first_name, last_name, title"     "  FROM employees JOIN titles USING (emp_no)"     " WHERE emp_no = %d", (emp_no,)) for row in cur:     print row[0], row[1], “,“, row[2] ● Provide sharding key ● Property: key ● Fabric will compute shard Join queries are sent to correct shard and executed there
  49. 49. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |50 Connector API: Global Update ● Provide tables in query ● Property: tables ● Fabric will compute map ● (Not necessary) conn.set_property(tables=['employees.titles'], scope='GLOBAL') cur = conn.cursor() cur.execute("ALTER TABLE employees.titles ADD nickname VARCHAR(64)") ● Set global scope ● Property: scope ● Query goes to global group
  50. 50. 51 Insert Picture Here Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 | Closing Remarks
  51. 51. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |52 What do we have now? ● MySQL Farm Management ● High-Availability ● Sharding ● High-Availability ● Group Concept ● Slave promotion ● Sharding ● Range and hash sharding ● Shard move and shard split ● Connector APIs ● Transaction properties ● “Virtual” connections ● Enhanced Connectors ● Connector/Python ● Connector/PHP ● Connector/J ● Command-line Interface ● XML-RPC Interfaces ● Distributed failure detector ● Connectors report failures ● Custom failure detectors ● Credentials ● RFC 2617 ● SSL support
  52. 52. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |53 Thoughts for the Future ● Connector multi-cast ● Scatter-gather ● UNION of result sets ● More complex operations? ● Extension interfaces ● Improve extension support ● Improve procedures support ● Command-line interface ● Improving usability ● Focus on ease-of-use ● More protocols ● MySQL-RPC Protocol? ● More frameworks? ● More connectors? ● C/C++? ● Fabric-unaware connectors? ● More HA group types ● DRBD ● MySQL Cluster
  53. 53. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |54 Thoughts for the Future ● “Transparent” Sharding ● Single-query transactions? ● Speculative execution? ● Cross-shard join? ● Multiple shard mappings ● Independent tables ● Multi-way shard split ● Efficient initial sharding ● Better use of resources ● High-availability executor ● Node failure stop execution ● Replicated State Machine – Paxos? – Raft? ● Continue execution on other Fabric node ● Session Consistency ● We have a distributed database ● It should look like a single database
  54. 54. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |55 Reading for the Interested ● MySQL Forum: Fabric, Sharding, HA, Utilities http://forums.mysql.com/list.php?144 ● MySQL Fabric Documentation http://dev.mysql.com/doc/mysql-utilities/1.4/en/fabric.html ● Migrating From an Unsharded to a Sharded Setup http://vnwrites.blogspot.com/2013/09/mysqlfabric-sharding-migration.html ● Configuring and running MySQL Fabric http://alfranio-distributed.blogspot.com/2014/03/mysqlfabric-installation.html
  55. 55. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |56 Want to contribute? ● Check it … and send us use-case and feature suggestions ● Test it … and send comments to the forum ● Break it … and send in bugs to http://bugs.mysql.com
  56. 56. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |57 Mats Kindahl Twitter: @mkindahl http://mysqlmusings.blogspot.com Alfranio Correia Twitter: @alfranio http://alfranio-distributed.blogspot.com Keeping in Touch Narayanan Venkateswaran Twitter: @vn_tweets http://vnwrites.blogspot.com Geert Vanderkelen Twitter: @geertjanvdk http://geert.vanderkelen.org
  57. 57. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.58 Graphic Section Divider MySQL Connect @ Oracle OpenWorld Sept 29 – Oct 2 Special Offer: Register Before May 2 and Save an Extra US$100! Code: DPER Call for Proposals Open Until April 15!
  58. 58. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.| Percona Live | April 3, 2014 |59 Thank you!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×