0
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1
Insert Picture Here
2013-09-21
MySQL Sharding:
Tools...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.2
2013-09-21
The following is intended to outline our ...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.3
Program Agenda
 Handling Scaling
 What is sharding...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.4
2013-09-21
What is
this
sharding?
What are
the benef...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.5
2013-09-21
It's all about scaling...
●
Start with a ...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.6
2013-09-21
It's all about scaling...
●
Start with a ...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.7
2013-09-21
It's all about scaling...
●
Start with a ...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.8
2013-09-21
It's all about scaling...
●
Start with a ...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.9
2013-09-21
It's all about scaling...
●
More and more...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.10
2013-09-21
It's all about scaling...
●
More and mor...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.11
2013-09-21
It's all about scaling...
●
More and mor...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.12
2013-09-21
It's all about scaling...
●
More and mor...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.13
2013-09-21
Benefits of Sharding
●
Write scalability...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.14
Insert Picture Here
Architecture of a
Sharding Solu...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.15
What do you need to consider?
 High level architec...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.16
2013-09-21
High-level Architecture
●
What component...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.17
2013-09-21
Components for a Sharding Solution
Shard...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.18
2013-09-21
Deployment of the Sharding Solution
Shar...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.19
2013-09-21
Deployment of the Sharding Solution
Shar...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.20
2013-09-21
Transaction and Shard Key Handling
●
How...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.21
2013-09-21
BEGIN
SELECT salary INTO @s FROM salarie...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.22
2013-09-21
Transaction Handling
●
Detecting transac...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.23
2013-09-21
Mapping the Sharding Key
●
What is a sha...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.24
2013-09-21
Granularity of Sharding
●
Can multiple t...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.25
2013-09-21
Sharded Tables
Table Rows
salaries 284 4...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.26
2013-09-21
Multi-table Query with Sharded Tables
SE...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.27
2013-09-21
Global Tables
Table Rows
salaries 284 40...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.28
2013-09-21
Multi-table Query with Global Tables
SEL...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.29
Insert Picture Here
Managing a
Sharded Database
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.30
Insert Picture Here
An extensible and easy-to-use
f...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.31
2013-09-21
MySQL Fabric: What is it?
●
“Farm” Manag...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.32
2013-09-21
MySQL Fabric: Features
●
Decision logic ...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.33
2013-09-21
Birds-eye View of a Sharded Database
Hig...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.34
2013-09-21
MySQL Fabric Node Architecture
MySQL
MyS...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.35
2013-09-21
MySQL Fabric: Prerequisites
●
MySQL Serv...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.36
2013-09-21
MySQL Fabric: Configuration
●
Backing St...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.37
2013-09-21
MySQL Fabric: Setup and Teardown
●
Creat...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.38
2013-09-21
MySQL Fabric: Starting and Stopping
●
St...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.39
2013-09-21
Sharding Architecture
Shards
MySQL
Fabri...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.40
2013-09-21
MySQL Fabric: Sharding Setup
●
Set up so...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.41
2013-09-21
MySQL Fabric: Create Groups and add Serv...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.42
2013-09-21
MySQL Fabric: Create Groups and add Serv...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.43
2013-09-21
MySQL Fabric: Set up Shard Mapping
●
Def...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.44
2013-09-21
MySQL Fabric: Add Shards
●
Add shards to...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.45
2013-09-21
MySQL Fabric: Moving and Splitting Shard...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.46
Insert Picture Here
Working with a
Sharded Database
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.47
2013-09-21
Fabric-aware Connector API
●
Support Tra...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.48
2013-09-21
Fabric-aware Connector API
import mysql....
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.49
2013-09-21
Digression: Computing Shards
●
Multiple ...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.50
2013-09-21
Connector API: Shard Specific Query
●
Pr...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.51
2013-09-21
Connector API: Shard Specific Query
●
Pr...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.52
2013-09-21
Connector API: Global Update
●
Provide t...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.53
Insert Picture Here
Closing Remarks
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.54
2013-09-21
Thoughts for the Future
●
Connector mult...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.55
2013-09-21
Thoughts for the Future
●
“More transpar...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.56
2013-09-21
MySQL Connect Sessions
● MySQL High Avai...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.57
2013-09-21
References
● MySQL Forum: Fabric, Shardi...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.58
2013-09-21
Mats Kindahl
Twitter: @mkindahl
Blog: ht...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.59
2013-09-21
Thank you!
Upcoming SlideShare
Loading in...5
×

MySQL Sharding: Tools and Best Practices for Horizontal Scaling

35,746

Published on

This presentation provides an introduction to what you need to consider when implementing a sharding solution and introduce the MySQL Fabric as a tool to help you to easy set up a sharded database.

Published in: Technology, News & Politics
4 Comments
78 Likes
Statistics
Notes
  • I create 3 groups, global-group, group_id-1 and group_id-2, then built a shard. Till now, everything is OK. But when i split the shard with a new group named group_id-3, there is an error occurred. The message is as follows:
    # mysqlfabric sharding split_shard 2 group_id-3 –split_value=100000
    Password for admin:
    Procedure :
    { uuid = a0eaff93-f6fe-4a7d-9c44-e7e53882fbfe,
    finished = True,
    success = False,
    return = BackupError: (‘Error while restoring the backup using the mysql client\n, %s’, “ERROR 1840 (HY000) at line 24 in file: ‘MySQL_132.228.239.19_3316.sql’: @@GLOBAL.GTID_PURGED can only be set when @@GLOBAL.GTID_EXECUTED is empty.\n”),
    activities =
    }
    I run the “reset master” command first, but the error is still there.what is this error?

    Regards
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Paulo, the connector API have been extended to support MySQL Fabric as well as web APIs in XML-RPC, but MySQL Table Partitioning does not really have anything to do with this.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • are there APIs available or do you have to make it manually. and does this apply for MySQL table partitioning?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • excellent presentation.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
35,746
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
727
Comments
4
Likes
78
Embeds 0
No embeds

No notes for slide

Transcript of "MySQL Sharding: Tools and Best Practices for Horizontal Scaling"

  1. 1. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1 Insert Picture Here 2013-09-21 MySQL Sharding: Tools and Best Practices for Horizontal Scaling Mats Kindahl (mats.kindahl@oracle.com) Alfranio Correia (alfranio.correia@oracle.com) Narayanan Venkateswaran (v.narayanan@oracle.com) 2013-09-21
  2. 2. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.2 2013-09-21 The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decision. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. Safe Harbor Statement
  3. 3. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.3 Program Agenda  Handling Scaling  What is sharding?  Managing a Sharded Database  Working with a Sharded Database
  4. 4. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.4 2013-09-21 What is this sharding? What are the benefits of sharding? How do I shard my database?
  5. 5. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.5 2013-09-21 It's all about scaling... ● Start with a single server The Growing Enterprise
  6. 6. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.6 2013-09-21 It's all about scaling... ● Start with a single server ● More and more page requests ● ...more and more reads ● What to do? ● Scale out! The Growing Enterprise
  7. 7. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.7 2013-09-21 It's all about scaling... ● Start with a single server ● More and more page requests ● ...more and more reads ● What to do? ● Scale out! ● Replicate to read servers The Growing Enterprise
  8. 8. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.8 2013-09-21 It's all about scaling... ● Start with a single server ● More and more page requests ● ...more and more reads ● What to do? ● Scale out! ● Replicate to read servers ● Perform a read-write split ● Writes go to master ● Reads go to read servers The Growing Enterprise
  9. 9. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.9 2013-09-21 It's all about scaling... ● More and more updates ● Write load increases ● What now? The Growing Enterprise
  10. 10. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.10 2013-09-21 It's all about scaling... ● More and more updates ● Write load increases ● What now? ● Add another master? The Growing Enterprise
  11. 11. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.11 2013-09-21 It's all about scaling... ● More and more updates ● Write load increases ● What now? ● Add another master? ● Doesn't work... ● Write load is replicated The Growing Enterprise
  12. 12. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.12 2013-09-21 It's all about scaling... ● More and more updates ● Write load increases ● What now? ● Add another master? ● Doesn't work... ● Write load is replicated ● Partition database ● Distribute writes ● Called sharding The Growing Enterprise UID 10000-20000 UID 20001-40000
  13. 13. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.13 2013-09-21 Benefits of Sharding ● Write scalability ● Can handle more writes ● Large data set ● Database too large ● Does not fit on single server ● Improved performance ● Smaller index size ● Smaller working set ● Improve performance UID 10000-20000 UID 20001-40000
  14. 14. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.14 Insert Picture Here Architecture of a Sharding Solution
  15. 15. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.15 What do you need to consider?  High level architecture  Transaction and sharding key handling  Granularity of sharding
  16. 16. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.16 2013-09-21 High-level Architecture ● What components do we need? ● How are they deployed?
  17. 17. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.17 2013-09-21 Components for a Sharding Solution Shard #2 Shard #1 Shard #3 Switch State Store Executor QUERY KEY KEY QUERY Contain decision logic for distributing queries Contain information about location of shards SHARD#
  18. 18. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.18 2013-09-21 Deployment of the Sharding Solution Shard #2 Shard #1 Shard #3 Switch State Store Executor Network hop! Performance? Protocol? Single Points of Failure! Caches can be used to avoid performance impact
  19. 19. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.19 2013-09-21 Deployment of the Sharding Solution Shard #2 Shard #1 Shard #3 Switch State Store Executor Deployed with application (e.g., inside connector) API simple to add
  20. 20. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.20 2013-09-21 Transaction and Shard Key Handling ● How are the transactions handled? ● How do get the shard key for a transaction? ● How to compute shard from key?
  21. 21. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.21 2013-09-21 BEGIN SELECT salary INTO @s FROM salaries WHERE emp_no = 20101; SET @s = 1.1 * @s; INSERT INTO salaries VALUES (20101, @s); COMMIT BEGIN INSERT INTO ...  COMMIT Sharding key? Ah, there it is! Session state? Hmm... looks like a read transaction Oops.. it was a write transaction! Transaction done! Clear session state? New transaction! Different shard? What about the session state? Transaction Handling
  22. 22. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.22 2013-09-21 Transaction Handling ● Detecting transaction boundaries ● Managing session state ● Move session state between servers – Easy to use – Expensive and error prone ● Reset state after each transaction – Transactions start with default session state What about crashes? Where do I store the session state?
  23. 23. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.23 2013-09-21 Mapping the Sharding Key ● What is a sharding key? ● Single column ● Multi column – Same table? – Different tables? ● How is the key transformed? ● Hash ● Range ● User-defined Compute Shard# KeyShard# (X) (X,Y,...) RANGE HASH Something else
  24. 24. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.24 2013-09-21 Granularity of Sharding ● Can multiple tables be sharded with the same key? ● Can we shard different tables different ways? ● Do we allow global tables? ● Do we allow cross-database queries?
  25. 25. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.25 2013-09-21 Sharded Tables Table Rows salaries 284 404 700 titles 44 330 800 employees 30 002 400 dept_emp 33 160 300 dept_manager 2 400 departments 900 In desperate need of sharding! Foreign keys
  26. 26. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.26 2013-09-21 Multi-table Query with Sharded Tables SELECT first_name, last_name, salary FROM salaries JOIN employees USING (emp_no) WHERE emp_no = 21012 AND CURRENT_DATE BETWEEN from_date AND to_date; ● Referential Integrity Constraint ● Example query joining salaries and employees ● Same key, same shard: co-locate rows for same user ● JOIN normally based on equality ● Using non-equality defeats purpose of foreign key
  27. 27. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.27 2013-09-21 Global Tables Table Rows salaries 284 404 700 titles 44 330 800 employees 30 002 400 dept_emp 33 160 300 dept_manager 2 400 departments 900 Do not really need to be sharded
  28. 28. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.28 2013-09-21 Multi-table Query with Global Tables SELECT first_name, last_name, GROUP_CONCAT(dept_name) FROM employees JOIN dept_emp USING (emp_no) JOIN departments USING (dept_no) WHERE emp_no = 21012 GROUP BY emp_no; ● JOIN with departments table ● Has no employee number, hence no sharding key ● Table need to be present on all shards ● How do we update global tables?
  29. 29. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.29 Insert Picture Here Managing a Sharded Database
  30. 30. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.30 Insert Picture Here An extensible and easy-to-use framework for managing a farm of MySQL server supporting high-availability and sharding MySQL Fabric
  31. 31. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.31 2013-09-21 MySQL Fabric: What is it? ● “Farm” Management System ● High-Availability ● Sharding ● Distributed ● Procedure Execution ● Extensible ● Written in Python ● Early alpha ● Long road ahead ● Open Source ● You can participate ● Suggest features ● Report bugs ● Contribute patches ● MySQL 5.6 is focus
  32. 32. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.32 2013-09-21 MySQL Fabric: Features ● Decision logic in connector ● Reducing network load ● Support Transactions ● API to provide sharding key ● Global Updates ● Global Tables ● Schema updates ● Procedure Executor ● Shard Multiple Tables ● Using same key ● Sharding Functions ● Range ● (Consistent) Hash ● Shard Operations ● Using built-in executor ● Shard move ● Shard split
  33. 33. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.33 2013-09-21 Birds-eye View of a Sharded Database High Availability Groups (Shards) MySQL Fabric Node Application XML-RPC SQL SQL Connector Connector Connector Operator
  34. 34. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.34 2013-09-21 MySQL Fabric Node Architecture MySQL MySQL Fabric Framework Executor State Store (Persister) Sh ?HA MySQLAMQP XML-RPC ?? Connector Connector Connector Protocols Extensions Backing Store
  35. 35. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.35 2013-09-21 MySQL Fabric: Prerequisites ● MySQL Servers (version 5.6.10 or later) ● Server for meta-data backing store ● Servers being managed ● Python 2.6 or 2.7 ● No support for 3.x yet ● MySQL Utilities 1.4.0 ● Available at http://labs.mysql.com/
  36. 36. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.36 2013-09-21 MySQL Fabric: Configuration ● Backing Store ● MySQL server ● Persistent storage for state ● Storage engine-agnostic ● Protocol ● Address where node will be ● Currently only XML-RPC ● Logging ● Chatty: INFO (default) ● Moderate: WARNING ● URL for rotating log [storage] address = localhost:3306 user = fabric password =  database = fabric connection_timeout = 6 [protocol.xmlrpc] address = localhost:8080 threads = 5 [logging] level = INFO url = file:///var/log/fabric.log
  37. 37. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.37 2013-09-21 MySQL Fabric: Setup and Teardown ● Create the necessary tables in backing store mysqlfabric manage setup ● Remove the tables from backing store mysqlfabric manage teardown ● Connect to database server in “storage” section ● Ensure that you have the necessary users and privileges
  38. 38. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.38 2013-09-21 MySQL Fabric: Starting and Stopping ● Start MySQL Fabric node in foreground – print log to terminal mysqlfabric manage start ● Start MySQL Fabric node in background – print log to file mysqlfabric manage start ­­daemonize  ● Stop MySQL Fabric node mysqlfabric manage stop
  39. 39. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.39 2013-09-21 Sharding Architecture Shards MySQL Fabric Node Application Connector Connector Connector Global Group Global Updates Shard Updates Replication Support global update for off-line shards
  40. 40. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.40 2013-09-21 MySQL Fabric: Sharding Setup ● Set up some groups ● my_global – for global updates ● my_group.* – for the shards ● Add servers to the groups ● Create a shard mapping ● A “distributed database” ● Mapping keys to shards ● Give information on what tables are sharded ● Add shards
  41. 41. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.41 2013-09-21 MySQL Fabric: Create Groups and add Servers ● Define a group mysqlfabric group create my_global ● Add servers to group mysqlfabric group add my_global global.example.com    mats xyzzy mysqlfabric group add my_global … User + Password (Likely to go away)
  42. 42. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.42 2013-09-21 MySQL Fabric: Create Groups and add Servers ● Promote one server to be primary mysqlfabric group promote my_global ● Tell failure detector to monitor group mysqlfabric group activate my_global
  43. 43. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.43 2013-09-21 MySQL Fabric: Set up Shard Mapping ● Define shard mapping mysqlfabric sharding define hash my_global ● Add tables that should be sharded mysqlfabric sharding add_mapping 1    employees.employees emp_no mysqlfabric sharding add_mapping 1    employees.salaries emp_no ● Tables not added are global Will show the shard map identifier (a number) Shard map identifier
  44. 44. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.44 2013-09-21 MySQL Fabric: Add Shards ● Add shards to shard mapping mysqlfabric sharding add_shard 1 my_group.1 enabled     .     .     . mysqlfabric sharding add_shard 1 my_group.N enabled Shard map identifier
  45. 45. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.45 2013-09-21 MySQL Fabric: Moving and Splitting Shards ● Moving a shard from one group to another mysqlfabric sharding move 5 my_group.5 ● Splitting a shard into two pieces (hash) mysqlfabric sharding split 5 my_group.6 Shard ID
  46. 46. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.46 Insert Picture Here Working with a Sharded Database
  47. 47. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.47 2013-09-21 Fabric-aware Connector API ● Support Transactions ● Sharding key out of band ● Fabric-aware Connectors ● Connector/J ● Connector/Python ● Connector/PHP ● Fabric-aware Frameworks ● Doctrine ● Hibernate ● Focus on Connector/Python
  48. 48. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.48 2013-09-21 Fabric-aware Connector API import mysql.connector.fabric as connector conn = connector.MySQLFabricConnection(     fabric={"host": "fabric.example.com", "port" : 8080},     user='mats', database="employees") ● Establish a “virtual” connection ● Real server connection established lazily ● Provide connection information for the Fabric node ● Fabric node will provide information about real servers
  49. 49. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.49 2013-09-21 Digression: Computing Shards ● Multiple Mappings ● Which mapping to use? ● Application don't care … but know tables in transaction ● Currently only one mapping ● Computing shard requires ● Tables + sharding key ● Extended Connector API ● Extra properties passed out-of-band Compute Shard# Key Shard#Map# Compute Map# Tables
  50. 50. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.50 2013-09-21 Connector API: Shard Specific Query ● Provide tables in query ● Property: tables ● Fabric will compute map conn.set_property(tables=['employees.employees', 'employees.titles'],                   key=emp_no) cur = conn.cursor() cur.execute("INSERT INTO employees VALUES (%s,%s,%s)",             (emp_no, first_name, last_name)) cur.execute("INSERT INTO titles(emp_no,title,from_date)"             “ VALUES (%s,%s,CURDATE())",             (emp_no, 'Intern')); conn.commit() ● Provide sharding key ● Property: key ● Fabric will compute shard Transactions work fine!
  51. 51. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.51 2013-09-21 Connector API: Shard Specific Query ● Provide tables in query ● Property: tables ● Fabric will compute map conn.set_property(tables=['employees.employees', 'employees.titles'],                   key=emp_no) cur = conn.cursor() cur.execute(     "SELECT first_name, last_name, title"     "  FROM employees JOIN titles USING (emp_no)"     " WHERE emp_no = %d", (emp_no,)) for row in cur:     print row[0], row[1], “,“, row[2] ● Provide sharding key ● Property: key ● Fabric will compute shard Join queries are sent to correct shard and executed there
  52. 52. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.52 2013-09-21 Connector API: Global Update ● Provide tables in query ● Property: tables ● Fabric will compute map conn.set_property(tables=['employees.titles'], scope='GLOBAL') cur = conn.cursor() cur.execute("ALTER TABLE employees.titles ADD nickname VARCHAR(64)") ● Set global scope ● Property: scope ● Query goes to global group
  53. 53. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.53 Insert Picture Here Closing Remarks
  54. 54. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.54 2013-09-21 Thoughts for the Future ● Connector multi-cast ● Scatter-gather ● Internal interfaces ● Improve extension support ● Improve procedures support ● Command-line interface ● Improving usability ● Focus on ease-of-use ● More protocols ● MySQL-RPC Protocol? ● AMQP? ● More frameworks? ● More HA group types ● DRBD ● MySQL Cluster ● Fabric-unaware connectors?
  55. 55. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.55 2013-09-21 Thoughts for the Future ● “More transparent” sharding ● Single-query transactions ● Cross-shard joins is a problem ● Multiple shard mappings ● Independent tables ● Multi-way shard split ● Efficient initial sharding ● Better use of resources ● High-availability executor ● Node failure stop execution ● Replicated State Machine ● Fail over to other Fabric node ● Distributed failure detector ● Connectors report failures ● Custom failure detectors
  56. 56. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.56 2013-09-21 MySQL Connect Sessions ● MySQL High Availability: Managing Farms of Distributed Servers ● September 22, 5:30pm-6:30pm in Imperial Ballroom B ● Scaling PHP Applications ● September 22, 10:00am-11:00am in Union Square Room ¾ ● MySQL Sharding, Replication, and HA ● September 21, 5:30-6:30pm in Imperial Ballroom B
  57. 57. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.57 2013-09-21 References ● MySQL Forum: Fabric, Sharding, HA, Utilities ● http://forums.mysql.com/list.php?144 ● A Brief Introduction to MySQL Fabric ● http://mysqlmusings.blogspot.com/2013/09/brief-introduction-to-mysql-fabric.html ● MySQL Fabric – Sharding – Introduction ● http://vnwrites.blogspot.com/2013/09/mysqlfabric-sharding-introduction.html ● Migrating From an Unsharded to a Sharded Setup ● http://vnwrites.blogspot.com/2013/09/mysqlfabric-sharding-migration.html
  58. 58. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.58 2013-09-21 Mats Kindahl Twitter: @mkindahl Blog: http://mysqlmusings.blogspot.com Alfranio Correia Twitter: @alfranio Blog: http://alfranio-distributed.blogspot.com Narayanan Venkateswaran Twitter: @vn_tweets Blog: http://vnwrites.blogspot.com Keeping in Touch
  59. 59. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.59 2013-09-21 Thank you!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×