Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

My sql fabric webinar tw2

673 views

Published on

  • Be the first to comment

My sql fabric webinar tw2

  1. 1. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1 MySQL Fabric: Easy Management of MySQL Servers 杜修文 Ivan.Tu@Oracle.Com MySQL Principal Sales Consultant Oracle LLC July 17, 2014
  2. 2. 2 Insert Picture Here Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 | An extensible and easy-to-use framework for managing a farm of MySQL server supporting high-availability and sharding MySQL Fabric
  3. 3. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |3 MySQL Fabric: Goals & Features  Connector API Extensions  Support Transactions  Support full SQL  Decision logic in connector  Reducing network load  Load Balancing  Read-Write Split  Distribute transactions  Global Updates  Global tables  Schema updates  Shard Multiple Tables  Using same key  Sharding Functions  Range  (Consistent) Hash  Shard Operations  Using built-in executor  Shard move  Shard split
  4. 4. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |4 Transaction Handling
  5. 5. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |5  Routing Transactions  Pre-declare properties of transactions  Detecting transaction boundaries  Push as much as possible to server  Managing Session State  Move session state between servers − Easy to use − Expensive and error prone  Reset state after each transaction − Transactions start with default session state Transaction Handling
  6. 6. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |6 Routing Transactions Shard #2 Shard #1 Shard #3 Switch State Store Executor QUERY KEY KEY QUERY Contain decision logic for routing queries Contain information about location of shards SHARD#
  7. 7. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |7 Birds-eye View High Availability Groups (Shards) MySQL Fabric Node Application XML-RPC SQL Connector Connector Connector Operator
  8. 8. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |8 MySQL Fabric: Prerequisites  MySQL Servers (version 5.6.10 or later)  Server for meta-data backing store  Servers being managed  Python 2.6 or 2.7  No support for 3.x yet  MySQL Utilities 1.4.0  Available at http://labs.mysql.com/
  9. 9. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |9 MySQL Fabric: Configuration  Backing Store  MySQL server  Persistent storage for state  Storage engine-agnostic  Protocol  Address where node will be  Currently only XML-RPC  Logging  Chatty: INFO (default)  Moderate: WARNING  URL for rotating log [storage] address = localhost:3306 user = fabric password = database = fabric connection_timeout = 6 [protocol.xmlrpc] address = localhost:8080 threads = 5 [logging] level = INFO url = file:///var/log/fabric.log
  10. 10. 10 Insert Picture Here Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 | Connecting to a MySQL Fabric Farm
  11. 11. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |11 Fabric-aware Connector API  Connector API Extensions  Support Transactions  Support full SQL  Decision logic in connector  Reducing network load  Load Balancing  Read-Write Split  Distribute transactions  Fabric-aware Connectors  Connector/J  Connector/Python  Connector/PHP  Fabric-aware Frameworks  Doctrine  Hibernate  Focus on Connector/Python
  12. 12. 12 Insert Picture Here Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 | Architecture for Sharding
  13. 13. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |13 Benefits of Sharding  Write scalability  Can handle more writes  Large data set  Database too large  Does not fit on single server  Improved performance  Smaller index size  Smaller working set  Improve performance UID 10000-20000 UID 20001-40000
  14. 14. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |14 MySQL Fabric: Sharding Goals & Features  Connector API Extensions  Support Transactions  Support full SQL  Decision logic in connector  Reducing network load  Shard Multiple Tables  Using same key  Global Updates  Global tables  Schema updates  Sharding Functions  Range  (Consistent) Hash  Shard Operations  Using built-in executor  Shard move  Shard split
  15. 15. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |15 Mapping the Sharding Key  What is a sharding key?  Single column  Multi column − Same table? − Different tables?  How is the key transformed?  Hash  Range  User-defined Compute Shard# KeyShard# (X) (X,Y,...) RANGE HASH Something else
  16. 16. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |16 Sharded Tables In desperate need of sharding! Foreign keys
  17. 17. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |17 Multi-table Query with Sharded Tables SELECT first_name, last_name, salary FROM salaries JOIN employees USING (emp_no) WHERE emp_no = 21012 AND CURRENT_DATE BETWEEN from_date AND to_date;  Referential Integrity Constraint  Example query joining salaries and employees  Same key, same shard: co-locate rows for same user  JOIN normally based on equality  Using non-equality defeats purpose of foreign key
  18. 18. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |18 Global Tables Does not need to be sharded
  19. 19. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |19 Multi-table Query with Global Tables SELECT first_name, last_name, GROUP_CONCAT(dept_name) FROM employees JOIN dept_emp USING (emp_no) JOIN departments USING (dept_no) WHERE emp_no = 21012 GROUP BY emp_no;  JOIN with departments table  Has no employee number, hence no sharding key  Table need to be present on all shards  How do we update global tables?
  20. 20. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |20 Sharding Architecture Shards MySQL Fabric Node Application Global Group Global Updates Shard Updates Replication Support global update for off-line shards
  21. 21. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |21 Demo
  22. 22. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |22 Demo Server Side – Startup Instances 3306 3606 Fabric Repository 3506 Fabric Node 32274 3308 3314 3408 3406
  23. 23. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |23 Demo Server Side – Configuration of Fabric /etc/mysql/fabric.cfg [DEFAULT] prefix = sysconfdir = /etc logdir = /var/log [storage] address = localhost:3506 user = fabric password = welcome1 database = fabric auth_plugin = mysql_native_password connection_timeout = 6 connection_attempts = 6 connection_delay = 1 [servers] user = fabric password = welcome1 [protocol.xmlrpc] address = localhost:32274 threads = 5 user = admin password =welcome1 disable_authentication = no realm = MySQL Fabric ssl_ca = ssl_cert = ssl_key = [executor] executors = 5 [logging] level = INFO url = file:///var/log/fabric.log
  24. 24. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |24 Startup Fabric •Fabric help command mysqlfabric help commands •Instantiate Fabric mysqlfabric manage setup •Bring up Fabric mysqlfabric manage start --deamonize
  25. 25. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |25 Demo – Configure by Fabric GP1 3306 GP-Global 3606 GP2 Fabric Repository 3506 Fabric Node 32274 3308 3314 3408 3406
  26. 26. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |26 Fabric Group Operation  Create 1 global group, 2 shard group mysqlfabric group create gp-global mysqlfabric group create gp1 mysqlfabric group create gp2  Assign Servers to groups mysqlfabric group add gp-global 127.0.0.1:3606 mysqlfabric group add gp1 127.0.0.1:3306 mysqlfabric group add gp1 127.0.0.1:3308 mysqlfabric group add gp1 127.0.0.1:3314 mysqlfabric group add gp2 127.0.0.1:3406 mysqlfabric group add gp2 127.0.0.1:3408  Pick the master server by Fabric mysqlfabric group promote gp1 mysqlfabric group promote gp2
  27. 27. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |27 Fabric Group Operations • Let Fabric manage auto-failover for you mysqlfabric group activate gp-global mysqlfabric group activate gp1 mysqlfabric group activate gp2 • Define Range shard on to global group mysqlfabric sharding create_definition RANGE gp-global • Add shard table – employees to the shard created (shard map id = 1) shard by id column mysqlfabric sharding add_table 1 employees.employees id • Assign rows with id value 1~999 to group gp1, 1000 above to group gp2 mysqlfabric sharding add_shard 1 gp1/1,gp2/1000 --state=ENABLED
  28. 28. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |28 Demo – Databases is Ready GP1 3306 GP-Global 3606 GP2 Fabric Repository 3506 Fabric Node 32274 3308 3314-M 3408-M 3406
  29. 29. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |29 Demo Application Side – Need Connector/J 5.1.30 or later GP1 3306 GP-Global 3606 GP2 Fabric Repository 3506 Fabric Node 32274 3308 3314-M 3408-M 3406 JavaConnector/J
  30. 30. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |30 Demo Application Side – Cache Fabric Configuration on creating DB Connection GP1 3306 GP-Global 3606 GP2 Fabric Repository 3506 Fabric Node 32274 3308 3314-M 3408-M 3406 JavaConnector/J Connection Object
  31. 31. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |31 Java Application - Global tables and DDLs are accessed by property fabricServerGroup import com.mysql.fabric.jdbc.FabricMySQLConnection; … rawConnection = DriverManager.getConnection( baseUrl + database + "?fabricServerGroup=gp-global“,user, password); statement = rawConnection.createStatement(); … statement.executeUpdate("create table employees.employees (emp_no int not null,first_name varchar(50),last_name varchar(50), department_id int,primary key (emp_no))"); … statement.executeUpdate("insert into employees.departments values (1,'Sales'),(2,'Financial')");
  32. 32. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |32 Demo Environment – Dispatch Request Based On Cached Configuration with Connection GP1 3306 GP-Global 3606 GP2 Fabric Repository 3506 Fabric Node 32274 3308 3314-M 3408-M 3406 JavaConnector/J Connection Object – Shard Table Write Sharded Table Write Global Table
  33. 33. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |33 Demo Environment – Dispatch Request Based On Cached Configuration with Connection GP1 3306 GP-Global 3606 GP2 Fabric Repository 3506 Fabric Node 32274 3308 3314-M 3408-M 3406 JavaConnector/J Connection Object – Shard Table Write Sharded Table Write Global Table
  34. 34. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |34 Demo Environment – Dispatch Request Based On Cached Configuration with Connection GP1 3306 GP-Global 3606 GP2 Fabric Repository 3506 Fabric Node 32274 3308 3314-M 3408-M 3406 JavaConnector/J Connection Object – Shard Table Write Sharded Table Write Global Table
  35. 35. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |35 Shared tables are accessed by property fabricShardTable and fabericShardKey rawConnection = DriverManager.getConnection( "jdbc:mysql:fabric://127.0. 0.1:32274/mysql?fabricShardTable=employees.employees“, user, password); … for (int i = 0; i < 4; ++i) { connection.setShardKey(ids[i].toString()); ps.setInt(1, ids[i]); ps.setString(2, firstNames[i]); ps.setString(3, lastNames[i]); ps.setString(4, departmentIds[i]); ps.executeUpdate(); }
  36. 36. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |36 Demo Environment – Read/Write Split within Shard GP1 3306 GP-Global 3606 GP2 Fabric Repository 3506 Fabric Node 32274 3308 3314-M 3408-M 3406 JavaConnector/J Connection Object – Shard Table Read Only Operations Read Global Table
  37. 37. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |37 Read Only Operations FabricMySQLConnection newConn = (FabricMySQLConnection) rawConnection; newConn.setReadOnly(true); ps = newConn.prepareStatement( "select emp_no, first_name, last_name, d.name, i.variable_value from employees.employees e, information_schema.global_variables i, employees.departments d where e.department_id = d.department_id and i.variable_name='port' and emp_no = ?");
  38. 38. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |38 Demo Environment – Failure Detected GP1 3306 GP-Global 3606 GP2 Fabric Repository 3506 Fabric Node 32274 3308 3314-M 3408-M 3406 JavaConnector/J Connection Object Write Global Table Change Master to 3606 Change Master to 3308 X
  39. 39. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |39 Demo Environment – Failover within Shard GP1 3306 GP-Global 3606 GP2 Fabric Repository 3506 Fabric Node 32274 3308-M 3314-M 3408-M 3406 JavaConnector/J Connection Object Write Shard Table Write Global Table X
  40. 40. 40 Insert Picture Here Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 | Closing Remarks
  41. 41. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.| Oracle Webinar | December 17, 2013 |41 Thoughts for the Future  Connector multi-cast  Scatter-gather  UNION of result sets  More complex operations?  Internal interfaces  Improve extension support  Improve procedures support  Command-line interface  Improving usability  Focus on ease-of-use  More protocols  MySQL-RPC Protocol?  AMQP?  More frameworks?  More HA group types  DRBD  MySQL Cluster  Fabric-unaware connectors?

×