Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building Scalable High Availability Systems using MySQL Fabric

1,468 views

Published on

Building scalable, high-availability systems offers several challenges: managing the redundancy in the farm using replication, monitoring the system to find hotspots and rebalancing the system, automating scaling reads and writes, and upgrades and replacement without downtime. MySQL Fabric is a framework for building scalable, high-availability systems that are easy to use and flexible. It uses existing MySQL features to manage a high-availability system, and can also be used with existing systems where some parts of the high-availability solution are already in place. In this presentation from Oracle Open World you will learn about the new features in MySQL Fabric and how you can use it to build scalable high availability system or enhance your existing system.

Published in: Technology

Building Scalable High Availability Systems using MySQL Fabric

  1. 1. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Building Scalable High Availability Systems using MySQL Fabric Mats Kindahl Senior Principal Software Developer Copyright © 2014, Oracle and/or its affiliates. All rights reserved.
  2. 2. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. 2015-10-28
  3. 3. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Session Outline Introduction to MySQL Fabric1 2 3 4 5 Elements of Scalable High-Availability Systems Building a High Availability System Scaling High Availability Systems Closing Remarks & Thoughts for the Future
  4. 4. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Introduction to MySQL Fabric
  5. 5. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 What is MySQL Fabric? An extensible and easy-to- use framework for managing a farm of MySQL servers supporting high-availability and sharding
  6. 6. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2014-10-02 What does all that mean? ● Management System – Manages a MySQL Farm – Distributed Framework ● Framework – Resilient executor – State store – Transaction routing ● Extensible – High-availability – “Semi-automatic” sharding ● Written in Python ● Open Source – You can participate – Suggest features – Report bugs – Contribute patches ● MySQL 5.6 is focus – For now
  7. 7. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2014-10-02 MySQL Fabric: Goals & Features ● Decision logic in connector – Eliminates network hop – Reduce network load – Eliminate single point of failure – Scale naturally: not a bottleneck ● Connector API extensions – Support transactions – Support full SQL ● MySQL Router – Support for legacy connectors ● Load balancing – Read-write split – Weighted round-robin ● Cloud integration – Support elasticity – Servers on-demand
  8. 8. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2014-10-02 MySQL Fabric: Goals & Features ● Multi-table sharding – Co-locate related rows ● Sharding functions – Range – (Consistent) hash ● Shard operations – Shard move and split ● Global updates – Global tables – Schema updates
  9. 9. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 MySQL Fabric 1.6 (alpha): What is new ● MySQL Router (2.0 GA) – Distributed separately – Support for legacy connectors – Connection-based routing – Connection fail-over support – Connection load balancing ● Multi-node Fabric Control – Geographical Redundancy – Resilient Execution – Load distribution ● Improved Password Management – Management passwords in state store – Management passwords encrypted using AES
  10. 10. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Birds-eye View MySQL Fabric Node Application Operator High-Availability Groups (Shards)
  11. 11. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2014-10-02 High-Level Components ● Connecting to the Farm – Enhanced Connector API – MySQL Router ● MySQL Fabric Node – Manage information about farm – Provide status information – Execute procedures ● MySQL Servers – Organized in high-availability groups – Handling application data High Availability Group Application Connector Connector Connector MySQL Fabric Node
  12. 12. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 MySQL Fabric Node Architecture MySQL MySQL Fabric Framework Executor State Store (Persister) Sh ?HA MySQL-RPC XML-RPC Connector Connector Connector Protocols Extensions Backing Store
  13. 13. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2014-10-02 MySQL Router ● Plugin-based architecture – Harness with general functionality ● Connection-based Routing – Connection-based routing plugin – Routing decision on connecting – Raw packet copy: very fast ● Fabric Info Plugin – Meta-data Cache – Use Fabric node to get meta-data Fabric Info Module Cache MySQL MySQL Router MySQL Fabric Routing Module
  14. 14. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Elements of Scalable High-Availability Systems
  15. 15. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2014-10-02 High-Availability Concepts ● Redundancy – Duplicate components ● Monitoring – Detecting failing components – Monitor load ● Procedures – Activate replacements – Distribute load – Decommission components – Deploy new components
  16. 16. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Database Scalability Concepts ● Scalability and Elasticity – The ability of being able to adapt to changes in load of the system ● Scaling Reads – Being able to cope with an increasing read load ● Scaling Writes – Being able to cope with an increasing write load ● Scaling Storage – Being able to cope with an increasing database size
  17. 17. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 “[Elasticity] is defined as the degree to which a system is able to adapt to workload changes by provisioning and deprovisioning resources in an autonomic manner” What is Elasticity?
  18. 18. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Database Read Scalability ● Coping with increased read traffic – Typical use-case for many web application – Usually multiple tiers: web caches and read servers ● Adding result set caches – Can offload database server for repeated queries – Queries have to be identical – Cannot handle complex queries ● Adding read servers – Replicate from primary server to dedicated read servers – Can handle complex queries
  19. 19. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Database Write Scalability ● Coping with increased write traffic – Typical use-case for many large-scale monitoring applications ● Monitoring database servers ● Data collection systems ● Buying better machines – Straightforward solution – Cost?
  20. 20. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Database Storage Scalability ● Coping with increased amount of data – Large-scale monitoring applications ● Monitoring network – Business Analysis Systems ● Collecting and analyzing click-streams ● Sharding the data – Database need to be sharded into independent partitions – Application need to be tailored to sharded data
  21. 21. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Building a High Availability System A case study
  22. 22. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2014-10-02 Basic system without redundancy ● Application on the Web – Deployed on separate hardware – Connector built into application – Connections opened for each request ● Database Backend – Store data from many applications ● High Availability?
  23. 23. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2014-10-02 Basic system without redundancy ● No High Availability – No redundancy – No monitoring – No way to activate replacement ● What can you do? – Add redundancy – Add monitoring – Add activation of replacement !
  24. 24. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2014-10-02 Basic system with redundancy ● Server Redundancy – Keep a secondary around – Replicate from primary ● MySQL Router – Deployed as intermediate – Monitor servers – Re-direct connections on primary failure
  25. 25. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Router Configuration for Static Failover ● Section header – Name + Key ● Bind port – Port number to listen on – Connections from localhost only by default [routing:failover] bind_port = 13306 destinations = srv1.example.com,srv2.example.com mode = read-write ● Destinations – Servers to open connections to ● Mode – Read-write queries on the connection
  26. 26. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2014-10-02 Basic system with redundancy ● Server crashes – Replication stops ● Notices server is gone – Router notices server disappears – Server is put in quarantine ● Find alternative server – Router picks new server from static list of candidates ! ?
  27. 27. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2014-10-02 Basic system with redundancy ● Activate replacement – New connections are sent to secondary – Existing connections will time out ?
  28. 28. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Basic system with redundancy ● Server Restored – The primary is restored into working state – Replication is set up again ● Server unquarantined – Router notices server is available again – Server is unquarantined – New connections go to primary – Old connections still go to secondary ● Closed once the request is finished
  29. 29. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Using Fabric for Server Management ● Fabric Control Node – Provide Information about farm to router – Monitor servers – Execute procedures – Add and remove servers ● High Availability Group – Collection of servers – All servers in group have same data Fabric Node High Availability Group
  30. 30. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Router Configuration with Fabric ● Fabric cache – Address of Fabric node [fabric_cache:my_cache] address = fabric.example.com:32275 [routing:failover] bind_port = 13306 destinations = fabric+cache://my_cache/group/my_group mode = read-write ● Destinations – Reference to Fabric cache instead of servers. – Fabric cache is authority in the URL – Groups under group key
  31. 31. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Using Fabric for Server Management ● Initially – Secondaries have @@READ_ONLY = 1 – Primary has @@READ_ONLY = 0 ● Primary is lost ● Fabric node notices – Built-in Failure Detector ! ?
  32. 32. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Using Fabric for Server Management ● Fabric node perform fail-over – Secondary is selected as new primary – Other secondaries are changed to replicated from new primary – @@READ_ONLY = 0 on new primary – Router informed of change ● Router change primary – New connections go to new primary – Old connections error out ?
  33. 33. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Using Fabric for Server Management ● Availability – Router caches information – Service not interrupted on Fabric node failure – Note: Failovers not done while Fabric node is down ● Meta Data Durability – Information stored in database backend – Fabric node just have to be restarted ● For example, using init.d script Fabric Node
  34. 34. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Geographical Redundancy ● Multiple data centers – One Fabric node in each ● Meta data locality – Node in data center to provide meta-data ● Resilient execution – Long-running procedures are not interrupted
  35. 35. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Geographical Redundancy ● Fabric Control Cluster – Multiple Fabric control nodes – One active node (leader), others passive – Passive nodes can be used as read replicas – … but also act as standby ● Elastic Control Cluster – Adding or removing nodes on demand – Added nodes automatically brought up to date when joining
  36. 36. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Geographical Redundancy ● Fabric Control Cluster Leader – There is one leader in the cluster – The leader can change ● Leader dies ● Leader is explicitly changed – Commands changing meta data have to be executed on leader ● Internal Forwarding – Commands that change meta-data are automatically forwarded to leader
  37. 37. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Scaling High Availability Systems A case study
  38. 38. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Scaling Reads ● When to scale for reads – Read-heavy workloads – Most web applications ● Result set caching – If queries are identical – Done at application layer – Not covered in this presentation ● Read servers – For more complex queries
  39. 39. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Scaling Reads ● Adding read servers to group – Servers added as spares – Fabric monitor spares as well ● Setup router – Add a read-only query port to router – Distribute load in round-robin fashion ● Split traffic – Updates sent to update port – Queries sent to query port Fabric Node Queries Updates
  40. 40. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Router Configuration for Static Load Balancing ● Section header – Name + Key ● Bind port – Port number to listen on – Connections from localhost only by default [routing:load_balancing] bind_port = 13307 destinations = slave1.example.com,slave2.example.com mode = read-only ● Destinations – Servers to open connections to ● Mode – Read-only queries on the connection – … can be load balanced
  41. 41. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Scaling Writes ● Write-heavy workloads – Data collection networks – Large-scale monitoring applications ● Workload Type – Large number of updates – Each update change few rows ● Replication does not help – Writes are replicated as well !
  42. 42. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Scaling Writes ● “Sharding” database – Horizontal partitioning of rows in tables – Each shard updated independently of others ● “Sharding” write stream – Write stream partitioned by key – Write streams are independent 1000-1999 1-999 2000-2999 3000-3999
  43. 43. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2014-10-02 Scaling Storage ● Data-intensive applications – Data collection networks – Business analysis applications ● On-Line Analytical Processing (OLAP) of large data sets ● Just too much data – Indexes too large – Backup time excessive – Analysis prohibitive ● “Sharding” the data – Horizontal partitioning of database based on sharding key – Same solution as for scaling writes ● Analytic Queries – Parallelization possible – Inter-shard queries? – Cross-shard queries?
  44. 44. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2014-10-02 Elasticity ● Scale hardware on demand – Automatically adapting to changes in workload – Useful for changes in read load, write load, and storage requirements – Require monitoring the load of the system ● Read load change – Adapt number of read slaves to read load ● Write load change – Adapt number of shards to incoming write traffic ● Storage requirement change – Adapt number of shards to storage requirements
  45. 45. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Scaling through Sharding in Fabric ● Use Fabric-aware connectors – … or doing key mapping in application – Sending transactions to correct shard – Key need to be given explicitly ● Provided together with the query ● Dispatching Analytical Queries – Done at application level – Fetch shard information from Fabric ● Using XML-RPC or MySQL-RPC interface – Dispatch queries to the shards – Collect and merge the results MySQL Fabric ServerApplication High-Availability Groups (Shards) Global Group
  46. 46. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Scaling through Sharding in Fabric ● Schema designed for sharding – Sharding key need to be picked carefully ● Sharded Tables – Application-level “objects” usually spread over several tables – Rows for one “object” need to be co- located on same shard ● Global Tables – Tables present on all shards – Replicated to all shards using the Global Group MySQL Fabric ServerApplication High-Availability Groups (Shards) Global Group
  47. 47. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 employee emp_id INT birth_date DATE first_name VARCHAR(14) last_name VARCHAR(16) gender ENUM('M','F') hire_date DATE titles emp_no INT title VARCHAR(50) from_date DATE to_date DATE salaries emp_no INT salary INT from_date DATE to_date DATE departments dept_no CHAR(4) dept_name VARCHAR(40) dept_emp emp_no INT dept_no CHAR(4) from_date DATE to_date DATE dept_manager dept_no CHAR(4) emp_no INT from_date DATE to_date DATE Table Rows salaries 284 404 700 titles 44 330 800 employees 30 002 400 dept_emp 33 160 300 dept_manager 2 400 departments 900 In desperate need of sharding! Foreign keySchema for sharding Global Table (replicated on all shards)
  48. 48. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 ● OpenStack Controller Node – Manages Identities – Manages Images – Manages Block Storage ● OpenStack Compute Node – (Virtual) machine host – Contain hypervisor – Spawn and destroy virtual machines Controller Compute (VM Host) Elasticity through Virtualization
  49. 49. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2014-10-02 MySQL Fabric and OpenStack MySQL Fabric Node OpenStack Instance Application High-Availability Groups / Shards
  50. 50. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Closing remarks
  51. 51. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Where are we now? ● High Availability Features – Slave promotion – Multi-node Fabric Control Cluster – Failure detectors ● Failure detection – Built-in Failure Detector – Failure reporting – Custom failure detectors ● Connector APIs – Transaction properties – “Virtual” connections – Legacy connector support ● Using router ● Interfaces – Command-line – XML-RPC – MySQL-RPC
  52. 52. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Where are we now? ● Execution – Node failure stop execution ● Execution restart on recovery – Replicated State Machine (RSM) – Fail-over execution ● Security – RFC 2617 for HTTP authentication – SSL support over XML-RPC – Secure Password Management ● Cloud integration – “Server providers” – OpenStack Nova ● Sharding – Range and hash sharding – Shard move and split
  53. 53. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Thoughts for the Future ● Custom Procedures – Better support for custom procedures – Easy way to write high availability procedures ● Cloud support – OpenStack Trove – Amazon AWS – Amazon RDS ● Frameworks – Django? – Symphony? ● More Fabric-aware connectors – C/C++? ● Improved provisioning – Using on-line backup? – Using filesystem snapshots?
  54. 54. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Thoughts for the Future ● Integration Router-Fabric – Better integration of Router and Fabric – Failure reporting in Router – Multi-node Fabric Support in Router ● Integration with Group Replication – Router to balance write load ● Enterprise Monitoring – Monitor farm status – Monitor Fabric nodes ● Dynamic Router Configuration – On-line configuration changes ● Transaction Multiplexing – Allow several incoming clients to use a single connection to the server – Transactions are interleaved
  55. 55. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Thoughts for the Future ● Router Monitoring – Statistics collection – Monitoring interfaces ● Query Consolidation – Consolidate result sets from identical queries “in flight” ● Router Logging – Logging plugin interface – Sending logs to different sinks ● Result set caching – Allow router to cache result sets – Transparent to application ● Session consistency – We have a distributed database – It should look like a single database – Read monotonicity – Write monotonicity
  56. 56. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 2015-10-28 Thoughts for the Future ● Automatic sharding – Single-query transactions? – Speculative execution? – Cross-shard queries? ● Shard aggregation – Cross-shard queries ● Connection-based Sharding – Sharding OLTP workload in router – Assign shard key range to router port – Router send session to shard based on port ● Multi-way shard split – Efficient initial sharding – Better use of resources
  57. 57. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Safe Harbor Statement The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. 2015-10-28

×