Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MySQL InnoDB Cluster / ReplicaSet - Making Provisioning & Troubleshooting as easy as pie

211 views

Published on

Session presented at pre-FOSDEM MySQL Days 2020.

MySQL InnoDB Cluster and ReplicaSet provide failover/high availability and scaling features baked in; providing an integrated end-to-end solution that is easy to use.

Recent enhancements and features added to MySQL Shell make the management of InnoDB Clusters / ReplicaSets even more powerful and effortless! Full instance provisioning and cluster troubleshooting became easy as 1, 2, 3.

Published in: Software

MySQL InnoDB Cluster / ReplicaSet - Making Provisioning & Troubleshooting as easy as pie

  1. 1. The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any statement in light of new information or future events. Safe Harbor Copyright © 2019 Oracle and/or its affiliates.
  2. 2. $ whoami Copyright © 2019 Oracle and/or its affiliates.
  3. 3. Miguel Araújo • MySQL Developer since 2011 • AdminAPI Tech Lead @ MySQL Shell team • Living in Portugal • http://about.me/miguel-araujo • @M1guelAraujo Copyright © 2019 Oracle and/or its affiliates.
  4. 4. MySQL Shell AdminAPI Copyright © 2019 Oracle and/or its affiliates.
  5. 5. Copyright © 2019 Oracle and/or its affiliates. MySQL Shell • Interactive multi-language console interface that supports development and administration for the MySQL Server: • Supports JavaScript, Python and SQL • Interactive and batch operations (scripting) • Supports Document and Relational models (X and MySQL protocols)
  6. 6. Copyright © 2019 Oracle and/or its affiliates. MySQL Shell • Exposes full Development and Administration APIs: • Scriptable “DevOps” APIS • Unified interface for developers and DBAs
  7. 7. Copyright © 2019 Oracle and/or its affiliates. MySQL Shell AdminAPI • Creation and Management of InnoDB Clusters / ReplicaSets • Available in both JavaScript and Python; and command-line • Simple and straight-forward • Flexible, powerful and secure!
  8. 8. Copyright © 2019 Oracle and/or its affiliates. •MySQL Group Replication •MySQL Shell •MySQL Router InnoDB Cluster “A single product – MySQL – with high availability and scaling features baked in; providing an integrated end-to-end solution that is easy to use.”
  9. 9. Copyright © 2019 Oracle and/or its affiliates. •MySQL Replication •MySQL Shell •MySQL Router “We take a strong technology that is very powerful but can be complex, and provide an easy-to-use AdminAPI for it in the MySQL Shell” InnoDB ReplicaSet
  10. 10. Copyright © 2019 Oracle and/or its affiliates. Creating a Cluster - AdminAPI // Configure Instances for Cluster usage mysqlsh-js> dba.configureInstance(“root@mysql1:3306”) // Create the Cluster mysqlsh-js> c root@mysql1:3306 mysqlsh-js> cluster = dba.createCluster(“myCluster”) // Add Instances to the Cluster mysqlsh-js> cluster.addInstance(“root@mysql2:3306”)
  11. 11. Copyright © 2019 Oracle and/or its affiliates. Creating a ReplicaSet- AdminAPI // Configure Instances for ReplicaSet usage mysqlsh-js> dba.configureReplicaSetInstance(“root@mysql1:3306”) // Create the ReplicaSet mysqlsh-js> c root@mysql1:3306 mysqlsh-js> replicaset = dba.createReplicaSet(“myReplicaSet”) // Add Instances to the ReplicaSet mysqlsh-js> replicaset.addInstance(“mysql2:3306”)
  12. 12. Couldn’t be easier! Powerful solutions supported by Shell’s ease of use. Copyright © 2019 Oracle and/or its affiliates.
  13. 13. Node Provisioning Copyright © 2019 Oracle and/or its affiliates.
  14. 14. Copyright © 2019 Oracle and/or its affiliates. Provisioning • Provisioning a server is the process of configuring and getting it ready to be used in an InnoDB Cluster/ReplicaSet. • In InnoDB Cluster, the server needs to have all clusters’ data to be able to join it • That’s typically done by: 1. Performing a backup of one of the cluster members 2. Restoring the image in the new member that wants to join
  15. 15. Copyright © 2019 Oracle and/or its affiliates. Distributed Recovery • Distributed recovery is a process that runs when a new member is added to the group and has missing transactions • A donor that can provide the missing data is automatically selected • The catch-up happens transparently and the server is synchronized with the rest of the cluster
  16. 16. Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision • Incremental recovery fetches only the missing data based on the GTIDs into - The cluster must be able to provide all the missing data • Relies on the classic Asynchronous Replication • Also provides group transactional information
  17. 17. Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision Some data is inserted in the DB
  18. 18. Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision The data modifications are stored in the binlog. And the transactions are delivered to every cluster member
  19. 19. Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision Group Replication is eventually synchronous. Guarantees that transactions are delivered to all members in the same order.
  20. 20. Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision The binlogs keep track of all data modifications.
  21. 21. Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision A new member wants to join the cluster
  22. 22. Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision A new member wants to join the cluster The missing data is fetched and the server synchronizes with the rest of the cluster
  23. 23. Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision A donor is automatically selected to fetch the missing data from, using plain Asynchronous Replication
  24. 24. Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision A donor is automatically selected to fetch the missing data from, using plain Asynchronous Replication
  25. 25. Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision A donor is automatically selected to fetch the missing data from, using plain Asynchronous Replication
  26. 26. Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision The instance successfully joins the cluster!
  27. 27. Incremental Recovery to Provision May take a considerable amount of time ! Copyright © 2019 Oracle and/or its affiliates.
  28. 28. Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision • But what if the cluster members lost all binary logs? • Or even... Just some data is missing that is not available on any binary log • What about space constraints? ?
  29. 29. Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision Some binary logs were purged. And the instance has no data yet.
  30. 30. Copyright © 2019 Oracle and/or its affiliates. Incremental Recovery to Provision Some binary logs were purged. And the instance has no data yet. Failed attempt to join the cluster !
  31. 31. Copyright © 2019 Oracle and/or its affiliates. MySQL Clone Plugin • Performs physical snapshots of databases Really fast! • Transfers snapshots over the network to provision servers Can be used remotely! • Replaces all server data Available in InnoDB Cluster and ReplicaSet !
  32. 32. Copyright © 2019 Oracle and/or its affiliates. Clone to Provision Common issues solved with Clone: • Very large datasesets • Incomplete binary log history • Re-provision diverged instances
  33. 33. Copyright © 2019 Oracle and/or its affiliates. Clone to Provision Some binary logs were purged. And the instance has no data yet.
  34. 34. Copyright © 2019 Oracle and/or its affiliates. Clone to Provision Some binary logs were purged. And the instance has no data yet. Clone required !
  35. 35. Copyright © 2019 Oracle and/or its affiliates. Clone to Provision A donor is automatically selected. And a full snapshot of its data is performed.
  36. 36. Copyright © 2019 Oracle and/or its affiliates. Clone to Provision The instance successfully joins the cluster!
  37. 37. Clone-based Recovery Live Demo Copyright © 2019 Oracle and/or its affiliates. InnoDB Cluster
  38. 38. Copyright © 2019 Oracle and/or its affiliates. •Incremental recovery is possible if: • At least 1 member can provide the joiner with the same set of transactions it has processed. • The joiner does not have errant transactions •Incremental recovery appropriate if: • No purged transactions • Joiner does not have an empty GTID set • GTID support and binary logs were always enabled Which recovery shall be used?
  39. 39. Incremental recovery safe and appropriate: Copyright © 2019 Oracle and/or its affiliates. AdminAPI Guidance
  40. 40. Copyright © 2019 Oracle and/or its affiliates. AdminAPI Guidance Clone-based recovery required:
  41. 41. Clone-based Recovery Live Demo Copyright © 2019 Oracle and/or its affiliates. InnoDB ReplicaSet
  42. 42. Copyright © 2019 Oracle and/or its affiliates. InnoDB Cluster/ReplicaSet • Built-in support for full instance provisioning! • AdminAPI transparently: ü Setups the cluster/replicaset to be ready for clone usage (plugin setup, user grants, etc.) ü Detects every possible scenario and guides the user accordingly to the best provisioning method ü Still allowing manual intervention to choose the provisioning method
  43. 43. Monitoring Copyright © 2019 Oracle and/or its affiliates.
  44. 44. Copyright © 2019 Oracle and/or its affiliates. Node Provisioning Monitoring • The AdminAPI was extended to provide monitoring features: ü For any type of recovery method: clone or incremental ü Real-time progress information ü Detailed stage status ü Asynchronous progress information from cluster.status()
  45. 45. Copyright © 2019 Oracle and/or its affiliates. Node Provisioning Monitoring Clone-based recovery progress information from: cluster.status()
  46. 46. Copyright © 2019 Oracle and/or its affiliates. Cluster Status Entry point for Cluster monitoring: ü Provides general info about the cluster ü Provides info about each member status mysqlsh-js> cluster.status()
  47. 47. Copyright © 2019 Oracle and/or its affiliates. Extended Status Provides information of: • GR Group name + protocol version • Cluster members UUIDs • Cluster members roles and states • Read-only settings mysqlsh-js> cluster.status({extended: 1})
  48. 48. Copyright © 2019 Oracle and/or its affiliates. Extended Status Provides information of: • Transactions processed by connection and applier threads mysqlsh-js> cluster.status({extended: 2})
  49. 49. Copyright © 2019 Oracle and/or its affiliates. Extended Status Provides information of: • Detailed stats about the replication machinery of each cluster member mysqlsh-js> cluster.status({extended: 3})
  50. 50. Copyright © 2019 Oracle and/or its affiliates. ReplicaSet Status Entry point for ReplicaSet monitoring: ü Provides general info about the replicaset ü Provides info about each member status mysqlsh-js> replicaset.status()
  51. 51. Copyright © 2019 Oracle and/or its affiliates. Router Monitoring Provides information of: • Router info: name, version • Last check-in • Hostname • RO/RW end ports mysqlsh-js> cluster.listRouters()
  52. 52. Maintenance and Troubleshooting Copyright © 2019 Oracle and/or its affiliates.
  53. 53. Anything that can go wrong will go wrong.”“ Copyright © 2019 Oracle and/or its affiliates.
  54. 54. Monitoring a Cluster/ReplicaSet can provide info/hints about: • Faulty Networks • Server crashes/failures • High latencies Copyright © 2019 Oracle and/or its affiliates. Maintenance • Total outages • … and much more!
  55. 55. Copyright © 2019 Oracle and/or its affiliates. Cluster Options Provides information of: • The cluster global configuration options • Each cluster member settings mysqlsh-js> cluster.options()
  56. 56. memberWeight Next primary in line when a failover happens autoRejoinTries Automatically join back group after leaving/being expelled expelTimeout Avoid frequent member expels due to less reliable networks consistency Transaction consistency guarantees Copyright © 2019 Oracle and/or its affiliates. Cluster Options
  57. 57. memberWeight Highest value will be first to be elected as primary in a failover autoRejoinTries Set to 3 expelTimeout Set to 5 Copyright © 2019 Oracle and/or its affiliates. Cluster Options – Best practices
  58. 58. Copyright © 2019 Oracle and/or its affiliates. Dynamically Changing Settings // Change the member-weight value of a member mysqlsh-js> cluster.setInstanceOption(“mysql2:3307”, “memberWeight”, 75) // Change the auto-rejoin-tries of the cluster mysqlsh-js> cluster.setOption(“autoRejoinTries”, 3) // Change the cluster’s consistency level mysqlsh-js> cluster.setOption(“consistency”, “AFTER”)
  59. 59. Copyright © 2019 Oracle and/or its affiliates. Topology Modes • The cluster’s primary instance can be changed, immediately:
  60. 60. Copyright © 2019 Oracle and/or its affiliates. Topology Modes • The topology-mode can be changed live: // Change the topology-mode to multi-primary mysqlsh-js> cluster.switchToMultiPrimaryMode() // Change back to single-primary-mode mysqlsh-js> cluster.switchToSinglePrimaryMode()
  61. 61. Copyright © 2019 Oracle and/or its affiliates. Security • Internal recovery accounts have randomly secure generated passwords • But security policies may enforce passwords lifetime ü Support to reset a cluster’s recovery accounts credentials added // Reset the recovery accounts credentials mysqlsh-js> cluster.resetRecoveryAccountsPassword()
  62. 62. Some scenarios may require intervention… • Network partitions • Complete outage • Server upgrades Copyright © 2019 Oracle and/or its affiliates. Troubleshooting • Crashed members of a Cluster or ReplicaSet
  63. 63. Copyright © 2019 Oracle and/or its affiliates. Network Partitions • A network partition can cause the loss of quorum • The cluster is not able to process any more transactions
  64. 64. Copyright © 2019 Oracle and/or its affiliates. Network Partitions • Using a single AdminAPI command the cluster can be easily unblocked mysqlsh-js> cluster.forceQuorumUsingPartitionOf(“clusterAdmin@server1:3306”)
  65. 65. Copyright © 2019 Oracle and/or its affiliates. Complete Outage • Very unlikely to happen! • But... • When it happens, restoring the instances back online is not enough • The Group Replication group is completely gone • Other configurations in place may have been lost
  66. 66. Copyright © 2019 Oracle and/or its affiliates. Complete Outage • Easily restore the Cluster back to its previously configured state using: mysqlsh-js> c clusterAdmin@server1:3306 mysqlsh-js> dba.rebootClusterFromCompleteOutage()
  67. 67. Copyright © 2019 Oracle and/or its affiliates. Rejoin Instance • An instance may not be able to automatically rejoin the cluster/replicaset: • Instance running MySQL 5.7 without settings persisted • Instance from minority partition • Invalidated primary from ReplicaSet
  68. 68. Copyright © 2019 Oracle and/or its affiliates. Rejoin Instance • Full detection of best provisioning method supported for rejoining instances mysqlsh-js> cluster.rejoinInstance(“server3:3306”) mysqlsh-js> replicaset.rejoinInstance(“server3:3306”, {recoveryMethod: “clone”})
  69. 69. Copyright © 2019 Oracle and/or its affiliates. Upgrade Checker • Verify whether instances are ready or not for upgrading to MySQL 8 • Available through the objectutil mysqlsh-js> util.checkForServerUpgrade(“root@localhost:3306”)
  70. 70. Copyright © 2019 Oracle and/or its affiliates. Upgrade Checker
  71. 71. Closing Copyright © 2019 Oracle and/or its affiliates.
  72. 72. Copyright © 2019 Oracle and/or its affiliates. Final Thoughts • InnoDB Cluster is THE built-in HA solution for MySQL • InnoDB ReplicaSet fully integrates MySQL Replication with Router and Shell brings ease of use! • MySQL Clone Plugin pushed InnoDB Cluster and ReplicaSet to a whole new level of usability! • MySQL Shell is the unified interface for developers and DBAs and the frontend manager for InnoDB Cluster and ReplicaSet
  73. 73. Copyright © 2019 Oracle and/or its affiliates. MOAR Info • https://dev.mysql.com/doc/refman/8.0/en/mysql-innodb- cluster-userguide.html (InnoDB cluster userguide) https://dev.mysql.com/doc/dev/mysqlsh-api-javascript/8.0/ (Shell APIs JS Ref manual) https://dev.mysql.com/doc/dev/mysqlsh-api-python/8.0/ (Shell APIs Python Ref manual) • https://mysqlhighavailability.com (Blogging) • http://mysqlserverteam.com/ (Blogging) • https://mysqlcommunity.slack.com (Community Slack)
  74. 74. Thank you! Copyright © 2019 Oracle and/or its affiliates.

×