MySQL InnoDB Cluster: Management and Troubleshooting with MySQL Shell

Copyright © 2019 Oracle and/or its affiliates.
MySQL InnoDB Cluster
Management and troubleshooting
using MySQL Shell
Senior Software Developer
MySQL Middleware and Clients
September 18, 2019
Miguel Araújo

The following is intended to outline our general product direction. It is intended for information purposes
only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code,
or functionality, and should not be relied upon in making purchasing decisions. The development,
release, timing, and pricing of any features or functionality described for Oracle’s products may change
and remains at the sole discretion of Oracle Corporation.
Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and
prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed
discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and
Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q
under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website
at http://www.oracle.com/investor. All information in this presentation is current as of September 2019
and Oracle undertakes no duty to update any statement in light of new information or future events.
Safe Harbor

$ whoami

Miguel Araújo
• MySQL Developer since 2011
• AdminAPI Tech Lead @ MySQL Shell team
• MySQL InnoDB Cluster evangelist
• Living in Portugal
• http://about.me/miguel-araujo
• @M1guelAraujo

1.
2.
3.
4.
5.
Introduction to InnoDB cluster
Node Provisioning
Monitoring
Maintenance and Troubleshooting
Final thoughts and resources
What’s Ahead

Introduction to
InnoDB Cluster

• MySQL Group Replication
• MySQL Shell
• MySQL Router
InnoDB cluster
“A single product – MySQL – with high
availability and scaling features baked in;
providing an integrated end-to-end solution
that is easy to use.”

One Product:
MySQL
1
MySQL InnoDB Cluster Goals
• All components
developed and integrated
together
• Full stack testing

One Product:
MySQL
1 2
• All components
together
• One client: MySQL Shell
• Easy packaging
• Integrated Orchestration
• Homogeneous servers
Easy to use!

One Product:
MySQL
Flexible and
Modern
1 2 3
• All components
together
• One client: MySQL Shell
• Easy packaging
• Integrated Orchestration
• Homogeneous servers
Easy to use!
• SQL and noSQL together:
X Protocol
• Protocol Buffers
• Asynchronous API
• Developer friendly

!
Completely Open Source --- GPL
No license required to have High Availability!

• Interactive multi-language console interface that supports
development and administration for the MySQL Server:
• Supports JavaScript, Python and SQL
• Interactive and batch operations (scripting)
• Supports Document and Relational models (X and MySQL protocols)
MySQL Shell

• Exposes full Development and Administration APIs:
• Scriptable “DevOps” APIS
• Unified interface for developers and DBAs
MySQL Shell

• Creation and Management of InnoDB clusters
• Available in both JavaScript and Python; and command-line
• Simple and straight-forward
• Flexible, powerful and secure!
MySQL AdminAPI

Creating a Cluster
// Configure servers for MySQL InnoDB cluster usage
mysqlsh-js> dba.configureInstance(“clusteradmin@mysql1:3306”)

Creating a Cluster
// Connect to target instance
mysqlsh-js> c clusteradmin@mysql1:3306

Creating a Cluster
// Create the cluster
mysqlsh-js> var cluster = dba.createCluster(“CodeOne19”)

Creating a Cluster
// Create the cluster
mysqlsh-js> var cluster = dba.createCluster(“CodeOne19”)
// Add instances to the cluster
mysqlsh-js> cluster.addInstance(“clusteradmin@mysql2:3306”)

• Transparent client routing between the application and the
backend servers:
ü Application connection failover
• Native support for InnoDB clusters
• Stateless
MySQL Router

Node Provisioning

• Provisioning a server is the process of configuring and getting it
ready to be used in an InnoDB cluster.
• The server needs to have all clusters’ data to be able to join it
• That’s typically done by:
1. Performing a backup of one of the cluster members
2. Restoring the image in the new member that wants to join
Provisioning

• Distributed recovery is a process that runs when a new member is
added to the group and has missing transactions
• A donor that can provide the missing data is automatically selected
• The catch-up happens transparently and the server is
synchronized with the rest of the cluster
Distributed Recovery

• Incremental recovery fetches only the missing data based on the
GTIDs into
- The cluster must be able to provide all the missing data
• Relies on the classic Asynchronous Replication
• Also provides group transactional information
Incremental Recovery to Provision

Some data is inserted in
the DB

The data modifications are
stored in the binlog.
And the transactions are
delivered to every cluster
member

Group Replication is
eventually synchronous.
Guarantees that
transactions are delivered
to all members in the
same order.

The binlogs keep track of
all data modifications.

A new member wants to
join the cluster

A new member wants to
join the cluster
The missing data is
fetched and the server
synchronizes with the rest
of the cluster

A donor is automatically
selected to fetch the
missing data from, using
plain Asynchronous
Replication

The instance successfully
joins the cluster!

!May take a considerable amount of time

• But what if the cluster members lost all binary logs?
• Or even... Just some data is missing that is not
available on any binary log
• What about space constraints?
?

Some binary logs were
purged.
And the instance has no
data yet.

purged.
data yet.
Failed attempt to join the
cluster!

• Performs physical snapshots of databases
ü Really fast!
• Transfers the snapshots over the network to provision servers
ü Can be used remotely!
• Replaces all the server data
MySQL Clone Plugin
!Available since 8.0.17

Distributed recovery = initial provisioning + online catch-up
• Seamless and automated solution for provisioning
• No external tools or scripting required
Clone to Provision

Common issues easily solved with clone:
• Very large datasets
• Incomplete binary log history
• Re-provision diverged instances
Clone to Provision

Clone to Provision
purged.
data yet.

Clone to Provision
purged.
data yet.
Clone required !

Clone to Provision
A donor is automatically
selected.
And a full snapshot of its
data is performed.

Clone to Provision
The instance successfully
joins the cluster!

Clone-based
Recovery Demo

• Incremental recovery is possible if:
• At least 1 member can provide the joiner
with the same set of transactions it has
processed.
• The joiner does not have errants
transactions
• Incremental recovery appropriate if:
• No purged transactions
• Joiner does not have an empty GTID set
• GTID support and binary logs were always
enabled
Which recovery shall be used?

AdminAPI Guidance
Incremental recovery safe and appropriate:

AdminAPI Guidance
Clone-based recovery required:

• Built-in support for full instance provisioning!
• AdminAPI transparently:
ü Setups the cluster to be ready for clone usage (plugin setup, user grants, etc.)
ü Detects every possible scenario and guides the user accordingly to the best provisioning
method
ü Still allowing manual intervention to choose the provisioning method
InnoDB cluster 8.0.17+

• Binlogs can now be purged frequently!
• But (at least) one cluster member should keep them for 2 days
(minimum) for recovery.
Just a tip!
!

Monitoring

Node Provisioning Monitoring
The AdminAPI was extended to provide monitoring features:
• For any type of recovery method: clone or incremental
• Real-time progress information
• Detailed stage status
• Asynchronous progress information from cluster.status()

ü Clone-based recovery real-time progress info

ü Clone-based recovery progress from cluster.status()

Cluster Status
Provides general information
about the cluster and each
member status
mysqlsh-js> cluster.status()

Extended Status
Provides information of:
• GR Group name
• Group protocol version
• Cluster members UUIDs
• Cluster members roles and states
• Read-only settings
mysqlsh-js>
cluster.status({extended: 1})

Extended Status
• Transactions processed by
connection and applier
mysqlsh-js>

Extended Status
• Detailed stats about the replication
machinery of each cluster member
mysqlsh-js>

• Built-in reports
• User-defined reports
Reporting Framework
// displays the result one time
mysqlsh-js> show query <query>
// executes repeatedly with screen refresh
mysqlsh-js> watch query <query>
// registers a user-defined-report
mysqlsh-js> shell.registerReport(<UDR>)

• Built-in reports
• User-defined reports
Reporting Framework
// displays the result one time
mysqlsh-js> show query <query>
// executes repeatedly with screen refresh
mysqlsh-js> watch query <query>
// registers a user-defined-report
mysqlsh-js> shell.registerReport(<UDR>)
! Monitor P_S to check which
secondary instance is slower

Maintenance and
Troubleshooting

Maintenance
Monitoring a cluster can provide info/hints about:
• Faulty networks
• High latencies
• Server failures
• ...

Cluster Options
• The cluster global configuration
options
• Each cluster member settings
mysqlsh-js> cluster.options()

expelTimeout
Avoid frequent member expels
due to faulty network
memberWeight
Next primary in line when a
failover happens
exitStateAction
Action when a server is expelled
from the cluster: abort, read-only,
offline-mode
autoRejoinTries
Rejoin attempts after an eviction
Cluster options

Changing Cluster Options
// Change the member-weight value of a member
mysqlsh-js> cluster.setInstanceOption(“mysql2:3307”, “memberWeight”, 75)

// Change the exit-state-action of a member
mysqlsh-js> cluster.setInstanceOption(“mysql3”, “exitStateAction”, “ABORT_SERVER”)

// Change the exit-state-action of a member
mysqlsh-js> cluster.setInstanceOption(“mysql3”, “exitStateAction”, “ABORT_SERVER”)
// Change the expel-timeout of the cluster
mysqlsh-js> cluster.setOption(“expelTimeout”, 60)

Consistency Levels
• InnoDB cluster operates, by default, in single-primary mode:
• Apps and users write to the primary instance
• What guarantees do we have that:
• We always read the most up-to-date data?
• We do not read stale data of an evicted server?
• Dirty reads do not happen?
?

Consistency Levels
// Create a cluster with “BEFORE” consistency level
// Ensures that reads always read the most up-to-data data
mysqlsh-js> cluster = dba.createCluster(“CodeOne19”, {consistency: “BEFORE”})

Consistency Levels
// Create a cluster with “BEFORE” consistency level
// Ensures that reads always read the most up-to-data data
mysqlsh-js> cluster = dba.createCluster(“CodeOne19”, {consistency: “BEFORE”})
// Change the consistency level at any time
// AFTER ensures that there are no dirty reads
mysqlsh-js> cluster.setOption(“consistency”, “AFTER”)

Topology Modes
• The cluster’s primary instance can be changed, immediately:

Topology Modes
• The topology-mode can be changed live:
// Change the topology-mode to multi-primary
mysqlsh-js> cluster.switchToMultiPrimaryMode()
// Change back to single-primary-mode
mysqlsh-js> cluster.switchToSinglePrimaryMode()

Security
• Internal recovery accounts have randomly secure generated
passwords
• But security policies may enforce passwords lifetime
ü Support to reset a cluster’s recovery accounts credentials added
// Reset the recovery accounts credentials
mysqlsh-js> cluster.resetRecoveryAccountsPassword()
Available in 8.0.18

“Anything that can go wrong will go wrong.”
Troubleshooting

“Anything that can go wrong will go wrong.”
Troubleshooting
• Shell’s interactive behavior provides the users with valuable
information, by default.
• But sometimes more information is needed for troubleshooting...

Logging
Logging levels:
• 1: none
• 2: internal
• 3: error
• 4: warning
• 5: info
• 6, 7, 8: debug
$ mysqlsh –-log-level=8

Logging
Verbose output level:
• [1, 4]
• Enables diagnostic
messages output
$ mysqlsh –-verbose=4

Logging
Log AdminAPI SQL:
• 0: no logging
• 1: log all queries
except SELECT and
SHOW
• 2: log all queries
$ mysqlsh –-dba-log-sql=2
Available in 8.0.18

• InnoDB cluster is THE built-in HA solution for MySQL
• MySQL Clone Plugin pushed InnoDB cluster to a whole new
level of usability!
• InnoDB Cluster features built-in support for full instance provisioning
• MySQL Shell is the unified interface for developers and DBAs
and the frontend manager for InnoDB cluster
Final Thoughts

Resources
• https://dev.mysql.com/doc/refman/8.0/en/mysql-innodb-
cluster-userguide.html (InnoDB cluster userguide)
https://dev.mysql.com/doc/dev/mysqlsh-api-javascript/8.0/ (Shell APIs JS Ref manual)
https://dev.mysql.com/doc/dev/mysqlsh-api-python/8.0/ (Shell APIs Python Ref manual)
• https://mysqlhighavailability.com (Blogging)
• http://mysqlserverteam.com/ (Blogging)
• https://mysqlcommunity.slack.com (Community Slack)

Thank you!

• MySQL Shell: The Best DBA Tool? How to Use MySQL Shell as a
Framework for DBAs [DEV2056]
• What’s New in MySQL Optimizer and Executor? [DEV2077]
• MySQL Connector/J 8.0 Feature Review and How to Upgrade from
Connector/J 5.1 [DEV6221]
• MySQL 8.0 Enterprise Backup [DEV4110]
• Group Replication: How I Stopped Worrying About Consistency and
Focused on Building Apps [DEV3016]
MOAR MySQL

Session Survey
Help us make the content
even better. Please complete
the session survey in the
Mobile App.

The preceding is intended to outline our general product direction. It is intended for information purposes
only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code,
or functionality, and should not be relied upon in making purchasing decisions. The development,
release, timing, and pricing of any features or functionality described for Oracle’s products may change
and remains at the sole discretion of Oracle Corporation.
Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and
prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed
discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and
Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q
under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website
at http://www.oracle.com/investor. All information in this presentation is current as of September 2019
and Oracle undertakes no duty to update any statement in light of new information or future events.
Safe Harbor

MySQL InnoDB Cluster: Management and Troubleshooting with MySQL Shell

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to MySQL InnoDB Cluster: Management and Troubleshooting with MySQL Shell

Similar to MySQL InnoDB Cluster: Management and Troubleshooting with MySQL Shell (20)

More from Miguel Araújo

More from Miguel Araújo (14)

Recently uploaded

Recently uploaded (20)

MySQL InnoDB Cluster: Management and Troubleshooting with MySQL Shell