Session delivered during Oracle MySQL Innovation Day 2019 in Tokyo. How to use MySQL Shell as a framework for DBAs

1. 1 / 69

2. MySQL Shell : the best DBA tool ?
How to use MySQL Shell as a framework for DBAs
Frédéric Descamps
Community Manager
MySQL
September 2019
2 / 69

3.
Safe Harbor
The following is intended to outline our general product direction. It is intended for information purpose only, and
may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality,
and should not be relied up in making purchasing decisions. The development, release, timing and pricing of any
features or functionality described for Oracle´s product may change and remains at the sole discretion of Oracle
Corporation.
Statement in this presentation relating to Oracle´s future plans, expectations, beliefs, intentions and ptospects
are "forward-looking statements" and are subject to material risks and uncertainties. A detailed discussion of
these factors and other risks that a ect our business is contained in Oracle´s Securities and Exchange
Commission (SEC) lings, including our most recent reports on Form 10-K and Form 10-Q under the heading
"Risk Factors". These lings are available on the SEC´s website or on Oracle´s website at
h p://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle
undertakes no duty to update any statement in light of new information or future events.
Copyright @ 2019 Oracle and/or its affiliates.
3 / 69

4. about.me/lefred
Who am I ?
Copyright @ 2019 Oracle and/or its affiliates.
4 / 69

5. Frédéric Descamps
@lefred
MySQL Evangelist
Managing MySQL since 3.23
devops believer
living in Belgium 🇧🇪
h ps://lefred.be
Copyright @ 2019 Oracle and/or its affiliates.
5 / 69

6. a new tool
MySQL Shell
Copyright @ 2019 Oracle and/or its affiliates.
6 / 69

7. MySQL Shell
The MySQL Shell is an interactive Javascript, Python, or SQL interface supporting
development and administration for the MySQL Server and is a component of the MySQL
Server. You can use the MySQL Shell to perform data queries and updates as well as
various administration operations.
Copyright @ 2019 Oracle and/or its affiliates.
7 / 69

8. MySQL Shell (2)
The MySQL Shell provides:
Copyright @ 2019 Oracle and/or its affiliates.
8 / 69

9. MySQL Shell (2)
The MySQL Shell provides:
Both Interactive and Batch operations
Copyright @ 2019 Oracle and/or its affiliates.
9 / 69

10. MySQL Shell (2)
The MySQL Shell provides:
Both Interactive and Batch operations
Document and Relational Models
Copyright @ 2019 Oracle and/or its affiliates.
10 / 69

11. MySQL Shell (2)
The MySQL Shell provides:
Both Interactive and Batch operations
Document and Relational Models
CRUD Document and Relational APIs via scripting
Copyright @ 2019 Oracle and/or its affiliates.
11 / 69

12. MySQL Shell (2)
The MySQL Shell provides:
Both Interactive and Batch operations
Document and Relational Models
CRUD Document and Relational APIs via scripting
Traditional Table, JSON, Tab Separated output results formats
Copyright @ 2019 Oracle and/or its affiliates.
12 / 69

13. MySQL Shell (2)
The MySQL Shell provides:
Both Interactive and Batch operations
Document and Relational Models
CRUD Document and Relational APIs via scripting
Traditional Table, JSON, Tab Separated output results formats
MySQL Standard and X Protocols
Copyright @ 2019 Oracle and/or its affiliates.
13 / 69

14. MySQL Shell (2)
The MySQL Shell provides:
Both Interactive and Batch operations
Document and Relational Models
CRUD Document and Relational APIs via scripting
Traditional Table, JSON, Tab Separated output results formats
MySQL Standard and X Protocols
and more...
Copyright @ 2019 Oracle and/or its affiliates.
14 / 69

15. MySQL Shell Overview
PY
JS
SQL
>
MySQL 8.0
Upgrade Checker
Auto Completion
&
Command History
Output
Formats
(table, json, tabs)
Prompt
Themes
Batch
Execution
5.7
8.0
JS
Document
Store
X dev API
SQL CLI InnoDB
Cluster
importJSON
JSJS
parallel
importTABLE
data
Copyright @ 2019 Oracle and/or its affiliates.
15 / 69

16. setup your environment
Starting with MySQL Shell
Copyright @ 2019 Oracle and/or its affiliates.
16 / 69

17. Install MySQL Shell
the package's name to install is mysql-shell
you always need to use the latest available even with a lower MySQL Server version
latest version if 8.0.17, use it even with MySQL 5.7.x
Shell
Copyright @ 2019 Oracle and/or its affiliates.
17 / 69

18. Start a beautiful Shell
The default prompts is basic, we provide several di erent prompt examples.
# cp /usr/share/mysqlsh/prompt/prompt_256pl+aw.json ~/.mysqlsh/prompt.json
Then start the mysqlsh:
Copyright @ 2019 Oracle and/or its affiliates.
18 / 69

19. Some configuration tips
These are some se ings I usually use:
mysql-js> shell.options.setPersist('history.autoSave', 1)
mysql-js> shell.options.setPersist('history.maxSize', 5000)
You can swithc to all 3 di erent modes using:
py for Python
js for Javascript
sql for SQL
If you have a preferred mode, you can specify it like this:
mysql-js> shell.options.setPersist('defaultMode', 'sql')
Copyright @ 2019 Oracle and/or its affiliates.
19 / 69

20. Util
The MySQL Shell comes with a global object called util. This object is used to perform
some dedicated DBA tasks like:
Checking if the server's con guration and schema design is compatible with next
MySQL version (checkForServerUpdrade())
Importing JSON documents, like exports from MondoDB (importJSON())
Importing table dumps in parallel (importTable())
Con guring OCI pro le (con gureOCI())
Copyright @ 2019 Oracle and/or its affiliates.
20 / 69

21. Admin API
The Admin API is used to con gure MySQL instances to be part of a MySQL InnoDB
Cluster.
It also creates and manage clusters.
More Info: h ps://lefred.be/content/mysql-innodb-cluster-from-scratch-even-more-
easy-since-8-0-17/
Copyright @ 2019 Oracle and/or its affiliates.
21 / 69

22. Admin API in action
Copyright @ 2019 Oracle and/or its affiliates.
22 / 69

23. we want more !
Extending MySQL Shell
Copyright @ 2019 Oracle and/or its affiliates.
23 / 69

24. MySQL Shell and Javascript
In JS mode, it's also possible to load some pure Javascript modules installed in sys.path:
Copyright @ 2019 Oracle and/or its affiliates.
24 / 69

25. MySQL Shell and Javascript (2)
The module has to be pure JS (not node.JS) modules:
Copyright @ 2019 Oracle and/or its affiliates.
25 / 69

26. MySQL Shell and Javascript (3)
Then your module can be loaded and used like this:
Copyright @ 2019 Oracle and/or its affiliates.
26 / 69

27. MySQL Shell and Python
When using the python mode in the Shell, it's even possible to use system modules (local).
Copyright @ 2019 Oracle and/or its affiliates.
27 / 69

28. MySQL Shell and Python (2)
Of course this can be any type of modules:
Copyright @ 2019 Oracle and/or its affiliates.
28 / 69

29. MySQL Shell and Python (2)
Of course this can be any type of modules:
Copyright @ 2019 Oracle and/or its affiliates.
29 / 69

30. Extending MySQL Shell
Since 8.0.17, you have 3 di erent ways to extend the MySQL Shell:
using the new Reporting Framework (>= 8.0.16)
create your own modules to extend MySQL Shell (obsolete)
create your own plugins that will be completely integrated in all supported interpreters
(>= 8.0.17)
8.0
Copyright @ 2019 Oracle and/or its affiliates.
30 / 69

31. MySQL Shell User-Defined Reports
You can create reports that are called once on demand (show) or constantly refreshed
(watch).
Example for checking the LOCKS:
Copyright @ 2019 Oracle and/or its affiliates.
31 / 69

32. MySQL Shell User-Defined Reports (2)
This is a Python le (locks_info.py) installed in ~/.mysqlsh/init.d:
It contains a de nition and a registration:
def locks_info(session):
query = session.sql("select ...")
...
return {'report': report}
shell.register_report(
'locks_info',
'list',
locks_info,
{
'brief': 'Shows Locks.',
'details': ['You need the SELECT privilege on sys.session view and the ..'],
'argc': '0'
}
)
Copyright @ 2019 Oracle and/or its affiliates.
32 / 69

33. MySQL Shell User-Defined Reports (3)
User-De ned Reports can be called from any MySQL Shell mode, even SQL:
Copyright @ 2019 Oracle and/or its affiliates.
33 / 69

34. MySQL Shell User-Defined Reports (4)
More info:
h ps://lefred.be/content/using-the-new-mysql-shell-reporting-framework-to-
monitor-innodb-cluster/
h ps://lefred.be/content/mysql-innodb-cluster-recovery-process-monitoring-with-
the-mysql-shell-reporting-framework/
Copyright @ 2019 Oracle and/or its affiliates.
34 / 69

35. MySQL Shell User-Defined Reports (4)
Sources of Examples:
h ps://github.com/lefred/mysql-shell-udr
Pull Requests welcome !
Copyright @ 2019 Oracle and/or its affiliates.
35 / 69

36. Why should I extend the MySQL Shell with
plugins?
For calling some long statements or group of operations or sometimes to replace a
missing functionality.
Copyright @ 2019 Oracle and/or its affiliates.
36 / 69

37. Why should I extend the MySQL Shell with
plugins?
For calling some long statements or group of operations or sometimes to replace a
missing functionality.
Recently, somebody pointed out that since the new DD it was not anymore possible to
delete all routines for a speci c schema.
Copyright @ 2019 Oracle and/or its affiliates.
37 / 69

38. Why should I extend the MySQL Shell with
plugins?
For calling some long statements or group of operations or sometimes to replace a
missing functionality.
Recently, somebody pointed out that since the new DD it was not anymore possible to
delete all routines for a speci c schema.
Jesper explained how the MySQL Shell could help here see
h ps://mysql.wisborg.dk/2018/12/02/mysql-8-drop-several-stored-events-procedures-
or-functions/
Copyright @ 2019 Oracle and/or its affiliates.
38 / 69

39. Using a plugin within MySQL Shell
Plugins are available from the ext global object:
Copyright @ 2019 Oracle and/or its affiliates.
39 / 69

40. Using a plugin within MySQL Shell (2)
Help is automatically available:
Copyright @ 2019 Oracle and/or its affiliates.
40 / 69

41. Using a plugin within MySQL Shell (3)
Plugins are available from JS and Python, regardless of the language in which they are
wri en:
Copyright @ 2019 Oracle and/or its affiliates.
41 / 69

42. Using a plugin wi thin MySQL Shell (4)
The help goes from the plugin to its methods:
Copyright @ 2019 Oracle and/or its affiliates.
42 / 69

43. Using a plugin within MySQL Shell (5)
And nally the plugin in action:
Copyright @ 2019 Oracle and/or its affiliates.
43 / 69

44. Using a plugin within MySQL Shell (5)
And nally the plugin in action:
Copyright @ 2019 Oracle and/or its affiliates.
44 / 69

45. How to use MySQL Shell Plugins ?
Plugins must be installed inside ~/.mysqlsh/plugins/ext directory.
Several existing plugins can be found at h ps://github.com/lefred/mysqlshell-plugins
Plugins are split in di erent categories:
audit innodb_cluster router
collations maintenance schema
con guration performance security
demo proxysql support
innodb ...
Copyright @ 2019 Oracle and/or its affiliates.
45 / 69

46. Overview of some plugins
Extending MySQL Shell with Plugins
Copyright @ 2019 Oracle and/or its affiliates.
46 / 69

47. MySQL Shell: Security
Retrieving the expiration period of passwords:
Copyright @ 2019 Oracle and/or its affiliates.
47 / 69

48. MySQL Shell: InnoDB
Which tables are potentially fragmented:
Copyright @ 2019 Oracle and/or its affiliates.
48 / 69

49. MySQL Shell: InnoDB (2)
And it can be improved using table space on disk size:
Copyright @ 2019 Oracle and/or its affiliates.
49 / 69

50. MySQL Shell: Schema
We already saw how we added methods to show and delete procedures. But recently
somebody complained about the complexity of knowing what are the default values of
columns when expressions are used (h ps://forums.mysql.com/read.php?
101,670682,670682):
Copyright @ 2019 Oracle and/or its affiliates.
50 / 69

51. MySQL Shell: Schema (2)
In MySQL 8.0, 0000-00-00 is not a valid value for a date eld anymore. We can check for
such dates in our dataset:
More Info: h ps://lefred.be/content/mysql-8-0-and-wrong-dates/
Copyright @ 2019 Oracle and/or its affiliates.
51 / 69

52. MySQL Shell: Maintenance
When using MySQL X Protocol, it's not allowed to shutdown MySQL. This plugin resolves
this limitation:
Copyright @ 2019 Oracle and/or its affiliates.
52 / 69

53. MySQL Shell: Audit
Copyright @ 2019 Oracle and/or its affiliates.
53 / 69

54. MySQL Shell: Audit (2)
Copyright @ 2019 Oracle and/or its affiliates.
54 / 69

55. MySQL Shell: Collations
When upgrading, the collation might be problematic for unique values. This plugin from
Bernt Marius Johnsen shows the non-unique values for a speci c collation:
Copyright @ 2019 Oracle and/or its affiliates.
55 / 69

56. Outside MySQL
It's also possible to interact with external tools
Copyright @ 2019 Oracle and/or its affiliates.
56 / 69

57. Monitoring MySQL Router
With the MySQL Shell and MySQL Router's REST API, it's even possible to create a plugin
to monitor the Router:
Copyright @ 2019 Oracle and/or its affiliates.
57 / 69

58. Monitoring MySQL Router (2)
Copyright @ 2019 Oracle and/or its affiliates.
58 / 69

59. MySQL Shell and ProxySQL
It's also possible to integrate 3rd party projects directly in MySQL Shell:
Copyright @ 2019 Oracle and/or its affiliates.
59 / 69

60. MySQL Shell and ProxySQL
Copyright @ 2019 Oracle and/or its affiliates.
60 / 69

61. More Info:
h ps://lefred.be/content/mysql-innodb-cluster-recovery-process-monitoring-with-
the-mysql-shell-reporting-framework/
h ps://lefred.be/content/using-the-new-mysql-shell-reporting-framework-to-
monitor-innodb-cluster/
h ps://lefred.be/content/mysql-router-8-0-17s-rest-api-mysql-shell-extensions/
h ps://lefred.be/content/mysqlrouter-8-0-17-and-the-rest-api/
h ps://lefred.be/content/how-to-integrate-proxysql-in-mysql-innodb-cluster/
Copyright @ 2019 Oracle and/or its affiliates.
61 / 69

62. I am a newbie
Writing your first MySQL Shell Plugin
Copyright @ 2019 Oracle and/or its affiliates.
62 / 69

63.
Let's write an oracle 8 ball plugin
Copyright @ 2019 Oracle and/or its affiliates.
63 / 69

64. demo/oracle8ball.py
from random import randrange
def tell_me():
"""Function that prints the so expected answer
Returns:
Nothing
"""
answers = ["It is certain.", "It is decidedly so.","Without a doubt.",
"Yes - de nitely.", "You may rely on it.",
"As I see it, yes.","Most likely.","Outlook good.",
"Yes.","Signs point to yes.","Reply hazy, try again.","Ask again later.",
"Better not tell you now.","Cannot predict now.",
"Concentrate and ask again.", "Don't count on it.",
"My reply is no.","My sources say no.",
"Outlook not so good.","Very doubtful."]
print answers[randrange(20)]
Copyright @ 2019 Oracle and/or its affiliates.
64 / 69

65. demo/init.py
from ext.mysqlsh_plugins_common import register_plugin
from ext.demo import oracle8ball as oracle_8_ball
[...]
try:
register_plugin("oracle8ball", oracle_8_ball.tell_me,
{
"brief": "Get the answer from the Oracle 8 Black Ball",
"parameters": []
},
"demo"
)
except Exception as e:
shell.log("ERROR", "Failed to register ext.demo.oracle8ball ({0}).".
format(str(e).rstrip())
Copyright @ 2019 Oracle and/or its affiliates.
65 / 69

66. Plugin's Help
Copyright @ 2019 Oracle and/or its affiliates.
66 / 69

67. Ready to contribute to MySQL Shell Plugins?
Get the code from h ps://github.com/lefred/mysqlshell-plugins and Pull Requests are
welcome !
Copyright @ 2019 Oracle and/or its affiliates.
67 / 69

68. Thank you !
Copyright @ 2019 Oracle and/or its affiliates.
68 / 69

69. Upgrade to MySQL 8.0
It's time to upgrade to
MySQL 8.0, the fastest MySQL
adoption release ever !
Copyright @ 2019 Oracle and/or its affiliates.
69 / 69