Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MySQL Shell - A DevOps-engineer day with MySQL’s development and administration tool

477 views

Published on

MySQL Shell is an interactive multi-language console interface that supports development and administration for the MySQL Server. It can be used to perform data queries or updates, and administration operations through scriptable DevOps APIs.

It is by nature a unified interface for all DevOps tasks related to MySQL, supporting SQL, JavaScript and Python, with an extensive set of features for both developers and DBAs.

This session demonstrates how can a DevOps-engineer day be better with the MySQL Shell. From the development tasks to the operations, culminating in the management of MySQL InnoDB Cluster (The HA solution for MySQL) with the most recent and advanced features.

Session presented at pre-FOSDEM MySQL Day 2019 (https://lefred.be/content/pre-fosdem-mysql-day-2019/)

Published in: Software
  • Be the first to comment

MySQL Shell - A DevOps-engineer day with MySQL’s development and administration tool

  1. 1. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. Miguel Araújo Senior Software Developer MySQL Middleware and Clients pre-FOSDEM’19 MySQL Day - February 01, 2019 MySQL Shell A DevOps-engineer day with MySQL’s development and administration tool
  2. 2. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. 2
  3. 3. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. 3 Who am I?
  4. 4. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. whoami Miguel Araújo • MySQL Senior Software Developer • Lead developer for the AdminAPI at the MySQL Shell team • MySQL InnoDB Cluster evangelist • Living in Portugal • http://about.me/miguel-araujo 4
  5. 5. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. 5 DevOps: What is it? As simple as:  Engineers (developers) and operations working together to: Automate and monitor processes  Share knowledge and collaborate Get things done faster Deliver faster!
  6. 6. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. DevOps-Engineer: …and that is? 6
  7. 7. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. DevOps-Engineer: …and that is? ¯_(ツ)_/¯ 7
  8. 8. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. DevOps-Engineer: …and that is? 8 OK, actually it’s someone that can:  Do system administration (sysadmin)  Code  Understand testing  Automate processes  Talk! (bridge between devs and ops)
  9. 9. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. MySQL Shell: Introduction • Interactive multi-language console interface that supports development and administration for the MySQL Server • Can be used to perform data queries or updates, and administration operations:  Interactively or through batch operations  With scriptable “DevOps” APIs  Unified Interface for MySQL Developers and DBAs: one tool! • Intuitive, flexible, powerful and easy to use! 9
  10. 10. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. MySQL Shell: Features (Dev) • Multi-language support – JavaScript, Python and SQL – Both interactive and batch operations (scripting) 10 MySQL Development
  11. 11. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. MySQL Shell: Features (Dev) • Multi-language support – JavaScript, Python and SQL – Both interactive and batch operations (scripting) • Document and relational model – Supports the classic relational model – Modern fluent API for the MySQL Document Store • CRUD and Relational 11 MySQL Development
  12. 12. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. 12 MySQL Shell: Features (Dev) • Scripting – Execute scripts from: • Interactive REPL interface • Directly from the cmdline • From Python, JS, or SQL script files – Import any installed module (in Python) MySQL Development
  13. 13. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. 13 MySQL Shell: Features (Dev/Ops) • APIs  MySQL DBs – ‘mysql’ object – Interaction with MySQL Server using the relational model  Document Store – ‘mysqlx’ object – Interact with MySQL Server using the X DevAPI MySQL Development
  14. 14. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. 14 MySQL Shell: Features (Dev/Ops) • APIs  ShellAPI – ‘shell’ object – General purpose functions and properties  AdminAPI – ‘dba’ object – Set up and management of InnoDB clusters  Shell utilities – ‘util’ object – Upgrade checker – Import JSON documents MySQL Administration
  15. 15. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. 15 A common use-case
  16. 16. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case Company X starts a new business unit 16 • A DBMS is needed
  17. 17. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case Company X starts a new business unit 17 • A DBMS is needed • The infraestructure must be prepared
  18. 18. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case Company X starts a new business unit 18 • A DBMS is needed • The infraestructure must be prepared • The DBMS must be deployed and managed
  19. 19. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case Company X starts a new business unit 19 • A DBMS is needed • The infraestructure must be prepared • The DBMS must be deployed and managed • The world’s most popular open-source database is suggested
  20. 20. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case Company X starts a new business unit 20 • A DBMS is needed • The infraestructure must be prepared • The DBMS must be deployed and managed • The world’s most popular open-source database is suggested  MySQL is approved!
  21. 21. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case Company X starts a new business unit 21 • A DBMS is needed • The infraestructure must be prepared • The DBMS must be deployed and managed • The world’s most popular open-source database is suggested  MySQL is approved! Call the devOps-engineer!
  22. 22. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. Meet Bill The devOps-engineer 22 This is Bill. Bill is a MySQL user. Bill knows the Shell is the devOps tool for MySQL. Bill uses the Shell to make his life easier. Bill is smart. Be like Bill.
  23. 23. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case Prepare the DBMS 23 1. Package installation 2. Directories management 3. Set up security 4. Configure server settings 5. Create accounts with the right permissions 6. ...
  24. 24. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case Prepare the DBMS 24 • Bill writes a JS script that can be used by others or even automated
  25. 25. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case Prepare the DBMS 25 • Interacts with the database through the session object
  26. 26. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case Prepare the DBMS 26 • Connect to localhost:3310 • Execute create_account.js with or without extra arguments forwarded to the script (sys.argv[]) • Rely on the Shell prompt utility
  27. 27. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case Upgrade the DBMS 27 • A major new MySQL version (8.0) is released with many improvements and features • Bill needs to upgrade the whole setup but is afraid it gets broken...
  28. 28. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case Upgrade the DBMS 28 • A major new MySQL version (8.0) is released with many improvements and features • Bill needs to upgrade the whole setup but is afraid it gets broken... Shell to the rescue!
  29. 29. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. MySQL Shell: Features (Ops) • Upgrade checker – Verify whether instances are ready or not for upgrading to MySQL 8 – Available through the util object 29 mysql-js> util.checkForServerUpgrade(“root@localhost:3306”); MySQL Administration
  30. 30. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case Upgrade the DBMS 30
  31. 31. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. MySQL Shell: Features (Ops) • API Command line integration – cmdline syntax to invoke built-in Shell commands – Requires minimal extra typing, quoting and escaping – Supported objects: dba, cluster, shell, shell.options and util Syntax: mysqlsh [OPTIONS] [URI] -- <object_name> <method_name> [argument_list] 31 $ mysqlsh root@myserver:33060 –- util check-for-server-upgrade MySQL Administration
  32. 32. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case Upgrade the DBMS 32 • Bill is smart • Bill is a devOps-engineer • Bill automates processes, by using continuous automation frameworks • How could Bill ensure that the MySQL Servers deployments are always using the latest available version?
  33. 33. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. Shell meets continuous automation 33 • Let’s use as an example: service { ‘mysql’: ensure => latest, enable => true, require => Package[‘mysql-server-community’], }
  34. 34. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. Shell meets continuous automation 34 What if current is 5.7 and latest is 8.0? Will it break everything? service { ‘mysql’: ensure => latest, enable => true, require => Package[‘mysql-server-community’], } ! • Let’s use as an example:
  35. 35. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. Shell meets continuous automation 35 service { ‘mysql’: ensure => latest, enable => true, require => Package[‘mysql-server-community’], before => Exec [‘upgrade checker’] } exec { “upgrade checker”: command => “mysqlsh -- util check-for-server-upgrade { --user=root -- host=localhost --port=3306} --password=‘myPwd’” require => Service[“mysqlsh”] } • Use MySQL Shell upgrade checker
  36. 36. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. • Customizable prompts – Display context information – Custom/dynamic tags that react to context • Eg. export PRODUCTION_SERVERS=192.168.42.16 MySQL Shell: The DBA toolbox Other features and tweaks 36
  37. 37. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. • Secure credential manager • Passwords queried from a secure password store (uses OS native APIs) • Optional auto-save for typed passwords (default will prompt) • History • Typed command history • Pager • Pass SQL results through an external screen paging program MySQL Shell: The DBA toolbox Other features and tweaks 37
  38. 38. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. • Built-in help – Documentation for included APIs, shell commands, SQL, etc. MySQL Shell: The DBA toolbox Other features and tweaks 38
  39. 39. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 2 The business unit is growing 39 • The systems are vulnerable to downtime • Downtime causes high losses... • Critical data can’t be lost and must ”always” be available! • High Availability becomes mandatory
  40. 40. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 2 The business unit is growing 40 • The systems are vulnerable to downtime • Downtime causes high losses... • Critical data can’t be lost and must ”always” be available! • High Availability becomes mandatory Bill wonders what kind of HA support MySQL has... ?
  41. 41. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. 41
  42. 42. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. 42 InnoDB cluster
  43. 43. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. 43 InnoDB cluster  MySQL Servers running Group Replication
  44. 44. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. 44 InnoDB cluster  MySQL Servers running Group Replication  App Servers with MySQL Router
  45. 45. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. 45 InnoDB cluster  MySQL Servers running Group Replication  App Servers with MySQL Router  MySQL Shell  Set up, Manage, Orchestrate IDC
  46. 46. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. 46 InnoDB cluster  MySQL Servers running Group Replication  App Servers with MySQL Router  MySQL Shell  Set up, Manage, Orchestrate IDC  Clients / Applications
  47. 47. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. MySQL Shell: Features (Ops) • Administration API: AdminAPI – Creation and Management of InnoDB Clusters – Hides the complexity of: • Configuration • Provisioning • Orchestration – Simple and straight-forward – Doesn’t require MySQL expertise – Flexible, powerful and secure – Available in both JavaScript and Python 47 MySQL Administration
  48. 48. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 2 The business unit is growing 48 • Bill needs to set up an HA deployment for the company needs • Using the Shell and the AdminAPI he can achieve that very easily!
  49. 49. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 2 The business unit is growing 49 • Bill needs to set up an HA deployment for the company needs • Using the Shell and the AdminAPI he can achieve that very easily! $ mysqlsh mysql-js> dba.configureInstance(“root@host1:3306”, {clusterAdmin: “bill”})
  50. 50. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 2 The business unit is growing 50 • Bill needs to set up an HA deployment for the company needs • Using the Shell and the AdminAPI he can achieve that very easily! $ mysqlsh mysql-js> dba.configureInstance(“root@host1:3306”, {clusterAdmin: “bill”}) mysql-js> var cluster = dba.createCluster(“Cluster1”)
  51. 51. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 2 The business unit is growing 51 • Bill needs to set up an HA deployment for the company needs • Using the Shell and the AdminAPI he can achieve that very easily! $ mysqlsh mysql-js> dba.configureInstance(“root@host1:3306”, {clusterAdmin: “bill”}) mysql-js> var cluster = dba.createCluster(“Cluster1”) mysql-js> cluster.addInstance(“bill@host2:3306”)
  52. 52. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 2 HA in 5 minutes! 52  JSON object describing the structure of the Cluster  Includes general cluster information  Topology:  ReplicaSets  Instance members
  53. 53. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 2 HA in 5 minutes! 53  JSON object describing the status of the Cluster  Includes general cluster status  Status information about each ReplicaSet member
  54. 54. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 3 The requirements change 54 • A strange behaviour was seen in the application leading to user complaints
  55. 55. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 3 The requirements change 55 • A strange behaviour was seen in the application leading to user complaints • Ops traced down the issue to the database... And found possible inconsistent reads
  56. 56. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 3 The requirements change 56 • A strange behaviour was seen in the application leading to user complaints • Ops traced down the issue to the database... And found possible inconsistent reads • Finally someone noticed that whenever a primary failover happens, stale data was being read from the newly elected primary causing the strange “inconsistencies”
  57. 57. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 3 The requirements change 57 • A strange behaviour was seen in the application leading to user complaints • Ops traced down the issue to the database... And found possible inconsistent reads • Finally someone noticed that whenever a primary failover happens, stale data was being read from the newly elected primary causing the strange “inconsistencies” • Direction asks for a solution even if it compromises performance!
  58. 58. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 3 Checking the cluster options 58  Verify the current cluster settings
  59. 59. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 3 Checking the cluster options 59  Verify the current cluster settings  JSON object listing configuration options of the Cluster  Includes global cluster options  Includes configuration options of each cluster member
  60. 60. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 3 Failover consistency 60 The failover consistency option can have 2 values: • EVENTUAL (default)  Read queries allowed in the new primary (even if backlog is being applied)  Writes are blocked  Possible stale reads • BEFORE_ON_PRIMARY_FAILOVER  Read and write queries blocked in the new primary (meanwhile backlog is being applied)  “read your writes”
  61. 61. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 3 Change the failover consistency 61 The AdminAPI allows changing a configuration option: 1) For the whole cluster  <Cluster.>setOption(option, value) 2) In a cluster member  <Cluster.>setInstanceOption(instance, option, value)
  62. 62. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 3 Change the failover consistency 62 The AdminAPI allows changing a configuration option: 1) For the whole cluster  <Cluster.>setOption(option, value) 2) In a cluster member  <Cluster.>setInstanceOption(instance, option, value)
  63. 63. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 4 Maintenance 63 • Maintenance tasks on servers are common... • Bill needs to perform a maintenance task on the current primary instance • Which implies some sort of disruption... • Idealy, Bill would rather manually elect a new primary to avoid any disruption
  64. 64. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 4 Maintenance 64 • Maintenance tasks on servers are common... • Bill needs to perform a maintenance task on the current primary instance • Which implies some sort of disruption... • Idealy, Bill would rather manually elect a new primary to avoid any disruption But... How? ?
  65. 65. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 4 Elect a new primary 65  <Cluster.>setPrimaryInstance(instance) By using the Shell and the AdminAPI!
  66. 66. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 5 The requirements change... again 66 • Analysis of the dataset and its usage, concluded that the workload could be spread accross the cluster members:
  67. 67. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 5 The requirements change... again 67 • Analysis of the dataset and its usage, concluded that the workload could be spread accross the cluster members: – No concurrent DDL ops in the same object
  68. 68. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 5 The requirements change... again 68 • Analysis of the dataset and its usage, concluded that the workload could be spread accross the cluster members: – No concurrent DDL ops in the same object – No foreign key constraints
  69. 69. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 5 The requirements change... again 69 • Analysis of the dataset and its usage, concluded that the workload could be spread accross the cluster members: – No concurrent DDL ops in the same object – No foreign key constraints – No large transactions, etc.
  70. 70. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 5 The requirements change... again 70 • Analysis of the dataset and its usage, concluded that the workload could be spread accross the cluster members: – No concurrent DDL ops in the same object – No foreign key constraints – No large transactions, etc. • Decision to use Multi-Primary mode, even if not recommended, is taken.
  71. 71. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 5 The requirements change... again 71 • Analysis of the dataset and its usage, concluded that the workload could be spread accross the cluster members: – No concurrent DDL ops in the same object – No foreign key constraints – No large transactions, etc. • Decision to use Multi-Primary mode, even if not recommended, is taken. But that implies the downtime of re-creating the whole cluster! !
  72. 72. Copyright © 2019, Oracle and/or its affiliates. All rights reserved.  <Cluster.>switchToMultiPrimaryMode() A common use-case: part 5 Switch to Multi-Primary mode 72 No restarts nor downtime!
  73. 73. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case: part 5 Switch to Multi-Primary mode 73 • All cluster members become writable (R/W)  Note: Switch to single-primary mode available with:  <Cluster.>switchToSinglePrimaryMode()
  74. 74. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. • Suddently somes instances are expelled from the cluster • ... even though ops report that the network is OK! • But further investigation concludes the network is actually flaky • Some members are being expelled due to very small network cuts of 2 minutes A common use-case: part 6 Troubleshooting 74
  75. 75. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. • Suddently somes instances are expelled from the cluster • ... even though ops report that the network is OK! • But further investigation concludes the network is actually flaky • Some members are being expelled due to very small network cuts of 2 minutes What can be done? A common use-case: part 6 Troubleshooting 75 ?
  76. 76. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. • Group Replication’s failure detector window is configurable • Allows delays of suspensions of previously active cluster members • Configurable using the AdminAPI! A common use-case: part 6 Defining the expel timeout 76
  77. 77. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. A common use-case Other useful features 77 • Define the next primary instance “in line” • memberWeight = x – Integer: [0, 100] • Define the behaviour of an instance whenever it drops out of the cluster • exitStateAction = y – ABORT_SERVER – READ_ONLY • Extended Cluster.status() • extended: true • queryMembers: true
  78. 78. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. Summary • MySQL Shell brings together developers and DBAs • Unified interface: one tool! • Development and Administration APIs and features • MySQL InnoDB Cluster is THE built-in HA solution for MySQL: • Full-stack: High Availability out-of-the-box • Easy to use: usability as a top concern • Full control of the Cluster using the AdminAPI 78
  79. 79. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. Resources • MySQL Shell Userguide: – https://dev.mysql.com/doc/refman/en/mysql-shell.html • MySQL InnoDB Cluster Userguide: – http://dev.mysql.com/doc/refman/en/mysql-innodb-cluster-userguide.html • APIs Reference manuals – JavaScript: https://dev.mysql.com/doc/dev/mysqlsh-api-javascript/ – Python: https://dev.mysql.com/doc/dev/mysqlsh-api-python/ • Blogging – http://mysqlserverteam.com/category/high-availability 79
  80. 80. Copyright © 2019, Oracle and/or its affiliates. All rights reserved. Thank you! Any questions? 80

×