dbdeployer
Giuseppe Maxia

Software Explorer
The universal MySQL installer
Who's this guy?
About me
‣ Giuseppe Maxia, a.k.a. "The Data Charmer"
‣ Software Explorer at VMware
‣ Several decades development and DB
experience
‣ Long timer MySQL community member.
‣ Blog: http://datacharmer.blogspot.com
‣ Twitter: @datacharmer
!2
Disclaimer
•None of what I say has
anything to do with my
company.

•I also don’t work for Oracle.
dbdeployer
• Command line tool

• No dependencies (single binary per O.S.)

• Interface similar to git, docker

• Fast!

• Runs single database, replication, group replication,
multiple deployments, multi-source replication.

• Integrated help

• Easy to extend.
https://dbdeployer.org
What can it do? (1)
• Install a single MySQL sandbox of ANY VERSION

• Separated from any existing MySQL server

• Completely in user space

• Reasonably isolated

• Easy to use and administer
dbdeployer deploy single 8.0
(Intermission)
WHY would you use it?
• Support to customers using different versions

• Consulting

• Testing of new versions

• Reporting bugs
What can it do? (2)
• Install a group of sandboxes

• Isolated from each other

• Without any relationship

• With binary logging active
dbdeployer deploy multiple 8.0
What can it do? (3)
• Install a master-slave replication cluster

• 1 master and 2 (or more) slaves

• Replication enabled

• Optional GTID

• Easy tools to run operations on all nodes at once
dbdeployer deploy replication 8.0 
--topology=master-slave
What can it do? (4)
• Install a group replication cluster (MySQL 5.7 and 8.0)

• 3 (or more) nodes

• Single-primary or multi-primary

• Group replication configured and enabled

• Easy tools to run operations on all nodes at once
dbdeployer deploy replication 8.0 
--topology=group
What can it do? (5)
• Install a multi-source replication cluster (MySQL 5.7-8.0)

• 3 (or more) nodes

• fan-in or all-masters topologies

• Easy tools to run operations on all nodes at once
dbdeployer deploy replication ndb7.6 
--topology=fan-in # or all-masters
What can it do? (6)
• Install a single or multiple TiDB sandbox
dbdeployer deploy single tidb3.0.0 
--client-from=5.7.25
dbdeployer deploy multiple tidb3.0.0 
--client-from=5.7.25
What can it do? (7)
• Install a Percona XtraDB cluster (5.7)

• 3 (or more) nodes

• Easy tools to run operations on all nodes at once
dbdeployer deploy replication pxc5.7 
--topology=pxc
What can it do? (8)
• Install a MySQL Cluster (NDB 7.6 and 8.0)

• 3 (or more) nodes

• Easy tools to run operations on all nodes at once
dbdeployer deploy replication ndb7.6 
--topology=ndb
What can it do? (9)
• Administer sandboxes

• List available binaries

• List installed sandboxes

• start, restart (with options), stop

• check status

• test

• delete
What can it do? (10)
• Customize sandboxes

• Customizable initialization and database start

• Provide templates for every script

• Allow on-the-fly and permanent template replacement

• Change most of the default values

• Make sandboxes permanent (= can't be deleted)
What can it do? (11)
• Find free ports automatically

• Test replication flow

• Expose MySQL 8 dictionary tables

• Use semi-sync replication

• Run SQL command before and after loading grants

• enable/disable X-protocol
Installation (1)
1. Go to https://github.com/datacharmer/dbdeployer

2. Find releases

3. Download the binary for your O.S.

4. Put it in a directory within your $PATH
5. Run it!
Installation (2)
(if you use go)
1. go get github.com/datacharmer/dbdeployer

2. Use it!
Using it
1. Download a MySQL binary tarball

2. Use dbdeployer to expand it

3. start creating sandboxes
Choose your download
Choose your download
MySQL 8.0.16+ has MINIMAL tarballs!
operations:
unpack once, use forever
$ dbdeployer unpack 
mysql-8.0.11-linux-glibc2.12-x86_64.tar.gz
Unpacking tarball mysql-8.0.11-linux-glibc2.12-
x86_64.tar.gz to $HOME/opt/mysql/8.0.11
$ dbdeployer versions
8.0.11
$ dbdeployer deploy single 8.0.11
Database installed in $HOME/sandboxes/msb_8_0_11
Principles
https://dev.mysql.com/downloads/mysql/get a MySQL
binary tarball
1
mysql-5.7.26-yourOS.tar.gz
dbdeployer unpack2
$HOME/opt/mysql/5.7.26
dbdeployer deploy single3
$HOME/sandboxes/msb_5_7_26
dbdeployer deploy multiple3
$HOME/sandboxes/multi_msb_5_7_26
What can it do? (12)
• List downloads for MacOS and Linux (dbdeployer 1.33.0)

• Download latest binaries directly
dbdeployer downloads list
dbdeployer downloads get mysql-5.7.26.tar.xz
dbdeployer downloads get-by version 8.0 
--newest --minimal
dbdeployer downloads get-unpack 
mysql-5.7.26.tar.xz
dbdeployer help
$ dbdeployer --help
dbdeployer makes MySQL server installation an easy task.
Runs single, multiple, and replicated sandboxes.
Usage:
dbdeployer [command]
Available Commands:
admin sandbox management tasks
cookbook Shows dbdeployer samples
defaults tasks related to dbdeployer defaults
delete delete an installed sandbox
delete-binaries delete an expanded tarball
deploy deploy sandboxes
downloads Manages remote tarballs
export Exports the command structure in JSON format
global Runs a given command in every sandbox
help Help about any command
info Shows information about dbdeployer environment samples
sandboxes List installed sandboxes
unpack unpack a tarball into the binary directory
usage Shows usage of installed sandboxes
versions List available versions
What else can it do? (13)
• Use MySQL 8.0 admin-port

• Replicate between sandboxes

• single/single

• Single to replication/group

• Replication to group/single

• NDB to NDB

• Upgrade a sandbox to a higher version

• Replication between sandboxes of different versions
DEMO
dbdeployer downloads list
dbdeployer downloads get
$ dbdeployer downloads get-by-version 5.0 
--newest --minimal
5.5 MB
File /home/msandbox/mysql-5.0.96.tar.xz
downloaded
Checksum matches
dbdeployer downloads get
$ dbdeployer downloads get-by-version 5.1 
--newest --minimal
10 MB
File /home/msandbox/mysql-5.1.72.tar.xz
downloaded
Checksum matches
$ dbdeployer downloads get-by-version 5.5 
--newest --minimal
6.6 MB
File /home/msandbox/mysql-5.5.62.tar.xz
downloaded
Checksum matches
dbdeployer downloads get
$ dbdeployer downloads get-by-version 5.6 --
newest --minimal
9.1 MB
File /home/msandbox/mysql-5.6.44.tar.xz
downloaded
Checksum matches
$ dbdeployer downloads get-by-version 5.7 --
newest --minimal
.. 23 MB
File /home/msandbox/mysql-5.7.26.tar.xz
downloaded
Checksum matches
dbdeployer downloads get
$ dbdeployer downloads get-by-version 8.0 
--newest --minimal
.... 44 MB
File /home/msandbox/mysql-8.0.16-linux-x86_64-
minimal.tar.xz downloaded
Checksum matches
dbdeployer unpack
$ for F in *.xz ; do dbdeployer unpack $F ; done
Unpacking tarball mysql-5.0.96.tar.xz to $HOME/opt/mysql/5.0.96
......64
Renaming directory /home/msandbox/opt/mysql/mysql-5.0.96 to /home/msandbox/opt/mysql/5.0.96
Unpacking tarball mysql-5.1.72.tar.xz to $HOME/opt/mysql/5.1.72
.......72
Renaming directory /home/msandbox/opt/mysql/mysql-5.1.72 to /home/msandbox/opt/mysql/5.1.72
Unpacking tarball mysql-5.5.62.tar.xz to $HOME/opt/mysql/5.5.62
......60
Renaming directory /home/msandbox/opt/mysql/mysql-5.5.62 to /home/msandbox/opt/mysql/5.5.62
Unpacking tarball mysql-5.6.44.tar.xz to $HOME/opt/mysql/5.6.44
.......72
Renaming directory /home/msandbox/opt/mysql/mysql-5.6.44 to /home/msandbox/opt/mysql/5.6.44
Unpacking tarball mysql-5.7.26.tar.xz to $HOME/opt/mysql/5.7.26
.........99
Renaming directory /home/msandbox/opt/mysql/mysql-5.7.26 to /home/msandbox/opt/mysql/5.7.26
Unpacking tarball mysql-8.0.16-linux-x86_64-minimal.tar.xz to $HOME/opt/mysql/8.0.16
.........100.........200.219
Renaming directory /home/msandbox/opt/mysql/mysql-8.0.16-linux-x86_64-minimal to /home/
msandbox/opt/mysql/8.0.16
dbdeployer deploy
$ dbdeployer deploy single 5.0
# 5.0 => 5.0.96
[...]
. sandbox server started
$ dbdeployer deploy single 5.7
# 5.7 => 5.7.26
[...]
. sandbox server started
$ dbdeployer deploy replication 5.7
[...]
Replication directory installed in $HOME/
sandboxes/rsandbox_5_7_26
dbdeployer deploy
$ dbdeployer deploy replication 8.0 
--concurrent
[...]
$ dbdeployer deploy replication 8.0 
--concurrent --topology=group
[...]
$ dbdeployer deploy replication 8.0 
--concurrent --topology=group 
--single-primary
dbdeployer sandboxes
$ dbdeployer sandboxes
group_msb_8_0_16 : group-multi-primary 8.0.16
[21617 21742 31617 21618 21743 31618 21619 21744 31619 ]
group_sp_msb_8_0_16 : group-single-primary 8.0.16
[22617 22742 32617 22618 22743 32618 22619 22744 32619 ]
msb_5_0_91 : single 5.0.91 [5091 ]
msb_5_7_26 : single 5.7.26 [5726 ]
msb_8_0_16 : single 8.0.16
[8016 18016 ]
rsandbox_5_7_26 : master-slave 5.7.26
[19327 19328 19329 ]
LIVE DEMO
Demo it yourself!
Dbdeployer cookbook list
Demo it yourself!
Dbdeployer cookbook list
`
dbdeployer uses semantic versioning (https://semver.org)

Current version: 1.33.0
Version
component
When does it change? example
major
Backward Incompatible API changes.

(Including new features)
1.21.3 => 2.0.0
minor
Backward compatible API changes.

Backward compatible new features
1.21.3 => 1.22.0
revision
Bug fixes. 

Code refactoring. 

No API changes
1.21.3 => 1.21.4
Parting thoughts
• dbdeployer is open source (Apache license)

• Contributions are welcome! (See
CONTRIBUTING.md)
https://dbdeployer.org
Q&A
https://dbdeployer.org

Dbdeployer, the universal installer

  • 1.
  • 2.
    Who's this guy? Aboutme ‣ Giuseppe Maxia, a.k.a. "The Data Charmer" ‣ Software Explorer at VMware ‣ Several decades development and DB experience ‣ Long timer MySQL community member. ‣ Blog: http://datacharmer.blogspot.com ‣ Twitter: @datacharmer !2
  • 3.
    Disclaimer •None of whatI say has anything to do with my company. •I also don’t work for Oracle.
  • 4.
    dbdeployer • Command linetool • No dependencies (single binary per O.S.) • Interface similar to git, docker • Fast! • Runs single database, replication, group replication, multiple deployments, multi-source replication. • Integrated help • Easy to extend. https://dbdeployer.org
  • 5.
    What can itdo? (1) • Install a single MySQL sandbox of ANY VERSION • Separated from any existing MySQL server • Completely in user space • Reasonably isolated • Easy to use and administer dbdeployer deploy single 8.0
  • 6.
    (Intermission) WHY would youuse it? • Support to customers using different versions • Consulting • Testing of new versions • Reporting bugs
  • 7.
    What can itdo? (2) • Install a group of sandboxes • Isolated from each other • Without any relationship • With binary logging active dbdeployer deploy multiple 8.0
  • 8.
    What can itdo? (3) • Install a master-slave replication cluster • 1 master and 2 (or more) slaves • Replication enabled • Optional GTID • Easy tools to run operations on all nodes at once dbdeployer deploy replication 8.0 --topology=master-slave
  • 9.
    What can itdo? (4) • Install a group replication cluster (MySQL 5.7 and 8.0) • 3 (or more) nodes • Single-primary or multi-primary • Group replication configured and enabled • Easy tools to run operations on all nodes at once dbdeployer deploy replication 8.0 --topology=group
  • 10.
    What can itdo? (5) • Install a multi-source replication cluster (MySQL 5.7-8.0) • 3 (or more) nodes • fan-in or all-masters topologies • Easy tools to run operations on all nodes at once dbdeployer deploy replication ndb7.6 --topology=fan-in # or all-masters
  • 11.
    What can itdo? (6) • Install a single or multiple TiDB sandbox dbdeployer deploy single tidb3.0.0 --client-from=5.7.25 dbdeployer deploy multiple tidb3.0.0 --client-from=5.7.25
  • 12.
    What can itdo? (7) • Install a Percona XtraDB cluster (5.7) • 3 (or more) nodes • Easy tools to run operations on all nodes at once dbdeployer deploy replication pxc5.7 --topology=pxc
  • 13.
    What can itdo? (8) • Install a MySQL Cluster (NDB 7.6 and 8.0) • 3 (or more) nodes • Easy tools to run operations on all nodes at once dbdeployer deploy replication ndb7.6 --topology=ndb
  • 14.
    What can itdo? (9) • Administer sandboxes • List available binaries • List installed sandboxes • start, restart (with options), stop • check status • test • delete
  • 15.
    What can itdo? (10) • Customize sandboxes • Customizable initialization and database start • Provide templates for every script • Allow on-the-fly and permanent template replacement • Change most of the default values • Make sandboxes permanent (= can't be deleted)
  • 16.
    What can itdo? (11) • Find free ports automatically • Test replication flow • Expose MySQL 8 dictionary tables • Use semi-sync replication • Run SQL command before and after loading grants • enable/disable X-protocol
  • 17.
    Installation (1) 1. Goto https://github.com/datacharmer/dbdeployer 2. Find releases 3. Download the binary for your O.S. 4. Put it in a directory within your $PATH 5. Run it!
  • 20.
    Installation (2) (if youuse go) 1. go get github.com/datacharmer/dbdeployer 2. Use it!
  • 21.
    Using it 1. Downloada MySQL binary tarball 2. Use dbdeployer to expand it 3. start creating sandboxes
  • 22.
  • 23.
    Choose your download MySQL8.0.16+ has MINIMAL tarballs!
  • 24.
    operations: unpack once, useforever $ dbdeployer unpack mysql-8.0.11-linux-glibc2.12-x86_64.tar.gz Unpacking tarball mysql-8.0.11-linux-glibc2.12- x86_64.tar.gz to $HOME/opt/mysql/8.0.11 $ dbdeployer versions 8.0.11 $ dbdeployer deploy single 8.0.11 Database installed in $HOME/sandboxes/msb_8_0_11
  • 25.
    Principles https://dev.mysql.com/downloads/mysql/get a MySQL binarytarball 1 mysql-5.7.26-yourOS.tar.gz dbdeployer unpack2 $HOME/opt/mysql/5.7.26 dbdeployer deploy single3 $HOME/sandboxes/msb_5_7_26 dbdeployer deploy multiple3 $HOME/sandboxes/multi_msb_5_7_26
  • 26.
    What can itdo? (12) • List downloads for MacOS and Linux (dbdeployer 1.33.0) • Download latest binaries directly dbdeployer downloads list dbdeployer downloads get mysql-5.7.26.tar.xz dbdeployer downloads get-by version 8.0 --newest --minimal dbdeployer downloads get-unpack mysql-5.7.26.tar.xz
  • 27.
    dbdeployer help $ dbdeployer--help dbdeployer makes MySQL server installation an easy task. Runs single, multiple, and replicated sandboxes. Usage: dbdeployer [command] Available Commands: admin sandbox management tasks cookbook Shows dbdeployer samples defaults tasks related to dbdeployer defaults delete delete an installed sandbox delete-binaries delete an expanded tarball deploy deploy sandboxes downloads Manages remote tarballs export Exports the command structure in JSON format global Runs a given command in every sandbox help Help about any command info Shows information about dbdeployer environment samples sandboxes List installed sandboxes unpack unpack a tarball into the binary directory usage Shows usage of installed sandboxes versions List available versions
  • 28.
    What else canit do? (13) • Use MySQL 8.0 admin-port • Replicate between sandboxes • single/single • Single to replication/group • Replication to group/single • NDB to NDB • Upgrade a sandbox to a higher version • Replication between sandboxes of different versions
  • 29.
  • 30.
  • 31.
    dbdeployer downloads get $dbdeployer downloads get-by-version 5.0 --newest --minimal 5.5 MB File /home/msandbox/mysql-5.0.96.tar.xz downloaded Checksum matches
  • 32.
    dbdeployer downloads get $dbdeployer downloads get-by-version 5.1 --newest --minimal 10 MB File /home/msandbox/mysql-5.1.72.tar.xz downloaded Checksum matches $ dbdeployer downloads get-by-version 5.5 --newest --minimal 6.6 MB File /home/msandbox/mysql-5.5.62.tar.xz downloaded Checksum matches
  • 33.
    dbdeployer downloads get $dbdeployer downloads get-by-version 5.6 -- newest --minimal 9.1 MB File /home/msandbox/mysql-5.6.44.tar.xz downloaded Checksum matches $ dbdeployer downloads get-by-version 5.7 -- newest --minimal .. 23 MB File /home/msandbox/mysql-5.7.26.tar.xz downloaded Checksum matches
  • 34.
    dbdeployer downloads get $dbdeployer downloads get-by-version 8.0 --newest --minimal .... 44 MB File /home/msandbox/mysql-8.0.16-linux-x86_64- minimal.tar.xz downloaded Checksum matches
  • 35.
    dbdeployer unpack $ forF in *.xz ; do dbdeployer unpack $F ; done Unpacking tarball mysql-5.0.96.tar.xz to $HOME/opt/mysql/5.0.96 ......64 Renaming directory /home/msandbox/opt/mysql/mysql-5.0.96 to /home/msandbox/opt/mysql/5.0.96 Unpacking tarball mysql-5.1.72.tar.xz to $HOME/opt/mysql/5.1.72 .......72 Renaming directory /home/msandbox/opt/mysql/mysql-5.1.72 to /home/msandbox/opt/mysql/5.1.72 Unpacking tarball mysql-5.5.62.tar.xz to $HOME/opt/mysql/5.5.62 ......60 Renaming directory /home/msandbox/opt/mysql/mysql-5.5.62 to /home/msandbox/opt/mysql/5.5.62 Unpacking tarball mysql-5.6.44.tar.xz to $HOME/opt/mysql/5.6.44 .......72 Renaming directory /home/msandbox/opt/mysql/mysql-5.6.44 to /home/msandbox/opt/mysql/5.6.44 Unpacking tarball mysql-5.7.26.tar.xz to $HOME/opt/mysql/5.7.26 .........99 Renaming directory /home/msandbox/opt/mysql/mysql-5.7.26 to /home/msandbox/opt/mysql/5.7.26 Unpacking tarball mysql-8.0.16-linux-x86_64-minimal.tar.xz to $HOME/opt/mysql/8.0.16 .........100.........200.219 Renaming directory /home/msandbox/opt/mysql/mysql-8.0.16-linux-x86_64-minimal to /home/ msandbox/opt/mysql/8.0.16
  • 36.
    dbdeployer deploy $ dbdeployerdeploy single 5.0 # 5.0 => 5.0.96 [...] . sandbox server started $ dbdeployer deploy single 5.7 # 5.7 => 5.7.26 [...] . sandbox server started $ dbdeployer deploy replication 5.7 [...] Replication directory installed in $HOME/ sandboxes/rsandbox_5_7_26
  • 37.
    dbdeployer deploy $ dbdeployerdeploy replication 8.0 --concurrent [...] $ dbdeployer deploy replication 8.0 --concurrent --topology=group [...] $ dbdeployer deploy replication 8.0 --concurrent --topology=group --single-primary
  • 38.
    dbdeployer sandboxes $ dbdeployersandboxes group_msb_8_0_16 : group-multi-primary 8.0.16 [21617 21742 31617 21618 21743 31618 21619 21744 31619 ] group_sp_msb_8_0_16 : group-single-primary 8.0.16 [22617 22742 32617 22618 22743 32618 22619 22744 32619 ] msb_5_0_91 : single 5.0.91 [5091 ] msb_5_7_26 : single 5.7.26 [5726 ] msb_8_0_16 : single 8.0.16 [8016 18016 ] rsandbox_5_7_26 : master-slave 5.7.26 [19327 19328 19329 ]
  • 39.
  • 40.
  • 41.
  • 42.
    ` dbdeployer uses semanticversioning (https://semver.org) Current version: 1.33.0 Version component When does it change? example major Backward Incompatible API changes. (Including new features) 1.21.3 => 2.0.0 minor Backward compatible API changes. Backward compatible new features 1.21.3 => 1.22.0 revision Bug fixes. Code refactoring. No API changes 1.21.3 => 1.21.4
  • 43.
    Parting thoughts • dbdeployeris open source (Apache license) • Contributions are welcome! (See CONTRIBUTING.md) https://dbdeployer.org
  • 44.