Using The Mysql Binary Log As A Change Stream

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. |
Using the MySQL Binary Log as a Change
Stream
Luis Soares
Software Development Director
MySQL Replication
1

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, timing, and pricing of any
features or functionality described for Oracle’s products may change and remains at the
sole discretion of Oracle Corporation.
2

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | 3
Agenda
MySQL Replication 101
The Binary Log
Change Capture Use Cases
Advanced Experiments and Plugins
Tips and Tricks
Conclusion
1
2
3
4
5
6

4

• This section will present:
– MySQL replication basic architecture.
– Different building blocks of MySQL replication.
– Different topologies that can built with MySQL replication.
5

MySQL Database Replication: Overview
INSERT ...
Server
B
binary log
INSERT ...
relay log
INSERT ...
Server
A
binary log
App
INSERT ...
Comm.
Framework

INSERT ...
Server
B
binary log
INSERT ...
relay log
INSERT ...
Server
A
binary log
App
INSERT ...
Persistent
log buffer Threaded
applier
Send, Receive,
ACK, NACK,
Heartbeating,
...
Comm.
Framework
Capture
statements
or data
changes.
Persistent
log buffer

INSERT ...
Server
B
binary log
INSERT ...
relay log
INSERT ...
Server
A
binary log
App
INSERT ...
Comm.
Framework
Master generates a binary log,
which contains changes.
Slave may also generate a
binary log.

9
Coordination Between Servers
A B
A B
semi-synchronous (plugin)
A CB
transactions
transactions
asynchronous (native)
acks
And in MySQL 8 as of 8.0.1
Since 5.7.17
group replication (plugin)

MySQL Database Replication: Topologies / Setups
10
Simple
SP

11
Chain
SP/SP

12
Tree
SP/SP
S
S

13
Star / Multi-source
SP
P
P
P
P
P

14
Circular
P/SP/S
P/S
P/S

15
Group
SSP

16
Group + External Secondary
SSP
ES

17
Group + External Primary
SSP
EP

18
But why is this important?
SSP
EP
And many other combinations of these topologies together...
... this is especially important in change capture to show how
easy and flexbile it is to deploy a MySQL replica dedicated to
capturing changes and expose them in its binary log, instead
of doing it on a primary or other more important
secondaries.

The Binary Log
19

The Binary Log
– What is the binary log and how it is organized.
– How changes are captured and then stored in the binary log.
– How to start a MySQL server.
– How to find out which files is the binary log stored in.
– How to inspect the contents of the binary log through a MySQL session.
– How to inspect the contents of the binary log using the mysqlbinlog command line
tool.
– What are the contents of the binary log.
20

The Binary Log: What is it?
• Sequential logical stream recording changes happening on the server.
– Persisted on disk in a set of files.
• Row or statement based format (may be intermixed).
– Row format enables change capture.
• Each transaction is split into groups of events, but all events of a
transaction are flushed to the log file together.
• Control events: Rotate, Format Description, Gtid, and more.
• Data events: Query_log_event, Update_rows_log_events, ...
21
Layout of the Binary Log.
BEGIN ...E1 E2 COMMIT BEGIN ...E1 E2 COMMITGTID GTID

The Binary Log: What is it used for?
Server
A
App
INSERT ...
INSERT ...
binary log
Master generates a binary log,
which records changes on the server.
• Not only replication... For instance:
– Point-in-time recovery
– Integration with other technologies
– Rolling upgrades
– And more!
• Row based format is rich in change
capture data.
– We will only focus on ROW format in this
tutorial.

The Binary Log: How is it created?
23
Simplified Capture Flow
Transaction
Begins
Commit?
Cache empty?
No
No
Execute
Statement
Yes
Yes
Changes
records?
No
Yes
Is next
record null?
Yes
No ha_update_row
ha_delete_row
ha_write_row
Write record
images to cache
Parse next
Statement
Prepare
Storage engine
Notify
before_commit
Flush to the binary log
and commit to storage engine
Notify
after_commit
Transaction
Ends

Capture is an in-memory operation (with overspill to temporary file if needed).
24
Transaction
Begins
Commit?
Cache empty?
No
No
Execute
Statement
Yes
Yes
Changes
records?
No
Yes
Is next
record null?
Yes
No ha_update_row
ha_delete_row
ha_write_row
Write record
images to cache
Parse next
Statement
Prepare
Storage engine
Notify
before_commit
Notify
after_commit
Transaction
Ends

Captured changes are written
to disk transactionally.
25
Transaction
Begins
Commit?
Cache empty?
No
No
Execute
Statement
Yes
Yes
Changes
records?
No
Yes
Is next
record null?
Yes
No ha_update_row
ha_delete_row
ha_write_row
Write record
images to cache
Parse next
Statement
Prepare
Storage engine
Notify
before_commit
Notify
after_commit
Transaction
Ends

The Binary Log: Where is it persisted?
• Collections of files
– content files (the file extension is
incremented by one on rotation)
– one index file listing content files
binlog.000002
26
Disk layout
Binary Log
binlog.000002
binlog.000001
binlog.000002
binlog.000003
binlog.000004
binlog.index binlog.000004
binlog.000003

The Binary Log: Hands-on
27
Initializing a data directory
tut6318 $ mysql-8.0.13/bin/mysqld --initialize-insecure --basedir=$PWD/mysql-8.0.13 --
datadir=$PWD/vardir/master
<TIMESTAMP> 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-
links (or equivalent) is the default. Consider not using this option as it' is deprecated and
will be removed in a future release.
<TIMESTAMP> 0 [System] [MY-013169] [Server] /ssd2/lsoares/tut6318/server/mysql-8.0.13/bin/mysqld
(mysqld 8.0.13) initializing of server in progress as process 67777
<TIMESTAMP> 5 [Warning] [MY-010453] [Server] root@localhost is created with an empty password !
Please consider switching off the --initialize-insecure option.
<TIMESTAMP> 0 [System] [MY-013170] [Server] /ssd2/lsoares/tut6318/server/mysql-8.0.13/bin/mysqld
(mysqld 8.0.13) initializing of server has completed
Note: we used --initialize-insecure
only to not clutter this presentation with passwords
and credentials management.

28
Launching the server
tut6318 $ mysql-8.0.13/bin/mysqld --datadir=$PWD/vardir/master
--socket=$PWD/vardir/master/master.sock
<TIMESTAMP> 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or
equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future
release.
<TIMESTAMP> 0 [System] [MY-010116] [Server] /ssd2/lsoares/tut6318/server/mysql-8.0.13/bin/mysqld (mysqld 8.0.13)
starting as process 69788
<TIMESTAMP> 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
<TIMESTAMP> 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/ssd2' in the path
is accessible to all OS users. Consider choosing a different directory.
<TIMESTAMP> 0 [System] [MY-010931] [Server] /ssd2/lsoares/tut6318/server/mysql-8.0.13/bin/mysqld: ready for
connections. Version: '8.0.13' socket: '/ssd2/lsoares/tut6318/server/vardir/master.sock' port: 3306 MySQL
Community Server - GPL.
<TIMESTAMP> 0 [ERROR] [MY-011300] [Server] Plugin mysqlx reported: 'Setup of socket: '/tmp/mysqlx.sock' failed,
can't open lock file /tmp/mysqlx.sock.lock'
<TIMESTAMP> 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060
• Start the server (the binary log is enabled by default in MySQL 8).

29
Connecting to the server and listing the binary logs
tut6318 $ mysql-8.0.13/bin/mysql -u root -h 127.0.0.1 -P 3306
Welcome to the MySQL monitor. Commands end with ; or g.
Your MySQL connection id is 8
Server version: 8.0.13 MySQL Community Server - GPL
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.
mysql> SHOW BINARY LOGS;

30
Connecting to the server and listing the binary logs
tut6318 $ mysql-8.0.13/bin/mysql -u root -h 127.0.0.1 -P 3306
Welcome to the MySQL monitor. Commands end with ; or g.
Your MySQL connection id is 8
Server version: 8.0.13 MySQL Community Server - GPL
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.
+---------------+-----------+
| Log_name | File_size |
+---------------+-----------+
| binlog.000001 | 155 |
+---------------+-----------+
1 row in set (0,00 sec) The binary log file name.
File offset where the next
entry shall be written to.

31
Inspecting the contents of the binary log
mysql> SHOW BINLOG EVENTS;

33
Forcing a log rotation
mysql> FLUSH BINARY LOGS;
Query OK, 0 rows affected (0,00 sec)
+---------------+-----------+
+---------------+-----------+
| binlog.000001 | 199 |
| binlog.000002 | 155 |
+---------------+-----------+
mysql>
[1]+ Stopped […]/mysql-8.0.13/bin/mysql -u root -h 127.0.0.1 -P 3306
tut6318 $ cat $PWD/vardir/master/binlog.index
./binlog.000001
./binlog.000002
tut6318 $ fg
[…]/mysql-8.0.13/bin/mysql -u root -h 127.0.0.1 -P 3306
mysql>

34
Forcing a log rotation
mysql> FLUSH BINARY LOGS;
+---------------+-----------+
+---------------+-----------+
| binlog.000001 | 199 |
| binlog.000002 | 155 |
+---------------+-----------+
mysql>
[1]+ Stopped […]/mysql-8.0.13/bin/mysql -u root -h 127.0.0.1 -P 3306
tut6318 $ cat $PWD/vardir/master/binlog.index
./binlog.000001
./binlog.000002
tut6318 $ fg
[…]/mysql-8.0.13/bin/mysql -u root -h 127.0.0.1 -P 3306
mysql>
New log file: binlog.000002
Checking the index file in the filesystem shows
twofiles as well. You should not mess around
with this file yourself as this is an internal file.
This is presented here only to demonstrate how
the filesystem layout is organized.

35
Creating a schema and a table.
mysql> CREATE DATABASE test;
Query OK, 1 row affected (0,07 sec)
mysql> use test
Database changed
mysql> CREATE TABLE t1 (c1 INT PRIMARY KEY);
mysql>

36
Creating a schema and a table and inspecting the binlog now.
mysql> CREATE DATABASE test;
mysql> use test
Database changed
mysql> CREATE TABLE t1 (c1 INT PRIMARY KEY);
mysql> SHOW BINLOG EVENTS IN ‘binlog.000002’;
+---------------+-----+----------------+-----------+-------------+---------------------------------------------------------------+
+---------------+-----+----------------+-----------+-------------+---------------------------------------------------------------+
| binlog.000002 | 155 | Anonymous_Gtid | 1 | 228 | SET @@SESSION.GTID_NEXT= 'ANONYMOUS' |
| binlog.000002 | 228 | Query | 1 | 336 | CREATE DATABASE test /* xid=8 */ |
| binlog.000002 | 409 | Query | 1 | 533 | use `test`; CREATE TABLE t1 (c1 INT PRIMARY KEY) /* xid=13 */ |
+---------------+-----+----------------+-----------+-------------+---------------------------------------------------------------+
New log file: binlog.000002Actual changes recorded.
Transaction specific metadata.
Contains transaction specific
metadata.

37
Populating the table and printing the changes captured.
mysql> FLUSH LOGS;
mysql> INSERT INTO t1 VALUES (1), (2);
Records: 2 Duplicates: 0 Warnings: 0
mysql> SHOW BINLOG EVENTS in 'binlog.000003';
+---------------+-----+----------------+-----------+-------------+--------------------------------------+
+---------------+-----+----------------+-----------+-------------+--------------------------------------+
| binlog.000003 | 230 | Query | 1 | 307 | BEGIN |
| binlog.000003 | 307 | Table_map | 1 | 355 | table_id: 89 (test.t1) |
| binlog.000003 | 355 | Write_rows | 1 | 400 | table_id: 89 flags: STMT_END_F |
| binlog.000003 | 400 | Xid | 1 | 431 | COMMIT /* xid=17 */ |
+---------------+-----+----------------+-----------+-------------+--------------------------------------+
Table metadata encoded in the change stream.
Specifies properties of the table that was modified.
MySQL 8 extended this metadata. More on that later.
The records inserted.

38
Populating the table and printing the changes captured.
mysql> UPDATE t1 SET c1 = 3 WHERE c1 = 1;
Rows matched: 1 Changed: 1 Warnings: 0
mysql> DELETE FROM t1 WHERE c1 = 2;
mysql> SHOW BINLOG EVENTS in 'binlog.000003';
[...]
| binlog.000003 | 640 | Update_rows | 1 | 686 | table_id: 89 flags: STMT_END_F |
| binlog.000003 | 917 | Delete_rows | 1 | 957 | table_id: 89 flags: STMT_END_F |
+---------------+-----+----------------+-----------+-------------+--------------------------------------+

• Decodes events in the binary log files;
• Good for debugging, recovery and even auditing;
• Outputs valid SQL syntax that a MySQL server can process:
– When piped through mysql client program can replay binary logs;
– Can be used as a tool for implementing point-in-time recovery;
– Can even act as a fake slave – enables complex workflows. Old example, but illustrates
the potential:
http://d2-systems.blogspot.com/2013/06/enabling-mysql-multi-source-replication.html
39
Command line tool: mysqlbinlog

40
tut6318 $ mysqlbinlog –vv $PWD/vardir/master/binlog.000003
# at 4
#181015 16:24:25 server id 1 end_log_pos 124 CRC32 0x4f091648 Start: binlog v 4, server v 8.0.13 created
181015 16:24:25
# Warning: this binlog is either in use or was not closed properly.
BINLOG '
GaPEWw8BAAAAeAAAAHwAAAABAAQAOC4wLjEzAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAEwANAAgAAAAABAAEAAAAYAAEGggAAAAICAgCAAAACgoKKioAEjQA
CgFIFglP
'/*!*/;
The format description event.

41
tut6318 $ mysqlbinlog –vv $PWD/vardir/master/binlog.000003
[...]
# at 124
#181015 16:24:25 server id 1 end_log_pos 155 CRC32 0x0e5119bb Previous-GTIDs
# [empty]
An event containing information about
transaction identifiers stored in preceding
binary log files.

42
[...]
# at 155
#181015 16:24:35 server id 1 end_log_pos 230 CRC32 0xa63f6ba6 Anonymous_GTID last_committed=0
sequence_number=1 rbr_only=yes original_committed_timestamp=1539613476111048
immediate_commit_timestamp=1539613476111048 transaction_length=276
/*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/;
# original_commit_timestamp=1539613476111048 (2018-10-15 16:24:36.111048 CEST)
# immediate_commit_timestamp=1539613476111048 (2018-10-15 16:24:36.111048 CEST)
/*!80001 SET @@session.original_commit_timestamp=1539613476111048*//*!*/;
SET @@SESSION.GTID_NEXT= 'ANONYMOUS'/*!*/;
[...]
# at 307
#181015 16:24:35 server id 1 end_log_pos 355 CRC32 0x2fce6489 Table_map: `test`.`t1` mapped to number 89
# at 355
#181015 16:24:35 server id 1 end_log_pos 400 CRC32 0xa03a40a8 Write_rows: table id 89 flags: STMT_END_F
[...]
### INSERT INTO `test`.`t1`
### SET
### @1=1 /* INT meta=0 nullable=0 is_null=0 */
### SET
Transaction specific metadata, that allows
the user to know when it was originally
committed and when it was committed in a
relay master. Moreover, it contains
information about the transaction identififer
and commit sequence number.
The actual values inserted (1 and 2).

43
[...]
# at 431
#181015 16:31:19 server id 1 end_log_pos 506 CRC32 0x4b322c42 Anonymous_GTID last_committed=1
# at 506
#181015 16:31:19 server id 1 end_log_pos 592 CRC32 0x51bfc235 Query thread_id=8 exec_time=0 error_code=0
SET TIMESTAMP=1539613879/*!*/;
/*!80013 SET @@session.sql_require_primary_key=0*//*!*/;
BEGIN
/*!*/;
# at 592
#181015 16:31:19 server id 1 end_log_pos 640 CRC32 0xc968fe53 Table_map: `test`.`t1` mapped to number 89
# at 640
#181015 16:31:19 server id 1 end_log_pos 686 CRC32 0xad89e32a Update_rows: table id 89 flags: STMT_END_F
### UPDATE `test`.`t1`
### WHERE
### SET
The actual row updated (1 becomes 3).

44
[...]
# at 717
#181015 16:31:43 server id 1 end_log_pos 792 CRC32 0xb654a2a4 Anonymous_GTID last_committed=2
# at 792
#181015 16:31:43 server id 1 end_log_pos 869 CRC32 0xfef26634 Query thread_id=8 exec_time=0
error_code=0
SET TIMESTAMP=1539613903/*!*/;
/*!80013 SET @@session.sql_require_primary_key=0*//*!*/;
BEGIN
/*!*/;
# at 869
#181015 16:31:43 server id 1 end_log_pos 917 CRC32 0xa4deb2e5 Table_map: `test`.`t1` mapped to number 89
# at 917
#181015 16:31:43 server id 1 end_log_pos 957 CRC32 0x16f6ca45 Delete_rows: table id 89 flags: STMT_END_F
[...]
### DELETE FROM `test`.`t1`
### WHERE
The actual row deleted (2).

• MySQL 8 can capture more table metadata if instructed to do it.
• This is governed by the following option:
• Details: https://mysqlhighavailability.com/more-metadata-is-written-into-binary-log/
45
Extended table metadata
Property Value
Name binlog_row_metadata
Scope GLOBAL
Type ENUM
Dynamic Yes
Values MINIMAL, FULL
Default MINIMAL

46
mysql> SET GLOBAL binlog_row_metadata = FULL;
mysql> q
Bye
tut6318 $ $PWD/mysql-8.0.13/bin/mysql -u root -h 127.0.0.1 -P 3306
[...]
mysql> use test
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> INSERT INTO t1 VALUES (1000);
mysql> q
Bye
tut6318 $ mysql-8.0.13/bin/mysqlbinlog -vv --print-table-metadata $PWD/vardir/master/binlog.000003
Explicit switch to display
additional metadata.

47
[...]
# at 1140
#181015 17:12:48 server id 1 end_log_pos 1196 CRC32 0xed25a26b Table_map: `test`.`t1` mapped to number
89
# Columns(`c1` INT NOT NULL)
# Primary Key(c1)
# at 1196
#181015 17:12:48 server id 1 end_log_pos 1236 CRC32 0xe49b76d8 Write_rows: table id 89 flags:
STMT_END_F
BINLOG '
cK7EWxMBAAAAOAAAAKwEAAAAAFkAAAAAAAEABHRlc3QAAnQxAAEDAAABAQAEAwJjMQgBAGuiJe0=
cK7EWx4BAAAAKAAAANQEAAAAAFkAAAAAAAEAAgAB/wDoAwAA2Hab5A==
'/*!*/;
### SET
[...]
In this case, column names and
primary key information is
displayed.

• mysqlbinlog can decode the contents of the binary log files or even from a
server directly, if it connects to it acting as a fake slave.
• mysqlbinlog relies on a general purpose decoder that is included in
libbinlogevents.
48

The Binary Log: New Metadata in MySQL 8
• Table Metadata:
– UNSIGNED flag of numeric columns
– Default character set of string columns
– Character set of string columns
– Column name [requires binlog_row_metadata=FULL]
– String value of SET columns [requires binlog_row_metadata=FULL]
– String value of ENUM columns [requires binlog_row_metadata=FULL]
– Primary key without prefix [requires binlog_row_metadata=FULL]
– Primary key with prefix [requires binlog_row_metadata=FULL]
– Geometry type of geometry columns
49
MySQL 8

The Binary Log: New Metadata in MySQL 8
• Transaction Metadata:
– Original execution commit timestamp in seconds since the epoch
– Relay server commit timestamp in seconds since the epoch
– Full transaction length
50
MySQL 8

51

– How the replication stream can be used for other use cases than just plain replication.
– Other technologies that rely on MySQL replication stream and use it to implement:
• change data capture
• extract, transform and load
• external triggers
• point-in-time recovery
• and more.
52

• Some technologies presented in this section are Community projects and
not owned by Oracle.
– ProxySQL, Gh-ost, Maxwell
• They are here to show how some of the MySQL users are relying on MySQL
infrastructure to build their own tooling and automation.
53
Disclaimer

Change Tracking
54

Change Tracking: ProxySQL’s consistent reads case
• Track which changes are where at a given point in time. Route application
traffic based on that information.
• The ProxySQL case (ProxySQL is a MySQL Proxy):
– Knows which transactions a session has “seen”.
– A process listens to the binary log stream and captures transaction identifiers.
Forwards that information to ProxySQL instances.
– ProxySQL routes incoming requests based on the knowledge of which transactions
have been applied to which servers.
– Read your writes consistency can then be implemented by routing transactions to
servers that already have the expected changes applied.
• https://proxysql.com/blog/proxysql-gtid-causal-reads
55

56
A B C
ProxySQL
App.

57
A B C
ProxySQL
App.
Binlog
reader
Binlog
reader
Binlog
reader
TransactionIdentifiers

Change Tracking: ProxySQL’s consistent reads Case
58
A B C
ProxySQL
App.
Binlog
reader
Binlog
reader
Binlog
reader
TransactionIdentifiers
Extract from the binary
log stream, which
transactions have been
applied on each server
Query routing is done
according to the consistency
requiremetns and the state of
each server.

Userland Online Schema Changes
59

Userland Online Schema Changes: Github’s gh-ost
• Github Online Schema Changes (gh-ost)
– https://github.com/github/gh-ost
– Userland tool to migrates a table from schema V to schema V+1.
– Operations: (preferred mode, but there are others)
60
PRIMARY SECONDARY
Binary Log
Replication
Gh-ost
(Convert records to V+1)
Load records
from V
Copies records
to V+1
Reads the Binary Log stream
for changes to V while
loading is taking place.

Userland Online Schema Changes: Github’s gh-ost
• Github Online Schema Changes (gh-ost)
– https://github.com/github/gh-ost
– Userland tool to migrates a table from schema V to schema V+1.
– Operations: (preferred mode, but there are others)
61
PRIMARY SECONDARY
Binary Log
Replication
Gh-ost
(Convert records to V+1)
Reads the Binary Log stream
for changes to V while
loading is taking place.Load records
from V
Copies records
to V+1
Binary log used as a
change stream, from
where changes are
extracted.
Records are loaded
into table with
version v+1.
Orchestrator
transforms the
records.

Online Rolling Upgrades
62

• Upgrades are operationally expensive
– Requires automation
– Always on, even when upgrading
• The binary log, as a change stream, and MySQL replication, prove
themselves invaluable to meet these requirements.
– Setup a secondary (version V+1) that replicates from the primary (version V).
– Cut application traffic on the primary, let secondary catch up.
– Switch application over to the secondary, which becomes the new primary.
– Nuke or upgrade old primary.
63

64
SP S
S
SP S
S
SP S
S
SP S
S
P* S
S S
New Version V+1
Old Version V
• Rolling upgrades means no
new features used until
upgrade is done (usually).
• Whatch out for defaults and
changed behavior.
• Test, Test, Test.
App
App App
App App

Data Integration
65

Data Integration: Oracle Golden Gate
• Replicate data from MySQL to Oracle and vice-versa.
• Use the binary log stream (row format) as a change log.
• Self-contained metadata in the binary log stream.
– Contains enough metadata to infer the original table structure.
– Contains enough metadata notifying table structure modifications.
– Metadata has been continuously enhaced. New metadata fields in MySQL 8.
• https://docs.oracle.com/goldengate/
66

67
MySQL Oracle
Oracle GoldenGate
Capture changes from
the binary log stream.
Apply to Oracle.
Capture from Oracle DB.
Apply to MySQL.

68
MySQL Oracle
Oracle GoldenGate
Capture changes from
the binary log stream.
Apply to Oracle.
Capture from Oracle DB.
Apply to MySQL.
Kafka
Publish to Kafka

Change Listener: Maxwell’s Daemon
• Reads the binary log stream
• Writes row changes, as JSON, to a streaming infrastructure:
– Kafka
– Kinesis
– RabbitMQ
• http://maxwells-daemon.io/
69

70
Maxwell
MySQL
App
insert into `test`.`maxwell` set id = 1,
daemon = 'Stanislaw Lem';
maxwell: {
"database": "test",
"table": "maxwell",
"type": "insert",
"ts": 1449786310,
"xid": 940752,
"commit": true,
"data": {
"id":1,
“daemon": "Stanislaw Lem“
}
}
Example taken from: https://github.com/zendesk/maxwell

71
Maxwell
MySQL
App
Replication
Row format
Kafka

Point-in-time Recovery
72

• Binary logs capture incremental changes to the server.
• Backup then restore and apply binary logs to get state up to a certain
[logical] point in time.
73
Primary
time
Backup
Primary
Crash
Backup
Copy of the
Primary
Restore
Backup
Copy of the
Primary
Incremental

74
# Starts replaying binlog from 16:31:43 to 17:12:48 of 15-Oct-2018
tut6318 $ mysql-8.0.13/bin/mysqlbinlog --start-datetime=“18-10-15 16:31:43” --stop-datetime="18-10-15 17:12:48"
$PWD/vardir/master/binlog.000003 | mysql –u root –P 3306 –h 127.0.0.1
[...]
# Starts replaying binlog from offset 431 to 988
tut6318 $ mysql-8.0.13/bin/mysqlbinlog --start-position=431 --stop-position=988 $PWD/vardir/master/binlog.000003
| mysql –u root –P 3306 –h 127.0.0.1
[...]
# Replay only the following transactions
tut6318 $ mysql-8.0.13/bin/mysqlbinlog –include-gtids=“9816148c-9c3f-4842-ae40-173a99ab45c9:3-10”
$PWD/vardir/master/binlog.000003 | mysql –u root –P 3306 –h 127.0.0.1
[...]
https://dev.mysql.com/doc/refman/8.0/en/point-in-time-recovery.html

Undelete
75

Undelete One or More Transactions
• The binary log captures the record images for inserts, deletes and updates.
• If one replays them in reverse order, in some cases, one is able to go back
to a point in time and undo an error.
– Some people have developed their own tooling to do it.
• Examples:
– http://thenoyes.com/littlenoise/?p=307
– https://github.com/lefred/MyUndelete
• Limited applicability (e.g., can’t revert DDL operations), but in some cases it
may be useful to undo a specific transaction instead of re-imaging a server.
76

77
BEGIN BI COMMITGTID AI BEGIN AI COMMITGTID BI

78
BEGIN BI COMMITGTID AI BEGIN AI COMMITGTID BI
App sends
offending
transaction.
User reverts the order of the change
images in the binary log for the
offending transaction.
User applies the
modified binlog as if
it was doing a point-
in-time recovery.
Offending transaction
was compensated.
Database is back to
the previous state.

79

• The next section will:
– Show how some plugins mine the binary log caches to intersect the captured
changes.
80

81
Over-simplified Capture Flow
Transaction
Begins
Commit?
Cache empty?
No
No
Execute
Statement
Yes
Yes
Changes
records?
No
Yes
Is next
record null?
Yes
No ha_update_row
ha_delete_row
ha_write_row
Write record
images to cache
Parse next
Statement
Prepare
Storage engine
Notify
before_commit
Notify
after_commit
Transaction
Ends
Refreshing our mind!

Group Replication Change Propagation
82

Change listener plugin: The Group Replication Plugin Case
• Built on top of proven technology!
– Shares many pieces of MySQL Replication.
• Built on reusable components!
– Layered implementation approach.
– Interface driven development.
– Decoupled from the server core.
– Plugin registers as listener to server events.
– Reuses the capture procedure from regular replication.
– Provides further decoupling from the communication infrasctructure.
83
MySQL Group Replication plugin is...

84
P P P P P
Replication Group
Applications

85
P P P P P
Replication Group
Applications
Group Com. Engine
API
Replication
Plugin
API
MySQL
Server

Group Com. Engine
86
Listens to a notification that a transaction is about to commit!
API
Replication
Plugin
API
MySQL
Server
Performance Schema Tables: Monitoring
MySQL
APIs: Lifecycle / Capture / Applier
InnoDB
Replication Protocol
Group Com. API
Group Com. Engine
Network
Plugin
Capture Applier
Conflicts
Handler
Group Com. Binding
Recovery

Group Com. Engine
87
API
Replication
Plugin
API
MySQL
Server
MySQL
InnoDB
Group Com. API
Group Com. Engine
Network
Plugin
Capture Applier
Conflicts
Handler
Group Com. Binding
Recovery
Intercepts the change buffer
and sends it out to the
group.

88
Over-simplified Capture Flow – Interception point
Transaction
Begins
Commit?
Cache empty?
No
No
Execute
Statement
Yes
Yes
Changes
records?
No
Yes
Is next
record null?
Yes
No ha_update_row
ha_delete_row
ha_write_row
Write record
images to cache
Parse next
Statement
Prepare
Storage engine
Notify
before_commit
Notify
after_commit
Transaction
Ends
This is where the interception takes
place. Note that at this point the
changes captured are still in the
capture cache.

89
Capture Interception Point
before_commit
Begin
RW Trx
execute capture encode
Next
stmt
buffer
Com
mit?
Rollback
Trx
no
noPrepare
OK?
Replicate
OK?
yes
yes
no
Flush
and sync
OK?
Commit
In SE
OK?
Commit
Trx
Panic /
Server
RO
no
yesyes
no
Binary Log
Persistence.
Distributed Commit.
Global externalization.
Group Replication Propagation
Local commit. Local
externalization.
The three stages of the binlog group
commit.
yes
More specifically, during the
commit phase, the
transaction is prepared
locally and then propagated
to the group.

90
Internal APIs used by the plugin
/**
This callback is called before transaction commit
This callback is called right before write binlog
cache to
binary log.
@param param The parameter for transaction
observers
@retval 0 Success
@retval 1 Failure
*/
typedef int (*before_commit_t)(Trans_param *param);
/**
Transaction observer parameter
*/
typedef struct Trans_param {
[...]
/*
Set on before_commit hook.
*/
Binlog_cache_storage *trx_cache_log;
Binlog_cache_storage *stmt_cache_log;
[...]
} Trans_param;
Callback definition Part of trans_param definition

91
before_commit hook implementation
int group_replication_trans_before_commit(Trans_param *param) {
[...]
// Broadcast the Transaction Message
send_error = gcs_module->send_message(transaction_msg);
[...]
}

• Interception happens in the capture pipeline.
– Right before the transaction terminates and is written to the log file.
– The cache is shared with the plugin.
– The plugin eventually sends the transaction changes to the group.
– Other servers in the receive and apply the changes.
• Key point:
– Capture and extraction are the same as for regular replication, except that it is
intercepted and exposed at a different point in time.
92
Take aways

Replication Observer Plugin
93

Group Com. Engine
Change listener plugin: An Experiment
94
API
Replication
Plugin
API
MySQL
Server
MySQL
InnoDB
Group Com. API
Group Com. Engine
Network
Plugin
Capture Applier
Conflicts
Handler
Group Com. Binding
Recovery

Change listener plugin: An Observer Plugin
95
Observing Changes
Observer
MySQL
Server
MySQL
InnoDB
Plugin
Capture

96
unsigned long before_commit_counter;
static int trans_before_commit(Trans_param *param MY_ATTRIBUTE((unused))) {
before_commit_counter++;
LogPluginErr(
INFORMATION_LEVEL, ER_LOG_PRINTF_MSG,
“before_commit_counter = %s“, before_commit_counter);
}
Trans_observer trans_observer = {
sizeof(Trans_observer),
trans_before_dml, trans_before_commit, trans_before_rollback,
trans_after_commit, trans_after_rollback,
};
static int replication_observers_example_plugin_init(MYSQL_PLUGIN plugin_info) {
[...]
register_trans_observer(&trans_observer, (void *)plugin_info_ptr);
[...]
}

97
LogPluginErr(
}
};
[...]
[...]
}
before_commit hook
implementation that counts
the number of times the
before_commit hooks was
called and prints the counter.

98
LogPluginErr(
}
};
[...]
[...]
}
before_commit hook
Transaction observer hooks
setup.

99
LogPluginErr(
}
};
[...]
[...]
}
before_commit hook
Transaction observer hooks
setup.
Registering the hooks when
the plugin is initialized.

100
LogPluginErr(
}
};
[...]
[...]
}
For a complete example/mock, see:
plugin/replication_observers_example/replication_observers_example.cc
(https://github.com/mysql/mysql-server)

Improving it Further
101

Capture Interfaces
• Some of the APIs shown here are really internal hooks.
• These should be made APIs registered in the new service infrastructure
that has debutted in MySQL 8.
– Promote them to proper public APIs.
– Make them properly documented.
– Allows everyone to extend MySQL more easily.
• But the basic infrastructure is there and already allows interesting things.
102

Tips and Tricks
103

Tips and Tricks
– How to control whether or not the original statement should be included as part of
the changes captured.
– How to control the amount of data logged in the record’s change images.
– How to control whether to log full documents or just logical diff of a document
change.
– How to control the amount of table metadata captured.
104

Tip #1: Capture Partial Rows
• Two optimizations:
– RBR can be configured to exclude BLOBS only (when not needed);
– RBR can also work with partial rows in addition to full rows:
• Before image: only fields required to find the row;
• After image: only fields that actually changed;
• Reduces memory footprint;
• Reduces network bandwidth usage;
• Reduces binary log files size;
• Default configuration: capture full row.
– This is the most interesting setup in a change capture scenario.
105

Before Image After Image
Full Rows
Rows without Blobs
Minimal Rows
Blob
Primary
Key
Changed
Columns

• Configure the behavior with:
– SET binlog_row_image = [ MINIMAL | NOBLOB | FULL ]
107
mysql> CREATE TABLE t3 (i1 INT PRIMARY KEY AUTO_INCREMENT, i2 INT, f FLOAT, t TEXT);
mysql> INSERT INTO t3 VALUES (1, 1, 1.0, 'Hello World!');

108
SET SESSION binlog_row_image = FULL;
mysql> UPDATE t3 SET i2=2 WHERE i1=1;
tut6318 $ mysql-8.0.13/bin/mysqlbinlog –vv $PWD/vardir/master/binlog.000004
[...]
# at 944
#181017 18:07:47 server id 1 end_log_pos 1034 CRC32 0xaff13579 Update_rows: table id 90 flags:
STMT_END_F
BINLOG ‘
[...]
'/*!*/;
### WHERE
### @3=1 /* FLOAT meta=4 nullable=1 is_null=0 */
### @4='Hello World!' /* BLOB/TEXT meta=2 nullable=1 is_null=0 */
### SET
### @4='Hello World!' /* BLOB/TEXT meta=2 nullable=1 is_null=0 */

109
SET SESSION binlog_row_image = NOBLOB;
[...]
# at 1299
#181017 18:11:15 server id 1 end_log_pos 1361 CRC32 0x9da2d84a Update_rows: table id 90 flags:
BINLOG '
I1/HWxMBAAAASQAAABMFAAAAAFoAAAAAAAEABHRlc3QAAnQzAAQDAwT8AgQCDgEBAAID/P8ABAoC
aTECaTIBZgF0CAEAaftKOg==
[...]
P0rYop0=
'/*!*/;
### WHERE
### SET

110
SET SESSION binlog_row_image = MINIMAL;
[...]
# at 1626
#181017 18:12:31 server id 1 end_log_pos 1672 CRC32 0x3bdf3247 Update_rows: table id 90 flags:
STMT_END_F
BINLOG '
b1/HWxMBAAAASQAAAFoGAAAAAFoAAAAAAAEABHRlc3QAAnQzAAQDAwT8AgQCDgEBAAID/P8ABAoC
aTECaTIBZgF0CAEADattrQ==
b1/HWx8BAAAALgAAAIgGAAAAAFoAAAAAAAEAAgAEAQIAAQAAAAAEAAAARzLfOw==
'/*!*/;
### WHERE
### SET

Tip #2: Capture Original Statement
• Log the original query along with the RBR events:
– Enhances auditing and debugging;
– Extra event preceding the Rows events;
– Replicated everywhere in the topology, together with the Rows events themselves;
– User can turn the behavior on and off.
– SET binlog_rows_query_log_events = [ON|OFF]
111

Tip #2: Capture Original Statement
112
SET SESSION binlog_rows_query_log_events=ON
mysql> SET SESSION binlog_rows_query_log_events=ON;
mysql> SHOW BINLOG EVENTS IN 'binlog.000005';
+---------------+-----+----------------+-----------+-------------+--------------------------------------+
+---------------+-----+----------------+-----------+-------------+--------------------------------------+
| binlog.000005 | 316 | Rows_query | 1 | 369 | # UPDATE t3 SET i2=5 WHERE i1=1 |
| binlog.000005 | 442 | Update_rows | 1 | 488 | table_id: 90 flags: STMT_END_F |
+---------------+-----+----------------+-----------+-------------+--------------------------------------+

Tip #3: Capture Only Logical Diffs for JSON columns
• Log the only the logical diff of a JSON document in the after image instead
of the full image.
– Less storage footprint.
– Less network bandwidth used.
– Faster apply procedure on the replica.
– SET binlog_row_value_options=[PARTIAL_JSON,’’]
113
MySQL 8

114
SET SESSION binlog_row_value_options=PARTIAL_JSON
mysql> CREATE TABLE t4 (i INT PRIMARY KEY, j JSON);
mysql> INSERT INTO t4 VALUES (1, '{"name": "strawberry", "price": 1}');
mysql> SET SESSION binlog_row_value_options=PARTIAL_JSON;
Query OK, 0 rows affected, 1 warning (0,00 sec)
mysql> SET SESSION binlog_row_image=MINIMAL;
mysql> UPDATE t4 SET j= JSON_REPLACE(j, '$.name', “orange");
MySQL 8

115
Replicate only the logical diff
of the json document in the
after image.
MySQL 8

116
Replicate only the logical diff
of the json document in the
after image.
Replicate only the primary key
in the before image.
MySQL 8

117
tut6318 $ mysql-8.0.13/bin/mysqlbinlog –v $PWD/vardir/master/binlog.000006
[...]
# at 2139
#181018 12:24:26 server id 1 end_log_pos 2204 CRC32 0x38ef3f91 Update_rows_partial: table id 95
flags: STMT_END_F
BINLOG '
Wl/IWxMBAAAAOwAAAFsIAAAAAF8AAAAAAAEABHRlc3QAAnQ1AAID9QEEAgEBAAQEAWkBaggBAKMk
v5I=
Wl/IWycBAAAAQQAAAJwIAAAAAF8AAAAAAAEAAgACAQIAAQAAAAEBABEAAAAABiQubmFtZQgMBm9y
YW5nZZE/7zg=
'/*!*/;
### WHERE
### @1=1
### SET
### @2=JSON_REPLACE(@2, '$.name', 'orange')
[...]
MySQL 8

118
Benchmarks
0
10000
20000
30000
40000
50000
60000
Entire JSON Partial JSON
Bytespertransaction
Binary Log Space Per Transaction
FULL MINIMAL
• Numbers are from a specially
designed benchmark:
• tables have 10 JSON fields,
• each transaction modifies
around 10% of the data
MySQL 8

Tip #4: Capture more table metadata
119
Easier to extract, transform and load into other systems.
[...]
Table_map event
Types, primary key, …
Sign, charset, …
Write_rows
[...]
Server
A
Binary log
App
INSERT ...
• New Metadata
– Easy to decode what is in the
binary log.
– Reduces impedance while
connecting MySQL to other
systems through the binary log
stream.
– SET binlog_row_metadata=[MINIMAL|FULL]
MySQL 8

Tip #4: Capture more table metadata
120
SET GLOBAL binlog_row_metadata = FULL;
mysql> CREATE TABLE t6 (c1 TINYINT, c2 INT NOT NULL, c3 CHAR(10) CHARSET latin1, c4 VARCHAR(10) CHARSET
utf8mb4, c5 ENUM ("e_v1", "e_v2", "e_v3"), PRIMARY KEY(c1,c2));
mysql> SET GLOBAL binlog_row_metadata = FULL;
mysql> INSERT INTO t6 VALUES (1,1,"luis“,“soares”,"e_v1");
tut6318 $ mysql-8.0.13/bin/mysqlbinlog –v $PWD/vardir/master/binlog.000005
[...]
# at 4339
#181018 12:49:46 server id 1 end_log_pos 4442 CRC32 0x14b877d5 Table_map: `test`.`t6` mapped to
number 99
# Columns(`c1` TINYINT NOT NULL, `c2` INT NOT NULL, `c3` CHAR(10) CHARSET latin1 COLLATE latin1_swedish_ci,
`c4` VARCHAR(10) CHARSET utf8mb4 COLLATE utf8mb4_0900_ai_ci, `c5` ENUM('e_v1', 'e_v2', 'e_v3'))
# Primary Key(c1, c2)
[...]
MySQL 8

Conclusion
121

Conclusion
• The binary log is a very useful building block.
– Contains a logical, self contained, representation of the changes happening in the
server.
– Primarily used for replication...
• Not only replication.
– Row-based format allows integration of MySQL in many advanced workflows.
– Change capture, change tracking, ETL, external triggers, and more.
• Plugins can tap into the transaction lifecycle events and access capture
caches.
• MySQL 8 adds more metadata into the change stream, enabling more use
cases.
122

Using The Mysql Binary Log As A Change Stream

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Using The Mysql Binary Log As A Change Stream

Similar to Using The Mysql Binary Log As A Change Stream (20)

Recently uploaded

Recently uploaded (20)

Using The Mysql Binary Log As A Change Stream