OLTP更新とバッチ処理の負荷シュミレーション
Copyrights LOCONDO,Inc. AllRights Reserved.
SYSBENCH用: 初期データ(4テーブル)
+----------+---------+------------+
| Database | Table | Size in MB |
+----------+---------+------------+
| sysbench | sbtest3 | 22426.69 |
| sysbench | sbtest1 | 22390.70 |
| sysbench | sbtest2 | 22383.67 |
| sysbench | sbtest4 | 22319.73 |
+----------+---------+------------+
Pri Sec Sec
Tool: mysqlslap (バッチ系の処理を確認)
大量にまとめてデータをINSERTして削除する等、一つのトランザクションで大量にデータを処理する試験に使用。
Tool: sysbench(更新処理内容:計測の為に各ベンチマークを10分間 実行)
INSERT INTO sbtest1 (id, k, c, pad) VALUES (75087935, 12463253, '13889104256-30791869017-66538184821-5…….
DELETE FROM sbtest1 WHERE id=75087935
UPDATE sbtest1 SET k=k+1 WHERE id=63874937
UPDATE sbtest4 SET c='51744764069-67682943268-15332003429-10934680309-42746352369-…………….' WHERE id=12627688
アーキテクチャ: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 40
Model name: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Memory : 128GB
Disk : SanDisk ioMemory PX Series
29.
OLTP:複数接続+細かいデータ更新処理
Copyrights LOCONDO,Inc. AllRights Reserved.
[benchmark]# ./mysqlslap_check_group_member_lag.sh
BENCHMARKデータベースに対して、100クライアント x 1000件実行します
Enter password:
Benchmark
Average number of seconds to run all queries: 0.012 seconds
Minimum number of seconds to run all queries: 0.010 seconds
Maximum number of seconds to run all queries: 0.213 seconds
Number of clients running queries: 100
Average number of queries per client: 1
SQL> SELECT * FROM performance_schema.replication_group_members;SELECT count_transactions_in_queue FROM
performance_schema.replication_group_member_stats;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | 42c3cbdc-f269-33j4-b56c-02011477005e | TST-db-12 | 3306 | ONLINE |
| group_replication_applier | e1731e29-f269-33j4-8646-020114770060 | TST-db-14 | 3306 | ONLINE |
| group_replication_applier | e53e17a3-f269-33j4-98c5-02011477005f | TST-db-13 | 3306 | ONLINE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
3 rows in set (0.00 sec)
細かくトランザクションが分かれているので、
トランザクションがQueueに溜まる事はあるが、
グループメンバー間に差は発生していない。
“insert”毎のコミット処理
30.
Copyrights LOCONDO,Inc. AllRights Reserved.
select CHANNEL_NAME,RECEIVED_TRANSACTION_SET from replication_connection_status where CHANNEL_NAME = "group_replication_applier"¥G
select TRANSACTIONS_COMMITTED_ALL_MEMBERS from performance_schema.replication_group_member_stats¥G
select CHANNEL_NAME,WORKER_ID,LAST_SEEN_TRANSACTION from replication_applier_status_by_worker where CHANNEL_NAME = "group_replication_applier";
select @last_exec:=SUBSTRING_INDEX(SUBSTRING_INDEX(SUBSTRING_INDEX( @@global.GTID_EXECUTED,':',-1),':',1),'-',-1) last_executed;
select @last_rec:=SUBSTRING_INDEX(SUBSTRING_INDEX(SUBSTRING_INDEX( Received_transaction_set,':',-1),':',1),'-',-1) last_received FROM
performance_schema.replication_connection_status WHERE Channel_name = 'group_replication_applier';
select (@last_rec - @last_exec) as real_lag;SELECT count_transactions_in_queue FROM performance_schema.replication_group_member_stats;
*************************** 1. row ***************************
CHANNEL_NAME: group_replication_applier
RECEIVED_TRANSACTION_SET: cb904887-a1fc-11e8-ae58-02011477005e:1-14035941
1 row in set (0.00 sec)
*************************** 1. row ***************************
TRANSACTIONS_COMMITTED_ALL_MEMBERS: cb904887-a1fc-11e8-ae58-02011477005e:1-13896262
1 row in set (0.00 sec)
<SNIP>
| group_replication_applier | 5 | cb904887-a1fc-11e8-ae58-02011477005e:14035940 |
| group_replication_applier | 6 | cb904887-a1fc-11e8-ae58-02011477005e:14035941 |
| group_replication_applier | 7 | cb904887-a1fc-11e8-ae58-02011477005e:14035937 |
| group_replication_applier | 8 | cb904887-a1fc-11e8-ae58-02011477005e:14035942 |
+---------------------------+-----------+-----------------------------------------------+
8 rows in set (0.00 sec)
+----------+
| real_lag |
+----------+
| 0 |
+----------+
1 row in set (0.00 sec)
+-----------------------------+
| count_transactions_in_queue |
+-----------------------------+
| 0 |
+-----------------------------+
1 row in set (0.00 sec)
この値は、Certificationモジュールによってグループ
メンバー全体の状況を確認する為に、一定の間隔で
更新される為、リアルタイムモニタリングには使えない。
受け取ったトランザクションはApplierスレッドが処理完了、
若しくは処理中。以下のクエリーの値が”0”を返す場合
は、まだコミットしていないトランザクションが残っている。
SELECT GTID_SUBSET(LAST_SEEN_TRANSACTION,
@@GLOBAL.GTID_EXECUTED) FROM
performance_schema.replication_applier_status_by_worker;
Queuingしているトランザクション数
大量に処理を流すと、Queueに溜まるが, 検証中は
問題になる程Queueは滞留する事はなかった。
受け取ったトランザクションセットの内容が
適用済みかどうか? 適用が完了していない、
トランザクションがあるかどうかの確認。
31.
Copyrights LOCONDO,Inc. AllRights Reserved.
[root@TST-db-12 oltp16th80G]# ../oltp_test_read_only
sysbench 0.5: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 16
Initializing random number generator from timer.
Random number generator seed is 0 and will be ignored Threads started!
OLTP test statistics:
queries performed:
read: 32271554
write: 0
other: 4610222
total: 36881776
transactions: 2305111 (3841.82 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 32271554 (53785.48 per sec.)
other operations: 4610222 (7683.64 per sec.)
General statistics:
total time: 600.0049s
total number of events: 2305111
total time taken by event execution: 9595.8084s
response time:
min: 2.97ms
avg: 4.16ms
max: 66.61ms
approx. 95 percentile: 4.67ms
Threads fairness:
events (avg/stddev): 144069.4375/3076.16
execution time (avg/stddev): 599.7380/0.01
※ MySQL Router経由での処理は別途確認だがスレーブは分散されるのでスケール可
SELECT
TPS:3841.82
[root@TST-db-12 oltp64th80G]# ../oltp_test_read_only
sysbench 0.5: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 64
Initializing random number generator from timer.
Random number generator seed is 0 and will be ignored Threads started!
OLTP test statistics:
queries performed:
read: 47745418
write: 0
other: 6820774
total: 54566192
transactions: 3410387 (5683.85 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 47745418 (79573.93 per sec.)
other operations: 6820774 (11367.70 per sec.)
General statistics:
total time: 600.0133s
total number of events: 3410387
total time taken by event execution: 38389.2534s
response time:
min: 3.41ms
avg: 11.26ms
max: 227.48ms
approx. 95 percentile: 18.26ms
Threads fairness:
events (avg/stddev): 53287.2969/16943.99
execution time (avg/stddev): 599.8321/0.05
※ MySQL Router経由での処理は別途確認だがスレーブは分散されるのでスケール可
TPS:5683.85
32.
Copyrights LOCONDO,Inc. AllRights Reserved.
[root@TST-db-12 write16th80G]# ../oltp_test_write
sysbench 0.5: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 16
Initializing random number generator from timer.
Random number generator seed is 0 and will be ignored Threads started!
OLTP test statistics:
queries performed:
read: 12396902
write: 3541972
other: 1770986
total: 17709860
transactions: 885493 (1475.79 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 15938874 (26564.29 per sec.)
other operations: 1770986 (2951.59 per sec.)
General statistics:
total time: 600.0114s
total number of events: 885493
total time taken by event execution: 9598.1602s
response time:
min: 5.93ms
avg: 10.84ms
max: 1178.88ms
approx. 95 percentile: 16.10ms
Threads fairness:
events (avg/stddev): 55343.3125/508.16
execution time (avg/stddev): 599.8850/0.00
※ count_transactions_in_queueは0~5の間で推移し遅延無し
DML
TPS:1475.79
[root@TST-db-12 write128th80G]# ../oltp_test_write
sysbench 0.5: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 128
Initializing random number generator from timer.
Random number generator seed is 0 and will be ignored Threads started!
OLTP test statistics:
queries performed:
read: 21410284
write: 6117224
other: 3058612
total: 30586120
transactions: 1529306 (2548.67 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 27527508 (45876.01 per sec.)
other operations: 3058612 (5097.33 per sec.)
General statistics:
total time: 600.0415s
total number of events: 1529306
total time taken by event execution: 76788.4301s
response time:
min: 10.29ms
avg: 50.21ms
max: 2539.81ms
approx. 95 percentile: 78.80ms
Threads fairness:
events (avg/stddev): 11947.7031/560.60
execution time (avg/stddev): 599.9096/0.06
※ count_transactions_in_queueは0~400の間で推移するが遅延無し
※ こちらのワークロードでは、512 Threadsまではトランザクションがスケールする事を確認。
※ 1G程度の小さいテーブルであれば、512 ThreadsでTPS: 3500/sec以上可能
TPS:2548.67
33.
Copyrights LOCONDO,Inc. AllRights Reserved.
OLTP test statistics:
queries performed:
read: 23119866
write: 6605676
other: 3302838
total: 33028380
transactions: 1651419 (2746.82 per sec.)
deadlocks: 0 (0.00 per sec.)
read/write requests: 29725542 (49442.71 per sec.)
other operations: 3302838 (5493.63 per sec.)
General statistics:
total time: 601.2118s
total number of events: 1651419
total time taken by event execution: 307199.4364s
response time:
min: 31.27ms
avg: 186.02ms
max: 3222.41ms
approx. 95 percentile: 257.07ms
Threads fairness:
events (avg/stddev): 3225.4277/81.55
execution time (avg/stddev): 599.9989/1.13
Number of threads: 512
6,605,676書き込み/600秒
= 約11,000/秒
合計: 2746.82トランザクション/秒
細かいトランザクションであれば、
マスターとスレーブ間は同期状態
処理数を増やしていっても
ギャップが発生しない事を確認 ◎
更新処理も通常のOLTP処理であれば、
3倍以上は捌けると判断。
参考: Additional Featurein MySQL8.0
WL#10379: Group Replication: consistent reads
https://dev.mysql.com/worklog/task/?id=10379
WL#11123: Group Replication: hold reads and writes when the new primary
has replication backlog to apply
https://dev.mysql.com/worklog/task/?id=11123
WL#10378: Group Replication: group single/multi primary mode change and
primary election
https://dev.mysql.com/worklog/task/?id=10378
Copyrights LOCONDO,Inc. All Rights Reserved.
●●
38.
バージョンアップに伴う追加メリット
Copyrights LOCONDO,Inc. AllRights Reserved.
MySQLのレプリケーションは1メジャーバージョンの違いはサポート対象
“MySQL supports replication from one release series to the next higher release series. ”
●●
39.
MySQL8.0:集計関数が利用可能(Windows関数・CTE)
Copyrights LOCONDO,Inc. AllRights Reserved.
Pri
5.7
Sec
5.7
Sec
5.7
Slave
MySQL8.0
mysql> select
-> rank() over w as 'rank',
-> Continent,Name,GNP,
-> row_number() over w as 'row_number',
-> dense_rank() over w as 'dense_rank'
-> from country
-> window w as (order by GNP desc) limit 10;
+------+---------------+----------------+------------+------------+------------+
| rank | Continent | Name | GNP | row_number | dense_rank |
+------+---------------+----------------+------------+------------+------------+
| 1 | North America | United States | 8510700.00 | 1 | 1 |
| 2 | Asia | Japan | 3787042.00 | 2 | 2 |
| 3 | Europe | Germany | 2133367.00 | 3 | 3 |
| 4 | Europe | France | 1424285.00 | 4 | 4 |
| 5 | Europe | United Kingdom | 1378330.00 | 5 | 5 |
| 6 | Europe | Italy | 1161755.00 | 6 | 6 |
| 7 | Asia | China | 982268.00 | 7 | 7 |
| 8 | South America | Brazil | 776739.00 | 8 | 8 |
| 9 | North America | Canada | 598862.00 | 9 | 9 |
| 9 | Europe | Spain | 598862.00 | 10 | 9 |
+------+---------------+----------------+------------+------------+------------+
10 rows in set (0.00 sec)
mysql> with t1 as (select * from city where CountryCode = 'JPN')
-> select * from t1 limit 5;
+------+---------------------+-------------+----------+------------+
| ID | Name | CountryCode | District | Population |
+------+---------------------+-------------+----------+------------+
| 1532 | Tokyo | JPN | Tokyo-to | 7980230 |
| 1533 | Jokohama [Yokohama] | JPN | Kanagawa | 3339594 |
| 1534 | Osaka | JPN | Osaka | 2595674 |
| 1535 | Nagoya | JPN | Aichi | 2154376 |
| 1536 | Sapporo | JPN | Hokkaido | 1790886 |
+------+---------------------+-------------+----------+------------+
5 rows in set (0.05 sec)
Windows関数による、容易なランキング集計
CTEによる、Viewや一時テーブル処理と比較したパフォーマンス改善
http://mysqlserverteam.com/mysql-8-0-improved-performance-with-cte/