MySQL Power Group 2017년 1차 오프라인 세미나 발표자료 입니다. MySQL InnoDB Cluster 에 대한 간단한 설명 및 Remote Cluster 구성하여 테스트한 내용입니다.

1. MySQL InnoDB Cluster
미리 보기
MySQL Power Group
(http://cafe.naver.com/mysqlp)
2017.01.14
유 승민 (smyoo0316@gmail.com)

2. MySQL InnoDB Cluster
New HA solution (available on labs)
Read-out Scalability
Combination of three Technologies
MySQL Group Replication
MySQL Router
MySQL Shell

3. MySQL Group Replication
 The plugin that brings multi-master update everywhere (atomic broadcasting)
MySQL Router
 Lightweight middle ware between application and MySQL Server
 Read-write connection redirect (first-available)
 Read-only connection distribution (round-robin)
 Connection fail-over (redirect the failed connection to available MySQL Server)
MySQL Shell
 New command interface for MySQL Server.
 Support three type languages : javascript, python, sql
 Support admin command for InnoDB Cluster
 Must install python 2.7

4. MySQL (3306)
PRIMARY : R/W
[MySQL Group Replication]
Application
MySQL (3306)
SECONDARY : R/O
MySQL (3306)
SECONDARY : R/O
[MySQL Shell]
shell> mysqlsh
mysqlsh-js> c root@localhost:3306
mysqlsh-js> c = dba.createCluster(“Test”)
mysqlsh-js> c.addInstance(‘root@server2:3306)
mysqlsh-js> c.addInstance(‘root@server3:3306)
mysqlsh-js> c.describe()
mysqlsh-js> c.status()
[MySQL Router]
R/W : 6446
R/O : 6447
READ-WRITE
READ-ONLY

5. MySQL (3306)
PRIMARY : R/W
[MySQL Group Replication]
Application
MySQL (3306)
PRIMARY : R/W
MySQL (3306)
SECONDARY : R/O
[MySQL Shell]
mysql-js> c.status();
{
"clusterName": "TestCluster",
"defaultReplicaSet": {
"status": "Cluster is NOT tolerant to any failures.",
"topology": {
“server2:3306": {
"address": “server2:3306",
"status": "ONLINE",
"role": "HA",
"mode": "R/W",
"leaves": {
“server3:3306": {
"address": “server3:3306",
"status": "ONLINE",
"role": "HA",
"mode": "R/O",
"leaves": {}
},
“server1:3306": {
"address": “server1:3306",
"status": "OFFLINE",
"role": "HA",
"mode": "R/O",
"leaves": {}
}
}
}
}
}
}
[MySQL Router]
R/W : 6446
R/O : 6447
READ-WRITE
READ-ONLY

7. PLUGIN
UPDATE EVERYWHERE
AUTO DISTRIBUTED RECOVERY
CLOUD-FRIENDLY
5.7.17 GA Released (2016.12.12)
MySQL MySQL MySQL MySQL MySQL
Group Replication Cluster

8. <이미지출처 : http://mysqlhighavailability.com/mysql-group-replication-hello-world/>
Atomic Broadcasting.
First writer wins rules.

9. SLAVESLAVE
MASTER
Replication
SERVER3SERVER2
SERVER1
Group Replication

10. <이미지출처 : http://mysqlhighavailability.com/order-from-chaos-member-coordination-in-group-replication/>
기존 replication framework 에서 사용하는 기능
Binary log caching infrastructure
GTID framework
Row-based replication
완전 새로운 기능 (Xcom)
Messaging
Group membership
Auto distributed recovery mechanism
Atomic broadcasting
Transaction coordination
- Check whether the transaction should commit or not
- Propagate the changes
- Conflict resolution follows the first writer wins rule

11. MYSQL> SHOW CREATE TABLE T1G;
*************************** 1. row ***************************
Table: T1
Create Table: CREATE TABLE `t1` (
`col1` int(11) NOT NULL,
`col2` varchar(100) DEFAULT NULL,
PRIMARY KEY (`col1`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8
1 row in set (0.00 sec)
ERROR:
No query specified
MYSQL> INSERT INTO T1 VALUES (100, 'AAAA');
ERROR 3098 (HY000): The table does not comply with the requirements by an external plugin.
Only Available on InnoDB storage engine (That’s why they named InnoDB Cluster)

12. Single PRIMARY mode
Beginning with 5.7.15
group_replication_single_primary_mode=TRUE|FALSE
Which enables/disables the single primary mode
group_replication_enforce_update_everywhere_checks=FALSE|TRUE
Which enables/disables strict consistency checks for multi-master update everywhere
PRIMARY 외 다른 인스턴스들은 모두 SUPER_READ_ONLY 상태가 됨.

13. (5.7.15 lab version)

16. Please specify an administrative MASTER key for the cluster 'test':
Creating InnoDB cluster 'test' on 'root@localhost:3310'...
Traceback (most recent call last):
File "/usr/lib64/python2.6/runpy.py", line 122, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib64/python2.6/runpy.py", line 34, in _run_code
exec code in run_globals
File "/home/mysql/programs/mysql-innodb-cluster-labs201609-linux-glibc2.5-x86_64/mysql-shell-1.0.5-labs-linux-glibc2.12-x86-64bit/bin/mysqlprovision/__main__.py", line 32,
in <module>
File "/home/mysql/programs/mysql-innodb-cluster-labs201609-linux-glibc2.5-x86_64/mysql-shell-1.0.5-labs-linux-glibc2.12-x86-
64bit/bin/mysqlprovision/mysql_gadgets/common/options.py", line 32, in <module>
File "/home/mysql/programs/mysql-innodb-cluster-labs201609-linux-glibc2.5-x86_64/mysql-shell-1.0.5-labs-linux-glibc2.12-x86-
64bit/bin/mysqlprovision/mysql_gadgets/common/connection_parser.py", line 254
if hostportsock[0] in {'"', "'"}:
^
SyntaxError: invalid syntax
ArgumentError: Dba.createCluster: Dba.createCluster: Error while executing mysqlprovision (return 1) at (shell):1:14
in cluster = dba.createCluster('test');
^
mysql-js> cluster = dba.createCluster('test',);
SyntaxError: Unexpected token ) at (shell):1:35
in cluster = dba.createCluster('test',);
Python 2.7로 업그레이드 먼저 해야 함.
JS문법을 써도 python버전 에러가 남.
내부 구현체는 모두 python 인 듯.
파이썬 2.7 업그레이드 방법은 “여기”

17. ERROR: Error executing the 'start-replicaset' command: The operation could not continue due to the following requirements not being met:
Some active options on server 'localhost@3310' are incompatible with Group Replication.
Please restart the server 'localhost@3310' with the updated options file and try again.
Option name Required Value Current Value Result
------------------------------- --------------- --------------- -----
binlog_checksum NONE CRC32 FAIL
binlog_format ROW MIXED FAIL
enforce_gtid_consistency ON OFF FAIL
gtid_mode ON OFF FAIL
log_slave_updates ON 0 FAIL
master_info_repository TABLE FILE FAIL
relay_log_info_repository TABLE FILE FAIL
transaction_write_set_extraction XXHASH64 OFF FAIL
ArgumentError: Dba.createCluster: Dba.createCluster: The operation could not continue due to the following requirements not being met:
at (shell):1:14
in cluster = dba.createCluster('test');
^
필수 Parameter 가 맞지 않으면 유효성 검사에서 에러남. 아래와 같이 my.cnf 에 추가하고 DB 재시작
########################################
# InnoDB Cluster requirements
########################################
binlog_checksum = NONE
binlog_format = ROW
enforce_gtid_consistency = ON
gtid_mode = ON
log_slave_updates = ON
master_info_repository = TABLE
relay_log_info_repository = TABLE
transaction_write_set_extraction = XXHASH64

18. shell> mysqlsh // mysql shell 실행
mysql-js> c root@192.168.237.81:3310 // 클러스터를 생성할 DB 접속
mysql-js> c = dba.createCluster('TestCluster'); // Group Replication whitelist 관련 “ERROR 발생”. 하단 에러 정보 참고 (리모트에서만 발생)
mysql-js> sql // SQL 모드로 변경
mysql-sql> show global variables like '%whitelist%';
mysql-sql> set global group_replication_ip_whitelist = ‘192.168.237.81,192.168.237.82,192.168.237.83'; // whitelist 추가
mysql-sql> js // Java Script 모드로 변경
mysql-js> c = dba.createCluster('TestCluster'); // 다시 클러스터 생성. 이때 입력하는 “클러스터패스워드”를 잘 기억해야 함.
mysql-js> c.describe(); // 클러스터 멤버 정보 확인
mysql-js> c.status(); // 클러스터 인스턴스 상태 확인
< ERROR LOG >
2016-11-16T06:36:36.582375Z 0 [Warning] Plugin group_replication reported: '[GCS] Connection attempt from IP address 192.168.237.81 refused. Address is not in
the IP whitelist.'
2016-11-16T06:36:36.582528Z 0 [ERROR] Plugin group_replication reported: '[GCS] Error connecting to the local group communication engine instance.'
2016-11-16T06:36:36.582557Z 0 [Note] Plugin group_replication reported: 'state 4127 action xa_exit'
2016-11-16T06:36:36.582682Z 0 [Note] Plugin group_replication reported: 'Exiting xcom thread'
2016-11-16T06:36:37.593237Z 0 [ERROR] Plugin group_replication reported: '[GCS] The member was unable to join the group. Local port: 13310'

19. MySQL> install plugin group_replication soname 'group_replication.so';
MySQL> set global group_replication_ip_whitelist = '192.168.237.81,192.168.237.82,192.168.237.83'; // whitelist 미리 추가
MySQL> set global group_replication_allow_local_disjoint_gtids_join = TRUE; // 하단 설명 참조.
MySQL> exit
$mysqlsh
mysql-js> c c root@192.168.237.81:3310 // PRIMARY 인스턴스 접속
mysql-js> dba.validateInstance('root@192.168.237.83:3310'); // 필수 parameter 점검
mysql-js> c.addInstance('root@192.168.237.83:3310'); // 인스턴스 추가
mysql-js> c.status(); // RECOERVING 상태로 대기함. root 유저 생성한 sql문 충돌 때문 (리모트에서만 발생)
mysql-js> sql // SQL 모드로 변경
stop group_replication;
drop user 'root'@'192.168.237.%'; // error log 에서 충돌 원인 확인하여 조치
start group_replication;
mysql-sql> js // Java Script 모드로 변경
c.status(); // ONLINE 상태로 보이면 정상
group_replication_allow_local_disjoint_gtids_join
인스턴스 추가 할 때 기존 클러스터 멤버 보다 최신 데이터가 감지 될 시 복구나 런타임 에러를 방지하기 위해 REJECT함.
무시해도 된다고 확신 할 경우 TRUE로 하고 강제로 추가.

20. mysql-js> c.status();
{
"clusterName": "test",
"defaultReplicaSet": {
"status": "Cluster tolerant to up to ONE failure.",
"topology": {
"192.168.237.81:3310": {
"address": "192.168.237.81:3310",
"status": "ONLINE",
"role": "HA",
"mode": "R/W",
"leaves": {
"192.168.237.83:3310": {
"address": "192.168.237.83:3310",
"status": "ONLINE",
"role": "HA",
"mode": "R/O",
"leaves": {}
},
"192.168.237.82:3310": {
"address": "192.168.237.82:3310",
"status": "ONLINE",
"role": "HA",
"mode": "R/O",
"leaves": {}
}
}
}
}
}
}
두 번째와 동일한 방식으로 추가하고 status()에서 모두
ONLINE 상태이면 정상.
# mysqlsh 세션 종류 후 클러스터 인스턴스 컨트롤 하는 법
mysql-sh> c = dba.getCluster(); //클러스터패스워드 입력
mysql-sh> c.help();

21. show grants for 'mysql_innodb_cluster_admin'@'192.168.237.81';
+-----------------------------------------------------------------------------------------------------------------+
| Grants for mysql_innodb_cluster_admin@192.168.237.81 |
+-----------------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'mysql_innodb_cluster_admin'@'192.168.237.81' |
| GRANT ALL PRIVILEGES ON `mysql_innodb_cluster_metadata`.* TO 'mysql_innodb_cluster_admin'@'192.168.237.81' |
| GRANT SELECT ON `performance_schema`.`replication_group_members` TO 'mysql_innodb_cluster_admin'@'192.168.237.81' |
+-----------------------------------------------------------------------------------------------------------------+
3 rows in set (0.00 sec)
// MySQL Router 에서 접속할 유저 생성
GRANT USAGE ON *.* TO 'mysql_innodb_cluster_admin'@'192.168.237.105';
GRANT ALL PRIVILEGES ON `mysql_innodb_cluster_metadata`.* TO 'mysql_innodb_cluster_admin'@'192.168.237.105';
GRANT SELECT ON `performance_schema`.`replication_group_members` TO 'mysql_innodb_cluster_admin'@'192.168.237.105';
set password for 'mysql_innodb_cluster_admin'@'192.168.237.105' = '1'; // 클러스터패스워드와 동일해야 함.
// Failover 테스트를 위한 일반 DB유저 생성
create user 'smyoo'@'192.168.237.105' identified by 'smyoo';
grant select, insert, update, delete on *.* to 'smyoo'@'192.168.237.105';

22. $ ./mysqlrouter --bootstrap 192.168.237.81:3310 // DB로부터 클러스터 메타 정보를 읽어서 mysqlrouter.conf 파일을 생성
Please enter the administrative MASTER key for the MySQL InnoDB cluster: // 클러스터 패스워드 입력
MySQL Router has now been configured for the InnoDB cluster 'test'.
The following connection information can be used to connect to the cluster.
Classic MySQL protocol connections to cluster 'test':
- Read/Write Connections: localhost:6446
- Read/Only Connections: localhost:6447
$
$ nohup ./mysqlrouter & // 백그라운드로 기동
[1] 43986
$ nohup: ignoring input and appending output to `nohup.out'
$
$ ps -ef | grep mysqlrouter
root 43986 8680 0 15:04 pts/0 00:00:00 ./mysqlrouter
root 44050 8680 0 15:04 pts/0 00:00:00 grep mysqlrouter
$

23. // 자동으로 생성된 mysqlrouter.conf 파일 내용
[DEFAULT]
plugin_folder=/root/programs/mysql-innodb-cluster-labs201609-linux-glibc2.5-x86_64/mysql-router-2.1.0-labs-linux-glibc2.12-x86-
64bit/lib/mysqlrouter
# logging_folder=/root/programs/mysql-innodb-cluster-labs201609-linux-glibc2.5-x86_64/mysql-router-2.1.0-labs-linux-glibc2.12-x86-
64bit/bin/../
[logger]
level = INFO
[metadata_cache]
bootstrap_server_addresses=mysql://192.168.237:3310,mysql://192.168.237.82:3310,mysql://192.168.237.83:3310
user=mysql_innodb_cluster_reader
password=,wY,JbZ9Saxf%}(S
metadata_cluster=test
ttl=300
metadata_replicaset=default
[routing:default_rw]
bind_port=6446
destinations=metadata-cache:///default?role=PRIMARY
mode=read-write
[routing:default_ro]
bind_port=6447
destinations=metadata-cache:///default?role=SECONDARY
mode=read-only

24. $ mysqlsh --uri smyoo@localhost:6446 // MySQL Router 로 PRIMARY 인스턴스로 접속
mysql-js> sql // SQL 모드로 변경
Switching to SQL mode... Commands end with ;
mysql-sql>
mysql-sql> select @@hostname; // PRIMARY 인스턴스 접속 확인
+------------+
| @@hostname |
+------------+
| mytest1 |
+------------+
1 row in set (0.00 sec)
mysql-sql>
mysql-sql> select @@hostname;
ERROR: 2013 (HY000): Lost connection to MySQL server during query // PRIMARY 인스턴스 DOWN 감지 (약 1초 정도 딜레이)
The global session got disconnected. // 접속 실패한 세션은 그냥 사라짐. 자동으로 재 접속 하지 않는다.
Attempting to reconnect to 'smyoo@localhost:6446'...
The global session was successfully reconnected.
mysql-sql>
mysql-sql> select @@hostname; // Failover 된 것 확인
+------------+
| @@hostname |
+------------+
| mytest2 |
+------------+
1 row in set (0.00 sec)

27. MYSQL> SELECT * FROM PERFORMANCE_SCHEMA.REPLICATION_GROUP_MEMBERS ORDER BY MEMBER_HOST;
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
| group_replication_applier | dae27541-b2e5-11e6-a22c-ac162d76e1c4 | mytest1 | 3310 | ONLINE |
| group_replication_applier | deeb5d42-b2e5-11e6-9ef5-002655e4a549 | mytest2 | 3310 | ONLINE |
| group_replication_applier | e2449c17-b2e5-11e6-b72a-002655e4a51d | mytest3 | 3310 | ONLINE |
+---------------------------+--------------------------------------+-------------+-------------+--------------+
3 rows in set (0.00 sec)
MYSQL> SELECT * FROM PERFORMANCE_SCHEMA.REPLICATION_GROUP_MEMBER_STATSG;
*************************** 1. row ***************************
CHANNEL_NAME: group_replication_applier
VIEW_ID: 14800615308848506:7
MEMBER_ID: e2449c17-b2e5-11e6-b72a-002655e4a51d
COUNT_TRANSACTIONS_IN_QUEUE: 0  큐에 누적된 트랜잭션 수. (동기화 지연 정도) 0을 유지하는 것이 바람직.
COUNT_TRANSACTIONS_CHECKED: 1217439
COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: dcd7ae4b-b2e6-11e6-b4a5-002655e4a51d:1-1217444,
e2449c17-b2e5-11e6-b72a-002655e4a51d:1-22
LAST_CONFLICT_FREE_TRANSACTION: dcd7ae4b-b2e6-11e6-b4a5-002655e4a51d:1217444
1 row in set (0.00 sec)
ERROR:
No query specified

28. (with Grafana & InfluxDB)

29. 1번 노드가 빠진 상태에서 다시 추가하기 전 큰 테
이블(1천만건) 미리 생성해 두고 sysbench트래픽이
유입되는 상태에서 노드 추가.
노드 추가 시 발생하는 자동 분산 복구 시 “아직은”
제대로 동작 안함.
DML 트래픽을 처리 못하고 트랜잭션 큐 지연 발생.
RECOVERING 상태에서 더 이상 진행이 안됨.

30.  요약
완전히 다른 동기화 메커니즘
구축 용이, 편리한 운영, 유연한 확장 (을 기대 합니다)
 현재상황
기본적인 기능은 무난하게 동작함. (Single PRIMARY에서의 동기화 처리 속도, Failover)
Remote 구성이 “아직은” 매끄럽지 않음.
mysqlsh은 조금 익숙해지면 편리함.
온라인 자동 복구는 “아직은” 제대로 동작 안 함.
 제약사항
InnoDB만 된다.
ROW 포멧
GTID 필수