MySQL InnoDB Cluster 미리보기 (remote cluster test)


Published on

MySQL Power Group 2017년 1차 오프라인 세미나 발표자료 입니다.
MySQL InnoDB Cluster 에 대한 간단한 설명 및 Remote Cluster 구성하여 테스트한 내용입니다.

Published in: Software
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

MySQL InnoDB Cluster 미리보기 (remote cluster test)

  1. 1. MySQL InnoDB Cluster 미리 보기 MySQL Power Group ( 2017.01.14 유 승민 (
  2. 2. MySQL InnoDB Cluster New HA solution (available on labs) Read-out Scalability Combination of three Technologies MySQL Group Replication MySQL Router MySQL Shell
  3. 3. MySQL Group Replication  The plugin that brings multi-master update everywhere (atomic broadcasting) MySQL Router  Lightweight middle ware between application and MySQL Server  Read-write connection redirect (first-available)  Read-only connection distribution (round-robin)  Connection fail-over (redirect the failed connection to available MySQL Server) MySQL Shell  New command interface for MySQL Server.  Support three type languages : javascript, python, sql  Support admin command for InnoDB Cluster  Must install python 2.7
  4. 4. MySQL (3306) PRIMARY : R/W [MySQL Group Replication] Application MySQL (3306) SECONDARY : R/O MySQL (3306) SECONDARY : R/O [MySQL Shell] shell> mysqlsh mysqlsh-js> c root@localhost:3306 mysqlsh-js> c = dba.createCluster(“Test”) mysqlsh-js> c.addInstance(‘root@server2:3306) mysqlsh-js> c.addInstance(‘root@server3:3306) mysqlsh-js> c.describe() mysqlsh-js> c.status() [MySQL Router] R/W : 6446 R/O : 6447 READ-WRITE READ-ONLY
  5. 5. MySQL (3306) PRIMARY : R/W [MySQL Group Replication] Application MySQL (3306) PRIMARY : R/W MySQL (3306) SECONDARY : R/O [MySQL Shell] mysql-js> c.status(); { "clusterName": "TestCluster", "defaultReplicaSet": { "status": "Cluster is NOT tolerant to any failures.", "topology": { “server2:3306": { "address": “server2:3306", "status": "ONLINE", "role": "HA", "mode": "R/W", "leaves": { “server3:3306": { "address": “server3:3306", "status": "ONLINE", "role": "HA", "mode": "R/O", "leaves": {} }, “server1:3306": { "address": “server1:3306", "status": "OFFLINE", "role": "HA", "mode": "R/O", "leaves": {} } } } } } } [MySQL Router] R/W : 6446 R/O : 6447 READ-WRITE READ-ONLY
  7. 7. <이미지출처 :> Atomic Broadcasting. First writer wins rules.
  8. 8. SLAVESLAVE MASTER Replication SERVER3SERVER2 SERVER1 Group Replication
  9. 9. <이미지출처 :> 기존 replication framework 에서 사용하는 기능 Binary log caching infrastructure GTID framework Row-based replication 완전 새로운 기능 (Xcom) Messaging Group membership Auto distributed recovery mechanism Atomic broadcasting Transaction coordination - Check whether the transaction should commit or not - Propagate the changes - Conflict resolution follows the first writer wins rule
  10. 10. MYSQL> SHOW CREATE TABLE T1G; *************************** 1. row *************************** Table: T1 Create Table: CREATE TABLE `t1` ( `col1` int(11) NOT NULL, `col2` varchar(100) DEFAULT NULL, PRIMARY KEY (`col1`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8 1 row in set (0.00 sec) ERROR: No query specified MYSQL> INSERT INTO T1 VALUES (100, 'AAAA'); ERROR 3098 (HY000): The table does not comply with the requirements by an external plugin. Only Available on InnoDB storage engine (That’s why they named InnoDB Cluster)
  11. 11. Single PRIMARY mode Beginning with 5.7.15 group_replication_single_primary_mode=TRUE|FALSE Which enables/disables the single primary mode group_replication_enforce_update_everywhere_checks=FALSE|TRUE Which enables/disables strict consistency checks for multi-master update everywhere PRIMARY 외 다른 인스턴스들은 모두 SUPER_READ_ONLY 상태가 됨.
  12. 12. (5.7.15 lab version)
  13. 13. Please specify an administrative MASTER key for the cluster 'test': Creating InnoDB cluster 'test' on 'root@localhost:3310'... Traceback (most recent call last): File "/usr/lib64/python2.6/", line 122, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib64/python2.6/", line 34, in _run_code exec code in run_globals File "/home/mysql/programs/mysql-innodb-cluster-labs201609-linux-glibc2.5-x86_64/mysql-shell-1.0.5-labs-linux-glibc2.12-x86-64bit/bin/mysqlprovision/", line 32, in <module> File "/home/mysql/programs/mysql-innodb-cluster-labs201609-linux-glibc2.5-x86_64/mysql-shell-1.0.5-labs-linux-glibc2.12-x86- 64bit/bin/mysqlprovision/mysql_gadgets/common/", line 32, in <module> File "/home/mysql/programs/mysql-innodb-cluster-labs201609-linux-glibc2.5-x86_64/mysql-shell-1.0.5-labs-linux-glibc2.12-x86- 64bit/bin/mysqlprovision/mysql_gadgets/common/", line 254 if hostportsock[0] in {'"', "'"}: ^ SyntaxError: invalid syntax ArgumentError: Dba.createCluster: Dba.createCluster: Error while executing mysqlprovision (return 1) at (shell):1:14 in cluster = dba.createCluster('test'); ^ mysql-js> cluster = dba.createCluster('test',); SyntaxError: Unexpected token ) at (shell):1:35 in cluster = dba.createCluster('test',); Python 2.7로 업그레이드 먼저 해야 함. JS문법을 써도 python버전 에러가 남. 내부 구현체는 모두 python 인 듯. 파이썬 2.7 업그레이드 방법은 “여기”
  14. 14. ERROR: Error executing the 'start-replicaset' command: The operation could not continue due to the following requirements not being met: Some active options on server 'localhost@3310' are incompatible with Group Replication. Please restart the server 'localhost@3310' with the updated options file and try again. Option name Required Value Current Value Result ------------------------------- --------------- --------------- ----- binlog_checksum NONE CRC32 FAIL binlog_format ROW MIXED FAIL enforce_gtid_consistency ON OFF FAIL gtid_mode ON OFF FAIL log_slave_updates ON 0 FAIL master_info_repository TABLE FILE FAIL relay_log_info_repository TABLE FILE FAIL transaction_write_set_extraction XXHASH64 OFF FAIL ArgumentError: Dba.createCluster: Dba.createCluster: The operation could not continue due to the following requirements not being met: at (shell):1:14 in cluster = dba.createCluster('test'); ^ 필수 Parameter 가 맞지 않으면 유효성 검사에서 에러남. 아래와 같이 my.cnf 에 추가하고 DB 재시작 ######################################## # InnoDB Cluster requirements ######################################## binlog_checksum = NONE binlog_format = ROW enforce_gtid_consistency = ON gtid_mode = ON log_slave_updates = ON master_info_repository = TABLE relay_log_info_repository = TABLE transaction_write_set_extraction = XXHASH64
  15. 15. shell> mysqlsh // mysql shell 실행 mysql-js> c root@ // 클러스터를 생성할 DB 접속 mysql-js> c = dba.createCluster('TestCluster'); // Group Replication whitelist 관련 “ERROR 발생”. 하단 에러 정보 참고 (리모트에서만 발생) mysql-js> sql // SQL 모드로 변경 mysql-sql> show global variables like '%whitelist%'; mysql-sql> set global group_replication_ip_whitelist = ‘,,'; // whitelist 추가 mysql-sql> js // Java Script 모드로 변경 mysql-js> c = dba.createCluster('TestCluster'); // 다시 클러스터 생성. 이때 입력하는 “클러스터패스워드”를 잘 기억해야 함. mysql-js> c.describe(); // 클러스터 멤버 정보 확인 mysql-js> c.status(); // 클러스터 인스턴스 상태 확인 < ERROR LOG > 2016-11-16T06:36:36.582375Z 0 [Warning] Plugin group_replication reported: '[GCS] Connection attempt from IP address refused. Address is not in the IP whitelist.' 2016-11-16T06:36:36.582528Z 0 [ERROR] Plugin group_replication reported: '[GCS] Error connecting to the local group communication engine instance.' 2016-11-16T06:36:36.582557Z 0 [Note] Plugin group_replication reported: 'state 4127 action xa_exit' 2016-11-16T06:36:36.582682Z 0 [Note] Plugin group_replication reported: 'Exiting xcom thread' 2016-11-16T06:36:37.593237Z 0 [ERROR] Plugin group_replication reported: '[GCS] The member was unable to join the group. Local port: 13310'
  16. 16. MySQL> install plugin group_replication soname ''; MySQL> set global group_replication_ip_whitelist = ',,'; // whitelist 미리 추가 MySQL> set global group_replication_allow_local_disjoint_gtids_join = TRUE; // 하단 설명 참조. MySQL> exit $mysqlsh mysql-js> c c root@ // PRIMARY 인스턴스 접속 mysql-js> dba.validateInstance('root@'); // 필수 parameter 점검 mysql-js> c.addInstance('root@'); // 인스턴스 추가 mysql-js> c.status(); // RECOERVING 상태로 대기함. root 유저 생성한 sql문 충돌 때문 (리모트에서만 발생) mysql-js> sql // SQL 모드로 변경 stop group_replication; drop user 'root'@'192.168.237.%'; // error log 에서 충돌 원인 확인하여 조치 start group_replication; mysql-sql> js // Java Script 모드로 변경 c.status(); // ONLINE 상태로 보이면 정상 group_replication_allow_local_disjoint_gtids_join 인스턴스 추가 할 때 기존 클러스터 멤버 보다 최신 데이터가 감지 될 시 복구나 런타임 에러를 방지하기 위해 REJECT함. 무시해도 된다고 확신 할 경우 TRUE로 하고 강제로 추가.
  17. 17. mysql-js> c.status(); { "clusterName": "test", "defaultReplicaSet": { "status": "Cluster tolerant to up to ONE failure.", "topology": { "": { "address": "", "status": "ONLINE", "role": "HA", "mode": "R/W", "leaves": { "": { "address": "", "status": "ONLINE", "role": "HA", "mode": "R/O", "leaves": {} }, "": { "address": "", "status": "ONLINE", "role": "HA", "mode": "R/O", "leaves": {} } } } } } } 두 번째와 동일한 방식으로 추가하고 status()에서 모두 ONLINE 상태이면 정상. # mysqlsh 세션 종류 후 클러스터 인스턴스 컨트롤 하는 법 mysql-sh> c = dba.getCluster(); //클러스터패스워드 입력 mysql-sh>;
  18. 18. show grants for 'mysql_innodb_cluster_admin'@''; +-----------------------------------------------------------------------------------------------------------------+ | Grants for mysql_innodb_cluster_admin@ | +-----------------------------------------------------------------------------------------------------------------+ | GRANT USAGE ON *.* TO 'mysql_innodb_cluster_admin'@'' | | GRANT ALL PRIVILEGES ON `mysql_innodb_cluster_metadata`.* TO 'mysql_innodb_cluster_admin'@'' | | GRANT SELECT ON `performance_schema`.`replication_group_members` TO 'mysql_innodb_cluster_admin'@'' | +-----------------------------------------------------------------------------------------------------------------+ 3 rows in set (0.00 sec) // MySQL Router 에서 접속할 유저 생성 GRANT USAGE ON *.* TO 'mysql_innodb_cluster_admin'@''; GRANT ALL PRIVILEGES ON `mysql_innodb_cluster_metadata`.* TO 'mysql_innodb_cluster_admin'@''; GRANT SELECT ON `performance_schema`.`replication_group_members` TO 'mysql_innodb_cluster_admin'@''; set password for 'mysql_innodb_cluster_admin'@'' = '1'; // 클러스터패스워드와 동일해야 함. // Failover 테스트를 위한 일반 DB유저 생성 create user 'smyoo'@'' identified by 'smyoo'; grant select, insert, update, delete on *.* to 'smyoo'@'';
  19. 19. $ ./mysqlrouter --bootstrap // DB로부터 클러스터 메타 정보를 읽어서 mysqlrouter.conf 파일을 생성 Please enter the administrative MASTER key for the MySQL InnoDB cluster: // 클러스터 패스워드 입력 MySQL Router has now been configured for the InnoDB cluster 'test'. The following connection information can be used to connect to the cluster. Classic MySQL protocol connections to cluster 'test': - Read/Write Connections: localhost:6446 - Read/Only Connections: localhost:6447 $ $ nohup ./mysqlrouter & // 백그라운드로 기동 [1] 43986 $ nohup: ignoring input and appending output to `nohup.out' $ $ ps -ef | grep mysqlrouter root 43986 8680 0 15:04 pts/0 00:00:00 ./mysqlrouter root 44050 8680 0 15:04 pts/0 00:00:00 grep mysqlrouter $
  20. 20. // 자동으로 생성된 mysqlrouter.conf 파일 내용 [DEFAULT] plugin_folder=/root/programs/mysql-innodb-cluster-labs201609-linux-glibc2.5-x86_64/mysql-router-2.1.0-labs-linux-glibc2.12-x86- 64bit/lib/mysqlrouter # logging_folder=/root/programs/mysql-innodb-cluster-labs201609-linux-glibc2.5-x86_64/mysql-router-2.1.0-labs-linux-glibc2.12-x86- 64bit/bin/../ [logger] level = INFO [metadata_cache] bootstrap_server_addresses=mysql://192.168.237:3310,mysql://,mysql:// user=mysql_innodb_cluster_reader password=,wY,JbZ9Saxf%}(S metadata_cluster=test ttl=300 metadata_replicaset=default [routing:default_rw] bind_port=6446 destinations=metadata-cache:///default?role=PRIMARY mode=read-write [routing:default_ro] bind_port=6447 destinations=metadata-cache:///default?role=SECONDARY mode=read-only
  21. 21. $ mysqlsh --uri smyoo@localhost:6446 // MySQL Router 로 PRIMARY 인스턴스로 접속 mysql-js> sql // SQL 모드로 변경 Switching to SQL mode... Commands end with ; mysql-sql> mysql-sql> select @@hostname; // PRIMARY 인스턴스 접속 확인 +------------+ | @@hostname | +------------+ | mytest1 | +------------+ 1 row in set (0.00 sec) mysql-sql> mysql-sql> select @@hostname; ERROR: 2013 (HY000): Lost connection to MySQL server during query // PRIMARY 인스턴스 DOWN 감지 (약 1초 정도 딜레이) The global session got disconnected. // 접속 실패한 세션은 그냥 사라짐. 자동으로 재 접속 하지 않는다. Attempting to reconnect to 'smyoo@localhost:6446'... The global session was successfully reconnected. mysql-sql> mysql-sql> select @@hostname; // Failover 된 것 확인 +------------+ | @@hostname | +------------+ | mytest2 | +------------+ 1 row in set (0.00 sec)
  22. 22. MYSQL> SELECT * FROM PERFORMANCE_SCHEMA.REPLICATION_GROUP_MEMBERS ORDER BY MEMBER_HOST; +---------------------------+--------------------------------------+-------------+-------------+--------------+ | CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | +---------------------------+--------------------------------------+-------------+-------------+--------------+ | group_replication_applier | dae27541-b2e5-11e6-a22c-ac162d76e1c4 | mytest1 | 3310 | ONLINE | | group_replication_applier | deeb5d42-b2e5-11e6-9ef5-002655e4a549 | mytest2 | 3310 | ONLINE | | group_replication_applier | e2449c17-b2e5-11e6-b72a-002655e4a51d | mytest3 | 3310 | ONLINE | +---------------------------+--------------------------------------+-------------+-------------+--------------+ 3 rows in set (0.00 sec) MYSQL> SELECT * FROM PERFORMANCE_SCHEMA.REPLICATION_GROUP_MEMBER_STATSG; *************************** 1. row *************************** CHANNEL_NAME: group_replication_applier VIEW_ID: 14800615308848506:7 MEMBER_ID: e2449c17-b2e5-11e6-b72a-002655e4a51d COUNT_TRANSACTIONS_IN_QUEUE: 0  큐에 누적된 트랜잭션 수. (동기화 지연 정도) 0을 유지하는 것이 바람직. COUNT_TRANSACTIONS_CHECKED: 1217439 COUNT_CONFLICTS_DETECTED: 0 COUNT_TRANSACTIONS_ROWS_VALIDATING: 0 TRANSACTIONS_COMMITTED_ALL_MEMBERS: dcd7ae4b-b2e6-11e6-b4a5-002655e4a51d:1-1217444, e2449c17-b2e5-11e6-b72a-002655e4a51d:1-22 LAST_CONFLICT_FREE_TRANSACTION: dcd7ae4b-b2e6-11e6-b4a5-002655e4a51d:1217444 1 row in set (0.00 sec) ERROR: No query specified
  23. 23. (with Grafana & InfluxDB)
  24. 24. 1번 노드가 빠진 상태에서 다시 추가하기 전 큰 테 이블(1천만건) 미리 생성해 두고 sysbench트래픽이 유입되는 상태에서 노드 추가. 노드 추가 시 발생하는 자동 분산 복구 시 “아직은” 제대로 동작 안함. DML 트래픽을 처리 못하고 트랜잭션 큐 지연 발생. RECOVERING 상태에서 더 이상 진행이 안됨.
  25. 25.  요약 완전히 다른 동기화 메커니즘 구축 용이, 편리한 운영, 유연한 확장 (을 기대 합니다)  현재상황 기본적인 기능은 무난하게 동작함. (Single PRIMARY에서의 동기화 처리 속도, Failover) Remote 구성이 “아직은” 매끄럽지 않음. mysqlsh은 조금 익숙해지면 편리함. 온라인 자동 복구는 “아직은” 제대로 동작 안 함.  제약사항 InnoDB만 된다. ROW 포멧 GTID 필수