1
MaxScale 운영 매뉴얼
㈜네오클로바
DB컨설팅
2
개정이력
버전 변경일 변경사유1
변경내용2
작성자 승인자
Ver. 1.0 2019.12.12 최초작성 문서 최초 작성 김경연
Ver 1.1 2021.1.16 버전추가 2.5의 협업모니터링 추가 김경연
1
변경 사유: 변경 내용이 이전 문서에 대해 최초작성/승인/추가/수정/삭제 중 선택 기입
2
변경 내용: 변경이 발생되는 위치와 변경 내용을 자세히 기록(장.절과 변경 내용을 기술한다.)
3
목차
1. 구성..............................................................................................................................................................................................5
1) 구성도................................................................................................................................................................................5
2) 구성요소...........................................................................................................................................................................6
3) GTID 사용시 옵션 ........................................................................................................................................................6
2. MariaDB Server Replication...............................................................................................................................................7
1) 계정생성...........................................................................................................................................................................7
2) /etc/my.cnf설정..............................................................................................................................................................7
3) Replication 구성............................................................................................................................................................8
1) 다운로드...........................................................................................................................................................................9
2) 설치/구성 .........................................................................................................................................................................9
2-1) /etc/maxscale.cnf.......................................................................................................................................... 10
2-2) DB User 추가................................................................................................................................................. 10
2-3) Maxscale 주요 Command....................................................................................................................... 11
3. AUTO_FAILOVER/AUTO_REJOIN with MaxScale.................................................................................................... 12
1) 작동방식........................................................................................................................................................................ 12
2) 주요 파라미터들 ( in Monitor Module)......................................................................................................... 12
3) 주의사항........................................................................................................................................................................ 12
4) Service auto_failover / auto_rejoin 테스트.................................................................................................... 13
4-1) auto_failover................................................................................................................................................... 13
4-2) auto_rejoin시 오류 ...................................................................................................................................... 13
4. MaxScale의 High Availability......................................................................................................................................... 14
1) Keepalived 활용 ......................................................................................................................................................... 14
2) Keepalived 설치 ......................................................................................................................................................... 15
4
2-1) Configuration File ........................................................................................................................................ 15
2-3) Maxscale Mode 통지 스크립트 ............................................................................................................ 17
2-4) Keepalived stop 통지 스크립트............................................................................................................ 17
3) MaxScale + Keepalived Failover Test................................................................................................................ 18
3-1) MAX1노드 maxscale stop........................................................................................................................ 18
3-2) MAX1노드 keepalived stop..................................................................................................................... 19
3-3) MAX2노드 maxscale stop........................................................................................................................ 20
3-4) MAX2노드 keepalived stop..................................................................................................................... 21
5
1. 구성
본 내용은 MariaDB 에 대한 Master/Slave 서비스구성에 대하여 Maxscale 을 통해서
- Read/Write Connection 의 Load Balancing 구성
- Maxscale 을 통한 Auto Failover/Switchover / Rejoin 구성
- Keepalived 를 통한 Maxscale 의 고가용성 구성
을 목적으로 합니다.
1) 구성도
Client
Master
MariaDB
Active
Maxscale
VIP
Passive
Maxscale
Slave
MariaDB
keepalived
status broadcast
Semi-Sync
Replication
base GTID
Database Query
R/W
R/W
R/O
R/O
Monitoring
Failover Management
Monitoring
6
2) 구성요소
HOST 구성요소 설명
DB1 MariaDB Server MariaDB GTID를 기반으로
Semi-Sync Replication Set을 구성합니다.
DB2 MariaDB Server
MAX1
MAX2
Maxscale auto_failover
auto_rejoin
Keepalived maxscale check script
mode notify script
notify_stop script
주의) Maxscale의 auto_failover / auto_rejoin등은 MariaDB기반의 GTID만 지원합니다.
MySQL GTID는 지원하지 않음으로 Connection Load Balancing만 지원합니다.
CF) Maxscale의 MySQL GTID HA기능 지원여부는 아래 URL에서 참고하시기 바랍니다.
https://jira.mariadb.org/browse/MXS-2814
현재는 2.6 버전에 반영할 계획으로 보입니다.
3) MariaDB GTID 사용시 옵션
- CHANGE MASTER TO ~,MASTER_USE_GTID=(current_pos / slave_pos)
+ current_pos : 노드가 Failover필요한 경우. maxscale의 auto_failover=true 경우 필수.
+ slave_pos : 단순 복제구성(영구적)인 경우.
4) MaxScale 2.5 버전의 협업모니터링 기능 지원에 따른 아키텍처 개선
- 2.4 이전까지는 autofailover=true 설정을 다수의 maxscale에 설정할 경우, Master 장애에 대한
Failover 수행처리 과정의 충돌로 서비스 장애가 발생할 가능성이 존재합니다.
- 2.5 버전부터 협업모니터링(Cooperative Monitoring)기능 지원으로 Maxscale 노드간에 장애조치
에 대한 충돌이 발생하지 않는다.
https://mariadb.com/ko/resources/blog/mariadb-maxscale-2-5-cooperative-monitoring/
- 주의할 점은 Backend 노드의 MariaDB 대수와 설정값에 따라서 협업모니터링의 효과가 발생한
다는 것이다.
7
2. MariaDB Server Replication
1) 계정생성
# CREATE USER ‘repl’@’192.168.100.%’ identified by ‘repl’;
# GRANT REPLICATION SLAVE on *.* to ‘repl’@’192.168.100.%’;
2) /etc/my.cnf설정
[mysqld]
log-bin=/LOG/mysql-bin.log
max_allowed_packet=1G #slave와 동일해야 함.
binlog_format=MIXED #ROW / MIXED
event_scheduler=on
##mariadb case##
gtid_strict_mode=on
log_slave_updates=1
rpl_semi_sync_master_enabled= 1
rpl_semi_sync_slave_enabled=1
rpl_semi_sync_master_timeout=10000
rpl_semi_sync_master_wait_point=after_sync
##mysql case##
#enforce_gtid_consistency = on
#gtid_mode=on
모든노드의 GTID 를 활성화 해야 합니다.
8
3) Replication 구성
## Semi-Sync Replication Install ##
## 10.3 버전 이하에서는 plugin이을 활성화해야 하며, 10.3부터는 별도로 설치할 필요가 없다.
## MariaDB 10.3 이하버전##
# INSTALL SONAME ‘semisync_master’; # INSTALL SONAME ‘semisync_slave’;
하거나
## my.cnf 에 plugin_load_add = semisync_master;semisync_slave
#MariaDB [(none)]> show global variables like '%rpl_semi%';
+---------------------------------------+------------+
| Variable_name | Value |
+---------------------------------------+------------+
| rpl_semi_sync_master_enabled | ON |
| rpl_semi_sync_master_timeout | 10000 |
| rpl_semi_sync_master_wait_no_slave | ON |
| rpl_semi_sync_master_wait_point | AFTER_SYNC |
| rpl_semi_sync_slave_enabled | ON |
MariaDB [(MASTER)]> show global status like '%rpl_semi%';
+--------------------------------------------+-------+
| Variable_name | Value |
+--------------------------------------------+-------+
| Rpl_semi_sync_master_status | ON |
| Rpl_semi_sync_slave_status | OFF |
MariaDB [(SLAVE)]> show global status like '%rpl_semi%';
+--------------------------------------------+-------+
| Variable_name | Value |
+--------------------------------------------+-------+
| Rpl_semi_sync_master_status | ON |
| Rpl_semi_sync_slave_status | ON |
#########MariaDB ##########
# Master
MariaDB [mysql]> show master status;
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000009 | 1155273 | | |
MariaDB [mysql]> select binlog_gtid_pos('mysql-bin.000009',1155273);
| binlog_gtid_pos('mysql-bin.000009',1155273) |
+---------------------------------------------+
| 0-11-4487 |
# Slave
# set global gtid_slave_pos =’0-11-4487’; ##Master 노드의 현재 GTID값을 Slave에 설정합니다.
# change master to ~, master_use_gtid=current_pos; ##주의) auto_rejoin시 current_pos로해야 문제발생 안합니다.
# start slave;
# show slave statusG;
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Seconds_Behind_Master: 0
Using_Gtid: Current_Pos
Gtid_IO_Pos: 0-11-12
9
2. MaxScale 2.4.x 설치
MaxScale은 MariaDB 서버의 고가용성, 확장성을 제공하는 DB Proxy입니다.
리눅스기반의 운영체제들을 지원합니다. (RHEL/ CentOS/ SLES/ Debian)
Process 구분 내용
maxscale OS 계정 maxscale.maxscale (auto)
DB 계정 maxscale
Install RPM Package
conf file /etc/maxscale.cnf
logdir /var/log/maxscale/maxscale.log
/var/log/messages
keepalived OS 계정 root
Install RPM Package
conf file /etc/keepalived/keepalived.conf
log file /var/log/messages
1) 다운로드
# https://mariadb.com/downloads/#mariadb_platform-mariadb_maxscale
해당 경로에서 원하는 버전과 OS를 선택 후 rpm package를 다운로드합니다.
wget https://downloads.mariadb.com/MaxScale/2.2.21/centos/7/x86_64/maxscale-2.2.21-1.centos.7.x86_64.rpm
wget https://dlm.mariadb.com/885363/MaxScale/2.3.15/centos/7/x86_64/maxscale-2.3.15.centos.7.tar.gz
wget https://dlm.mariadb.com/879619/MaxScale/2.4.4/centos/7/x86_64/maxscale-2.4.4-1.centos.7.x86_64.rpm
# https://downloads.mariadb.com/MariaDB/mariadb_repo_setup
MariaDB Package Repository를 사용할 수도 있습니다.
3가지중에 MariaDB MaxScale Repository를 이용하세요.
# tar 설치
멀티 버전을 동일노드에서 운영하는 것이 아니라면, rpm 설치를 권장합니다.
2) 설치/구성
# yum localinstall maxscale-2.4.4-1.centos.7.x86_64.rpm
설치 시 OS에 maxscale 계정이 없을 경우 자동 생성합니다.
# 구성 전 결정사항
- Master / Slave에 대한 분산 처리 – readconnroute Module
- Master / Slave에 대한 R/W 자동분산 처리 - readwritesplit Module
- Master / Slave에 대한 auto_failover & auto_rejoin – Must use GTID Replication (gtid_strict_mode=ON)
.
10
2-1) /etc/maxscale.cnf
[maxscale]
threads=auto
#-------------------------------------------------------------------
[MaxAdmin-Listener]
type=listener
service=MaxAdmin-Service
protocol=maxscaled
socket=default
[MaxAdmin-Service]
type=service
router=cli
#-------------------------------------------------------------------
[MariaDB-Monitor]
type=monitor
module=mariadbmon
servers=server1,server2
user=maxscale
password=maxscale
monitor_interval=1000
auto_failover = true
auto_rejoin = true
replication_user=repl
replication_password=repl
#-------------------------------------------------------------------
[RW-Listener]
type=listener
service=RW-Service
protocol=MySQLClient
port=3306
address=0.0.0.0
[RW-Service]
type=service
router=readconnroute
router_options=master
servers=server1,server2
user=maxscale
password=maxscale
enable_root_user=1
#-------------------------------------------------------------------
[RO-Listener]
type=listener
service=RO-Service
protocol=MySQLClient
port=3307
address=0.0.0.0
[RO-Service]
type=service
router=readconnroute
servers=server1,server2
user=maxscale
password=maxscale
router_options=slave
enable_root_user=1
#-------------------------------------------------------------------
[server1]
type=server
address=192.168.100.11
port=3306
protocol=MariaDBBackend
[server2]
type=server
address=192.168.100.12
port=3306
protocol=MariaDBBackend
#-------------------------------------------------------------------
2-2) DB User 추가
# CREATE USER ‘maxscale’@’192.168.100.%’ IDENTIFIED BY ‘maxscale’;
# GRANT ALL ON *.* TO ‘maxscale’@’192.168.100.%’;
 monitor 섹션의 user / password 파라미터에 사용합니다.
CF) REPLICATION용 계정정보 확인필요( CHANGE MASTER TO MASTER_USER=’repl’, MASTER_PASSWORD=’repl’; )
 monitor 섹션의 replication_user / replication_password 파리미터에 사용합니다.
11
2-3) Maxscale 주요 Command
# service maxscale start / stop / status / restart
# systemctl start/stop/status/restart maxscale.service
# maxctrl show maxscale
주의) maxadmin은 maxctrl로 대체되었습니다. 향후에는 관리Utility는 maxctrl을 이용하세요.
12
3. AUTO_FAILOVER/AUTO_REJOIN with MaxScale
1) 작동방식
Maxscale은 Monitor module을 통해서 장애발생 시 자동조치를 수행할 수 있습니다.
auto_failover=true 이며 서비스구조(master/slave or galera)에 참여하는 각 노드를 주기적으로
health check를 수행하며 일정 횟수이상 통신이상이 확인되면 Slave중 한대를 New Master로 전환
하고 다른 Slave들을 New Master로 연결을 자동전환 합니다.
# cat /etc/maxscale.cnf
…
[MariaDB-Monitor]
type=monitor
module=mariadbmon
servers=server1,server2
monitor_interval=1000
auto_failover = true
auto_rejoin = true
handle_events=true
…
2) 주요 파라미터들 ( in Monitor Module)
Parameter 설명 기본값 권고값
auto_failover false true
auto_rejoin false true
switchover_on_low_disk_space Master 디스크공간 부족 시 failover 처리 여부 비활성화 비활성화
failover_timeout 장애/재결합 조치 제한시간(초) 90
switchover_timeout 전환조치 제한시간(초) 90
verify_master_failure Master 장애확인 검증 기능 활성화 여부. true true
master_failure_timeout Master 장애확인시 시간초과(초) 한계값 10 5
servers_no_promotion Candidate Master중 제외할 목록(no-master그룹?) “,”로 구분
promotion_sql_file New Master로 전환 후, New Master에서 실행할 수 있는 SQL 파일
demotion_sql_slave Old Master가 rejoin되는 경우 Old Master에서 실행할 callback 파일
handle_server_events
handle_events
mysql.event 에 대한 Object 상태제어.(SLAVE_SIDE_DISABLED ->ENABLED)
event_scheduler에 대한 제어가 아님.
on on
**2019-12-11 handle_server_evnets (in 2.3.15, 2.4.4) 테스트결과 unknown parameter로 확인.(https://jira.mariadb.org/browse/MXS-2797)
MariaDB를 통해 handle_events로 확인됨.(공식문서 오류)
3) 주의사항
maxscale의 auto_failover & auto_rejoin은 MaxScale 2.2 부터 지원됩니다.
handle_events 는 Maxscale 2.3 부터 지원됩니다. 따라서 event_scheduler를 활용하시는 서비스구
조라면 MaxScale 2.3.x 이후버전의 사용을 추천합니다.
13
4) Service auto_failover / auto_rejoin 테스트
4-1) auto_failover
tail –f /var/log/maxscale/maxscale.log
error : Monitor was unable to connect to server server2[192.168.100.11:3306] : 'Can't connect to MySQL server on '192.168.100.11' (115)'
notice : Server changed state: server1[192.168.100.11:3306]: master_down. [Master, Running] -> [Down]
warning: [mariadbmon] Master has failed. If master status does not change in 4 monitor passes, failover begins.
notice : [mariadbmon] Selecting a server to promote and replace 'server1'. Candidates are: 'server2'.
warning: [mariadbmon] Slave 'server2' has log_slave_updates disabled. It is a valid candidate but replication will break for lagging slaves
if 'server2' is promoted.
notice : [mariadbmon] Selected 'server2'.
notice : [mariadbmon] Performing automatic failover to replace failed master 'server1'.
notice : [mariadbmon] Event 'test.test_es_1' on server 'server2' set to 'ENABLE'.
notice : [mariadbmon] Failover 'server1' -> 'server2' performed.
notice : Server changed state: server2[192.168.100.12:3306]: new_master. [Slave, Running] -> [Master, Running]
정상수행
4-2) auto_rejoin시 오류
Slave> show slave statusG;
…
Last_IO_Errno: 1236
Last_IO_Error : Got fatal error 1236 from master when reading data from binary log: 'Error: connecting slave requested to start from GTID
0-12-29, which is not in the master's binlog. Since the master's binlog contains GTIDs with higher sequence numbers, it probably means
that the slave has diverged due to executing extra erroneous transactions'
Using_Gtid: Current_Pos
…
- 마스터의 binary log가 Slave에 미처 복제되기전에 Slave가 신규 마스터로 승격되는 경우, 발생하는 문제.
- async replication구성에서 innodb_flush_log_at_trx_commit= 1, sync_binlog= 1 에서도 동일증상 재현됨.
- 이 문제처럼 Master/Slave간 switch_over나 re_join에 문제가 발생하지 않으려면 Semi-Sync Replication을 구성해야 함.
semi-sync 구성후 정상수행.
14
4. MaxScale의 High Availability
MaxScale은 BSL v2.0 License정책을 따르면서 사용 노드수에 대한 제한이 없어졌습니다. 따라서
다수의 MaxScale을 운영해도 됩니다.
단일노드의 MaxScale 이중화 하는 방법은 가장 간단하게 keepalived를 활용하는 방식이 있습니다.
1) Keepalived 활용
keepalived는 vrrp 프로토콜을 이용하여 노드간 장애를 감지/검출합니다. 또한 VIP를 할당/해제할
수 있으며, 이를 통해서 운영중인 노드에 문제가 생겼을때 예비노드로 VIP를 전환해서 서비스의
연속성을 제공합니다.
MAX1 : 192.168.100.11
MAX2 : 192.168.100.12
VIP : 192.168.100.10
Keepalived version: 2.0.7
15
2) Keepalived 설치
설치가능한 버전을 선택합니다.
# wget https://rpmfind.net/linux/centos/7.7.1908/os/x86_64/Packages/keepalived-1.3.5-16.el7.x86_64.rpm
# yum localinstall keepalived-1.3.5-16.el7.x86_64.rpm
# wget http://www.nosuchhost.net/~cheese/fedora/packages/epel-7/x86_64//keepalived-2.0.7-1.el7.x86_64.rpm
# yum localinstall keepalived-2.0.7-1.el7.x86_64.rpm
2-1) Configuration File
keepalived는 root계정으로 기동 및 내부 callback script를 수행하는 것이 좋습니다.
MAX1~2 모두 동일한 conf를 사용합니다.(다만 priority값만 다릅니다.)
관리 편의성을 위해서 keepalived의 callback script들 위치는 “/usr/libexec/keepalived/” 로 합니다.
# cat /etc/keepalived/keealived.conf
global_defs {
script_user root
enable_script_security
}
vrrp_script chk_maxscale {
script "/usr/libexec/keepalived/maxscale_running_check.sh"
interval 2 # check every 2 seconds
fall 2 # require 2 failures for KO
rise 2 # require 2 successes for OK
}
vrrp_instance VRRP_1 {
state MASTER
interface enp0s3
virtual_router_id 10
priority 150
advert_int 1
authentication {
auth_type PASS
auth_pass mypass
}
virtual_ipaddress {
192.168.100.10
}
track_script {
chk_maxscale
}
notify /usr/libexec/keepalived/maxscale_mode_check.sh
notify_stop /usr/libexec/keepalived/notify_stop.sh
}
16
2-2) Maxscale 상태 체크 스크립트
keepalived에서 maxscale의 기동상태를 체크하는 스크립트가 필요합니다.
keepalived는 자체 health check를 통해서 한노드는 MASTER, 다른노드는 BACKUP노드의 상태가
됩니다.
[root@localhost keepalived]# ll /usr/libexec/keepalived/
-rwxr-xr-x. 1 root root 704 11월 9 17:39 maxscale_mode_check.sh
-rwxr-xr-x. 1 root root 530 11월 9 20:30 maxscale_running_check.sh
# cat /usr/libexec/keepalived/maxscale_running_check.sh
#!/bin/bash
fileName="/usr/local/maxadmin_output.log"
rm $fileName
timeout 2s maxadmin list servers > $fileName
to_result=$?
if [ $to_result -ge 1 ]
then
echo Timed out or error, timeout returned $to_result
exit 3
else
echo MaxAdmin success, rval is $to_result
echo Checking maxadmin output sanity
grep1=$(grep server1 $fileName)
grep2=$(grep server2 $fileName)
if [ "$grep1" ] && [ "$grep2" ]
then
echo All is fine
exit 0
else
echo Something is wrong
service keepalived stop
exit 3
fi
fi
17
2-3) Maxscale Mode 통지 스크립트
이 state정보를 제어하여 maxscale에서 auto_failover기능을 한쪽 maxscale에서만 처리해야 한다.
그렇지 않으면 두대의 maxscale의 auto_failover=ON인경우 충돌이 발생합니다.
이를 위해서 maxscale의 passive state를 제어하는 스크립트가 필요합니다.
[root@localhost keepalived]# ll /usr/libexec/keepalived/
-rwxr-xr-x. 1 root root 704 11월 9 17:39 maxscale_mode_check.sh
-rwxr-xr-x. 1 root root 530 11월 9 20:30 maxscale_running_check.sh
# cat /usr/libexec/keepalived/maxscale_mode_check.sh
#!/bin/bash
TYPE=$1
NAME=$2
STATE=$3
OUTFILE=/usr/local/maxscale_state.log
case $STATE in
"MASTER") echo "Setting this MaxScale node to active mode" > $OUTFILE
maxctrl alter maxscale passive false
exit 0
;;
"BACKUP") echo "Setting this MaxScale node to passive mode" > $OUTFILE
maxctrl alter maxscale passive true
exit 0
;;
"FAULT") echo "MaxScale failed the status check." > $OUTFILE
maxctrl alter maxscale passive true
exit 0
;;
*) echo "Unknown state" > $OUTFILE
exit 1
;;
esac
2-4) Keepalived stop 통지 스크립트
keepalived가 stop되면 BACKUP모드 중 하나로 VIP가 전환되고, 해당 노드의 maxscale의 passive
state를 true에서 false로 변경시킵니다. 그러나 keepalived가 stop된 노드의 maxscale의 passive
state가 false로 남아있어서 문제가 됩니다. 이를 위해서 notify_stop script를 이용합니다.
[root@localhost keepalived]# ll /usr/libexec/keepalived/
-rwxr-xr-x. 1 root root 704 11월 9 17:39 maxscale_mode_check.sh
-rwxr-xr-x. 1 root root 530 11월 9 20:30 maxscale_running_check.sh
# cat /usr/libexec/keepalived/notify_stop.sh
#!/bin/bash
maxctrl alter maxscale passive true
18
3) MaxScale + Keepalived Failover Test
3-1) MAX1노드 maxscale stop
MAX1 MAX2
[root@MAX1 keepalived]# maxctrl show maxscale | grep passive
"passive": false,
[root@MAX2 keepalived]# maxctrl show maxscale | grep passive
"passive": true,
[root@MAX1 keepalived]# ip addr
…
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:8d:12:9d brd ff:ff:ff:ff:ff:ff
inet 192.168.100.13/24 brd 192.168.100.255 scope global
noprefixroute enp0s3
valid_lft forever preferred_lft forever
inet 192.168.100.10/32 scope global enp0s3
valid_lft forever preferred_lft forever
[root@MAX2 local]# ip addr
…
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:28:54:b3 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.14/24 brd 192.168.100.255 scope global
noprefixroute enp0s3
valid_lft forever preferred_lft forever
[root@localhost keepalived]# service keepalived status | grep Active
Active: active (running) since 수 2019-12-11 22:30:16 KST; 26min
ago
[root@localhost local]# service keepalived status | grep Active
Active: active (running) since 수 2019-12-11 22:57:03 KST; 41s ago
[root@MAX1 keepalived]# service maxscale stop
Redirecting to /bin/systemctl stop maxscale.service
[root@localhost keepalived]# service maxscale status | grep Active
Redirecting to /bin/systemctl status maxscale.service
Active: inactive (dead)
[root@localhost keepalived]# ip addr
…
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:8d:12:9d brd ff:ff:ff:ff:ff:ff
inet 192.168.100.13/24 brd 192.168.100.255 scope global
noprefixroute enp0s3
valid_lft forever preferred_lft forever
[root@localhost keepalived]# ip addr
…
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:28:54:b3 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.14/24 brd 192.168.100.255 scope global
noprefixroute enp0s3
valid_lft forever preferred_lft forever
inet 192.168.100.10/32 scope global enp0s3
valid_lft forever preferred_lft forever
[root@MAX1 keepalived]# maxctrl show maxscale | grep passive
[root@MAX1 keepalived]#
[root@MAX2 keepalived]# maxctrl show maxscale | grep passive
"passive": false,
# 정리
- VIP 전환 : MAX1  MAX2 전환 OK.
- Maxscale MAX1 : 중지 OK
- Maxscale MAX2 : (passive:true)  (passive:false) OK
Service OK.
19
3-2) MAX1노드 keepalived stop
MAX1 MAX2
[root@MAX1 keepalived]# maxctrl show maxscale | grep passive
"passive": false,
[root@MAX2 keepalived]# maxctrl show maxscale | grep passive
"passive": true,
[root@MAX1 keepalived]# ip addr
…
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:8d:12:9d brd ff:ff:ff:ff:ff:ff
inet 192.168.100.13/24 brd 192.168.100.255 scope global
noprefixroute enp0s3
valid_lft forever preferred_lft forever
inet 192.168.100.10/32 scope global enp0s3
valid_lft forever preferred_lft forever
[root@MAX2 local]# ip addr
…
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:28:54:b3 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.14/24 brd 192.168.100.255 scope global
noprefixroute enp0s3
valid_lft forever preferred_lft forever
[root@localhost keepalived]# service keepalived status |grep Active
Redirecting to /bin/systemctl status keepalived.service
Active: active (running) since 수 2019-12-11 22:30:16 KST;
33min ago
[root@localhost local]# service keepalived status |grep Active
Redirecting to /bin/systemctl status keepalived.service
Active: active (running) since 수 2019-12-11 22:57:03 KST; 7min
ago
[root@MAX1 keepalived]# service keepalived stop
Redirecting to /bin/systemctl stop keepalived.service
[root@MAX1 keepalived]# ps -ef |grep keepalived
[root@MAX1 keepalived]#
[root@localhost keepalived]# ip addr
…
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:8d:12:9d brd ff:ff:ff:ff:ff:ff
inet 192.168.100.13/24 brd 192.168.100.255 scope global
noprefixroute enp0s3
valid_lft forever preferred_lft forever
[root@localhost keepalived]# ip addr
…
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:28:54:b3 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.14/24 brd 192.168.100.255 scope global
noprefixroute enp0s3
valid_lft forever preferred_lft forever
inet 192.168.100.10/32 scope global enp0s3
valid_lft forever preferred_lft forever
[root@MAX1 keepalived]# maxctrl show maxscale | grep passive
"passive": true,
[root@MAX2 keepalived]# maxctrl show maxscale | grep passive
"passive": false,
# 정리
- VIP 전환 : MAX1  MAX2 전환 OK.
- Maxscale MAX1 : (passive:false)  (passive:true)
- Maxscale MAX2 : (passive:true)  (passive:false)
Service OK.
20
3-3) MAX2노드 maxscale stop
MAX1 MAX2
[root@MAX1 keepalived]# maxctrl show maxscale | grep passive
"passive": false,
[root@MAX2 keepalived]# maxctrl show maxscale | grep passive
"passive": true,
[root@MAX1 keepalived]# ip addr
…
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:8d:12:9d brd ff:ff:ff:ff:ff:ff
inet 192.168.100.13/24 brd 192.168.100.255 scope global
noprefixroute enp0s3
valid_lft forever preferred_lft forever
inet 192.168.100.10/32 scope global enp0s3
valid_lft forever preferred_lft forever
[root@MAX2 local]# ip addr
…
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:28:54:b3 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.14/24 brd 192.168.100.255 scope global
noprefixroute enp0s3
valid_lft forever preferred_lft forever
[root@localhost keepalived]# service keepalived status | grep Active
Active: active (running) since 수 2019-12-11 22:30:16 KST; 26min
ago
[root@localhost local]# service keepalived status | grep Active
Active: active (running) since 수 2019-12-11 22:57:03 KST; 41s ago
[root@MAX2 local]# service maxscale stop
Redirecting to /bin/systemctl stop maxscale.service
[root@MAX2 local]# service maxscale status | grep Active
Redirecting to /bin/systemctl status maxscale.service
Active: inactive (dead)
[root@localhost keepalived]# ip addr
…
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:8d:12:9d brd ff:ff:ff:ff:ff:ff
inet 192.168.100.13/24 brd 192.168.100.255 scope global
noprefixroute enp0s3
valid_lft forever preferred_lft forever
inet 192.168.100.10/32 scope global enp0s3
valid_lft forever preferred_lft forever
[root@localhost keepalived]# ip addr
…
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:28:54:b3 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.14/24 brd 192.168.100.255 scope global
noprefixroute enp0s3
valid_lft forever preferred_lft forever
[root@MAX1 keepalived]# maxctrl show maxscale | grep passive
"passive": false,
[root@MAX2 keepalived]# maxctrl show maxscale | grep passive
[root@MAX2 keepalived]#
# 정리
- VIP 전환 : MAX1 현상유지 OK.
- Maxscale MAX1 : (passive:false) 현상유지 OK
- Maxscale MAX2 : 중지 OK
Service OK.
21
3-4) MAX2노드 keepalived stop
MAX1 MAX2
[root@MAX1 keepalived]# maxctrl show maxscale | grep passive
"passive": false,
[root@MAX2 keepalived]# maxctrl show maxscale | grep passive
"passive": true,
[root@MAX1 keepalived]# ip addr
…
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:8d:12:9d brd ff:ff:ff:ff:ff:ff
inet 192.168.100.13/24 brd 192.168.100.255 scope global
noprefixroute enp0s3
valid_lft forever preferred_lft forever
inet 192.168.100.10/32 scope global enp0s3
valid_lft forever preferred_lft forever
[root@MAX2 local]# ip addr
…
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:28:54:b3 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.14/24 brd 192.168.100.255 scope global
noprefixroute enp0s3
valid_lft forever preferred_lft forever
[root@localhost keepalived]# service keepalived status |grep Active
Redirecting to /bin/systemctl status keepalived.service
Active: active (running) since 수 2019-12-11 22:30:16 KST;
33min ago
[root@localhost local]# service keepalived status |grep Active
Redirecting to /bin/systemctl status keepalived.service
Active: active (running) since 수 2019-12-11 22:57:03 KST; 7min
ago
[root@MAX2 keepalived]# service keepalived stop
Redirecting to /bin/systemctl stop keepalived.service
[root@MAX2 keepalived]# ps -ef |grep keepalived
[root@MAX2 keepalived]#
[root@localhost keepalived]# ip addr
…
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:8d:12:9d brd ff:ff:ff:ff:ff:ff
inet 192.168.100.13/24 brd 192.168.100.255 scope global
noprefixroute enp0s3
valid_lft forever preferred_lft forever
inet 192.168.100.10/32 scope global enp0s3
valid_lft forever preferred_lft forever
[root@localhost keepalived]# ip addr
…
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:28:54:b3 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.14/24 brd 192.168.100.255 scope global
noprefixroute enp0s3
valid_lft forever preferred_lft forever
[root@MAX1 keepalived]# maxctrl show maxscale | grep passive
"passive": false,
[root@MAX2 keepalived]# maxctrl show maxscale | grep passive
"passive": true,
# 정리
- VIP 전환 : MAX1 현상유지 OK.
- Maxscale MAX1 : (passive:false) 현상유지 OK
- Maxscale MAX2 : (passive:true) 현상유지 OK
Service OK.

Maxscale_메뉴얼

  • 1.
  • 2.
    2 개정이력 버전 변경일 변경사유1 변경내용2 작성자승인자 Ver. 1.0 2019.12.12 최초작성 문서 최초 작성 김경연 Ver 1.1 2021.1.16 버전추가 2.5의 협업모니터링 추가 김경연 1 변경 사유: 변경 내용이 이전 문서에 대해 최초작성/승인/추가/수정/삭제 중 선택 기입 2 변경 내용: 변경이 발생되는 위치와 변경 내용을 자세히 기록(장.절과 변경 내용을 기술한다.)
  • 3.
    3 목차 1. 구성..............................................................................................................................................................................................5 1) 구성도................................................................................................................................................................................5 2)구성요소...........................................................................................................................................................................6 3) GTID 사용시 옵션 ........................................................................................................................................................6 2. MariaDB Server Replication...............................................................................................................................................7 1) 계정생성...........................................................................................................................................................................7 2) /etc/my.cnf설정..............................................................................................................................................................7 3) Replication 구성............................................................................................................................................................8 1) 다운로드...........................................................................................................................................................................9 2) 설치/구성 .........................................................................................................................................................................9 2-1) /etc/maxscale.cnf.......................................................................................................................................... 10 2-2) DB User 추가................................................................................................................................................. 10 2-3) Maxscale 주요 Command....................................................................................................................... 11 3. AUTO_FAILOVER/AUTO_REJOIN with MaxScale.................................................................................................... 12 1) 작동방식........................................................................................................................................................................ 12 2) 주요 파라미터들 ( in Monitor Module)......................................................................................................... 12 3) 주의사항........................................................................................................................................................................ 12 4) Service auto_failover / auto_rejoin 테스트.................................................................................................... 13 4-1) auto_failover................................................................................................................................................... 13 4-2) auto_rejoin시 오류 ...................................................................................................................................... 13 4. MaxScale의 High Availability......................................................................................................................................... 14 1) Keepalived 활용 ......................................................................................................................................................... 14 2) Keepalived 설치 ......................................................................................................................................................... 15
  • 4.
    4 2-1) Configuration File........................................................................................................................................ 15 2-3) Maxscale Mode 통지 스크립트 ............................................................................................................ 17 2-4) Keepalived stop 통지 스크립트............................................................................................................ 17 3) MaxScale + Keepalived Failover Test................................................................................................................ 18 3-1) MAX1노드 maxscale stop........................................................................................................................ 18 3-2) MAX1노드 keepalived stop..................................................................................................................... 19 3-3) MAX2노드 maxscale stop........................................................................................................................ 20 3-4) MAX2노드 keepalived stop..................................................................................................................... 21
  • 5.
    5 1. 구성 본 내용은MariaDB 에 대한 Master/Slave 서비스구성에 대하여 Maxscale 을 통해서 - Read/Write Connection 의 Load Balancing 구성 - Maxscale 을 통한 Auto Failover/Switchover / Rejoin 구성 - Keepalived 를 통한 Maxscale 의 고가용성 구성 을 목적으로 합니다. 1) 구성도 Client Master MariaDB Active Maxscale VIP Passive Maxscale Slave MariaDB keepalived status broadcast Semi-Sync Replication base GTID Database Query R/W R/W R/O R/O Monitoring Failover Management Monitoring
  • 6.
    6 2) 구성요소 HOST 구성요소설명 DB1 MariaDB Server MariaDB GTID를 기반으로 Semi-Sync Replication Set을 구성합니다. DB2 MariaDB Server MAX1 MAX2 Maxscale auto_failover auto_rejoin Keepalived maxscale check script mode notify script notify_stop script 주의) Maxscale의 auto_failover / auto_rejoin등은 MariaDB기반의 GTID만 지원합니다. MySQL GTID는 지원하지 않음으로 Connection Load Balancing만 지원합니다. CF) Maxscale의 MySQL GTID HA기능 지원여부는 아래 URL에서 참고하시기 바랍니다. https://jira.mariadb.org/browse/MXS-2814 현재는 2.6 버전에 반영할 계획으로 보입니다. 3) MariaDB GTID 사용시 옵션 - CHANGE MASTER TO ~,MASTER_USE_GTID=(current_pos / slave_pos) + current_pos : 노드가 Failover필요한 경우. maxscale의 auto_failover=true 경우 필수. + slave_pos : 단순 복제구성(영구적)인 경우. 4) MaxScale 2.5 버전의 협업모니터링 기능 지원에 따른 아키텍처 개선 - 2.4 이전까지는 autofailover=true 설정을 다수의 maxscale에 설정할 경우, Master 장애에 대한 Failover 수행처리 과정의 충돌로 서비스 장애가 발생할 가능성이 존재합니다. - 2.5 버전부터 협업모니터링(Cooperative Monitoring)기능 지원으로 Maxscale 노드간에 장애조치 에 대한 충돌이 발생하지 않는다. https://mariadb.com/ko/resources/blog/mariadb-maxscale-2-5-cooperative-monitoring/ - 주의할 점은 Backend 노드의 MariaDB 대수와 설정값에 따라서 협업모니터링의 효과가 발생한 다는 것이다.
  • 7.
    7 2. MariaDB ServerReplication 1) 계정생성 # CREATE USER ‘repl’@’192.168.100.%’ identified by ‘repl’; # GRANT REPLICATION SLAVE on *.* to ‘repl’@’192.168.100.%’; 2) /etc/my.cnf설정 [mysqld] log-bin=/LOG/mysql-bin.log max_allowed_packet=1G #slave와 동일해야 함. binlog_format=MIXED #ROW / MIXED event_scheduler=on ##mariadb case## gtid_strict_mode=on log_slave_updates=1 rpl_semi_sync_master_enabled= 1 rpl_semi_sync_slave_enabled=1 rpl_semi_sync_master_timeout=10000 rpl_semi_sync_master_wait_point=after_sync ##mysql case## #enforce_gtid_consistency = on #gtid_mode=on 모든노드의 GTID 를 활성화 해야 합니다.
  • 8.
    8 3) Replication 구성 ##Semi-Sync Replication Install ## ## 10.3 버전 이하에서는 plugin이을 활성화해야 하며, 10.3부터는 별도로 설치할 필요가 없다. ## MariaDB 10.3 이하버전## # INSTALL SONAME ‘semisync_master’; # INSTALL SONAME ‘semisync_slave’; 하거나 ## my.cnf 에 plugin_load_add = semisync_master;semisync_slave #MariaDB [(none)]> show global variables like '%rpl_semi%'; +---------------------------------------+------------+ | Variable_name | Value | +---------------------------------------+------------+ | rpl_semi_sync_master_enabled | ON | | rpl_semi_sync_master_timeout | 10000 | | rpl_semi_sync_master_wait_no_slave | ON | | rpl_semi_sync_master_wait_point | AFTER_SYNC | | rpl_semi_sync_slave_enabled | ON | MariaDB [(MASTER)]> show global status like '%rpl_semi%'; +--------------------------------------------+-------+ | Variable_name | Value | +--------------------------------------------+-------+ | Rpl_semi_sync_master_status | ON | | Rpl_semi_sync_slave_status | OFF | MariaDB [(SLAVE)]> show global status like '%rpl_semi%'; +--------------------------------------------+-------+ | Variable_name | Value | +--------------------------------------------+-------+ | Rpl_semi_sync_master_status | ON | | Rpl_semi_sync_slave_status | ON | #########MariaDB ########## # Master MariaDB [mysql]> show master status; | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | +------------------+----------+--------------+------------------+ | mysql-bin.000009 | 1155273 | | | MariaDB [mysql]> select binlog_gtid_pos('mysql-bin.000009',1155273); | binlog_gtid_pos('mysql-bin.000009',1155273) | +---------------------------------------------+ | 0-11-4487 | # Slave # set global gtid_slave_pos =’0-11-4487’; ##Master 노드의 현재 GTID값을 Slave에 설정합니다. # change master to ~, master_use_gtid=current_pos; ##주의) auto_rejoin시 current_pos로해야 문제발생 안합니다. # start slave; # show slave statusG; Slave_IO_Running: Yes Slave_SQL_Running: Yes Seconds_Behind_Master: 0 Using_Gtid: Current_Pos Gtid_IO_Pos: 0-11-12
  • 9.
    9 2. MaxScale 2.4.x설치 MaxScale은 MariaDB 서버의 고가용성, 확장성을 제공하는 DB Proxy입니다. 리눅스기반의 운영체제들을 지원합니다. (RHEL/ CentOS/ SLES/ Debian) Process 구분 내용 maxscale OS 계정 maxscale.maxscale (auto) DB 계정 maxscale Install RPM Package conf file /etc/maxscale.cnf logdir /var/log/maxscale/maxscale.log /var/log/messages keepalived OS 계정 root Install RPM Package conf file /etc/keepalived/keepalived.conf log file /var/log/messages 1) 다운로드 # https://mariadb.com/downloads/#mariadb_platform-mariadb_maxscale 해당 경로에서 원하는 버전과 OS를 선택 후 rpm package를 다운로드합니다. wget https://downloads.mariadb.com/MaxScale/2.2.21/centos/7/x86_64/maxscale-2.2.21-1.centos.7.x86_64.rpm wget https://dlm.mariadb.com/885363/MaxScale/2.3.15/centos/7/x86_64/maxscale-2.3.15.centos.7.tar.gz wget https://dlm.mariadb.com/879619/MaxScale/2.4.4/centos/7/x86_64/maxscale-2.4.4-1.centos.7.x86_64.rpm # https://downloads.mariadb.com/MariaDB/mariadb_repo_setup MariaDB Package Repository를 사용할 수도 있습니다. 3가지중에 MariaDB MaxScale Repository를 이용하세요. # tar 설치 멀티 버전을 동일노드에서 운영하는 것이 아니라면, rpm 설치를 권장합니다. 2) 설치/구성 # yum localinstall maxscale-2.4.4-1.centos.7.x86_64.rpm 설치 시 OS에 maxscale 계정이 없을 경우 자동 생성합니다. # 구성 전 결정사항 - Master / Slave에 대한 분산 처리 – readconnroute Module - Master / Slave에 대한 R/W 자동분산 처리 - readwritesplit Module - Master / Slave에 대한 auto_failover & auto_rejoin – Must use GTID Replication (gtid_strict_mode=ON) .
  • 10.
    10 2-1) /etc/maxscale.cnf [maxscale] threads=auto #------------------------------------------------------------------- [MaxAdmin-Listener] type=listener service=MaxAdmin-Service protocol=maxscaled socket=default [MaxAdmin-Service] type=service router=cli #------------------------------------------------------------------- [MariaDB-Monitor] type=monitor module=mariadbmon servers=server1,server2 user=maxscale password=maxscale monitor_interval=1000 auto_failover =true auto_rejoin = true replication_user=repl replication_password=repl #------------------------------------------------------------------- [RW-Listener] type=listener service=RW-Service protocol=MySQLClient port=3306 address=0.0.0.0 [RW-Service] type=service router=readconnroute router_options=master servers=server1,server2 user=maxscale password=maxscale enable_root_user=1 #------------------------------------------------------------------- [RO-Listener] type=listener service=RO-Service protocol=MySQLClient port=3307 address=0.0.0.0 [RO-Service] type=service router=readconnroute servers=server1,server2 user=maxscale password=maxscale router_options=slave enable_root_user=1 #------------------------------------------------------------------- [server1] type=server address=192.168.100.11 port=3306 protocol=MariaDBBackend [server2] type=server address=192.168.100.12 port=3306 protocol=MariaDBBackend #------------------------------------------------------------------- 2-2) DB User 추가 # CREATE USER ‘maxscale’@’192.168.100.%’ IDENTIFIED BY ‘maxscale’; # GRANT ALL ON *.* TO ‘maxscale’@’192.168.100.%’;  monitor 섹션의 user / password 파라미터에 사용합니다. CF) REPLICATION용 계정정보 확인필요( CHANGE MASTER TO MASTER_USER=’repl’, MASTER_PASSWORD=’repl’; )  monitor 섹션의 replication_user / replication_password 파리미터에 사용합니다.
  • 11.
    11 2-3) Maxscale 주요Command # service maxscale start / stop / status / restart # systemctl start/stop/status/restart maxscale.service # maxctrl show maxscale 주의) maxadmin은 maxctrl로 대체되었습니다. 향후에는 관리Utility는 maxctrl을 이용하세요.
  • 12.
    12 3. AUTO_FAILOVER/AUTO_REJOIN withMaxScale 1) 작동방식 Maxscale은 Monitor module을 통해서 장애발생 시 자동조치를 수행할 수 있습니다. auto_failover=true 이며 서비스구조(master/slave or galera)에 참여하는 각 노드를 주기적으로 health check를 수행하며 일정 횟수이상 통신이상이 확인되면 Slave중 한대를 New Master로 전환 하고 다른 Slave들을 New Master로 연결을 자동전환 합니다. # cat /etc/maxscale.cnf … [MariaDB-Monitor] type=monitor module=mariadbmon servers=server1,server2 monitor_interval=1000 auto_failover = true auto_rejoin = true handle_events=true … 2) 주요 파라미터들 ( in Monitor Module) Parameter 설명 기본값 권고값 auto_failover false true auto_rejoin false true switchover_on_low_disk_space Master 디스크공간 부족 시 failover 처리 여부 비활성화 비활성화 failover_timeout 장애/재결합 조치 제한시간(초) 90 switchover_timeout 전환조치 제한시간(초) 90 verify_master_failure Master 장애확인 검증 기능 활성화 여부. true true master_failure_timeout Master 장애확인시 시간초과(초) 한계값 10 5 servers_no_promotion Candidate Master중 제외할 목록(no-master그룹?) “,”로 구분 promotion_sql_file New Master로 전환 후, New Master에서 실행할 수 있는 SQL 파일 demotion_sql_slave Old Master가 rejoin되는 경우 Old Master에서 실행할 callback 파일 handle_server_events handle_events mysql.event 에 대한 Object 상태제어.(SLAVE_SIDE_DISABLED ->ENABLED) event_scheduler에 대한 제어가 아님. on on **2019-12-11 handle_server_evnets (in 2.3.15, 2.4.4) 테스트결과 unknown parameter로 확인.(https://jira.mariadb.org/browse/MXS-2797) MariaDB를 통해 handle_events로 확인됨.(공식문서 오류) 3) 주의사항 maxscale의 auto_failover & auto_rejoin은 MaxScale 2.2 부터 지원됩니다. handle_events 는 Maxscale 2.3 부터 지원됩니다. 따라서 event_scheduler를 활용하시는 서비스구 조라면 MaxScale 2.3.x 이후버전의 사용을 추천합니다.
  • 13.
    13 4) Service auto_failover/ auto_rejoin 테스트 4-1) auto_failover tail –f /var/log/maxscale/maxscale.log error : Monitor was unable to connect to server server2[192.168.100.11:3306] : 'Can't connect to MySQL server on '192.168.100.11' (115)' notice : Server changed state: server1[192.168.100.11:3306]: master_down. [Master, Running] -> [Down] warning: [mariadbmon] Master has failed. If master status does not change in 4 monitor passes, failover begins. notice : [mariadbmon] Selecting a server to promote and replace 'server1'. Candidates are: 'server2'. warning: [mariadbmon] Slave 'server2' has log_slave_updates disabled. It is a valid candidate but replication will break for lagging slaves if 'server2' is promoted. notice : [mariadbmon] Selected 'server2'. notice : [mariadbmon] Performing automatic failover to replace failed master 'server1'. notice : [mariadbmon] Event 'test.test_es_1' on server 'server2' set to 'ENABLE'. notice : [mariadbmon] Failover 'server1' -> 'server2' performed. notice : Server changed state: server2[192.168.100.12:3306]: new_master. [Slave, Running] -> [Master, Running] 정상수행 4-2) auto_rejoin시 오류 Slave> show slave statusG; … Last_IO_Errno: 1236 Last_IO_Error : Got fatal error 1236 from master when reading data from binary log: 'Error: connecting slave requested to start from GTID 0-12-29, which is not in the master's binlog. Since the master's binlog contains GTIDs with higher sequence numbers, it probably means that the slave has diverged due to executing extra erroneous transactions' Using_Gtid: Current_Pos … - 마스터의 binary log가 Slave에 미처 복제되기전에 Slave가 신규 마스터로 승격되는 경우, 발생하는 문제. - async replication구성에서 innodb_flush_log_at_trx_commit= 1, sync_binlog= 1 에서도 동일증상 재현됨. - 이 문제처럼 Master/Slave간 switch_over나 re_join에 문제가 발생하지 않으려면 Semi-Sync Replication을 구성해야 함. semi-sync 구성후 정상수행.
  • 14.
    14 4. MaxScale의 HighAvailability MaxScale은 BSL v2.0 License정책을 따르면서 사용 노드수에 대한 제한이 없어졌습니다. 따라서 다수의 MaxScale을 운영해도 됩니다. 단일노드의 MaxScale 이중화 하는 방법은 가장 간단하게 keepalived를 활용하는 방식이 있습니다. 1) Keepalived 활용 keepalived는 vrrp 프로토콜을 이용하여 노드간 장애를 감지/검출합니다. 또한 VIP를 할당/해제할 수 있으며, 이를 통해서 운영중인 노드에 문제가 생겼을때 예비노드로 VIP를 전환해서 서비스의 연속성을 제공합니다. MAX1 : 192.168.100.11 MAX2 : 192.168.100.12 VIP : 192.168.100.10 Keepalived version: 2.0.7
  • 15.
    15 2) Keepalived 설치 설치가능한버전을 선택합니다. # wget https://rpmfind.net/linux/centos/7.7.1908/os/x86_64/Packages/keepalived-1.3.5-16.el7.x86_64.rpm # yum localinstall keepalived-1.3.5-16.el7.x86_64.rpm # wget http://www.nosuchhost.net/~cheese/fedora/packages/epel-7/x86_64//keepalived-2.0.7-1.el7.x86_64.rpm # yum localinstall keepalived-2.0.7-1.el7.x86_64.rpm 2-1) Configuration File keepalived는 root계정으로 기동 및 내부 callback script를 수행하는 것이 좋습니다. MAX1~2 모두 동일한 conf를 사용합니다.(다만 priority값만 다릅니다.) 관리 편의성을 위해서 keepalived의 callback script들 위치는 “/usr/libexec/keepalived/” 로 합니다. # cat /etc/keepalived/keealived.conf global_defs { script_user root enable_script_security } vrrp_script chk_maxscale { script "/usr/libexec/keepalived/maxscale_running_check.sh" interval 2 # check every 2 seconds fall 2 # require 2 failures for KO rise 2 # require 2 successes for OK } vrrp_instance VRRP_1 { state MASTER interface enp0s3 virtual_router_id 10 priority 150 advert_int 1 authentication { auth_type PASS auth_pass mypass } virtual_ipaddress { 192.168.100.10 } track_script { chk_maxscale } notify /usr/libexec/keepalived/maxscale_mode_check.sh notify_stop /usr/libexec/keepalived/notify_stop.sh }
  • 16.
    16 2-2) Maxscale 상태체크 스크립트 keepalived에서 maxscale의 기동상태를 체크하는 스크립트가 필요합니다. keepalived는 자체 health check를 통해서 한노드는 MASTER, 다른노드는 BACKUP노드의 상태가 됩니다. [root@localhost keepalived]# ll /usr/libexec/keepalived/ -rwxr-xr-x. 1 root root 704 11월 9 17:39 maxscale_mode_check.sh -rwxr-xr-x. 1 root root 530 11월 9 20:30 maxscale_running_check.sh # cat /usr/libexec/keepalived/maxscale_running_check.sh #!/bin/bash fileName="/usr/local/maxadmin_output.log" rm $fileName timeout 2s maxadmin list servers > $fileName to_result=$? if [ $to_result -ge 1 ] then echo Timed out or error, timeout returned $to_result exit 3 else echo MaxAdmin success, rval is $to_result echo Checking maxadmin output sanity grep1=$(grep server1 $fileName) grep2=$(grep server2 $fileName) if [ "$grep1" ] && [ "$grep2" ] then echo All is fine exit 0 else echo Something is wrong service keepalived stop exit 3 fi fi
  • 17.
    17 2-3) Maxscale Mode통지 스크립트 이 state정보를 제어하여 maxscale에서 auto_failover기능을 한쪽 maxscale에서만 처리해야 한다. 그렇지 않으면 두대의 maxscale의 auto_failover=ON인경우 충돌이 발생합니다. 이를 위해서 maxscale의 passive state를 제어하는 스크립트가 필요합니다. [root@localhost keepalived]# ll /usr/libexec/keepalived/ -rwxr-xr-x. 1 root root 704 11월 9 17:39 maxscale_mode_check.sh -rwxr-xr-x. 1 root root 530 11월 9 20:30 maxscale_running_check.sh # cat /usr/libexec/keepalived/maxscale_mode_check.sh #!/bin/bash TYPE=$1 NAME=$2 STATE=$3 OUTFILE=/usr/local/maxscale_state.log case $STATE in "MASTER") echo "Setting this MaxScale node to active mode" > $OUTFILE maxctrl alter maxscale passive false exit 0 ;; "BACKUP") echo "Setting this MaxScale node to passive mode" > $OUTFILE maxctrl alter maxscale passive true exit 0 ;; "FAULT") echo "MaxScale failed the status check." > $OUTFILE maxctrl alter maxscale passive true exit 0 ;; *) echo "Unknown state" > $OUTFILE exit 1 ;; esac 2-4) Keepalived stop 통지 스크립트 keepalived가 stop되면 BACKUP모드 중 하나로 VIP가 전환되고, 해당 노드의 maxscale의 passive state를 true에서 false로 변경시킵니다. 그러나 keepalived가 stop된 노드의 maxscale의 passive state가 false로 남아있어서 문제가 됩니다. 이를 위해서 notify_stop script를 이용합니다. [root@localhost keepalived]# ll /usr/libexec/keepalived/ -rwxr-xr-x. 1 root root 704 11월 9 17:39 maxscale_mode_check.sh -rwxr-xr-x. 1 root root 530 11월 9 20:30 maxscale_running_check.sh # cat /usr/libexec/keepalived/notify_stop.sh #!/bin/bash maxctrl alter maxscale passive true
  • 18.
    18 3) MaxScale +Keepalived Failover Test 3-1) MAX1노드 maxscale stop MAX1 MAX2 [root@MAX1 keepalived]# maxctrl show maxscale | grep passive "passive": false, [root@MAX2 keepalived]# maxctrl show maxscale | grep passive "passive": true, [root@MAX1 keepalived]# ip addr … 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:8d:12:9d brd ff:ff:ff:ff:ff:ff inet 192.168.100.13/24 brd 192.168.100.255 scope global noprefixroute enp0s3 valid_lft forever preferred_lft forever inet 192.168.100.10/32 scope global enp0s3 valid_lft forever preferred_lft forever [root@MAX2 local]# ip addr … 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:28:54:b3 brd ff:ff:ff:ff:ff:ff inet 192.168.100.14/24 brd 192.168.100.255 scope global noprefixroute enp0s3 valid_lft forever preferred_lft forever [root@localhost keepalived]# service keepalived status | grep Active Active: active (running) since 수 2019-12-11 22:30:16 KST; 26min ago [root@localhost local]# service keepalived status | grep Active Active: active (running) since 수 2019-12-11 22:57:03 KST; 41s ago [root@MAX1 keepalived]# service maxscale stop Redirecting to /bin/systemctl stop maxscale.service [root@localhost keepalived]# service maxscale status | grep Active Redirecting to /bin/systemctl status maxscale.service Active: inactive (dead) [root@localhost keepalived]# ip addr … 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:8d:12:9d brd ff:ff:ff:ff:ff:ff inet 192.168.100.13/24 brd 192.168.100.255 scope global noprefixroute enp0s3 valid_lft forever preferred_lft forever [root@localhost keepalived]# ip addr … 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:28:54:b3 brd ff:ff:ff:ff:ff:ff inet 192.168.100.14/24 brd 192.168.100.255 scope global noprefixroute enp0s3 valid_lft forever preferred_lft forever inet 192.168.100.10/32 scope global enp0s3 valid_lft forever preferred_lft forever [root@MAX1 keepalived]# maxctrl show maxscale | grep passive [root@MAX1 keepalived]# [root@MAX2 keepalived]# maxctrl show maxscale | grep passive "passive": false, # 정리 - VIP 전환 : MAX1  MAX2 전환 OK. - Maxscale MAX1 : 중지 OK - Maxscale MAX2 : (passive:true)  (passive:false) OK Service OK.
  • 19.
    19 3-2) MAX1노드 keepalivedstop MAX1 MAX2 [root@MAX1 keepalived]# maxctrl show maxscale | grep passive "passive": false, [root@MAX2 keepalived]# maxctrl show maxscale | grep passive "passive": true, [root@MAX1 keepalived]# ip addr … 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:8d:12:9d brd ff:ff:ff:ff:ff:ff inet 192.168.100.13/24 brd 192.168.100.255 scope global noprefixroute enp0s3 valid_lft forever preferred_lft forever inet 192.168.100.10/32 scope global enp0s3 valid_lft forever preferred_lft forever [root@MAX2 local]# ip addr … 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:28:54:b3 brd ff:ff:ff:ff:ff:ff inet 192.168.100.14/24 brd 192.168.100.255 scope global noprefixroute enp0s3 valid_lft forever preferred_lft forever [root@localhost keepalived]# service keepalived status |grep Active Redirecting to /bin/systemctl status keepalived.service Active: active (running) since 수 2019-12-11 22:30:16 KST; 33min ago [root@localhost local]# service keepalived status |grep Active Redirecting to /bin/systemctl status keepalived.service Active: active (running) since 수 2019-12-11 22:57:03 KST; 7min ago [root@MAX1 keepalived]# service keepalived stop Redirecting to /bin/systemctl stop keepalived.service [root@MAX1 keepalived]# ps -ef |grep keepalived [root@MAX1 keepalived]# [root@localhost keepalived]# ip addr … 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:8d:12:9d brd ff:ff:ff:ff:ff:ff inet 192.168.100.13/24 brd 192.168.100.255 scope global noprefixroute enp0s3 valid_lft forever preferred_lft forever [root@localhost keepalived]# ip addr … 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:28:54:b3 brd ff:ff:ff:ff:ff:ff inet 192.168.100.14/24 brd 192.168.100.255 scope global noprefixroute enp0s3 valid_lft forever preferred_lft forever inet 192.168.100.10/32 scope global enp0s3 valid_lft forever preferred_lft forever [root@MAX1 keepalived]# maxctrl show maxscale | grep passive "passive": true, [root@MAX2 keepalived]# maxctrl show maxscale | grep passive "passive": false, # 정리 - VIP 전환 : MAX1  MAX2 전환 OK. - Maxscale MAX1 : (passive:false)  (passive:true) - Maxscale MAX2 : (passive:true)  (passive:false) Service OK.
  • 20.
    20 3-3) MAX2노드 maxscalestop MAX1 MAX2 [root@MAX1 keepalived]# maxctrl show maxscale | grep passive "passive": false, [root@MAX2 keepalived]# maxctrl show maxscale | grep passive "passive": true, [root@MAX1 keepalived]# ip addr … 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:8d:12:9d brd ff:ff:ff:ff:ff:ff inet 192.168.100.13/24 brd 192.168.100.255 scope global noprefixroute enp0s3 valid_lft forever preferred_lft forever inet 192.168.100.10/32 scope global enp0s3 valid_lft forever preferred_lft forever [root@MAX2 local]# ip addr … 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:28:54:b3 brd ff:ff:ff:ff:ff:ff inet 192.168.100.14/24 brd 192.168.100.255 scope global noprefixroute enp0s3 valid_lft forever preferred_lft forever [root@localhost keepalived]# service keepalived status | grep Active Active: active (running) since 수 2019-12-11 22:30:16 KST; 26min ago [root@localhost local]# service keepalived status | grep Active Active: active (running) since 수 2019-12-11 22:57:03 KST; 41s ago [root@MAX2 local]# service maxscale stop Redirecting to /bin/systemctl stop maxscale.service [root@MAX2 local]# service maxscale status | grep Active Redirecting to /bin/systemctl status maxscale.service Active: inactive (dead) [root@localhost keepalived]# ip addr … 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:8d:12:9d brd ff:ff:ff:ff:ff:ff inet 192.168.100.13/24 brd 192.168.100.255 scope global noprefixroute enp0s3 valid_lft forever preferred_lft forever inet 192.168.100.10/32 scope global enp0s3 valid_lft forever preferred_lft forever [root@localhost keepalived]# ip addr … 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:28:54:b3 brd ff:ff:ff:ff:ff:ff inet 192.168.100.14/24 brd 192.168.100.255 scope global noprefixroute enp0s3 valid_lft forever preferred_lft forever [root@MAX1 keepalived]# maxctrl show maxscale | grep passive "passive": false, [root@MAX2 keepalived]# maxctrl show maxscale | grep passive [root@MAX2 keepalived]# # 정리 - VIP 전환 : MAX1 현상유지 OK. - Maxscale MAX1 : (passive:false) 현상유지 OK - Maxscale MAX2 : 중지 OK Service OK.
  • 21.
    21 3-4) MAX2노드 keepalivedstop MAX1 MAX2 [root@MAX1 keepalived]# maxctrl show maxscale | grep passive "passive": false, [root@MAX2 keepalived]# maxctrl show maxscale | grep passive "passive": true, [root@MAX1 keepalived]# ip addr … 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:8d:12:9d brd ff:ff:ff:ff:ff:ff inet 192.168.100.13/24 brd 192.168.100.255 scope global noprefixroute enp0s3 valid_lft forever preferred_lft forever inet 192.168.100.10/32 scope global enp0s3 valid_lft forever preferred_lft forever [root@MAX2 local]# ip addr … 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:28:54:b3 brd ff:ff:ff:ff:ff:ff inet 192.168.100.14/24 brd 192.168.100.255 scope global noprefixroute enp0s3 valid_lft forever preferred_lft forever [root@localhost keepalived]# service keepalived status |grep Active Redirecting to /bin/systemctl status keepalived.service Active: active (running) since 수 2019-12-11 22:30:16 KST; 33min ago [root@localhost local]# service keepalived status |grep Active Redirecting to /bin/systemctl status keepalived.service Active: active (running) since 수 2019-12-11 22:57:03 KST; 7min ago [root@MAX2 keepalived]# service keepalived stop Redirecting to /bin/systemctl stop keepalived.service [root@MAX2 keepalived]# ps -ef |grep keepalived [root@MAX2 keepalived]# [root@localhost keepalived]# ip addr … 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:8d:12:9d brd ff:ff:ff:ff:ff:ff inet 192.168.100.13/24 brd 192.168.100.255 scope global noprefixroute enp0s3 valid_lft forever preferred_lft forever inet 192.168.100.10/32 scope global enp0s3 valid_lft forever preferred_lft forever [root@localhost keepalived]# ip addr … 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:28:54:b3 brd ff:ff:ff:ff:ff:ff inet 192.168.100.14/24 brd 192.168.100.255 scope global noprefixroute enp0s3 valid_lft forever preferred_lft forever [root@MAX1 keepalived]# maxctrl show maxscale | grep passive "passive": false, [root@MAX2 keepalived]# maxctrl show maxscale | grep passive "passive": true, # 정리 - VIP 전환 : MAX1 현상유지 OK. - Maxscale MAX1 : (passive:false) 현상유지 OK - Maxscale MAX2 : (passive:true) 현상유지 OK Service OK.