Intro KaKao ADT (Almighty Data Transmitter)

A.D.T
성동찬 (Chan @KakaoBank)
한수호(Gordon @Kakao)

About Speakers
• 성동찬
• KTH, 티몬, 카카오, (현)카카오뱅크 DBA
• 특이사항: ADT 프로젝트 도중 은행으로 튐
• A.k.a 배신자 (장난입니다.ㅎㅎ)
• 한수호
• 2007: (주)아이씨유 공동 창업
• 2012: 카카오에 인수됨(카카오랩으로 사명 변경)
• 카카오에서 계속 잘 지내는 중

What is ADT?
Almighty
Data
Transmitter

History
• 2015년 중반
- 일부 서비스의 MySQL 샤드 재구성 필요성 대두

Range Sharding (1/3)
Shard1 Shard2 Shard3
ID: 1 - 1000
ID: 1001 - 2000
ID: 2001 - 3000

ID: 1 - 1000
10,000,000 rows
5,000 QPS
ID: 1001 - 2000
5,000,000 rows
1,000 QPS
ID: 2001 - 3000
1,000 rows
10,000 QPS
new shard

- PROS.
- CONS.
Unbalanced rows and traffic
Easy to attach next shard  
without any side effect

Modulus Sharding (1/3)
ID mod 3 : 0
ID mod 3 : 1
ID mod 3 : 2

ID % 3 == 0 ID % 3 == 1 ID % 3 == 2
ID % 4 == 0 ID % 4 == 1 ID % 4 == 2
Shard4
ID % 4 == 3
Add New?

- PROS.
- CONS.
Difficult to attach new shard
Better resource balancing

History
• 2015년 여름
• 2015년 하반기: 프로젝트 시작
- 목표: MySQL 샤드 재구성 (시작은 이러했으나...)

– Chan
“MySQL Binary Log를 이용해서 할 수 있는 것이
샤드 재구성 외에도 많지 않을까요?”

History
• 2015년 여름
- 목표: MySQL 샤드 재구성
- 목표: 다양한 마이그레이션 (ETL+CDC ?)

– 어느 서비스의 어느 MySQL 서버
수백MB/min의 속도로 자라나는 Binlog를
니가 감당할 수 있을까?ㅋ

History
• 2015년 여름
- 목표: MySQL 샤드 재구성
- 다양한 마이그레이션
- 빠른 처리 속도가 필요함

Goals
Dynamic Migration
Performance
Support only MySQL in first release

Dynamic Migration
• One-time Migration
- e.g. Shard Rebalancing
• Real-time Migration (as a Service?)
- e.g. MySQL => HBase in Real-time

Ex: Add new shard
Shard1 Shard2
Shard1 Shard3Shard2

Ex: Change shard rule
ID: 1 - 1000
ID: 1001 - 2000
ID: 2001 - 3000
Range
ID mod 3: 0
ID mod 3: 1
ID mod 3: 2
Modulus

Ex: Copy to diff. DBMS
MySQL
HBase
MongoDB
NO side effect

Ex: Copy to diff. schema
MySQL
MySQL
MySQL
ID AGE CNT V
1 30 5 msg1
2 29 10 msg2
ID CNT V
1 5 msg1
ID CNT V
2 10 msg2
NO side effect

Ex: Write w/ custom processing
MySQL
MySQL
MySQL
Events
ID AGE CNT V
1 30 5 msg1
2 29 10 msg2
1) REQ
INFO2)
3) GET INFO
4) MERGE
ID FRIENDS
1 4,5,6,7,8,9
2 3,4,5,8,10

Performance (1/3)
Source
Dest1 Dest2
Data Copy Layer

Performance (2/3)
Single-threaded Applier
Dest1 Dest2
1ms
1ms
2ms
2ms
2ms

Performance (3/3)
Multi-threaded Applier
Dest1 Dest2

Parallel Processing (1/3)
Different Row ID
Parallel Processing
Same Row ID
Sequential Processing

AS-IS (SQL Thread)
1 : a=a+1
3 : a=a+2
2 : a=a+1
3 : a=a+4
2 : a=a+1
1 : a=a+2
Binary Log
1 : a=a+1
3 : a=a+2
2 : a=a+1
3 : a=a+4
2 : a=a+1
1 : a=a+2
Sequential Apply

TO-BE (ADT)
1 : a=a+1
3 : a=a+2
2 : a=a+1
3 : a=a+4
2 : a=a+1
1 : a=a+2
1 : a=a+1 3 : a=a+22 : a=a+1
3 : a=a+42 : a=a+11 : a=a+2
Thread1 Thread2 Thread3
Binary Log

Features
• Table Crawler
• SELECT 쿼리의 반복 
SELECT * FROM ? [ WHERE id > ? ] LIMIT ?;
• Binlog Receiver
• MySQL Replication 프로토콜
• Custom Data Handler
• 수집한 데이터의 처리 부분 
e.g. Shard reconstruction handler
• 여러 스레드에 의해 동시에 실행됨

Features: Binlog Processor
Binlog
Recv
Queue #1
Custom
Data
Handler
Queue #2
Queue #3
Queue #n
…
ADT Binlog Processor
•Binlog Receiver
1. PK/UK 값을 읽음
2. 읽은 값을 기준으로 적절한
Queue에 넣음
3. 해당 Queue를 처리할 담당
Thread 지정
•1 Thread / 1 Queue
- 같은 Queue: 순차 처리
- 다른 Queue: 병렬 처리

Features: Data Crawler
1. SELECT … LIMIT n
2. 파이프라이닝
•SELECT한 값을 처리하기
전에 다른 쓰레드를 깨워 다
음 SELECT 실행
•SELECT한 순서대로
Custom Handler가 호출되
지는 않음 
Select Crawl from Last PK
Custom Data Handler
ADT Data Crawler
Thread
Pool

Req 1. Row Format
각 Binlog는 Before, After 값이 필요합니다
N/A 1 : a=1, b=2, c=3
Before After
INSERT
1 : a=1, b=99, c=99 1 : a=1, b=2, c=3UPDATE
1 : a=1, b=99, c=99 N/ADELETE

Req 2. Primary Key
DELETE
Source Destination
INSERT
UPDATE
Delete Old/Insert New
Delete Old/Insert New
Delete Old
모든 Table은 Primary Key가 존재해야 합니다

ADT
Req 3. Custom Data Handler
MySQL
Binlog
Processor
MySQL
Data
Crawler
Custom Data Handler
(예: Shard 재분배 핸들러)
New
Masters
Load
Current
Master
Data
이 부분을 직접 구현해야 합니다

기타 요구/제약 사항들
• Millisecond 사용 불가
• Alter Table 실시간 반영 안 됨
• 기타 등등...

Types of Integrity
• Domain Integrity
• Column Type, NOT NULL, CHECK, …
• Entity Integrity
• Primary Key, Unique Key
• Referential Integrity
• Foreign Key
다른 data의 DML에
영향을 주는
Integrity만 고려

Constraints
Unique Key
Same PK(Row ID) or UK
Sequential Processing
Foreign Key
Not Considered 
set foreign_key_checks = 0

Why Ignore FK?
• 샤드 재구성 하려고 했던 곳이 성능을 위해 FK를 안 쓰는 곳이어서
• FK constraint 체크는 not null, check와 같이 master에서 이미 했
으므로 slave에서 할 필요 없다고 판단
• FK가 데이터 변경을 유발하지 않는 경우: 고려 X
• FK로 인해 다른 테이블 값 변경되는 경우
• table이 다르면 병렬 처리 시 conflict 가능성이 없음
• FK로 인해 같은 테이블 값 변경되는 경우
• 어차피 parent, child row 둘 다 다른 constraint에서 문제가 없
을 경우에만 binlog에 기록되므로 PK, UK만 고려

– Chan & Gordon
“운영이 항상 내 마음대로 되는 것도 아니고,
여러 상황에 대해 다 고려할 필요 없이 간단하게 갑시다.”

그 간단한 방법은 이렇습니다
최근 완료 시점부터 재시작
에러 감지
모두 덮어쓰기로 다시 작업 시작

How Data Crawler Works
• SELECT
• SELECT using PK of ex-selected rows
• INSERT
• INSERT IGNORE is required 
if Binlog Processor runs together

How to Handle Binlog? (1/4)
Row Event
Type
Query to Dest. (normally)
WRITE insert( after )
DELETE delete( before )
UPDATE update( before, after )
Normally binlog events are handled like this.

• However, we should consider…
- Unexpected restart
- Data inserted by Crawler
Overwriting!

Row Event
Type
Query to Dest.
(Overwriting)
WRITE replace( after )
DELETE delete( before )
UPDATE
if( before.pk!=after.pk ){
delete( before )
}
replace( after )

• Normal Query 
UPDATE … SET @1=after.1, @2=after.2,… 
WHERE pk_col=before.pk
• Transformation 1: Unrolling 
DELETE FROM … WHERE pk_col=before.pk;
INSERT INTO … VALUES(after.1, after.2,…);
• Transformation 2: Overwriting 
DELETE FROM … WHERE pk_col=before.pk;
REPLACE INTO … VALUES(after.1, after.2,…);
• Transformation 3: Reducing
• Delete [before] only if PK is changed

Strategy 1
• Run sequentially
1. Crawl Data
2. Process Binary Log (after 1 is finished)
• Binlog file (created before starting) is required
• If crawling takes more than 3 days, then…?

Strategy 2
• Run parallel with master DB
1. Start binlog processor
2. Start Data Crawler (ASAP after 1 is started) 
with INSERT IGNORE
• Problem: conflicts

Strategy 2: Conflicts
src
dest
Binlog Processor Data Crawler
SELECT PK:1
INSERT PK:1
PK:1 is deleted
DELETE PK:1

Strategy 2-1
• Data Crawler
- SELECT … FOR UPDATE when crawling
- ROLLBACK after INSERT IGNORE
• Binlog Processor
- Just act normally 
Because there’s no logs for locked rows
• Problem: Multi row lock is dangerous for master DB

Strategy 2-2
• Binlog Processor
- Cache deleted history during a few minutes
• Data Cralwer
- If delete history exists, no INSERT
• Problems
- Complicated: Lock is neccesary for history cache

Strategy 2-3
• Same as Strategy 2-1, except using slave DB
• If sync is finished with slave,  
restart with new config
- Receive binlog from master

Test Scenario
•Split into 2 shards from 1 master DB
• Binary log only
•Query many updates into master DB
• 1K active sessions with random DML & data
•Make errors
• Master DB: ifdown —> mysql restart —> ifup
•Compare data

Test DML List
INSERT INTO …
INSERT IGNORE INTO …
INSERT INTO … ON DUPLICATE KEY …
REPLACE INTO …
UPDATE …
DELETE …

Test Table Schema
Column
Name
Type
Table 1 Table 2
Value 
(when exec. DML)Primary Unique Primary Unique
no int 1 2 1 Random(1~n)
seq int 2 Random(1~n)
uk int 1 1 Random(1~n)
update_cnt int Incr. when update
random_val text Random String
(이하 생략)

Step1. Split into 2 Shards
MySQL
Source
Query
Tool
ADT
Binlog
Events
Write
MySQL
Dest.
Split
Shards

Step2. Make Error
MySQL
Source
ADT
Binlog
Events
ifdown eth0
service mysql restart
ifup eth0
If any problem is detected,
ADT restarts and re-sync

Step3. Compare Data
MySQL
Source
Comparator MySQL
Dest.
Lock &
Select All
Remained
Binlogs
Select All
Compare
Unlock
Compare Periodically
Write

Test result is…
No Error
during 2 weeks

Wish to Apply for…
Shard reconstruction (default)
MySQL binary log —> NoSQL
Copy data change history into OLAP
MySQL binary log —> Push Notification
Re-construct shards by GPS Point (Kakao Taxi?)
……

A.D.T
Change Replicate Rule
Sharded by USER_ID
Sharded by Location
4-nodes
1-nodes
2-nodes
Ex: Shard By GPS Point

Next Dev. Plans
• Change language: Java —> GoLang
• Control Tower: Admin & Monitoring
• Is ADT alive?
• Save checkpoint for ungraceful restart
• Support Multiple DB Types
• Redis, PgSQL, ……

Intro KaKao ADT (Almighty Data Transmitter)

More Related Content

What's hot

Similar to Intro KaKao ADT (Almighty Data Transmitter)

More from I Goo Lee

Intro KaKao ADT (Almighty Data Transmitter)