RedisConf19

1. PRESENTED BY
History of Redis replication and
future prospects
Zhao Zhao
@solostoy
Alibaba Cloud, senior engineer

2. • Redis in Alibaba Cloud
• Redis replication history
• Redis replication prospect
• Replication based on AOF
• Multi-active Redis

3. Standard Ali-Cluster Split-rw
Redis Mode Stand-alone Stand-alone Stand-alone
High Availability Yes Yes Yes
Proxy No Yes Yes
Scalable Up Up & Out Up & Out
Read/Write Splitting No No Yes
Redis in Alibaba Cloud

4. • Simple architecture
• Easy to use
• High availability
• Full compatibility
VIP
Client
replication
master replica
HA
Standard

5. • Proxy based cluster
• Easy to use
• High availability
• Good compatibility
VIP
Client
Proxy Proxy Proxy
master
replica
master
replica
master
replica
HA
CS
Ali-Cluster

6. • Replication chain
• Easy to use
• High availability
• Full compatibility
VIP
Client
Proxy Proxy Proxy
master read-only
replica
read-only
replica
HA
CS
Split-rw

7. REPLICATION

8. 1. Prepare stage
• Build connection
2. Full sync stage
• Transfer snapshot(RDB)
3. Incremental sync stage
• Propagate commands
master
replica
replication
replica client
master client
Replication

9. SYNC

10. master replica
No.1 Connect to master
No.2 PING to master
No.3 AUTH to master
No.4 Send port to master
• Prepare stage
SYNC

11. master replica
No.1 SYNC to master
No.2 Save RDB
Feed replica client
output buffer
No.3 Send RDB to replica
No.4 Load RDBNo.4 Put replica online
• Full sync stage
SYNC

12. master replica
Ping replica by configuration
Propagate commands
• Incremental sync stage
SYNC

13. master replica
replication
• Disadvantages
1. Prepare stage
2. Full sync stage
3. Incremental sync stage
SYNC

14. PSYNC

15. • New features
• Run ID
• Random ID like process ID
• Replication offset
• Record the replication status
• Replication backlog
• Cache the replication data
master
replica
replication
master client: run_id: ABC
reploff: 123
· run_id: ABC
· repl_offset: 123
· run_id: ?
· repl_offset: ?
replication backlog
PSYNC

16. • Replication information
master replica
PSYNC

17. master replica
replication
• Handle network failure
PSYNC ABC 123+1
+CONTINUE & 124 ~ 200
PING & commands
REPL slave_repl_offset
• run_id: ABC
• repl_backlog:
• 100 ~ 200
• master client
• run_id: ABC
• reploff: 123
PSYNC

18. masterreplica
• Disadvantages
replicamaster
• Failover
• run_id doesn’t match
• full resync
run_id: XYZrun_id: ABC
master client is NULL
PSYNC

19. • Disadvantages
master replica
• Change master
• run_id doesn’t match
• offset doesn’t satisfy consistency
• full resync
replica
• run_id: ABC
• repl_offset: 100
PING every 10s PING every 1s
• run_id: OPQ
• repl_offset: 200
• run_id: ?
• repl_offset: ?
• master.run_id: ABC
• master.reploff: 100
• master.run_id: OPQ
• master.reploff: 200
PSYNC

20. PSYNC2

21. • New features
master
replica
replication
master client: repl_id: ABC
reploff: 123
· repl_id: ABC
· repl_offset: 123
· repl_id: ABC
· repl_offset: 123
replication backlog
replication backlog
• Replication ID
• Same in replication chain
• Replication ID2/offset2
• Record the previous replication ID and offset
• Replica works as a proxy
• Replica just transfer data from master
• Saving replication info in RDB
• RDB contains replication ID and offset
PSYNC2

22. • Replication information
• master • replica
PSYNC2

23. master replica
replication
• Resync
PSYNC ABC 123+1
+CONTINUE repl_id & 124 ~ 200
+FULLRESYNC repl_id master_repl_offset
PING & commands
REPL slave_repl_offset
• repl_id: ABC
• repl_backlog:
• 100 ~ 200
• repl_id: ABC
• master client
• repl_id: ABC
• reploff: 123
PSYNC2

24. repl_id: ABC
repl_offset: 123
repl_id: ABC
repl_offset: 123
• Failover
replicamaster
repl_id: XYZ
repl_id2: ABC
repl_offset2: 124
• master
master.repl_id: ABC
master.reploff: 123
• replica
PSYNC ABC 123+1
+CONTINUE XYZreplid: XYZ
master.repl_id: XYZ
RDB
repl_id: ABC
repl_offset: 123
replica of no one
PSYNC2

25. • Change master
master replica replica
1. all the replication IDs are same
2. offset satisfies consistency
PING every 1s transfer PING from master
• repl_id: ABC
• repl_offset: 100
• repl_id: ABC
• repl_offset: 100
• repl_id: ABC
• repl_offset: 100
• master.repl_id: ABC
• master.reploff: 100
• master.repl_id: ABC
• master.reploff: 100
PSYNC2

26. PSYNC3

27. • Next improvements to Redis replication
https://github.com/antirez/redis/issues/4357
• AOF annotations
• AOF contains replication infomations
• Detecting reboot instances as failing
• Check run id to failover
• Replica ability to reconnect master with partial sync
• Same as PSYNC2
PSYNC3

28. AOF-PSYNC

29. • AOF annotations—oplog
• AOF OPINFO command
OPINFO
Redis Cmd
*2rn$6rnOPINFOrn$16rn***rn
*3rn$3rSETrn$3rnKEYrn$5rnVALUErn
AOF-PSYNC

30. AOF
*2
$6
OPINFO
$16
***
dbid
opid
timestamp
*3
$3
SET
$3
KEY
$5
VALUE
OPINFO
Redis Command
OPINFO
Redis Command
OPINFO
Redis Command
OPINFO
Redis Command
AOF-PSYNC

31. master replica
replication
• Resync
PSYNC ABC 123+1 40+1
+AOFCONTINUE repl_id
PING & commands
REPL slave_repl_offset
• repl_id: ABC
• repl_backlog:
• 200 ~ 300
• oplog in AOF:
• 1 ~ 50
• repl_id: ABC
• master client
• repl_id: ABC
• reploff: 123
• current opid: 40
send command from 41 to 50
SYNCREPLOFFSET
AOF-PSYNC

32. • Avoid AOF rewrite
RDB1 AOF1 AOF2
timeline
RDB2 AOF3 AOF4
AOF-PSYNC

33. Multi-active

34. Multi-active

35. • Improve oplog
Multi-active

36. AOF
OPINFO
Redis Command
*2
$6
OPINFO
$32
***
server_id
dbid
opid
src_opid
timestamp
*3
$3
SET
$3
KEY
$5
VALUE
OPINFO
Redis Command
OPINFO
Redis Command
OPINFO
Redis Command
Multi-active

37. • OPGET & OPAPPLY
middle ware
OPGET from A OPAPPLY to B
OPGET from BOPAPPLY to A
• OPGET <server_id> <op_id> <count>
• Get n commands from op_id with server_id in AOF
• OPAPPLY
• Give this client a special mark
Multi-active

38. • Avoid cycle
server_id op_id src_opid cmd
A 100 -1 Cmd1
server_id op_id src_opid cmd
B 20 -1 Cmd2
server_id op_id src_opid cmd
A 100 -1 Cmd1
B 101 20 Cmd2
server_id op_id src_opid cmd
A * -1 Cmd1
server_id op_id src_opid cmd
B 19 -1 Cmd1
Multi-active

39. • Conflict
SET key value1
SET key value2 SET key value3
• Conflict Solution
• Users POV - avoid conflict
• Using different DB
• Keys with different prefix
• CRDT - solve conflict
• Most data structures satisfy
• It’s on the way, but it is not the Silver Bullet
Multi-active

40. Open Source & Alibaba Cloud
https://github.com/alibaba/ApsaraCache
https://www.alibabacloud.com/product/apsaradb-for-redis
https://github.com/alibaba/RedisShake

41. • https://github.com/soloestoy
• https://twitter.com/soloestoy6
• zhaozhao.zz@alibaba-inc.com
Contact me

42. Thank you!

43. PRESENTED BY