Redis sentinelinternals deview

5,726 views

Published on

Published in: Technology, Self Improvement
2 Comments
27 Likes
Statistics
Notes
No Downloads
Views
Total views
5,726
On SlideShare
0
From Embeds
0
Number of Embeds
2,271
Actions
Shares
0
Downloads
147
Comments
2
Likes
27
Embeds 0
No embeds

No notes for slide

Redis sentinelinternals deview

  1. 1. Redis Sentinel charsyam@naver.com
  2. 2. Redis.io
  3. 3. Redis/Twemproxy Contributor 카카오 홈 개발자
  4. 4. Redis Monitoring?
  5. 5. Checking Liveness
  6. 6. Checking Liveness servers = [ ('localhost', 6379), ('localhost', 6380), ('localhost', 6381) ] conns = [] def check(): for conn in conns: print conn.ping()
  7. 7. Checking it works
  8. 8. Checking it works def check(): for conn in conns: i = conn.info() print i['run_id'], i['used_memory_human'], i['total_commands_processed']
  9. 9. Is it Reliable?
  10. 10. Is Server really Failed?
  11. 11. Split Network.
  12. 12. Server is busy.
  13. 13. Monitoring is not the one we really want.
  14. 14. HA High Availity
  15. 15. What do you do? When your servers failed
  16. 16. 1. Recognize Redis is down.
  17. 17. 2. choose new master candidate
  18. 18. 3. Promote slave to Master.
  19. 19. 4. make client connect to new master
  20. 20. What Redis Sentinel do?
  21. 21. 1. Check Redis Liveness
  22. 22. 1. Check Redis Liveness -Subjective Down -Objective Down
  23. 23. 2. Choose good slave.
  24. 24. Good Slave NOT SDOWN, ODOWN, DISCONNECTED NOT DEMOTE Ping reply > info_validity_time Slave_priority != 0 Info reply > info_validity_time
  25. 25. Choose new master Sort Slave Priority Runid
  26. 26. 3. Promote it
  27. 27. Promoting Send Slaveof no one to new master Send Slaveof [new master ip] [addr] to Other redis Notify new master to clietns +switch-master Set DEMOTE mark to old-Master
  28. 28. Failover Scenario
  29. 29. Sentinel Master Slave Slave Pub/Sub Client Client
  30. 30. Sentinel Master Slave A Slave B Pub/Sub Client Client
  31. 31. Choose Slave A as New Master Master Slave A Slave B Sentinel Pub/Sub Client Client
  32. 32. Send to Slave B Slaveof Slave A Master Slave A Slave B Sentinel Pub/Sub Client Client
  33. 33. Send “Slaveof no one” to Slave A Master Slave A Slave B Sentinel Pub/Sub Client Client
  34. 34. Send “Slaveof Slave A” to Slave B Master Master Slave B Sentinel Pub/Sub Client Client
  35. 35. Sentinel Master Master Slave B Pub/Sub: +switch-master Client Client
  36. 36. Sentinel Internal
  37. 37. Sentinel Failover State Machine State 내용 NONE WAIT_START SELECT_SLAVE SEND_SLAVEOF WAIT_PROMOTION RECONF_SLAVES DETECT_END Next State SELECT_SLAVE SEND_SLAVEOF WAIT_PROMOTION
  38. 38. Starting Point Of Sentinel • sentinelTimer in sentinel.c –Called every 100ms –sentinelHandleRedisInstance
  39. 39. SentinelRedisInstance • master –If it is master, it is null. • slaves • sentinels
  40. 40. sentinelHandleDictOfRedisInstances while((de = dictNext(di)) != NULL) { sentinelRedisInstance *ri = dictGetVal(de); sentinelHandleRedisInstance(ri); if (ri->flags & SRI_MASTER) { sentinelHandleDictOfRedisInstances(ri->slaves); sentinelHandleDictOfRedisInstances(ri->sentinels); …… } }
  41. 41. sentinelHandleRedisInstance • Reconnect to Instances • Ping • Asking Master State to other Sentinel • Check SDOWN • Check ODOWN
  42. 42. How to check Subjective Down • When Sentinel get Ping Reply –Set last_avail_time as current • Check –e = current –last_avail_time –e > last_avail_time •Subjective Down.
  43. 43. How to check Objective Down • Ask to other sentinels is it down. –sentinelAskMasterStateToOtherSentinels • If other sentinels reply it is down. –Count them, if it is bigger than Quorum. –It is objective Down.
  44. 44. How to find Redis
  45. 45. INFO Command # Replication role:master connected_slaves:1 slave0:ip=127.0.0.1,port=6380,state=online,offset=1,lag=0 master_repl_offset:1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:2 repl_backlog_histlen:0
  46. 46. INFO Command # Replication role:slave master_host:127.0.0.1 master_port:6379
  47. 47. +redirect-master
  48. 48. How to find Sentinel
  49. 49. Find Sentinel Subscribe SENTINEL_HELLO_CHANNEL In Master Publish Sentinel Information In sentinelPingInstance Add New Sentinel In sentinelReceiveHelloMessages
  50. 50. Using Pub/sub *4 $8 pmessage $1 * $18 __sentinel__:hello $58 127.0.0.1:26379:7d717f945afde99e6f82f825de052f17cab7e6f3:1
  51. 51. Is Sentinel Really Good?
  52. 52. Not Mature #1
  53. 53. Sentinel Conf port 26379 sentinel monitor mymaster 127.0.0.1 6379 2 sentinel down-after-milliseconds mymaster 2000 sentinel can-failover mymaster yes sentinel parallel-syncs mymaster 1 sentinel failover-timeout mymaster 900000
  54. 54. Quorum Count sentinel monitor mymaster 127.0.0.1 6379 2 We should check this count.
  55. 55. Sentinel Conf sentinel down-after-milliseconds mymaster 2000 If this value is Too small, It can cause some trouble. -sdown/+sdown loop
  56. 56. Compare with Zookeeper Conf tickTime=2000 dataDir=/var/zookeeper clientPort=2181 initLimit=5 syncLimit=2 server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888
  57. 57. Not Mature #2
  58. 58. Master Slave Sentinel
  59. 59. Master Slave Sentinel
  60. 60. Master Master Sentinel
  61. 61. Master Master Sentinel
  62. 62. Case #1 Master Sentinel Master It’s OK.
  63. 63. Case #1 Master Master Sentinel
  64. 64. Case #2 Master Slave Sentinel
  65. 65. Case #2 Master Slave Sentinel Sentinel can’t promote Slave to Master
  66. 66. Why?
  67. 67. Sentinel Mechanism. When master is changed to Slave, Reconf with new Master.
  68. 68. Sentinel Mechanism. When master is changed to Slave, Reset information with new Master. But, if new master failed, sentinel will not get any information from it.
  69. 69. Sentinel Mechanism. if ((ri->flags & SRI_MASTER) && role == SRI_SLAVE && ri->slave_master_host) { sentinelEvent(REDIS_WARNING,"+redirect-to-master",ri, "%s %s %d %s %d", ri->name, ri->addr->ip, ri->addr->port, ri->slave_master_host, ri->slave_master_port); sentinelResetMasterAndChangeAddress(ri,ri->slave_master_host, ri->slave_master_port); return; /* Don't process anything after this event. */ }
  70. 70. Not Mature #3
  71. 71. If redis is down, when redis is loading data(RDB, AOF), sentinel can’t register redis.
  72. 72. Conclusion
  73. 73. Even though, Sentinel is not mature. It is useful.
  74. 74. Redis Sentinel Tips
  75. 75. 1. Subscribe all Sentinels And ignore duplicated message
  76. 76. 2. Sentinel has explicit failover command sentinel mymaster failover
  77. 77. But can’t select new master
  78. 78. Hacking is easy. https://github.com/antirez/redis/pull/1126
  79. 79. 3. Make Sentinel.conf same.
  80. 80. Sentinels can’t share their status
  81. 81. Deploy conf to all sentinels with Script
  82. 82. And restart all sentinels.
  83. 83. 4. If you want not to promote specific server to master
  84. 84. Set Slave-Priority = 0
  85. 85. Thank you!
  86. 86. Just One more thing
  87. 87. Twemprox Redis Redis-sentinel Sharded Redis
  88. 88. Redis-Sentinel TwemProxy Agent http://www.codeproject.com/Articles/656965/R edis-Sentinel-TwemProxy-Agent
  89. 89. Agent Pub/Sub Shard Redis Master Restart VRRP Twemproxy HA Proxy Redis Slave Redis Slave Twemproxy HA Proxy Twemproxy Shard Sentinel Redis Master Redis Slave Redis Slave

×