Your SlideShare is downloading. ×
Redis sentinelinternals deview
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Redis sentinelinternals deview

4,336
views

Published on

Published in: Technology, Self Improvement

2 Comments
20 Likes
Statistics
Notes
  • (11) Checking if it works
    (13) Has the server really failed?
    (16) what we really want
    (17) High Availability
    (18) when your servers fail
    (19) Recognize that redis is down
    (20) Choose a new master candidate
    (26) Choose a good slave
    (28) Choose a new master
    (42) null if it is the master
    (46) Ask other sentinels if it is down
    (76) Even though sentinel is not mature, it is useful
    (78) Subscribe to all sentinels and ignore duplicate messages
    (86) If you don't want to
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • 한글이 없다니 ㅠ_ㅠ
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
4,336
On Slideshare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
99
Comments
2
Likes
20
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Redis Sentinel charsyam@naver.com
  • 2. Redis.io
  • 3. Redis/Twemproxy Contributor 카카오 홈 개발자
  • 4. Redis Monitoring?
  • 5. Checking Liveness
  • 6. Checking Liveness servers = [ ('localhost', 6379), ('localhost', 6380), ('localhost', 6381) ] conns = [] def check(): for conn in conns: print conn.ping()
  • 7. Checking it works
  • 8. Checking it works def check(): for conn in conns: i = conn.info() print i['run_id'], i['used_memory_human'], i['total_commands_processed']
  • 9. Is it Reliable?
  • 10. Is Server really Failed?
  • 11. Split Network.
  • 12. Server is busy.
  • 13. Monitoring is not the one we really want.
  • 14. HA High Availity
  • 15. What do you do? When your servers failed
  • 16. 1. Recognize Redis is down.
  • 17. 2. choose new master candidate
  • 18. 3. Promote slave to Master.
  • 19. 4. make client connect to new master
  • 20. What Redis Sentinel do?
  • 21. 1. Check Redis Liveness
  • 22. 1. Check Redis Liveness -Subjective Down -Objective Down
  • 23. 2. Choose good slave.
  • 24. Good Slave NOT SDOWN, ODOWN, DISCONNECTED NOT DEMOTE Ping reply > info_validity_time Slave_priority != 0 Info reply > info_validity_time
  • 25. Choose new master Sort Slave Priority Runid
  • 26. 3. Promote it
  • 27. Promoting Send Slaveof no one to new master Send Slaveof [new master ip] [addr] to Other redis Notify new master to clietns +switch-master Set DEMOTE mark to old-Master
  • 28. Failover Scenario
  • 29. Sentinel Master Slave Slave Pub/Sub Client Client
  • 30. Sentinel Master Slave A Slave B Pub/Sub Client Client
  • 31. Choose Slave A as New Master Master Slave A Slave B Sentinel Pub/Sub Client Client
  • 32. Send to Slave B Slaveof Slave A Master Slave A Slave B Sentinel Pub/Sub Client Client
  • 33. Send “Slaveof no one” to Slave A Master Slave A Slave B Sentinel Pub/Sub Client Client
  • 34. Send “Slaveof Slave A” to Slave B Master Master Slave B Sentinel Pub/Sub Client Client
  • 35. Sentinel Master Master Slave B Pub/Sub: +switch-master Client Client
  • 36. Sentinel Internal
  • 37. Sentinel Failover State Machine State 내용 NONE WAIT_START SELECT_SLAVE SEND_SLAVEOF WAIT_PROMOTION RECONF_SLAVES DETECT_END Next State SELECT_SLAVE SEND_SLAVEOF WAIT_PROMOTION
  • 38. Starting Point Of Sentinel • sentinelTimer in sentinel.c –Called every 100ms –sentinelHandleRedisInstance
  • 39. SentinelRedisInstance • master –If it is master, it is null. • slaves • sentinels
  • 40. sentinelHandleDictOfRedisInstances while((de = dictNext(di)) != NULL) { sentinelRedisInstance *ri = dictGetVal(de); sentinelHandleRedisInstance(ri); if (ri->flags & SRI_MASTER) { sentinelHandleDictOfRedisInstances(ri->slaves); sentinelHandleDictOfRedisInstances(ri->sentinels); …… } }
  • 41. sentinelHandleRedisInstance • Reconnect to Instances • Ping • Asking Master State to other Sentinel • Check SDOWN • Check ODOWN
  • 42. How to check Subjective Down • When Sentinel get Ping Reply –Set last_avail_time as current • Check –e = current –last_avail_time –e > last_avail_time •Subjective Down.
  • 43. How to check Objective Down • Ask to other sentinels is it down. –sentinelAskMasterStateToOtherSentinels • If other sentinels reply it is down. –Count them, if it is bigger than Quorum. –It is objective Down.
  • 44. How to find Redis
  • 45. INFO Command # Replication role:master connected_slaves:1 slave0:ip=127.0.0.1,port=6380,state=online,offset=1,lag=0 master_repl_offset:1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:2 repl_backlog_histlen:0
  • 46. INFO Command # Replication role:slave master_host:127.0.0.1 master_port:6379
  • 47. +redirect-master
  • 48. How to find Sentinel
  • 49. Find Sentinel Subscribe SENTINEL_HELLO_CHANNEL In Master Publish Sentinel Information In sentinelPingInstance Add New Sentinel In sentinelReceiveHelloMessages
  • 50. Using Pub/sub *4 $8 pmessage $1 * $18 __sentinel__:hello $58 127.0.0.1:26379:7d717f945afde99e6f82f825de052f17cab7e6f3:1
  • 51. Is Sentinel Really Good?
  • 52. Not Mature #1
  • 53. Sentinel Conf port 26379 sentinel monitor mymaster 127.0.0.1 6379 2 sentinel down-after-milliseconds mymaster 2000 sentinel can-failover mymaster yes sentinel parallel-syncs mymaster 1 sentinel failover-timeout mymaster 900000
  • 54. Quorum Count sentinel monitor mymaster 127.0.0.1 6379 2 We should check this count.
  • 55. Sentinel Conf sentinel down-after-milliseconds mymaster 2000 If this value is Too small, It can cause some trouble. -sdown/+sdown loop
  • 56. Compare with Zookeeper Conf tickTime=2000 dataDir=/var/zookeeper clientPort=2181 initLimit=5 syncLimit=2 server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888
  • 57. Not Mature #2
  • 58. Master Slave Sentinel
  • 59. Master Slave Sentinel
  • 60. Master Master Sentinel
  • 61. Master Master Sentinel
  • 62. Case #1 Master Sentinel Master It’s OK.
  • 63. Case #1 Master Master Sentinel
  • 64. Case #2 Master Slave Sentinel
  • 65. Case #2 Master Slave Sentinel Sentinel can’t promote Slave to Master
  • 66. Why?
  • 67. Sentinel Mechanism. When master is changed to Slave, Reconf with new Master.
  • 68. Sentinel Mechanism. When master is changed to Slave, Reset information with new Master. But, if new master failed, sentinel will not get any information from it.
  • 69. Sentinel Mechanism. if ((ri->flags & SRI_MASTER) && role == SRI_SLAVE && ri->slave_master_host) { sentinelEvent(REDIS_WARNING,"+redirect-to-master",ri, "%s %s %d %s %d", ri->name, ri->addr->ip, ri->addr->port, ri->slave_master_host, ri->slave_master_port); sentinelResetMasterAndChangeAddress(ri,ri->slave_master_host, ri->slave_master_port); return; /* Don't process anything after this event. */ }
  • 70. Not Mature #3
  • 71. If redis is down, when redis is loading data(RDB, AOF), sentinel can’t register redis.
  • 72. Conclusion
  • 73. Even though, Sentinel is not mature. It is useful.
  • 74. Redis Sentinel Tips
  • 75. 1. Subscribe all Sentinels And ignore duplicated message
  • 76. 2. Sentinel has explicit failover command sentinel mymaster failover
  • 77. But can’t select new master
  • 78. Hacking is easy. https://github.com/antirez/redis/pull/1126
  • 79. 3. Make Sentinel.conf same.
  • 80. Sentinels can’t share their status
  • 81. Deploy conf to all sentinels with Script
  • 82. And restart all sentinels.
  • 83. 4. If you want not to promote specific server to master
  • 84. Set Slave-Priority = 0
  • 85. Thank you!
  • 86. Just One more thing
  • 87. Twemprox Redis Redis-sentinel Sharded Redis
  • 88. Redis-Sentinel TwemProxy Agent http://www.codeproject.com/Articles/656965/R edis-Sentinel-TwemProxy-Agent
  • 89. Agent Pub/Sub Shard Redis Master Restart VRRP Twemproxy HA Proxy Redis Slave Redis Slave Twemproxy HA Proxy Twemproxy Shard Sentinel Redis Master Redis Slave Redis Slave