Redis sentinelinternals deview
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Redis sentinelinternals deview

on

  • 4,178 views

 

Statistics

Views

Total Views
4,178
Views on SlideShare
2,745
Embed Views
1,433

Actions

Likes
18
Downloads
92
Comments
2

8 Embeds 1,433

http://deview.kr 1363
http://tedwon.com 30
https://twitter.com 29
https://www.facebook.com 5
http://115.68.2.182 2
http://local.deview.kr 2
http://www.linkedin.com 1
http://deview.admin.nhn.com 1
More...

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • (11) Checking if it works
    (13) Has the server really failed?
    (16) what we really want
    (17) High Availability
    (18) when your servers fail
    (19) Recognize that redis is down
    (20) Choose a new master candidate
    (26) Choose a good slave
    (28) Choose a new master
    (42) null if it is the master
    (46) Ask other sentinels if it is down
    (76) Even though sentinel is not mature, it is useful
    (78) Subscribe to all sentinels and ignore duplicate messages
    (86) If you don't want to
    Are you sure you want to
    Your message goes here
    Processing…
  • 한글이 없다니 ㅠ_ㅠ
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Redis sentinelinternals deview Presentation Transcript

  • 1. Redis Sentinel charsyam@naver.com
  • 2. Redis.io
  • 3. Redis/Twemproxy Contributor 카카오 홈 개발자
  • 4. Redis Monitoring?
  • 5. Checking Liveness
  • 6. Checking Liveness servers = [ ('localhost', 6379), ('localhost', 6380), ('localhost', 6381) ] conns = [] def check(): for conn in conns: print conn.ping()
  • 7. Checking it works
  • 8. Checking it works def check(): for conn in conns: i = conn.info() print i['run_id'], i['used_memory_human'], i['total_commands_processed']
  • 9. Is it Reliable?
  • 10. Is Server really Failed?
  • 11. Split Network.
  • 12. Server is busy.
  • 13. Monitoring is not the one we really want.
  • 14. HA High Availity
  • 15. What do you do? When your servers failed
  • 16. 1. Recognize Redis is down.
  • 17. 2. choose new master candidate
  • 18. 3. Promote slave to Master.
  • 19. 4. make client connect to new master
  • 20. What Redis Sentinel do?
  • 21. 1. Check Redis Liveness
  • 22. 1. Check Redis Liveness -Subjective Down -Objective Down
  • 23. 2. Choose good slave.
  • 24. Good Slave NOT SDOWN, ODOWN, DISCONNECTED NOT DEMOTE Ping reply > info_validity_time Slave_priority != 0 Info reply > info_validity_time
  • 25. Choose new master Sort Slave Priority Runid
  • 26. 3. Promote it
  • 27. Promoting Send Slaveof no one to new master Send Slaveof [new master ip] [addr] to Other redis Notify new master to clietns +switch-master Set DEMOTE mark to old-Master
  • 28. Failover Scenario
  • 29. Sentinel Master Slave Slave Pub/Sub Client Client
  • 30. Sentinel Master Slave A Slave B Pub/Sub Client Client
  • 31. Choose Slave A as New Master Master Slave A Slave B Sentinel Pub/Sub Client Client
  • 32. Send to Slave B Slaveof Slave A Master Slave A Slave B Sentinel Pub/Sub Client Client
  • 33. Send “Slaveof no one” to Slave A Master Slave A Slave B Sentinel Pub/Sub Client Client
  • 34. Send “Slaveof Slave A” to Slave B Master Master Slave B Sentinel Pub/Sub Client Client
  • 35. Sentinel Master Master Slave B Pub/Sub: +switch-master Client Client
  • 36. Sentinel Internal
  • 37. Sentinel Failover State Machine State 내용 NONE WAIT_START SELECT_SLAVE SEND_SLAVEOF WAIT_PROMOTION RECONF_SLAVES DETECT_END Next State SELECT_SLAVE SEND_SLAVEOF WAIT_PROMOTION
  • 38. Starting Point Of Sentinel • sentinelTimer in sentinel.c –Called every 100ms –sentinelHandleRedisInstance
  • 39. SentinelRedisInstance • master –If it is master, it is null. • slaves • sentinels
  • 40. sentinelHandleDictOfRedisInstances while((de = dictNext(di)) != NULL) { sentinelRedisInstance *ri = dictGetVal(de); sentinelHandleRedisInstance(ri); if (ri->flags & SRI_MASTER) { sentinelHandleDictOfRedisInstances(ri->slaves); sentinelHandleDictOfRedisInstances(ri->sentinels); …… } }
  • 41. sentinelHandleRedisInstance • Reconnect to Instances • Ping • Asking Master State to other Sentinel • Check SDOWN • Check ODOWN
  • 42. How to check Subjective Down • When Sentinel get Ping Reply –Set last_avail_time as current • Check –e = current –last_avail_time –e > last_avail_time •Subjective Down.
  • 43. How to check Objective Down • Ask to other sentinels is it down. –sentinelAskMasterStateToOtherSentinels • If other sentinels reply it is down. –Count them, if it is bigger than Quorum. –It is objective Down.
  • 44. How to find Redis
  • 45. INFO Command # Replication role:master connected_slaves:1 slave0:ip=127.0.0.1,port=6380,state=online,offset=1,lag=0 master_repl_offset:1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:2 repl_backlog_histlen:0
  • 46. INFO Command # Replication role:slave master_host:127.0.0.1 master_port:6379
  • 47. +redirect-master
  • 48. How to find Sentinel
  • 49. Find Sentinel Subscribe SENTINEL_HELLO_CHANNEL In Master Publish Sentinel Information In sentinelPingInstance Add New Sentinel In sentinelReceiveHelloMessages
  • 50. Using Pub/sub *4 $8 pmessage $1 * $18 __sentinel__:hello $58 127.0.0.1:26379:7d717f945afde99e6f82f825de052f17cab7e6f3:1
  • 51. Is Sentinel Really Good?
  • 52. Not Mature #1
  • 53. Sentinel Conf port 26379 sentinel monitor mymaster 127.0.0.1 6379 2 sentinel down-after-milliseconds mymaster 2000 sentinel can-failover mymaster yes sentinel parallel-syncs mymaster 1 sentinel failover-timeout mymaster 900000
  • 54. Quorum Count sentinel monitor mymaster 127.0.0.1 6379 2 We should check this count.
  • 55. Sentinel Conf sentinel down-after-milliseconds mymaster 2000 If this value is Too small, It can cause some trouble. -sdown/+sdown loop
  • 56. Compare with Zookeeper Conf tickTime=2000 dataDir=/var/zookeeper clientPort=2181 initLimit=5 syncLimit=2 server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888
  • 57. Not Mature #2
  • 58. Master Slave Sentinel
  • 59. Master Slave Sentinel
  • 60. Master Master Sentinel
  • 61. Master Master Sentinel
  • 62. Case #1 Master Sentinel Master It’s OK.
  • 63. Case #1 Master Master Sentinel
  • 64. Case #2 Master Slave Sentinel
  • 65. Case #2 Master Slave Sentinel Sentinel can’t promote Slave to Master
  • 66. Why?
  • 67. Sentinel Mechanism. When master is changed to Slave, Reconf with new Master.
  • 68. Sentinel Mechanism. When master is changed to Slave, Reset information with new Master. But, if new master failed, sentinel will not get any information from it.
  • 69. Sentinel Mechanism. if ((ri->flags & SRI_MASTER) && role == SRI_SLAVE && ri->slave_master_host) { sentinelEvent(REDIS_WARNING,"+redirect-to-master",ri, "%s %s %d %s %d", ri->name, ri->addr->ip, ri->addr->port, ri->slave_master_host, ri->slave_master_port); sentinelResetMasterAndChangeAddress(ri,ri->slave_master_host, ri->slave_master_port); return; /* Don't process anything after this event. */ }
  • 70. Not Mature #3
  • 71. If redis is down, when redis is loading data(RDB, AOF), sentinel can’t register redis.
  • 72. Conclusion
  • 73. Even though, Sentinel is not mature. It is useful.
  • 74. Redis Sentinel Tips
  • 75. 1. Subscribe all Sentinels And ignore duplicated message
  • 76. 2. Sentinel has explicit failover command sentinel mymaster failover
  • 77. But can’t select new master
  • 78. Hacking is easy. https://github.com/antirez/redis/pull/1126
  • 79. 3. Make Sentinel.conf same.
  • 80. Sentinels can’t share their status
  • 81. Deploy conf to all sentinels with Script
  • 82. And restart all sentinels.
  • 83. 4. If you want not to promote specific server to master
  • 84. Set Slave-Priority = 0
  • 85. Thank you!
  • 86. Just One more thing
  • 87. Twemprox Redis Redis-sentinel Sharded Redis
  • 88. Redis-Sentinel TwemProxy Agent http://www.codeproject.com/Articles/656965/R edis-Sentinel-TwemProxy-Agent
  • 89. Agent Pub/Sub Shard Redis Master Restart VRRP Twemproxy HA Proxy Redis Slave Redis Slave Twemproxy HA Proxy Twemproxy Shard Sentinel Redis Master Redis Slave Redis Slave