Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

RedisDay London 2018 - How to Count at Scale

66 views

Published on

RedisDay London 2018

Published in: Technology
  • Be the first to comment

RedisDay London 2018 - How to Count at Scale

  1. 1. How to count at scale
  2. 2. What is Pusher Channels?
  3. 3. It’s like Redis Pub/Sub ...
  4. 4. It’s like Redis Pub/Sub ... … with millionsof concurrentclients … on the Internet … using WebSocketconnections … hosted at pusher.com … plus extra features (presence, encryption, debug console …)
  5. 5. <script src="//js.pusher.com/4.3/pusher.min.js"></script> <script> var pusher = new Pusher('abc123'); var channel = pusher.subscribe('donuts'); channel.bind('new-price', price => alert('New price: ' + price)); </script>
  6. 6. <?php $pusher = new Pusher('abc123', 'd34db33', '123456'); $pusher->trigger('donuts', 'new-price', '0.99');
  7. 7. $ pusher channels channel-info --channel donuts --app-id 180207 { "Name": "donuts", "occupied": true, "subscription_count": 2 }
  8. 8. How to count subscriptions
  9. 9. We start with clients. client client
  10. 10. These clients connect to a Channels service. The clients get “socket ids”. client 2345 Channels client 234
  11. 11. Clients create subscriptions to channels. client 2345 user-jim Channels donuts client 234 donuts btc-usd
  12. 12. The Channels box must keep a map of channel name to clients. client 2345 user-jim Channels donuts client 234 donuts btc-usd user-jim donuts btc-usd
  13. 13. client 2345 user-jim Channels donuts client 234 donuts btc-usd user-jim donuts btc-usd Let’s count subscriptions. How many subscriptions are there for the channel “donuts”? Easy: find the “donuts” key of the map, then count the number of connections. (The answer is: 2.)
  14. 14. Scaling out
  15. 15. client 2345 user-jim Channels donuts client 234 donuts btc-usd user-jim donuts btc-usd This is a monolith! It can’t be scaled in this form.
  16. 16. client 123.2345 user-jim socket 123 socket 124 socket 125 donuts client 123.234 donuts client 124.23 donuts btc-usd client 125.1 donutsbtc-usd user-jim donuts btc-usd donuts btc-usd donuts To scale out: add more processes! We call these “socket processes”.
  17. 17. client 123.2345 user-jim socket 123 socket 124 socket 125 donuts client 123.234 donuts client 124.23 donuts btc-usd client 125.1 donutsbtc-usd user-jim donuts btc-usd donuts btc-usd donuts redis-pubsub After adding more processes, where should you publish to? These socket processes need to communicate somehow. For this, we use Redis.
  18. 18. client 123.2345 user-jim socket 123 socket 124 socket 125 donuts client 123.234 donuts client 124.23 donuts btc-usd client 125.1 donutsbtc-usd user-jim donuts btc-usd donuts btc-usd donuts redis-pubsub user-jim donuts btc-usd Specifically, we use the Redis Pub/Sub feature. Hence, we call this Redis “redis-pubsub”. Notice redis-pubsub looks a lot like a socket process! Redis Pub/Sub provides a very similar mapping from “channel names” to connections.
  19. 19. Counting subscriptions across boxes
  20. 20. client 123.2345 user-jim socket 123 socket 124 socket 125 donuts client 123.234 donuts client 124.23 donuts btc-usd client 125.1 donutsbtc-usd user-jim donuts btc-usd donuts btc-usd donuts redis-pubsub user-jim donuts btc-usd Let’s count subscriptions again. How many subscriptions are there for the channel “donuts”? We can ask Redis: 127.0.0.1:6381[2]> pubsub numsub donuts 1) "donuts" 2) "3" Wrong! Socket 123 has two end-user subscriptions for “donuts”, so the answer should be “4”.
  21. 21. client 123.2345 user-jim socket 123 socket 124 socket 125 donuts client 123.234 donuts client 124.23 donuts btc-usd client 125.1 donutsbtc-usd user-jim donuts btc-usd donuts btc-usd donuts redis-pubsub user-jim donuts btc-usd Instead of counting connections to socket processes, we must keep separate counters for the number of connections to clients. Unfortunately, Redis Pub/Sub doesn’t provide this hypothetical feature, so ... 1 2 4
  22. 22. redis-main client 123.2345 user-jim socket 123 socket 124 socket 125 donuts client 123.234 donuts client 124.23 donuts btc-usd client 125.1 donutsbtc-usd user-jim donuts btc-usd donuts btc-usd donuts redis-pubsub user-jim donuts btc-usd … for subscription counts, we have another Redis we call “redis-main”. All socket processes connect to this redis, too.
  23. 23. redis-main client 123.2345 user-jim socket 123 socket 124 socket 125 donuts client 123.234 donuts client 124.23 donuts btc-usd client 125.1 donutsbtc-usd user-jim donuts btc-usd donuts btc-usd donuts redis-pubsub user-jim donuts btc-usd user-jim donuts btc-usd 1 4 2 channels:count:global redis-main keeps the per-channel subscription counts in a Redis hash “channels:count:global”. When a subscription to “donuts” is added or removed on a socket box, we do: HINCRBY channels:count:global donuts 1 HINCRBY channels:count:global donuts -1
  24. 24. What happens when a socket process goes away?
  25. 25. redis-main client 123.2345 user-jim socket 123 socket 124 socket 125 donuts client 123.234 donuts client 124.23 donuts btc-usd client 125.1 donutsbtc-usd user-jim donuts btc-usd donuts btc-usd donuts redis-pubsub user-jim donuts btc-usd user-jim donuts btc-usd 1 4 2 channels:count:global Consider this situation.
  26. 26. redis-main client 123.2345 user-jim socket 123 socket 124 socket 125 donuts client 123.234 donuts client 124.23 donuts btc-usd client 125.1 donutsbtc-usd user-jim donuts btc-usd donuts btc-usd donuts redis-pubsub user-jim donuts btc-usd user-jim donuts btc-usd 1 4 2 channels:count:global Suddenly, socket 123 goes away! Along with all of its clients ...
  27. 27. redis-main socket 124 socket 125 client 124.23 donuts btc-usd client 125.1 donuts donuts btc-usd donuts redis-pubsub donuts btc-usd user-jim donuts btc-usd 1 4 2 channels:count:global Redis-pubsub was able to clean up, but what about redis-main? Let’s count subscriptions again. How many subscriptions are there for the channel “donuts”? Not four, but two! The per-channel subscription counters are wrong! We need to subtract the two client subscriptions to “donuts” from the counter, but that number is now lost ...
  28. 28. redis-main client 123.2345 user-jim socket 123 socket 124 socket 125 donuts client 123.234 donuts client 124.23 donuts btc-usd client 125.1 donutsbtc-usd user-jim donuts btc-usd donuts btc-usd donuts redis-pubsub user-jim donuts btc-usd user-jim donuts btc-usd 1 4 2 channels:count:global user-jim donuts btc-usd 1 2 1 channels:count:123 donuts btc-usd 2 1 channels:count:124 donuts 1 channels:count:125 We must keep the per-channel connection counts for each socket process, so that when the socket process dies, its counts can be subtracted. For each socket process, we keep another Redis hash of its subscription counts. For example, socket process 123 has the hash “channels:count:123”.
  29. 29. MULTI HINCRBY channels:count:global donuts 1 HINCRBY channels:count:123 donuts 1 EXEC
  30. 30. redis-main client 123.2345 user-jim socket 123 socket 124 socket 125 donuts client 123.234 donuts client 124.23 donuts btc-usd client 125.1 donutsbtc-usd user-jim donuts btc-usd donuts btc-usd donuts redis-pubsub user-jim donuts btc-usd user-jim donuts btc-usd 1 4 2 channels:count:global user-jim donuts btc-usd 1 2 1 channels:count:123 donuts btc-usd 2 1 channels:count:124 donuts 1 channels:count:125 Is this enough for cleaning up? See what happens when socket 123 goes away ...
  31. 31. redis-main socket 124 socket 125 client 124.23 donuts btc-usd client 125.1 donuts donuts btc-usd donuts redis-pubsub donuts btc-usd user-jim donuts btc-usd 1 4 2 channels:count:global user-jim donuts btc-usd 1 2 1 channels:count:123 donuts btc-usd 2 1 channels:count:124 donuts 1 channels:count:125 Redis-pubsub is able to clean up, as before. But redis-main still doesn’t do anything! Why? Unfortunately, redis-main doesn’t know the relationship between the lost connection and the id “123”. Instead, we must detect dead socket processes ourselves, and clean up after them.
  32. 32. Detecting dead socket processes
  33. 33. redis-main client 123.2345 user-jim socket 123 socket 124 socket 125 donuts client 123.234 donuts client 124.23 donuts btc-usd client 125.1 donutsbtc-usd user-jim donuts btc-usd donuts btc-usd donuts redis-pubsub user-jim donuts btc-usd 125 123 124 17:35 17:35 17:35 socket:process:all user-jim donuts btc-usd 1 4 2 channels:count:global user-jim donuts btc-usd 1 2 1 channels:count:123 donuts btc-usd 2 1 channels:count:124 donuts 1 channels:count:125 To identify dead socket processes, we maintain a “last seen” time for each socket process id in a Redis hash. Each socket process continually updates this hash.
  34. 34. redis-main socket 124 socket 125 client 124.23 donuts btc-usd client 125.1 donuts donuts btc-usd donuts redis-pubsub donuts btc-usd 125 123 124 17:46 17:35 17:46 socket:process:all user-jim donuts btc-usd 1 4 2 channels:count:global user-jim donuts btc-usd 1 2 1 channels:count:123 donuts btc-usd 2 1 channels:count:124 donuts 1 channels:count:125 When socket 123 goes away, its “last seen” time gets older. Then, after ten minutes, we consider it dead. client 123.2345 user-jim socket 123 donuts client 123.234 donuts btc-usd user-jim donuts btc-usd user-jim
  35. 35. redis-main socket 124 socket 125 client 124.23 donuts btc-usd client 125.1 donuts donuts btc-usd donuts redis-pubsub donuts btc-usd 125 123 124 17:46 17:35 17:46 socket:process:all user-jim donuts btc-usd 1 4 2 channels:count:global user-jim donuts btc-usd 1 2 1 channels:count:123 donuts btc-usd 2 1 channels:count:124 donuts 1 channels:count:125 But still nothing happens! Redis-main doesn’t know about our socket process list. Instead, we must check it ourselves ...
  36. 36. redis-main socket 124 socket 125 client 124.23 donuts btc-usd client 125.1 donuts donuts btc-usd donuts redis-pubsub donuts btc-usd cleanup-dead 125 123 124 17:46 17:35 17:46 socket:process:all user-jim donuts btc-usd 1 4 2 channels:count:global user-jim donuts btc-usd 1 2 1 channels:count:123 donuts btc-usd 2 1 channels:count:124 donuts 1 channels:count:125 We do this with a separate process, “cleanup-dead”.
  37. 37. redis-main socket 124 socket 125 client 124.23 donuts btc-usd client 125.1 donuts donuts btc-usd donuts redis-pubsub donuts btc-usd cleanup-dead 125 124 17:46 17:46 socket:process:all donuts btc-usd 2 1 channels:count:global donuts btc-usd 2 1 channels:count:124 donuts 1 channels:count:125 Cleanup-dead runs a job every 4 minutes. If a socket has been inactive for more than 10 minutes, it considers it dead. It decrements the socket’s counts from the global counts. Then it deletes the socket’s counts. Then it deletes the socket from the “last seen” times.
  38. 38. TESTAMENT SET EVALSHA 123abc 1 socket:123
  39. 39. redis Client C Local set A Partial global index Application code Client B Local set Partial global index Application code Set C Set B Client A Application State A Global indexes: A + B + C Application code Important State A Global indexes: A + B + C
  40. 40. We’re hiring! pusher.com/careers jim@pusher.com

×