1. Learning Redis
The hard way…
Audrius Bugas
Chief architect @ HomeToGo
audrius@hometogo.com

2. * Kiekvienas komandos narys svarbiausias

4. 4

5. redis.io
“Redis is an open source (BSD licensed), in-memory data structure
store, used as a database, cache and message broker. It supports
data structures such as strings, hashes, lists, sets, sorted sets with
range queries, bitmaps, hyperloglogs and geospatial indexes with
radius queries”
5

6. Stage #1
•5-6 application servers
•Single dedicated redis server
•Graceful degradation
•RDB persistence
6

7. Stage #2
•> 20 application servers
•Search workers starts writing data to
redis
7

8. Problem #1 - system slowdown
•CPU [OK]
•Disk IOPS [OK]
•Redis response time > 200ms [Slow]
•Connection rate [High]
8

9. Solution - redis sentinel
9

10. Solution - stop using databases
10
• SELECT is still an
operations
• flushDB actually ended
up bad. Twice…

11. Problem #2 - sentinel down
•CPU [OK]
•Disk IOPS [OK]
•Memory [OK]
•Connection rate [High]
11

12. https://github.com/nrk/predis/pull/527
Really???

13. Solution - extend
13

14. Problem #3 - write operations timeout
• CPU [OK]
• Disk IOPS [OK]
• Connection rate [OK]
• Redis response time [OK]
• Write operations timeouts rate > 5% [High]
14

15. Solution - use reds pipelining
15

16. Solution - use reds pipelining
16

17. Write operations timeouts still there…
• Redis write operation performance 6ms vs
10ms before [OK]
• Write operations timeouts rate > 5% [High]
17

18. Solution - disconnect when done

19. Problem #4- periodical redis response time
spike
• CPU [OK]
• CPU spikes. ~3 h intervals [Warning]
• Connection rate [OK]
• Redis response time spikes > 500ms. ~3 h
intervals [Warning]
19

20. Solution - stop using KEYS, SCAN on prod
• On large databases operations leads to
blocking redis
• Change key salt instead of trying find and
delete records
• Always use ttl. Redis has pretty good eviction
policies
20

21. Problem #5 - periodical redis response time
spike
• CPU [OK]
• CPU spikes. ~1 min intervals [Warning]
• Connection rate [OK]
• Redis response time spikes > 400ms. ~1 min
intervals [Warning]
21

22. RDB snapshotting
22

23. Solution - use redis cluster
• Smaller resident set size
• Complicated redis sentinel setup operations
• Exceptions while syncing
• Timeouts while syncing
23

24. 24

25. Cluster gotchas
• Choosing right node is entirely client responsibility
• Cluster delivers redirects in case wrong node used for
operation
• Potential connection rate problem again!!!
• Map caching/updating is also client responsibility
• Multi-key operation problem
25

26. phpredis extension

27. Problem #6 - WTF???
27

28. “This looks like a classic proxy problem”
28

29. twemproxy
29

30. twemproxy
https://github.com/twitter/twemproxy
• Does not support cluster
• Implements own sharding
• No rebalancing

31. Let’s build our own proxy!

32. Proxy features
32
• FAST Response time 7ms vs 4ms without proxy
• jruby
• multithreaded
• Supports Redis Cluster
• Auto hash map refresh
• For now only commands subset

33. https://github.com/hometogo/redis-proxy

34. Thank you
Audrius Bugas
Chief architect @ HomeToGo
audrius@hometogo.com
www.linkedin.com/in/audriusb