Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Learning redis

54 views

Published on

There are many reasons to appreciate Redis: advanced data structures, high performance and scalability. It might be most popular in-memory data structure store. However, we’ve faced quite few challenges when developing the world's largest vacation rental search engine HomeToGo. I will talk about pushing technology to the limits and things beyond official documentation.

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Learning redis

  1. 1. Learning Redis The hard way… Audrius Bugas Chief architect @ HomeToGo audrius@hometogo.com
  2. 2. * Kiekvienas komandos narys svarbiausias
  3. 3. 4
  4. 4. redis.io “Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius queries” 5
  5. 5. Stage #1 •5-6 application servers •Single dedicated redis server •Graceful degradation •RDB persistence 6
  6. 6. Stage #2 •> 20 application servers •Search workers starts writing data to redis 7
  7. 7. Problem #1 - system slowdown •CPU [OK] •Disk IOPS [OK] •Redis response time > 200ms [Slow] •Connection rate [High] 8
  8. 8. Solution - redis sentinel 9
  9. 9. Solution - stop using databases 10 • SELECT is still an operations • flushDB actually ended up bad. Twice…
  10. 10. Problem #2 - sentinel down •CPU [OK] •Disk IOPS [OK] •Memory [OK] •Connection rate [High] 11
  11. 11. https://github.com/nrk/predis/pull/527 Really???
  12. 12. Solution - extend 13
  13. 13. Problem #3 - write operations timeout • CPU [OK] • Disk IOPS [OK] • Connection rate [OK] • Redis response time [OK] • Write operations timeouts rate > 5% [High] 14
  14. 14. Solution - use reds pipelining 15
  15. 15. Solution - use reds pipelining 16
  16. 16. Write operations timeouts still there… • Redis write operation performance 6ms vs 10ms before [OK] • Write operations timeouts rate > 5% [High] 17
  17. 17. Solution - disconnect when done
  18. 18. Problem #4- periodical redis response time spike • CPU [OK] • CPU spikes. ~3 h intervals [Warning] • Connection rate [OK] • Redis response time spikes > 500ms. ~3 h intervals [Warning] 19
  19. 19. Solution - stop using KEYS, SCAN on prod • On large databases operations leads to blocking redis • Change key salt instead of trying find and delete records • Always use ttl. Redis has pretty good eviction policies 20
  20. 20. Problem #5 - periodical redis response time spike • CPU [OK] • CPU spikes. ~1 min intervals [Warning] • Connection rate [OK] • Redis response time spikes > 400ms. ~1 min intervals [Warning] 21
  21. 21. RDB snapshotting 22
  22. 22. Solution - use redis cluster • Smaller resident set size • Complicated redis sentinel setup operations • Exceptions while syncing • Timeouts while syncing 23
  23. 23. 24
  24. 24. Cluster gotchas • Choosing right node is entirely client responsibility • Cluster delivers redirects in case wrong node used for operation • Potential connection rate problem again!!! • Map caching/updating is also client responsibility • Multi-key operation problem 25
  25. 25. phpredis extension
  26. 26. Problem #6 - WTF??? 27
  27. 27. “This looks like a classic proxy problem” 28
  28. 28. twemproxy 29
  29. 29. twemproxy https://github.com/twitter/twemproxy • Does not support cluster • Implements own sharding • No rebalancing
  30. 30. Let’s build our own proxy!
  31. 31. Proxy features 32 • FAST Response time 7ms vs 4ms without proxy • jruby • multithreaded • Supports Redis Cluster • Auto hash map refresh • For now only commands subset
  32. 32. https://github.com/hometogo/redis-proxy
  33. 33. Thank you Audrius Bugas Chief architect @ HomeToGo audrius@hometogo.com www.linkedin.com/in/audriusb

×