This document discusses using Redis as both a cache and primary data store. Redis is described as fast, simple, and easily scalable. It can be used as a cache for things like user profiles, with hot data stored in Redis and cold data stored elsewhere, like LMDB. Redis is also used as a primary store for tracking metrics like pageviews and purchases. The document provides examples of storing hyperloglog data in Redis to track unique counts and expiries. It also discusses techniques for load balancing and aggregating Redis data.

1. My Other Car is a Redis
Cluster
Ramon Snir
Dynamic Yield
https://www.dynamicyield.com/

2. Dynamic Yield
100% Awesomeness
Sixth-level Synergy
Great Market Cap
Bleeding Edge Innovation

6. We love Apache. We love Elastic.
That is how we see ourselves.

7. Our tech stack

8. Why Redis? ● Fast
● Simple
● Easy naïve scalability
Yes, you all know this.
Let's just give it a minute.

9. Cache or Database?

10. Redis as a Secondary Store

11. Cache
The first Redis use-case.
● Redis is very easy to set up.
● Redis is fast.
● master->slave replication
allows scaling reads very
easily.

12. Two points on Redis as a cache
Expiry
Fixed TTL (expire in 5 minutes) is easy, but it
might allow data to be cached after its validity.
Dynamic TTL (expire at a round hour, midnight,
etc.) is not much harder, but it requires knowing
when the data expires.
Adding an extra parameter to the cache key,
along with TTL, can be a good compromise.
Bucketing
Redis has a high overhead per key. This is
especially true for smaller keys. When you scale
up to use Redis on Flash it becomes even more
important.
HSET instead of SET, HGET instead of GET.
Basically free money.

13. Index
The first Redis use-case.
● Redis is very easy to set up.
● Redis is fast.
● master->slave replication
allows scaling reads very
easily.

14. Index
Fine.
● Cache stampedes can be
dangerous for your database.
● Cache misses can be
expensive for your application.
● RAM isn't free, but…
● Cross data center replication is
hard, but it's easier with Redis.

15. Redis as a Primary Store

16. User Profiles
● Pageviews
● Clicks
● Purchases
● Interactions
● Real-time updates
● Sparse data
● Very long tail

17. HBase is not good for user
profiles
● Slow random access
● Inconsistent latency
Easy solution: stuff them into Redis.
… but RAM is expensive! Only put hot data into Redis, find an
alternative for cold data.
We chose LMDB, which we build
daily from collected data with
Spark.

18. How we join profile data from LMDB and Redis
https://bit.ly/iknowtime
LMDB
Today's
Redis
● Prefer Redis data over LMDB data
● Sum counters
● NEVER use arithmetics for unique
counts
Yesterday's
Redis

19. Key expiry is a feature, not a GC trick
Set your expiry to the last moment you'll need the data.
Trust your expiry to clear stale keys.
Key expiry is cheap.
Key expiry is the only bad thing about hash-bucketing. Bucket wisely.

20. Data collection: direct or through a queue?
Direct
Faster
"Less" points of failure in the flow
Queue (Kafka & Flink)
More stable
Less likely to harm other data flows
Easier to scale

21. Incrementing msgpack counters
HINCRBY doesn't work with bucketing.
DY 💖 msgpack.
DY's bank account 💗 msgpack.
Increment has to be an atomic operation.
Lua to the rescue!

22. I promise, I won't show any more code

23. Product
Popularity
“84 people bought this product in
the past 24 hours.”
“240 people viewed this product in
the past 7 days.”

24. April 22nd, 2014
Thank you Salvatore!
… and thank you Philippe Flajolet!
HyperLogLog operations in Redis

25. Prepare the HLLs
PFADD [Nice Jacket]:hour:[Wednesday 4 p.m.] 550-00-6379
PFADD [Nice Jacket]:day:[April 25th, 2018] 550-00-6379
PFADD [Nice Jacket]:week:[4th week of April 2018] 550-00-6379
… and of course, EXPIREAT

26. Read the HLLs
PFCOUNT [Nice Jacket]:hour:[Tuesday 4 p.m.]
...
[Nice Jacket]:hour:[Wednesday 12 a.m.]
...
[Nice Jacket]:hour:[Wednesday 4 p.m.]

27. WTF
● We needed to patch.
● We couldn't redo everything.
● Our application interfaces were
well isolated.
● I don't have to be proud of it in
order for it to be cool.

28. HTTP with Load Balancing
Internal Client Load Balancer
Server
Server
Server

29. Redis and Load Balancing
Internal Client Redis
Server
Server
Server
SUBSCRIBE resp:83634
LPUSH reqs "83634:nice-key"
MESSAGE resp:83634 "good-value" PUBLISH resp:83634 "good-value"
BRPOP reqs

30. In-Email
Recommendations
NewFresh is always better.

31. In-Email
Recommendations
NewFresh is always better.
● Consistent content across
"slots"
● Consistent content between
"opens"
● Up-to-date pricing

32. settings
index
distributed
lock
decisions
store
products
catalog index
recommendations
engine

33. Thank You