Caching can bring speed to the slow systems in your enterprise. In this session, we'll explore how to avoid expensive computations and reduce database (or other systems) load with caching. Considerations include when to cache (cache policy), how to cache, and when/how to evict. This session will also explore and discuss problems and gotchas like cache sizing, eviction mistakes, and where your cache should live. Finally, we'll look at how caching is implemented in the "memory-first" Couchbase architecture.

3. C.R.E.A.M. get the memory (dollar dollar bill ya'll)
January 2024
Matthew D. Groves | DevRel Engineer
aka “Passionate Bravo”
Cache Rules Everything
Around Me

4. 4
Latency Numbers Every Programmer Should Know
https://stackoverflow.com/questions/4087280/approximate-cost-to-access-various-caches-and-main-memory/4087315#4087315

5. 5
The Need for Speed
https://www.slideshare.net/devonauerswald/walmart-pagespeedslide

6. 6
Cache Rules Everything Around Me
Dollar dollar bill ya'll
Android apps iPhone apps
CPU cache
CDNs
Web browser
DNS

7. 01/
02/
03/
04/
05/
06/
What is caching?
Use Cases
Reading from Cache
Writing into Cache
Cache Invalidation
Gotchas
Agenda

8. 8
ho am I?
• Matthew D. Groves
• Microsoft MVP, author at Pluralsight, Manning, Apress
• DevRel Engineer at Couchbase
• https://twitter.com/mgroves
• https://www.linkedin.com/in/mgroves/
• C# Advent: https://csadvent.christmas/

9. 9
1 hat is Caching?

10. 10
Caching
Collection of duplicated data to serve future requests faster.
• First formally described by Maurice Wilkes in 1965
• "Slave Memories and Dynamic Storage Allocation" https://bit.ly/WilkesPaper
What is Caching?
Key Data
1 Proteck Ya Neck
2 C.R.E.A.M.
3 Brooklyn Zoo
4 Criminology
… …
Key Data
2 C.R.E.A.M.
4 Criminology
Records Cache

11. 11
Powerpoint as a Cache
What is Caching?
Key Value
What year was Wu-Tang
Clan Formed?
1992

12. 12
2 Caching Use Cases

13. 13
Caching Use Cases
What is Caching used for?

14. 14
Caching Use Case 1: Faster Database Access
• Databases can rely heavily on disk
• Reading the same data over again
could mean accessing the disk over
again
• Store frequently accessed data in a
cache, to reduce pressure on
database/disk

15. 15
Caching Use Case 1: Faster Database Access
DATABASE
APP
LAN
CACHE
CACHE
CACHE

16. 16
Caching Use Case 2: Faster Web Browsing
• Any given URL could result in
hundreds of files that are loaded
over HTTP
• A single JS file may be loaded by
every page on a site.
• Cache static assets (JS, images,
sound, icons, etc) on a user's laptop
to avoid loading them over the
network every time

17. 17
Caching Use Case 2: Faster Web Browsing
CACHE

18. 18
Caching Use Case 3: Efficient API Use
• API usage may involve an internet
call (with latency)
• API calls may cost per use, or may
be limited
• Storing results in cache may help
reduce latency AND reduce costs

19. 19
Caching Use Case 3: Efficient API Use
API APP
LAN/WAN
CACHE
CACHE
CACHE

20. 20
Caching Use Case 4: Microservice Availability
• Retrieving data from other
processes
• Latency is secondary concern

21. 21
Caching Use Case 4: Microservice Availability
CACHE

22. 22
Cache Rules Everything Around Me
Dollar dollar bill ya'll
Disconnected Availability
Slow
Repetition
Constrained
Costs

23. 23
3 Reading From Cache

24. 24
Using a Cache 101
Your Code
Key Value
A Protect Ya Neck
B C.R.E.A.M.
C Brooklyn Zoo
Cache
Key Value
A Protect Ya Neck
B C.R.E.A.M.
C Brooklyn Zoo
D Criminology
E Triumph
F Uzi
G Method Man
H Wu-Tang Clan Ain't . . .
. . . . . .
Database
1. Lookup "A"
2. Lookup "B"
3. Lookup "C"
HIT!
HIT!
MISS!
4. Lookup "C" again?
WALK OF SHAME

25. 25
4 Writing Into Cache

26. 26
What about writes?
• Data is duplicated, stored in cache and record
• The system is responsible for keeping the cache updated
• Commons methods:
• write-through
• write-around
• write-back
Stuff needs to get in the cache somehow

27. 27
write-through
1. System gets a new/updated piece of data
2. Write to cache and record "at the same time"
a) Write to record
b) If that didn't succeed, the write failed.
c) Write to cache
d) If that didn't succeed, the write failed. Rollback (a).
e) If the system reaches this point, the write succeeded!
Why or why not use this?

28. 28
write-around
1. System gets a new/updated piece of data
2. Write the data to the record ONLY
3. Invalidate the record in the cache
Why or why not use this?

29. 29
write-back
1. System gets a new/updated piece of data
2. Write the data to the cache.
3. Update the record asynchronously (don't wait on it to finish)
4. If the cache write succeeded, then consider the write succeeded.
Why or why not use this?

30. 30
Cache Writing Methods
Method Write performance Write integrity Best for…?
write-through
➖ ➕ High % reads
write-around
➖ ➕ High % read later
write-back
➕ ❔ Mixed reads / writes

31. 31
Couchbase: A write-back implementation
A database with a built-in managed cache
https://docs.couchbase.com/server/current/learn/buckets-memory-and-storage/memory-and-storage.html
Other
Couchbase
Server
nodes

32. 32
Couchbase and Writing to the Cache
await collection.InsertAsync("A", new { title = "Protect Ya Neck" });
await collection.InsertAsync("A", new { title = "Protect Ya Neck" }, options =>
{
options.Durability(DurabilityLevel.None);
// OR: options.Durability(DurabilityLevel.Majority);
// OR: options.Durability(DurabilityLevel.MajorityAndPersistToActive);
// OR: options.Durability(DurabilityLevel.PersistToMajority);
});

33. 33
Durability (with 3 replicas)
None
Memory
Disk
Node 1 Node 2 Node 3
✔
⚠
⚠
⚠
⚠
⚠
Node 4
⚠
⚠

34. 34
Durability (with 3 replicas)
Majority
Memory
Disk
Node 1 Node 2 Node 3
✔
⚠
⚠
⚠
✔
⚠
Node 4
✔
⚠

35. 35
Durability (with 3 replicas)
MajorityAndPersistToActive
Memory
Disk
Node 1 Node 2 Node 3
✔
✔
⚠
⚠
✔
⚠
Node 4
✔
⚠

36. 36
Durability (with 3 replicas)
PersistToMajority
Memory
Disk
Node 1 Node 2 Node 3
✔
✔
⚠
⚠
✔
✔
Node 4
✔
✔

37. 37
Latency Numbers Every Programmer Should Know
https://stackoverflow.com/questions/4087280/approximate-cost-to-access-various-caches-and-main-memory/4087315#4087315

38. 38
5 Cache Invalidation

39. 39
Powerpoint as a Cache
Key Value
What year was Wu-Tang Clan
Formed?
1992
What was Wu-Tang's first album
called?
Enter the Wu-Tang
What year was Maurice Wilkes
born?
1913
What year did Maurice Wilkes
die?
2010
What is the most recent Wu-
Tang studio album?
Once Upon a Time in Shaolin

40. 40
Cache Invalidation is Hard
There are only two hard things in Computer Science: cache
invalidation and naming things.
-- Phil Karlton
How to decide what to invalidate (what data to "evict"):
cache policy https://www.nndb.com/people/400/000031307/

41. 41
Cache Policy 1: LRU (Least Recently Used)
Key How new?
A 0
B 1 3
C 2
D 4
Cache max size: 3
Order of reads: A, B, C, B, D
X
---------------------------------------------------------------------------------

42. 42
Cache Policy 2: NRU (Not Recently Used)
Lower scores evicted (at random)
Key Referenced? Modified? Score
A 0 0 0
B 0 0 0
C 0 1 1
D 1 0 2
E 1 1 3

43. 43
Cache Policy 3: None
Don't evict anything

44. 44
More Cache Policies
• RR – random replacement
• FIFO and LIFO – treat a cache like a queue
• MRU – MOST recently used
• LFU – least frequently used
• Belady's algorithm – not possible to implement
• Machine learning
• https://en.wikipedia.org/wiki/Cache_replacement_policies

45. 45
6 Gotchas

46. 46
Gotcha 1: Cache Size
• Problem: too small or too big cache
• Symptom: High turnover and churn (many evictions) or long lifetimes (few
evictions)
• Solutions:
• Monitoring
• Size the cache
• "Value" vs "Full" eviction

47. 47
Gotcha 2: "Close" Caching
• Problem: running the cache on the same machine as the application
• Symptom: redundancy, hot spots, inconsistency, availability
• Solutions:
• Use a distributed cache
• Running on its own machine(s) / cluster
• Couchbase, of course!

48. 48
Gotcha 3: Bad Eviction Policy
• Problem: using the wrong eviction policy
• Symptom: lots of overhead, cache checking
• Solutions:
• Don't Write an Eviction Policy (or better yet don't write a Cache)
• Benchmarking: "hit ratio" and "latency"
• "Hit ratio" – how often a hit vs miss (e.g. "45% hit ratio")
• "Latency" – how long it takes from request to response (e.g. 10μs latency)

49. THANK YOU

50. 50
References
• https://www.geeksforgeeks.org/not-recently-used-nru-page-replacement-algorithm/
• https://shahriar.svbtle.com/Understanding-writethrough-writearound-and-writeback-
caching-with-python
• https://en.wikipedia.org/wiki/Maurice_Wilkes
• https://safari.ethz.ch/digitaltechnik/spring2021/lib/exe/fetch.php?media=wilkes.pdf
• https://stackoverflow.com/a/4087315/40015
• https://www.techtarget.com/searchstorage/definition/cache
• https://en.wikipedia.org/wiki/Cache_replacement_policies
• https://docs.couchbase.com/dotnet-sdk/current/howtos/kv-operations.html#durability
• https://docs.couchbase.com/dotnet-sdk/current/concept-docs/durability-replication-failure-
considerations.html#durable-writes