The document discusses Ristretto, a high performance concurrent cache for Go. It aims to have fast access times, high concurrency and resistance to contention, while bounding memory usage. It uses TinyLFU for admission control and sampled LFU for eviction. Stores are sharded across mutex-wrapped maps for concurrency. Hashing uses memhash for speed. Gets and Sets use lossy ring buffers and channels to balance performance and scalability across goroutines. Metrics avoid false sharing overhead. Usage involves creating a cache, setting and getting values, and deleting values.

1. Ristretto
Viney
2019/10/14 1

2. Ristretto
A High Performance, Concurrent, Memory-Bound Go
cache
2019/10/14 2

3. Why?
Dgraph need it
2019/10/14 3

4. Workaround
Groupcache’s LRU + mutex locks
!
2019/10/14 4

5. Coffeine
2019/10/14 5

6. Ristretto
Better Half of Espresso
2019/10/14 6

7. Requirement
4 Concurrent
4 High cache-hit ratio
4 Memory-bounded (limit to configurable max memory
usage)
4 Scale well as the number of cores and goroutines
increases
2019/10/14 7

8. Guiding Principles
2019/10/14 8

9. 4 Fast Accesses
4 High Concurrency and Contention Resistance
4 Memory Bounding
2019/10/14 9

10. Memory Bounding
Key Cost
2019/10/14 10

11. Key Cost
4 An infinitely large cache is practically impossible
4 Attach a cost to every key-value
4 Heavy item could displace many lightweight items
2019/10/14 11

12. Memory Bounding
Admission Policy via TinyLFU
2019/10/14 12

13. CountMin Sketch
4 DoorKeeper: BloomFilter
2019/10/14 13

14. TinyLFU
2019/10/14 14

15. Eviction Policy via Sampled LFU
// Snippet from the Admission and Eviction Algorithm
incHits := p.admit.Estimate(key)
for ; room < 0; room = p.evict.roomLeft(cost) {
sample = p.evict.fillSample(sample)
minKey, minHits, minId := uint64(0), int64(math.MaxInt64), 0
for i, pair := range sample {
if hits := p.admit.Estimate(pair.key); hits < minHits {
minKey, minHits, minId = pair.key, hits, i
}
}
if incHits < minHits {
p.stats.Add(rejectSets, key, 1)
return victims, false
}
p.evict.del(minKey)
sample[minId] = sample[len(sample)-1]
sample = sample[:len(sample)-1]
victims = append(victims, minKey)
}
2019/10/14 15

16. Fast Access
Store
2019/10/14 16

17. Store
4 Due to CSP
4 No thread-local storage in Go
4 No lockless concurrent hashmap
2019/10/14 17

18. Store (cont.)
4 sync.Map
4 sharded mutex-wrapped Go maps
4 256 shards in 64-core server
2019/10/14 18

19. Fast Access
Hash
2019/10/14 19

20. Hash
4 runtime.memhash
4 non-persistent cache.
BenchmarkMemHash-32 200000000 8.88 ns/op
BenchmarkFarm-32 100000000 17.9 ns/op
BenchmarkSip-32 30000000 41.1 ns/op
BenchmarkFnv-32 20000000 70.6 ns/op
2019/10/14 20

21. Hash (cont.)
4 stored key
4 which shard the key should go into
2019/10/14 21

22. Concurrency
Contention Resistance
2019/10/14 22

23. balancing the performance and scalability of the cache
across goroutines
4 BP-Wrapper
4 prefetching
4 batching
2019/10/14 23

24. Concurrency
Get
2019/10/14 24

25. Get
4 striped, lossy ring buffers
4 sync.Pool
2019/10/14 25

26. Get (cont.)
AddToLossyBuffer(key):
stripe := b.pool.Get().(*ringStripe)
stripe.Push(key)
b.pool.Put(stripe)
Once buffer fills up, push to channel:
select {
case p.itemsCh <- keys:
p.stats.Add(keepGets, keys[0], uint64(len(keys)))
return true
default:
p.stats.Add(dropGets, keys[0], uint64(len(keys)))
return false
}
p.itemCh processing:
func (p *tinyLFU) Push(keys []uint64) {
for _, key := range keys {
p.Increment(key)
}
}
2019/10/14 26

27. Concurrency
Set
2019/10/14 27

28. Set
4 channel
select {
case c.setBuf <- &item{key: hash, val: val, cost: cost}:
return true
default:
// drop the set and avoid blocking
c.stats.Add(dropSets, hash, 1)
return false
}
2019/10/14 28

29. Metrics
4 prevent False Sharing
4 10% overhead
Add:
valp := p.all[t]
// Avoid false sharing by padding at least 64 bytes of space between two
// atomic counters which would be incremented.
idx := (hash % 25) * 10
atomic.AddUint64(valp[idx], delta)
Read:
valp := p.all[t]
var total uint64
for i := range valp {
total += atomic.LoadUint64(valp[i])
}
return total
2019/10/14 29

30. How to
func main() {
cache, err := ristretto.NewCache(&ristretto.Config{
NumCounters: 1e7, // Num keys to track frequency of (10M).
MaxCost: 1 << 30, // Maximum cost of cache (1GB).
BufferItems: 64, // Number of keys per Get buffer.
})
if err != nil {
panic(err)
}
cache.Set("key", "value", 1) // set a value
// wait for value to pass through buffers
time.Sleep(10 * time.Millisecond)
value, found := cache.Get("key")
if !found {
panic("missing value")
}
fmt.Println(value)
cache.Del("key")
}
2019/10/14 30

31. Search
2019/10/14 31

32. Database
2019/10/14 32

33. Looping
2019/10/14 33

34. CODASYL
2019/10/14 34

35. Mixed: 25% Writes, 75% Reads
2019/10/14 35

36. Read: 100% Reads
2019/10/14 36

37. Write: 100% Writes
2019/10/14 37

38. TODO2019/10/14 38

39. Q&A2019/10/14 39