Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2h34ibv. Yao Yue introduces Pelikan, a framework to implement distributed caches such as Memcached and Redis. Filmed at qconsf.com. Yao Yue is a Distributed Systems Engineer Working on Cache at Twitter.

1. CURB TAIL LATENCY
IN-MEMORY CACHING:
WITH PELIKAN

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
pelikan

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. ABOUT ME
• 6 years at Twitter, on cache
• maintainer of Twemcache (OSS), Twitter’s Redis fork
• operations of thousands of machines
• hundreds of (internal) customers
• Now working on Pelikan, a next-gen cache framework to replace the above @twitter
• Twitter: @thinkingfish

5. CACHE PERFORMANCE
THE PROBLEM:

6. CACHE
RULES
EVERYTHING
AROUND
ME
CACHE DB
SERVICE

7. CACHE
RUINS
EVERYTHING
AROUND
ME
CACHE DB
SERVICE
😣
SENSITIVE!
😣

8. GOOD CACHE PERFORMANCE
=
PREDICTABLE LATENCY

9. GOOD CACHE PERFORMANCE
=
PREDICTABLE TAIL LATENCY

10. “MILLIONS OF QPS PER MACHINE”
“SUB-MILLISECOND LATENCIES”
“NEAR LINE-RATE THROUGHPUT”
…
KING OF PERFORMANCE

11. “USUALLY PRETTY FAST”
“HICCUPS EVERY ONCE IN A WHILE”
“TIMEOUT SPIKES AT THE TOP OF THE HOUR”
“SLOW ONLY WHEN MEMORY IS LOW”
…
GHOSTS OF PERFORMANCE

12. I SPENT FIRST 3 MONTHS AT TWITTER
LEARNING CACHE BASICS…
…AND THE NEXT 5 YEARS CHASING
GHOSTS

14. MINIMIZE
INDETERMINISTIC
BEHAVIOR
CONTAIN GHOSTS
=

15. HOW?
IDENTIFY
AVOID MITIGATE

16. CACHING IN DATACENTER
A PRIMER:

17. CONTEXT
• geographically centralized
• highly homogeneous network
• reliable, predictable infrastructure
• long-lived connections
• high data rate
• simple data/operations

18. MAINLY:
REQUEST → RESPONSE
CACHE IN PRODUCTION
INITIALLY:
CONNECT
ALSO (BECAUSE WE ARE ADULTS):
STATS, LOGGING, HEALTH CHECK…

19. CACHE: BIRD’S VIEW
HOST
event-driven
server
protocol
data
storage
OS
network infrastructure

20. HOW DID WE UNCOVER THE
UNCERTAINTIES?

21. ”
“
BANDWIDTH UTILIZATION WENT WAY
UP, BUT REQUEST RATE WAY DOWN.

22. SYSCALLS

23. CONNECTING IS SYSCALL-HEAVY
read
event
accept config register4+ syscalls

24. REQUEST IS SYSCALL-LIGHT
read
event
IO
(read)
post-
read
parse process compose
write
event
IO
(write)
post-
write
3 syscalls*
*: event loop returns multiple read events at once, I/O syscalls can be further amortized by batching/pipelining

25. TWEMCACHE IS MOSTLY SYSCALLS
• 1-2 µs overhead per call
• dominate CPU time in simple cache
• What if we have 100k conns / sec?
source

26. CONNECTION STORM
culprit:

27. ”
“
…TWEMCACHE RANDOM HICCUPS,
ALWAYS AT THE TOP OF THE HOUR.

28. DISK
⏱
cache
tworker
logging
cron job “x”
I/O

29. BLOCKING I/O
culprit:

30. ”
“
WE ARE SEEING SEVERAL “BLIPS”
AFTER EACH CACHE REBOOT…

31. LOCKING FACTS
• ~25ns per operation
• more expensive on NUMA
• much more costly when contended
source

32. MEMCACHE RESTART
…
EVERYTHING IS FINE
REQUESTS SUDDENLY GET SLOW/TIMED-OUT
CONNECTION STORM
CLIENTS TOPPLE
SLOWLY RECOVER
(REPEAT A FEW TIMES)
…
STABILIZE
A TIMELINE
lock!
lock!

33. LOCKING
culprit:

34. ”
“
HOSTS WITH LONG RUNNING CACHE
TRIGGERS OOM WHEN LOAD SPIKE.

35. ”
“
REDIS INSTANCES WERE KILLED BY
SCHEDULER.

36. MEMORY
culprit:

37. CONNECTION STORM
BLOCKING I/O
LOCKING
MEMORY
SUMMARY

38. HOW TO MITIGATE?

39. DATA PLANE,
CONTROL PLANE

40. PUT OPERATIONS OF DIFFERENT NATURE / PURPOSE
ON SEPARATE THREADS
HIDE EXPENSIVE OPS

41. LISTENING (ADMIN CONNECTIONS)
STATS AGGREGATION
STATS EXPORTING
LOG DUMP
SLOW: CONTROL PLANE

42. FAST: DATA PLANE / REQUEST
read
event
IO
(read)
post-
read
parse process compose
write
event
IO
(write)
post-
write
:
tworker

43. FAST: DATA PLANE / CONNECT
read
event
accept config
read
event
register
:
tworker
:
tserver
dispatch

44. LATENCY-ORIENTED THREADING
tworker
tserver tadmin
new
connection
logging,
stats update
logging,
stats update
REQUESTS
CONNECTS OTHER

45. WHAT TO AVOID?

46. LOCKING

47. WHAT WE KNOW
• inter-thread communication in cache
• stats
• logging
• connection hand-off
• locking propagates blocking/delay
between threads
tworker
tserver tadmin
new
connection
logging,
stats update
logging,
stats update

48. MAKE STATS UPDATE LOCKLESS
LOCKLESS OPERATIONS
w/ atomic instructions

49. MAKE LOGGING WAITLESS
LOCKLESS OPERATIONS
RING/CYCLIC BUFFER
read
position
writer
reader
write
position

50. MAKE CONNECTION HAND-OFF LOCKLESS
LOCKLESS OPERATIONS
RING ARRAY
read
position
writer
reader
write
position
… …

51. MEMORY

52. WHAT WE KNOW
• alloc-free cause fragmentation
• internal vs external fragmentation
• OOM/swapping is deadly
• memory alloc/copy relatively
expensive
source

53. AVOID EXTERNAL FRAGMENTATION
CAP ALL MEMORY RESOURCES
PREDICTABLE FOOTPRINT

54. REUSE BUFFER
PREALLOCATE
PREDICTABLE RUNTIME

55. PELIKAN CACHE
IMPLEMENTATION

56. WHAT IS PELIKAN CACHE?
• (Datacenter-) Caching framework
• A summary of Twitter’s cache ops
• Perf goal: deterministically fast
• Clean, modular design
• Open-source
waitless logging lockless metrics composed config
channels buffers timer alarm
poo
ling
streams events
data store
parse/compose/tracedata model
request response
server
orchestration
threading
common
core
cache
process
pelikan.io

57. A COMPARISON
PERFORMANCE DESIGN DECISIONS
latency-oriented
threading
Memory/
fragmentation
Memory/
buffer caching
Memory/
pre-allocation, cap
locking
Memcached partial internal partial partial yes
Redis no->partial external no partial no->yes
Pelikan yes internal yes yes no

58. MEMCACHED REDIS
TO BE FAIR…
• multiple worker threads
• binary protocol + SASL
• rich set of data structures
• master-slave replication
• redis-cluster
• modules
• tools

59. ALWAYS FAST
THE BEST CACHE IS…

60. QUESTIONS?

61. Watch the video with slide synchronization on
InfoQ.com!
https://www.infoq.com/presentations/pelikan