Three engineers, at various points, each take their own approach adding Rust to a C codebase, each being more and more ambitious. I initially just wanted to replace the server’s networking and event loop with an equally fast Rust implementation. We’d reuse many core components that were in C and just call into them from Rust. Surely it wouldn’t be that much code… Pelikan is Twitter’s open source and modular framework for in-memory caching, allowing us to replace Memcached and Redis forks with a single codebase and achieve better performance. At Twitter, we operate hundreds of cache clusters storing hundreds of terabytes of small objects in memory. In-memory caching is critical, and demands performance, reliability, and efficiency. In this talk, I’ll share my adventures in working on Pelikan and how rewriting it in Rust can be more than just a meme.

1. Brought to you by
Whoops!
I Rewrote It in Rust
Brian Martin (he/him)
Software Engineer at Twitter

2. Brian Martin (he/him)
Software Engineer at Twitter
■ At Twitter for 7 years, using Rust for 6 years
■ Open source benchmarking, telemetry agent, cache server
■ p99, p999, p9999 are all important
■ Volunteer Search and Rescue

3. Caching
■ distributed key-value storage
● Memcached
● Redis
■ volatile
● expiration (TTL)
● eviction
■ protect databases / backend services

4. Pelikan
■ open source caching framework
● https://pelikan.io
● https://github.com/twitter/pelikan
■ predictable performance
■ single codebase, multiple
solutions

5. Use Rust to:
■ add TLS support to Pelikan
■ match performance of C implementation

6. Rust in Pelikan
■ C server with Rust storage library - 2018
● quick development
● first commit of Rust into the project
■ Rust pingserver & Rust Twemcache server with C storage - 2019
● buffers, logging, metrics, parsers, … all still in C
● showed we could glue it all together

7. Performance Testing
■ maximize throughput
■ latency @ p999 < 1 ms

8. p999 / 99.9th percentile
■ fanout = 1:
● 0.1% of requests slower than p999

9. p999 / 99.9th percentile
■ fanout = 1:
● 0.1% of requests slower than p999
■ fanout = 10:
● 1% of requests slower than p999

10. p999 / 99.9th percentile
■ fanout = 1:
● 0.1% of requests slower than p999
■ fanout = 10:
● 1% of requests slower than p999
■ fanout = 100:
● 10% of requests slower than p999

11. p999 / 99.9th percentile
■ fanout = 1:
● 0.1% of requests slower than p999
■ fanout = 10:
● 1% of requests slower than p999
■ fanout = 100:
● 10% of requests slower than p999
Tail latency occurs more often than you think!

12. Pelikan Twemcache in Rust: 2019
■ Memcached compatible
■ Tokio / async for networking
■ uses C storage library
Can we use this to add TLS?

13. Performance Problems
■ Throughput 10-15% slower
● need more instances to match throughput requirements
■ Latency 25-30% higher (@ p999 / 99.9th Percentile)
● could cause timeouts/errors in production

14. Pelikan Twemcache C vs Rust

15. Pingserver

16. Pingserver
% telnet 127.0.0.1 12321
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
PING
PONG
PING
PONG
QUIT
Connection closed by foreign host.

17. Why Pingserver?
■ prove out performance of design
■ buffers / parsers / metrics / logging
■ can add TLS support

18. Pingserver Rewrite
■ Replace “tokio” with “mio”
● switch from async runtime to epoll event loop
■ Replace C components with Rust
● metrics/logging/buffers/…

19. Pingserver C vs Rust

20. Pelikan Twemcache

21. Twemcache Prototype
■ Memcache protocol compatible server
● temporary storage based on std::collections::HashMap
■ benchmark
● performance looked promising
■ real storage
● switch to the C library

22. Oh no!
■ Metrics and logging caused issues wrapping the C library

23. Rewrite it in Rust?

24. No. Don’t do it.
■ self-referential data structures
■ memory allocations!
■ linked lists!
■ pointers everywhere!
■ unsafe { }

25. But… maybe yes?
■ start with single-threaded
■ only hashtable in Rust?
■ use C slab allocator?

26. Whoops! All Rust
■ implemented new research storage design
● easier to port to Rust
■ single-threaded
● easy mode
■ managing all memory allocations in Rust

27. Rust Segment Storage
Pros
■ 100% Rust
■ Re-write helped improve code
■ Added new ideas to the design
Cons
■ ~2 months of work
■ Not 100% feature complete yet.

28. Rust Benefits
“empowering everyone to build reliable and efficient software”
■ high performance
■ code with confidence in reliability
■ awesome language features and tools
■ zero cost abstraction

29. Rust Benefits
“The Rust implementation finally made Pelikan modules what they ought to be, but
couldn’t before, due to the limitation of the C language. This feels exactly right and
is just as fast”
- Yao (@thinkingfish)

30. Modular
// Segcache is Memcache protocol + Seg storage
type Parser = MemcacheRequestParser;
type Request = MemcacheRequest;
type Response = MemcacheResponse;
type Storage = Seg;
...
let process_builder = ProcessBuilder::<Storage,
Parser, Request, Response>::new(...);

31. Rust has awesome tools!
■ cargo
■ fuzzers
■ benchmarks
■ rustfmt
■ clippy

32. Pelikan Twemcache C vs Segcache Rust

33. Memcached vs Segcache Rust

34. Memcached vs Segcache Rust
Memcached has higher
latency!

35. Next Steps
Path to Production
■ feature complete
■ more testing
■ production canary
■ deployment
Further Work
■ improve multi-threaded
performance
■ optimization
■ replace Redis
■ ...

36. Conclusion
■ Rust can compete with C implementations
■ rewriting is not free
■ on-track for production deployment
■ Pelikan has an exciting future with Rust

37. Brought to you by
Brian Martin (he/him)
@brayniac