1.
Brought to you by
Whoops!
I Rewrote It in Rust
Brian Martin (he/him)
Software Engineer at Twitter

2.
Brian Martin (he/him)
Software Engineer at Twitter
■ At Twitter for 7 years, using Rust for 6 years
■ Open source benchmarking, telemetry agent, cache server
■ p99, p999, p9999 are all important
■ Volunteer Search and Rescue

3.
Caching
■ distributed key-value storage
● Memcached
● Redis
■ volatile
● expiration (TTL)
● eviction
■ protect databases / backend services

4.
Pelikan
■ open source caching framework
● https://pelikan.io
● https://github.com/twitter/pelikan
■ predictable performance
■ single codebase, multiple
solutions

5.
Use Rust to:
■ add TLS support to Pelikan
■ match performance of C implementation

6.
Rust in Pelikan
■ C server with Rust storage library - 2018
● quick development
● first commit of Rust into the project
■ Rust pingserver & Rust Twemcache server with C storage - 2019
● buffers, logging, metrics, parsers, … all still in C
● showed we could glue it all together

7.
Performance Testing
■ maximize throughput
■ latency @ p999 < 1 ms

8.
p999 / 99.9th percentile
■ fanout = 1:
● 0.1% of requests slower than p999

9.
p999 / 99.9th percentile
■ fanout = 1:
● 0.1% of requests slower than p999
■ fanout = 10:
● 1% of requests slower than p999

10.
p999 / 99.9th percentile
■ fanout = 1:
● 0.1% of requests slower than p999
■ fanout = 10:
● 1% of requests slower than p999
■ fanout = 100:
● 10% of requests slower than p999

11.
p999 / 99.9th percentile
■ fanout = 1:
● 0.1% of requests slower than p999
■ fanout = 10:
● 1% of requests slower than p999
■ fanout = 100:
● 10% of requests slower than p999
Tail latency occurs more often than you think!

12.
Pelikan Twemcache in Rust: 2019
■ Memcached compatible
■ Tokio / async for networking
■ uses C storage library
Can we use this to add TLS?

13.
Performance Problems
■ Throughput 10-15% slower
● need more instances to match throughput requirements
■ Latency 25-30% higher (@ p999 / 99.9th Percentile)
● could cause timeouts/errors in production

14.
Pelikan Twemcache C vs Rust

15.
Pingserver

16.
Pingserver
% telnet 127.0.0.1 12321
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
PING
PONG
PING
PONG
QUIT
Connection closed by foreign host.

17.
Why Pingserver?
■ prove out performance of design
■ buffers / parsers / metrics / logging
■ can add TLS support

18.
Pingserver Rewrite
■ Replace “tokio” with “mio”
● switch from async runtime to epoll event loop
■ Replace C components with Rust
● metrics/logging/buffers/…

19.
Pingserver C vs Rust

20.
Pelikan Twemcache

21.
Twemcache Prototype
■ Memcache protocol compatible server
● temporary storage based on std::collections::HashMap
■ benchmark
● performance looked promising
■ real storage
● switch to the C library

22.
Oh no!
■ Metrics and logging caused issues wrapping the C library

23.
Rewrite it in Rust?

24.
No. Don’t do it.
■ self-referential data structures
■ memory allocations!
■ linked lists!
■ pointers everywhere!
■ unsafe { }

25.
But… maybe yes?
■ start with single-threaded
■ only hashtable in Rust?
■ use C slab allocator?

26.
Whoops! All Rust
■ implemented new research storage design
● easier to port to Rust
■ single-threaded
● easy mode
■ managing all memory allocations in Rust

27.
Rust Segment Storage
Pros
■ 100% Rust
■ Re-write helped improve code
■ Added new ideas to the design
Cons
■ ~2 months of work
■ Not 100% feature complete yet.

28.
Rust Benefits
“empowering everyone to build reliable and efficient software”
■ high performance
■ code with confidence in reliability
■ awesome language features and tools
■ zero cost abstraction

29.
Rust Benefits
“The Rust implementation finally made Pelikan modules what they ought to be, but
couldn’t before, due to the limitation of the C language. This feels exactly right and
is just as fast”
- Yao (@thinkingfish)

30.
Modular
// Segcache is Memcache protocol + Seg storage
type Parser = MemcacheRequestParser;
type Request = MemcacheRequest;
type Response = MemcacheResponse;
type Storage = Seg;
...
let process_builder = ProcessBuilder::<Storage,
Parser, Request, Response>::new(...);

31.
Rust has awesome tools!
■ cargo
■ fuzzers
■ benchmarks
■ rustfmt
■ clippy

32.
Pelikan Twemcache C vs Segcache Rust

33.
Memcached vs Segcache Rust

34.
Memcached vs Segcache Rust
Memcached has higher
latency!

35.
Next Steps
Path to Production
■ feature complete
■ more testing
■ production canary
■ deployment
Further Work
■ improve multi-threaded
performance
■ optimization
■ replace Redis
■ ...

36.
Conclusion
■ Rust can compete with C implementations
■ rewriting is not free
■ on-track for production deployment
■ Pelikan has an exciting future with Rust

37.
Brought to you by
Brian Martin (he/him)
@brayniac