1. HERD: Using RDMA
Efficiently for Key-Value
Services
Presented by Hanjun Xiao
CIS 800/003, Jan 27th, 2015

2. Background
 Key-Value stores (Omitted)
 Remote Direct Memory Access (RDMA)
 Remote Network Interface Controller (RNIC)
 Comparison with ‘classical Ethernet’
 Hardware-based network stack
 Kernel bypass
 Nearly same cost

3. More about RDMA
 Memory semantics vs. Channel semantics
 One-sided: READ/WRITE
 Two-sided: SEND/RECV
 Queue pair
 Send/Receive queue
 Completion queue
 Transport types
 Reliable Connection (RC)
 Unreliable Connection (UC)
 Unreliable Datagrams (UD)

4. Asymmetric system model
 Server is the bottleneck
 Not at maximum performance possible
 Current state-of-the-art
 Aim for low CPU use (Pilaf)
 Use RDMA reads as a building block (Pilaf, FaRM)
 Channel verbs are thought to be slower
 What’s NOT the goal
 Fault-tolerance

5. Approach
 Scalable (~200) and consistency
 Build on MICA (NSDI ‘14, to be presented)
 High-performance
 Be critical about conventional thoughts
 READ-based?
 Identify the bottleneck and workaround
 Comprehensive experiments
 CPU-interrupts
 RTT

6. Life of a WRITE

7. Optimized WRITEs
 Inlined
 Unreliable
 Unsignalled

8. WRITE is better!
Inbound throughput Outbound throughput

9. More optimization
 Batching
 Pipelining
 Prefetching
 Pre-allocation

10. Results: Throughput

11. Latency vs. Throughput

12. Latency vs. Throughput

13. Limitations
 Processes pinned to cores
 Asymmetric system model
 Lack of generality
 More like an engineering effort?

14. Conclusions
 Some design principles
 Experiment and analyze
 Bypass the bottleneck
 Optimize for the common case
 Challenge the convention thoughts
 Holistic architectural design of single data-center
 Distributed storage, CPUs, I/O path
 May be good to take CIS 501!

15. Future work
 “There is no best design. Name your goals. ”
 Recipes for various combinations of goals
 Examine bottlenecks

16. Thank you!
Discussion