1.
DPDK performance
How to not just do a demo with the DPDK
Stephen Hemminger
stephen@networkplumber.org
@networkplumber

2.
Agenda
●
DPDK
– Background
– Examples
●
Performance
– Lessons learned
●
Open Issues

3.
60 160 260 360 460 560 660 760 860 960 1060 1160 1260 1360 1460
0
200
400
600
800
1000
1200
1400
1600
1800
2000
Packet size (bytes)
Time(ns)
Server
Packet
Network
Infrastructure
Packet time vs size

4.
Demo Applications
●
Examples
– L2fwd → Bridge/Switch
– L3fwd → Router
– L3fwd-acl → Firewall
– Load_balancer
– qos_sched → Quality Of Service

5.
L3fwd
Route
Lookup
Transmit
Transmit
Transmit
TransmitReceive
Receive
Receive
Receive
Receive

6.
Forwarding thread
Read Burst
Process Burst
Statistics:
Received Packets
Transmit Packets
Iterations
Packets processed

7.
Time Budget
●
Packet
– 67.2ns = 201 cycles @ 3Ghz
●
Cache
– L3 = 8 ns
– L2 = 4.3
●
Atomic operations
– Lock = 8.25 ns
– Lock/Unlock = 16.1
Network stack challenges at increasing speeds – LCA 2015
Jesper Dangaard Brouer

8.
Architecture choices
●
Legacy
– Existing proprietary code
●
BSD clone
●
DIY
– Build forwarding engine from scratch

9.
Test fixture
Freebsd
(netmap)
Router
Linux
desktop
10G Fibre
192.18.1.0/27
10G Cat6
192.18.0.0/27
Management network

10.
Dataplane CPU activity
Core Interface RX Rate TX Rate Idle
--------------------------------------------------------
1 p1p1 14.9M 0
2 p1p1 0 250
3 p33p1 0 250
4 p33p1 1 250
5 p1p1 0 250
6 p33p1 11.9M 1
Internal Instrumentation

11.
Linux perf
●
Perf tool part of kernel source
●
Can look at kernel and application
– Drill down if has symbols

12.
Perf – active thread
Samples: 16K of event 'cycles', Event count (approx.): 11763536471
14.93% dataplane [.] ip_input
10.04% dataplane [.] ixgbe_xmit_pkts
7.69% dataplane [.] ixgbe_recv_pkts
7.05% dataplane [.] T.240
6.82% dataplane [.] fw_action_in
6.61% dataplane [.] fifo_enqueue
6.44% dataplane [.] flow_action_fw
6.35% dataplane [.] fw_action_out
3.92% dataplane [.] ip_hash
3.69% dataplane [.] cds_lfht_lookup
2.45% dataplane [.] send_packet
2.45% dataplane [.] bit_reverse_ulong

13.
Original model
Rx0.0
Rx0.1
Core 1
Core 2
Tx0.0
Rx1.0
Rx1.1
Core 3
Core 4
Tx0.1
Tx0.2
Tx0.3
Tx1.0
Tx1.1
Tx1.2
Tx1.3

14.
Split thread model
Rx0.0
Rx0.1
Core 1
Core 5
Tx1
Core 2
Core 6
Tx1
Rx1.0
Rx1.1
Core 3
Core 4

15.
Speed killer's
●
I/O
●
VM exit's
●
System call's
●
PCI access
●
HPET
●
TSC
●
Floating Point
●
Cache miss
●
CPU pipeline stall

16.
TSC counter
while(1)
cur_tsc = rte_rdtsc();
diff_tsc = cur_tsc – prev_tsc;
if (unlikely(diff_tsc > drain_tsc)) {
for (portid = 0; portid < RTE_MAX_ETHPORTS;
portid++) {
send_burst(qconf,
qconf­>tx_mbufs[portid].len,
portid);
CPU stall
Heisenburg: observing performance slows it down

17.
fw_action_in
│ struct ip_fw_args fw_args = {
.m = m,
│ .client = client,
│ .oif = NULL };
1.54 │1d: movzbl %sil,%esi
0.34 │ mov %rsp,%rdi
0.04 │ mov $0x13,%ecx
0.16 │ xor %eax,%eax
57.66 │ rep stos %rax,%es:(%rdi)
4.68 │ mov %esi,0x90(%rsp)
20.45 │ mov %r9,(%rsp)

18.
Why is QoS slow?
static inline void
rte_sched_port_time_resync(struct rte_sched_port *port)
{
uint64_t cycles = rte_get_tsc_cycles();
uint64_t cycles_diff = cycles ­ port­>time_cpu_cycles;
double bytes_diff = ((double) cycles_diff) /
port­>cycles_per_byte;
/* Advance port time */
port­>time_cpu_cycles = cycles;
port­>time_cpu_bytes += (uint64_t) bytes_diff;

19.
The value of idle
●
All CPU's are not equal
– Hyperthreading
– Cache sharing
– Thermal
If one core is idle, others can go faster

20.
Idle sleep
●
100% Poll → 100% CPU
– CPU power limits
– No Turbo boost
– PCI bus overhead
●
Small sleep's
– 0 - 250us
– Based on activity

21.
Memory Layout
●
Cache killers
– Linked lists
– Poor memory layout
– Global statistics
– Atomic
●
Doesn't help
– Prefetching
– Inlining

22.
Mutual Exclusion
●
Locking
– Reader/Writer lock is expensive
– Read lock more overhead than spin lock
●
Userspace RCU
– Don't modify, create and destroy
– Impacts thread model

23.
Longest Prefix Match
Nexthop
1.1.1.1
/24
table
1.1.1.X
If = dp0p9p1
gw = 2.2.33.5
1.1.3.6

24.
LPM issues
●
Prefix → 8 bit next hop
●
Missing barriers
●
Rule update
●
Fixed size /8 table

25.
PCI passthrough
●
I/O TLB size
– Hypervisor uses IOMMU to map guest
– IOMMU has small TLB cache
– Guest I/O exceeds TLB
●
Solution
– 1G hugepage on host KVM
– Put Guest in huge pages
– Only on KVM – requires manual configuration

26.
DPDK Issues
●
Static configuration
– Features
– CPU architecture
– Table sizes
●
Machine specific initialization
– # of Cores, Memory Channels
●
Poor device model
– Works for Intel E1000 like devices

27.
Conclusion
●
DPDK can be used to build fast router
– 12M pps per core
●
Lots of ways to go slow
– Fewer ways to go fast

28.
Q & A

29.
Thank you
Stephen Hemminger
stephen@networkplumber.org
@networkplumber