Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1VfmmLC. David Riddoch talks about the technologies that make very high performance networking possible on commodity servers and networks, with a special focus on kernel bypass technologies including sockets acceleration and NFV. These techniques give user-space applications direct access the network adapter hardware, making possible sub-microsecond latencies and millions of messages per second per thread. Filmed at qconlondon.com. David Riddoch leads the development of the Solarflare Open IP stack.

1. Copyright © 2016 Solarflare Communications, Inc. All rights reserved.
Much Faster Networking
David Riddoch
driddoch@solarflare.com

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
sockets-nfv

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon London
www.qconlondon.com

4. What is kernel bypass?

5. The standard receive path

6. The standard receive path

7. The standard receive path

8. The standard receive path

9. Kernel-bypass receive

10. Kernel-bypass transmit

11. Kernel-bypass transmit

12. Kernel-bypass transmit

13. Kernel-bypass transmit – even faster

14. What does all of this
cleverness achieve?

15. Better performance!
• Fewer CPU instructions for network operations
• Better cache locality
• Faster response (lower latency)
• Higher throughput (higher bandwidth/message rate)
• Reduced contention between threads
• Better core scaling
• Reduced latency jitter

17. Solarflare’s OpenOnload
• Sockets acceleration using kernel bypass
• Standard Ethernet, IP, TCP and UDP
• Standard BSD sockets API
• Binary compatible with existing applications

18. OpenOnload intercepts network calls

19. Single thread throughput and latency

20. UDP receive throughput (small messages)

21. Kernel stack OpenOnload

22. A much more challenging application

23. HAProxy performance and scaling
0
10
20
30
40
50
1 2 3 4 5 6 7 8 9
Connections/sec(000s)
CPU Cores
100K Byte
Solarflare Net Driver
Intel Net Driver
Solarflare OpenOnload
0
100
200
300
400
500
600
1 2 3 4 5 6 8 10 12
Connections/sec(000s)
CPU Cores
1K Byte
Solarflare Net Driver
Intel Net Driver
Solarflare OpenOnload
Solarflare kernel
Other kernel
OpenOnload
Solarflare kernel
Other kernel
OpenOnload
1 KiB message size
100 KiB message size
Connections/s(1000s)Connections/s(1000s)

24. Why doesn’t performance scale
when using the kernel stack?

25. Better question:
How come it scales as
well as it does?

26. CPU cores

27. Received packet is delivered into
memory (or L3 cache)

28. Interrupt triggers packet
handling on this CPU core

29. Application
calls recv()

30. Socket state pulled from one
cache to another; Inefficient
Single core bottleneck
for interrupt handling

31. Multiple receive channels
(up to one per core)
channel_id = hash(4tuple) % n_cores;

32. Multiple flows

33. Hopefully!

34. Usually

35. channel_id, n = lookup(4tuple);
if( n > 1 )
channel_id += hash(4tuple) % n_cores;

36. SO_REUSEPORT to the rescue
• Multiple listening sockets on the same TCP port
• One listening socket per worker thread
• Each gets a subset of incoming connection requests
• New connections go to the worker running on the core that
the flow hashes to
• Connection establishment scales with the number of
workers
• Received packets are delivered to the ‘right’ core

37. Problem solved?

41. 0
100
200
300
400
500
600
1 2 3 4 5 6 8 10 12
Connections/sec(000s)
CPU Cores
1K Byte
Solarflare Net Driver
Intel Net Driver
Solarflare OpenOnload
Solarflare kernel
Other kernel
OpenOnload
1 KiB message size
Connections/s(1000s)
Number of CPU cores

42. So much for sockets

43. Let’s get closer to the metal…

44. Layer-2 APIs
1-6 Mpps/core
Many protocols
End-host applications
1-60 Mpps/core
All protocols
Any network function

45. 60 million
pkt/s?

46. 17 ns

47. Some tips for achieving
really fast networking…

48. Tip 1. The faster your application
is already, the more speedup
you’ll get from kernel bypass
(This is just Ahmdal’s law)

49. Tip 2. NUMA locality applies
doubly to I/O devices

55. • DMA transfers will use the L3 cache if:
• The targeted cache line is resident
• Or if not then up to 10% of L3 is available for write-allocate
• Therefore
• If you want consistent high performance, DMA buffers must
be resident in L3 cache
• To achieve that
• Small set of DMA buffers recycled quickly
• (Even if that means doing an extra copy)

56. Tip 3.
Queue management is critical

57. Queues exist mostly to
handle mismatch between
arrival rate and service rate

58. • Buffers in switches and routers
• Descriptor rings in network adapters
• Socket send and receive buffers
• Shared memory queues, locks etc.
• Run queue in the kernel task scheduler

59. What happens when queues
start to fill?

60. Service rate < arrival rate
Queue fill level increases
(latency++)
Working-set size
increases
(efficiency--)
Service rate drops

61. Service rate < arrival rate
Queue fills
DROP!

62. Drops are bad, m’kay
Make SO_RCVBUF bigger!

63. Dilemma!
Small buffers:
Necessary for stable performance when overloaded
Large buffers:
Necessary for absorbing bursts without loss

64. For stable performance when overloaded
Limit working set size
• Limit the sizes of pools, queues, socket buffers etc.
Shed excess load early (Tip 3.1)
• Before you’ve wasted time on requests you’re going to have
to drop anyway

65. Interrupt moves packets from
descriptor ring to sockets
App thread consumes
from socket buffer

66. Drop newest data
(only at very high rates)
Drop newest data
Do work for every packet,
whether dropped or not

67. Drop newest data

68. Tip 3.2 Drop old data for better response time

69. Last example: Cache locality

70. recv()
OpenOnload stack; includes sockets, DMA buffers,
TX and RX rings, control plane etc.

71. recv()

72. Problem: Send a small (eg. 200 bytes) reply from a
different thread

73. send()
send()
Or…

74. send()
• Passing a small message to another thread:
• A few cache misses
• send() on a socket last accessed on another core:
• Dozens of cache misses

75. Thank you!

76. Watch the video with slide synchronization on
InfoQ.com!
https://www.infoq.com/presentations/sockets-
nfv