By Diego Pino García. Slides at https://people.igalia.com/dpino/gore20/network-functions-with-snabb/ (c) ES.NOG 20th http://www.esnog.net/gore20.html

1. SNABB
A TOOLKIT FOR BUILDING USER-SPACE NETWORK
FUNCTIONS

3. ABOUT IGALIA
Consultancy specialized in open-source
Base in Coruña but distributed all over the world (>60 people working from 15
different countries)
Contributors to projects such as WebKit, Chromium V8, etc
Other areas: Graphics, Multimedia, Networking

4. https://www.igalia.com/networking

5. AGENDA
What is Snabb?
How it works?
Catalog and programs
Use case: lwAFTR

6. WHAT IS SNABB?
Snabb is a toolkit for developing high-performance network functions in user-
space

7. WHAT IS A NETWORK FUNCTION?
A program that manipulates traffic data
Basic operations: read, forward, drop, modify, create...
Combining these primitives we can build any network function

8. EXAMPLES
Firewall: read incoming packets, compare to table of rules and execute an
action(forward or drop)
NAT: read incoming packets, modify headers and forward packet
Tunelling: read incoming packets, create a new packet, embed packet into new
one and send it

9. WHY SNABB?
Increasing improvement of commodity hardware: 10Gbps NICs at very
affordable prices
High-performance equipment is still very expensive
Idea: build an analog high-performance router using commodity hardware

10. WHY SNABB?
What software to put into this hardware?
Common intuition: Linux
Drawback: Linux is not suitable for high-performance networking

11. WHY NOT LINUX?
General-purpose operating system
An OS abstracts hw resources to offer high-level interfaces: filesystems,
processes, sockets...
Our network function will be divided into two lands: user-space and kernel-
space
Colorary: processing a packet has an inheritent cost => the cost of the OS

12. HIGH-PERFORMANCE NETWORKING
NIC: 10Gbps
Avg Packet-size: 550-byte
PPS: 2272727,27
1 packet every 440ns ((1/2272727,27)*10^9)
CPU: 2,5 Ghz
1100 cycles to process one packet (2,5 cycles/sec * 440 ns)

13. HIGH-PERFORMANCE NETWORKING
Packet-size: 64-byte: 51 ns per packet; 128 cycles per packet
Lock/Unlock: 16ns; Cache-miss: 32 ns
Source: Jonathan Corbet's
Small packet size => More packets per second => worse
Faster CPU => better
"Improving Linux networking performance"

14. USER-SPACE DRIVER
Do a kernel by-pass and manage the hardware directly from user-space:
Tell Linux not to manage the PCI device (unbind)
Do a mmap of the registers of the PCI device into addressable memory
Whenever we read/write the addressable memory, we're actually poking the
registers of the NIC
Follow the NIC's datasheet to implement operations such as initialize,
receive, transmit, etc

15. USER-SPACE NETWORKING
Snabb is not an isolated case of user-space networking:
Snabb (2012)
DPDK (2012)
VPP/fd.io (2016)
DPDK (Data-plane Development Kit, Intel)
VPP (Vector Packet Processing, Cisco)

16. RING-BUFFER
Very important to avoid packet drops

17. INSIDE SNABB

18. SNABB
Project started by Luke Gorrie
User-space networking benefit: freedom of programming language
Snabb is mostly written in Lua
Network functions are also written in Lua
Fast to run, fast to develop
Snabb means fast in Swedish :)

19. ABOUT LUA
Started in 1993 at University of Rio de Janeiro (PUC Rio)
Very similar to JavaScript, easy to learn
Very small and compact, it's generally embeded in other systems
Use cases: microcontrollers (NodeMCU), videogames (Grim Fandango), IA
(Torch7)

20. ABOUT LUAJIT
Just-in-time compiler for Lua
Extremely fast virtual machine!!
Very good integration with C thanks to FFI (Foreign Function Interface)

21. FFI: EXAMPLE
ffi.cdef[[
void syslog(int priority, const char*format, ...);
]]
ffi.C.syslog(2, "error:...");
local ether_header_t = ffi.typeof [[
/* All values in network byte order. */
struct {
uint8_t dhost[6];
uint8_t shost[6];
uint16_t type;
} __attribute__((packed))
]]

22. SNABB IN A NUTSHELL
A snabb program is an app graph
Apps are conected together via links
Snabb processes the program in units called breadths

23. NF: APP GRAPH

24. BREADTHS
A breadth has two steps:
inhale a batch of packets into the graph
process those packets
To inhale, the method pull of the apps is executed (if defined)
To process, the method push of the apps is executed (if defined)

25. # Pull function of included Intel 82599 driver
function Intel82599:pull ()
for i = 1, engine.pull_npackets do
if not self.dev:can_receive() then break end
local pkt = self.dev:receive()
link.transmit(self.output.tx, pkt)
end
end

26. # Push function of included PcapFilter
function PcapFilter:push ()
while not link.empty(self.input.rx) do
local p = link.receive(self.input.rx)
if self.accept_fn(p.data, p.length) then
link.transmit(self.output.tx, p)
else
packet.free(p)
end
end
end

27. PACKET PROCESSING
Normally only one app of the app graph introduces packets into the graph
The method push gives an opportunity to every app to do something with a
packet

28. APP GRAPH DEFINITION
local c = config.new()
-- App definition.
config.add(c, "nic", Intel82599, {
pci = "0000:04:00.0"
})
config.add(c, "filter", PcapFilter, "src port 80")
config.add(c, "writer", Pcap.PcapWriter, "output.pcap")
-- Link definition.
config.link(c, "nic.tx -> filter.input")
config.link(c, "filter.output -> writer.input")
engine.configure(c)
engine.main({duration=1})

29. PACKETS
struct packet {
uint16_t length;
unsigned char data[10*1024];
};

30. LINKS
struct link {
struct packet *packets[1024];
// the next element to be read
int read;
// the next element to be written
int write;
};

31. SNABB: APP CATALOG AND PROGRAMS

32. INVENTARY
apps: software components that developers combine together to build network
functions
programs: complete network functions

33. APPS I/O
Intel i210/i350/82599/XL710
Mellanox Connectx-4/5
Virtio host y guest
UNIX socket
Linux: tap and "raw" (e.g: eth0)
Pcap files

34. APPS L2
Flooding and learning bridge
VLAN insert/remove
ARP/NDP

35. APPS L3
IPv4/v6 fragmentation and reassembly
IPv4/v6 splitter
ICMPv4/v6 echo responder
Control-plane delegation (nh_fwd)

36. APPS L4
IPsec ESP
Lightweight 4-over-6 AFTR
Keyed IPv6 Tunnel

37. APPS MONITORING
IPFix capturer and exporter
L7 monitor/filtering (libndpi)
Pcap expressions filter (with own backend for code generation)

38. APPS TESTING
Lots of load generators: loadgen, packetblaster, loadbench...

39. USE CASE: LWAFTR

40. CONTEXT
2012-2014: Several RIRs run out of IPv4 public addresses
2008: IPv6 adoption starts to peak up
Still big dependency on IPv4: services, websites, programs, etc

41. SOLUTIONS
Carrier-Grade NAT: temporal solution for IPv4 address exhaustion problem
Deployment of Dual-Stack networks (IPv4 e IPv6)
Dual-Stack implies increasing complexity and costs (maintenance of two
separated networks)
Dual-Stack Lite (IPv6-only network which also offers IPv4 connectivity relying
on CGN)
Lightweight 4over6: iteration over Dual-Stack

42. LIGHTWEIGHT 4OVER6

43. LW4O6 - GOALS
RFC7596 fully complaint (lwAFTR part)
Performance: 2MPPS; 550-byte (packet-size); Binding-table: 1M subscribers.
No packet drops

44. LW4O6 - DEVELOPMENT
Version 1:
Prototype
Basic functionality (encapsulating/decapsulating)
Small binding-table (own format)
Development of tools to measure performance

45. LW4O6 - DEVELOPMENT
Version 2
Production quality
Fully standard compliant
Big binding-table: 1M subscribers (still customized format but much closer
to standard)
Add support for other necessary protocols: ARP, NDP, fragmentation,
reassembly, ping
Tons of optimizations (use of AVX instructions to speed up lookups)

46. LW4O6 - DEVELOPMENT
Version 3:
Added YANG support to Snabb
Support binding-table format according to standard
Support of execution as leader/worker (leader: control-plane/worker: data-
plane)

47. LW4O6 - DEVELOPMENT
Version 4:
Multiprocess (one leader, multiple workers)
Improvement of the Intel 10Gbps driver (added support for RSS, Received
Side Scaling)
Added alarms support according to latest draft

48. LIGHTWEIGHT 4OVER6 - TALKS
Juniper's vMX Lightweight 4over6 VNF
Charla:
Kostas Zordabelos's A real-world scale network VF using Snabb for lw4o6
Charla:
Juniper Tech Club, Marzo 2017
SDN Meetup, Abril 2017

49. OTHER PROGRAMS

50. PROGRAM: PACKET BLASTER
Generally useful tool: fill TX buffer of NIC with packets and transmit them over
and over again
Measures received traffic too
Easily saturates 10Gbps links
snabb packetblaster replay packets.pcap 82:00.1

51. PROGRAM: SNABBWALL
L7 firewall that optionally uses nDPI
Collaboration betwen Igalia and NLnet Foundation
Landed upstream in 2017
Website: http://snabbwall.org

52. PROGRAM: IPFIX
NETFLOW collector and exporter (v9 and IPFIX)
Line-rate speed on a single core. Further improvement: parallel processing via
RSS
Landed upstream very recently

53. PROGRAM: L2VPN
L2VPN over IPv6 (developed by Alexander Gall from SWITCH)
Pending to land upstream; used in production
Ideal Snabb use case: programmer/operator builds bespoke tool

54. PROGRAM: YOUR VNF
Snabb upstream open to include new network functions
Repository will grow as people will build new things
Igalia can build one for you

55. LAST NOTES ABOUT PERFORMANCE

56. CONSIDERATIONS
Isolcpus: Prevents the kernel to take a CPU to schedule processes
Dishable HyperThreading
Use HugePages (2MB) (Linux default is 4Kb)
Do not neglect NUMA when launching programs
Make use of SIMD instructions (AVX, AVX2) to speed up computations
(checksum)
Keep an eye on regressions: profile often

57. SUMMARY
Toolkit for developing high-performance network functions in user-space
Snabb provides apps which can be combined together forming a graph (network
function)
Snabb provides programs, complete network functions ready to use
Snabb provides libraries, to easy the development of new network functions
Completely written in Lua: easy to extend
Fast: kernel-by pass + high-level language + fast VM (LuaJIT)

58. THANKS!
Email: dpino@igalia.com
Twitter: @diepg
$ git clone https://github.com/snabbco/snabb.git
$ cd snabb
$ make