Open vSwitch (OVS) has long been a critical component of the Neutron's reference implementation, offering reliable and flexible virtual switching for cloud environments. Being an early adopter of the OVS technology, Neutron's reference implementation made some compromises to stay within the early, stable featureset OVS exposed. In particular, Security Groups (SG) have been so far implemented by leveraging hybrid Linux Bridging and IPTables, which come at a significant performance overhead. However, thanks to recent developments and ongoing improvements within the OVS community, we are now able to implement feature-complete security groups directly within OVS. In this talk we will summarize the existing Security Groups implementation in Neutron and compare its performance with the Open vSwitch-only approach. We hope this analysis will form the foundation of future improvements to the Neutron Open vSwitch reference design.

1. Taking Security Groups
to Ludicrous Speed
with Open vSwitch
OpenStack Summit
Vancouver, 2015
Miguel Angel Ajo
@mangel_ajo
Ivar Lazzaro
@ivarlazzaro
Thomas Graf
@tgraf__
Justin Pettit
@Justin_D_Pettit

2. Agenda
Problem Statement
– Status Quo – a.k.a “The Bridge Mess”
Possible Solution
– OVS + Stateful services (+ OVN)
Results
– Performance Numbers
Q&A

3. Status Quo

5. Mess of Bridges.
br-eth1
(Open vSwitch)
OpenFlow
table
OVS
bridge
br-int
(Open vSwitch)

6. Mess of Bridges.
br-eth1
(Open vSwitch)
veth
OpenFlow
table
OVS
bridge
br-int
(Open vSwitch)

7. Mess of Bridges.
br-eth1
(Open vSwitch)
qbr
(Linux Bridge)
qbr
(Linux Bridge)
qbr
(Linux Bridge)
veth
iptables
rules
OpenFlow
tableOVS
bridge
Linux
bridge
br-int
(Open vSwitch)

8. Mess of Bridges. Why?
VM
br-eth1
(Open vSwitch)
qbr
(Linux Bridge)
qbr
(Linux Bridge)
qbr
(Linux Bridge)
VM lxc
tap
veth
iptables
rules
OpenFlow
table
OVS
bridge
Linux
bridge
4-5 network devices per guest in host!
br-int
(Open vSwitch)

10. Possible Solution
Stacking Things Properly
(c) Karen Sagovac

11. Can we have a pure OVS Model?
br-int
(Open vSwitch)
VM
br-eth1
(Open vSwitch)
VM lxc
Tap, veth, or
internal port
OpenFlow table
with security groups
OVS
bridge
1 network device per guest in host!
Makes VMs and containers equally happy.

12. Some Background
(OVS, OVN, Kernel CT)

13. ● Highly scaleable multi layer virtual switch for hypervisors
– Apache License (User Space), GPL (Kernel)
● Extensive flow table programming capabilities
– OpenFlow 1.0 – 1.5 (some partial)
– Vendor Extensions
● Designed to manage overlay networks
– VXLAN (+ extensions), GRE, Geneve, LISP, STT, VLAN, ...
● Remote management protocol (OVSDB)
● Monitoring capabilities
Open vSwitch

14. ● Virtual Networking for OVS
– Developed by same team that made OVS
– Works on same platorms (Linux, Containers, Hyper-V)
● Provides L2/L3 virtual networking
– Logical switches and routers
– Conntrack-based security groups
– L2/L3/L4 ACLs
– Physical and DPDK-based logical-physical gateways
● Integrated with OpenStack and other CMSs
OVN

15. Implementing a Firewall with OVS
● OVS has traditionally only supported stateless matches
● As an example, currently, two ways to implement a firewall in OVS
– Match on TCP flags (Enforce policy on SYN, allow ACK|RST)
● Pro: Fast
● Con: Allows non-established flow through with ACK or RST
set, only TCP
– Use “learn” action to setup new flow in reverse direction
● Pro: More “correct”
● Con: Forces every new flow to OVS userspace, reducing flow
setup by orders of magnitude
– Neither approach supports “related” flows or TCP window
enforcement

16. Connection Tracking
● We are adding the ability to use the conntrack module from Linux
– Stateful tracking of flows
– Supports ALGs to punch holes for related “data” channels
● FTP, TFTP, SIP
● Implement a distributed firewall with enforcement at the edge
– Better performance
– Better visibility
● Introduce new OpenFlow extensions:
– Action to send to conntrack
– Match fields on state of connection
● Have prototype working. Expect to ship as part of OVS in next
release.

17. Netfilter Conntrack Integration
OVS Flow Table
Netfilter
Connection Tracker
CT
Table
Userspace Netlink API
Create & Update
CT entries
Connection State (conn_state=)
conntrack()
Recirculation
1
2
3
4

18. Zone 1
Connection Tracking Zones
OVS Flow Table
CT
Table
Zone 2
CT
Table
Netfilter
Connection Tracker

19. OVSFirewallDriver

20. OVSFirewallDriver
● Original proposal from Amir Sadoughi
– https://review.openstack.org/#/c/89712
● Stable/kilo (just a POC)
– https://review.openstack.org/#/c/183725/

21. Example HTTP Request
VM 1 VM 2
HTTP req
response
GLOSARY of OF actions
NORMAL = “do like a normal switch”
ct(commit) = “push this packet to CT”
ct(recirc) = “grab any ct info we have, set
+trk, and send to T0”

22. SG OpenFlow Table structure
+trk(+est/+rel)
→ NORMAL
ARP (with filters)
→ NORMAL
(…)
SG rules in OF:
ip
Egress T1Input T0 ct(commit,
recirc)
(…)
SG rules:
tp_dst=80
Ingress T2
From VM(n)
(MAC+in_port -trk)
-trk → ct(recirc)
ToVM(n)(MAC)
match
ct(commit),NORMAL
VM1
VM2

23. SG OpenFlow Table structure
+trk(+est/+rel)
→ NORMAL
ARP (with filters)
→ NORMAL
(…)
SG rules in OF:
ip
Egress T1Input T0 ct(commit,
recirc)
(…)
SG rules:
tp_dst=80
Ingress T2
From VM(n)
(MAC+in_port -trk-trk)
-trk → ct(recirc)
ToVM(n)(MAC)
match
ct(commit),NORMAL
VM1

24. SG OpenFlow Table structure
+trk(+est/+rel)
→ NORMAL
ARP (with filters)
→ NORMAL
(…)
SG rules in OF:
ip
Egress T1Input T0 ct(commit,
recirc)
(…)
SG rules:
tp_dst=80
Ingress T2
From VM(n)
(MAC+in_port -trk)
-trk → ct(recirc)
ToVM(n)(MAC)
match
ct(commit),NORMAL

25. SG OpenFlow Table structure
+trk(+est or +rel)
→ NORMAL
ARP (with filters)
→ NORMAL
(…)
SG rules in OF:
ip
Egress T1Input T0 ct(commit,
recirc)
(…)
SG rules:
tp_dst=80
Ingress T2
From VM(n)
(MAC+in_port -trk)
-trk → ct(recirc)
ToVM(n)(MAC)
match
ct(commit),NORMAL

26. SG OpenFlow Table structure
+trk(+est/+rel)
→ NORMAL
ARP (with filters)
→ NORMAL
(…)
SG rules in OF:
ip
Egress T1Input T0 ct(commit,
recirc)
(…)
SG rules:
tp_dst=80
Ingress T2
From VM(n)
(MAC+in_port -trk)
-trk → ct(recirc)
ToVM(n)(MAC)
match
ct(commit),NORMAL
VM2

27. SG OpenFlow Table structure
+trk(+est/+rel)
→ NORMAL
ARP (with filters)
→ NORMAL
(…)
SG rules in OF:
ip
Egress T1Input T0 ct(commit,
recirc)
(…)
SG rules:
tp_dst=80
Ingress T2
From VM(n)
(MAC+in_port -trk)
-trk → ct(recirc)
ToVM(n)(MAC)
match
ct(commit),NORMAL
VM2

28. SG OpenFlow Table structure
+trk(+est/+rel)
→ NORMAL
ARP (with filters)
→ NORMAL
(…)
SG rules in OF:
ip
Egress T1Input T0 ct(commit,
recirc)
(…)
SG rules:
tp_dst=80
Ingress T2
From VM(n)
(MAC+in_port -trk)
-trk → ct(recirc)
ToVM(n)(MAC)
match
ct(commit),NORMAL

29. SG OpenFlow Table structure
+trk(+est/+rel)
→ NORMAL
ARP (with filters)
→ NORMAL
(…)
SG rules in OF:
ip
Egress T1Input T0 ct(commit,
recirc)
(…)
SG rules:
tp_dst=80
Ingress T2
From VM(n)
(MAC+in_port -trk)
-trk → ct(recirc)
ToVM(n)(MAC)
match
ct(commit),NORMAL
VM1

30. SG OpenFlow Table structure
+trk(+est/+rel)
→ NORMAL
ARP (with filters)
→ NORMAL
(…)
SG rules in OF:
ip
Egress T1Input T0 ct(commit,
recirc)
(…)
SG rules:
tp_dst=80
Ingress T2
From VM(n)
(MAC+in_port -trk)
-trk → ct(recirc)
ToVM(n)(MAC)
match
ct(commit),NORMAL

31. openvswitch_firewall.py
● update_security_group_{rules, members}
● prepare_port_filter
● update_port_filter
● remove_port_filter
● filter_defer_apply_{on,off}
neutron.agent.linux.firewall.FirewallDriver
neutron.agent.linux.openvswitch_firewall.OVSFirewallDriver

32. Performance Numbers

33. Test Setup Explained
System: 2 Socket, 24 core, IvyBridge
CPU: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz
Kernel: 3.10.0-229.1.2.el7.x86_64
Test: Netperf with TCP_STREAM and TCP_RR
Notes: Virt overhead eliminated, netperf/netserver runs baremetal
Compute 1
netperf
Compute 2
10GiB Link
netserver
netserver
local
Multi node

34. TCP Stream, Local, 1 netperf thread
64 128 512 1024 9000 64000
0
50
100
150
200
250
300
350
400
450
500
0
5,000
10,000
15,000
20,000
25,000
TCP stream Local, 1 netperf threads
sub-title
iptables throughput
OVS throughput
iptables cycles
OVS cycles
Packet Size
CPUMegacyclesperMbit
Mbit

35. TCP Stream, Local, 16 netperf threads
64 128 512 1024 9000 64000
0
50
100
150
200
250
300
350
400
0
50,000
100,000
150,000
200,000
250,000
TCP stream Local, 16 netperf threads
sub-title
iptables throughput
OVS throughput
iptables cycles
OVS cycles
Packet Size
CPUMegacyclesperMbit
Mbit

36. TCP Stream, Multi Node, 8 netperf threads
64 128 512 1024 9000 64000
0
50
100
150
200
250
0
1,000
2,000
3,000
4,000
5,000
6,000
7,000
8,000
9,000
10,000
TCP stream node-to-node, 8 netperf threads
iptables throughput
OVS throughput
iptables cycles
OVS cycles
Packet Size
CPUMegacyclesperMbit
Mbit

37. TCP Requests, Local, 1 netperf thread
64 128 512 1024 9000 64000
0
50
100
150
200
250
300
350
400
450
500
0
5,000
10,000
15,000
20,000
25,000
TCP stream Local, 1 netperf threads
sub-title
iptables throughput
OVS throughput
iptables cycles
OVS cycles
Packet Size
CPUMegacyclesperMbit
Mbit

38. TCP Requests, Local, 64K packets
1 4 8 16
0
10
20
30
40
50
60
70
0
20,000
40,000
60,000
80,000
100,000
120,000
140,000
160,000
TCP Requests/s Local, 64K packets
sub-title
iptables requests/s
OVS requests/s
iptables cycles
OVS cycles
Number netperf threads
CPUMegacyclesperMbit
Requests/s

39. TCP Requests, Multi Node, 1 netperf thread
64 128 512 1024 9000 64000
0
2
4
6
8
10
12
14
16
18
20
0
5,000
10,000
15,000
20,000
25,000
30,000
TCP Requests/s node-to-node, 1 netperf threads
sub-title
iptables requests/s
OVS requests/s
iptables cycles
OVS cycles
Packet Size
CPUMegacyclesperMbit
Requests/s

40. TCP Requests, Multi Node, 64K packets
1 4 8 16
0
5
10
15
20
25
30
35
40
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
16,000
18,000
20,000
TCP Requests/s node-to-node, 64K packets
sub-title
iptables requests/s
OVS requests/s
iptables cycles
OVS cycles
Number netperf threads
CPUMegacyclesperMbit
Requests/s

41. Conclusion
● Both throughput and latency are considerably improve (Up to 6x in
some situations).
● If limited by wire speed, pure OVS approach generally consumes
less CPU cycles for the same result, leaving more resources for
actual workload.
● Issue for specific packet sizes to be investigated and resolved before
merge.

42. Next Steps
● Convert ML2 PoC to a patch that can be merged
– Write functional tests
– Optimize OF rules/manipulation
● Complete upstream merge of connection tracking
support in Open vSwitch in the Linux kernel
● Consider and realize OVN integration of this work
● Hopefully ready for Liberty

43. Q&A
● OVS w/ CT Neutron ML2 plugin
– https://github.com/mangelajo/vagrant-rdo-juno-ovs-ct
-firewall
● Open vSwitch
– http://openvswitch.org/
● Conntrack code on GitHub
– https://github.com/justinpettit/ovs/tree/conntrack
● Stateful Connection Tracking & Stateful NAT (OVS
conference)
– http://www.openvswitch.org/support/ovscon2014/17
/1030-conntrack_nat.pdf

44. Thank You!