Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Taking Security Groups
to Ludicrous Speed
with Open vSwitch
OpenStack Summit
Vancouver, 2015
Miguel Angel Ajo
@mangel_ajo
...
Agenda
Problem Statement
– Status Quo – a.k.a “The Bridge Mess”
Possible Solution
– OVS + Stateful services (+ OVN)
Result...
Status Quo
Mess of Bridges.
br-eth1
(Open vSwitch)
OpenFlow
table
OVS
bridge
br-int
(Open vSwitch)
Mess of Bridges.
br-eth1
(Open vSwitch)
veth
OpenFlow
table
OVS
bridge
br-int
(Open vSwitch)
Mess of Bridges.
br-eth1
(Open vSwitch)
qbr
(Linux Bridge)
qbr
(Linux Bridge)
qbr
(Linux Bridge)
veth
iptables
rules
OpenF...
Mess of Bridges. Why?
VM
br-eth1
(Open vSwitch)
qbr
(Linux Bridge)
qbr
(Linux Bridge)
qbr
(Linux Bridge)
VM lxc
tap
veth
i...
Possible Solution
Stacking Things Properly
(c) Karen Sagovac
Can we have a pure OVS Model?
br-int
(Open vSwitch)
VM
br-eth1
(Open vSwitch)
VM lxc
Tap, veth, or
internal port
OpenFlow ...
Some Background
(OVS, OVN, Kernel CT)
● Highly scaleable multi layer virtual switch for hypervisors
– Apache License (User Space), GPL (Kernel)
● Extensive flow...
● Virtual Networking for OVS
– Developed by same team that made OVS
– Works on same platorms (Linux, Containers, Hyper-V)
...
Implementing a Firewall with OVS
● OVS has traditionally only supported stateless matches
● As an example, currently, two ...
Connection Tracking
● We are adding the ability to use the conntrack module from Linux
– Stateful tracking of flows
– Supp...
Netfilter Conntrack Integration
OVS Flow Table
Netfilter
Connection Tracker
CT
Table
Userspace Netlink API
Create & Update...
Zone 1
Connection Tracking Zones
OVS Flow Table
CT
Table
Zone 2
CT
Table
Netfilter
Connection Tracker
OVSFirewallDriver
OVSFirewallDriver
● Original proposal from Amir Sadoughi
– https://review.openstack.org/#/c/89712
● Stable/kilo (just a PO...
Example HTTP Request
VM 1 VM 2
HTTP req
response
GLOSARY of OF actions
NORMAL = “do like a normal switch”
ct(commit) = “pu...
SG OpenFlow Table structure
+trk(+est/+rel)
→ NORMAL
ARP (with filters)
→ NORMAL
(…)
SG rules in OF:
ip
Egress T1Input T0 ...
SG OpenFlow Table structure
+trk(+est/+rel)
→ NORMAL
ARP (with filters)
→ NORMAL
(…)
SG rules in OF:
ip
Egress T1Input T0 ...
SG OpenFlow Table structure
+trk(+est/+rel)
→ NORMAL
ARP (with filters)
→ NORMAL
(…)
SG rules in OF:
ip
Egress T1Input T0 ...
SG OpenFlow Table structure
+trk(+est or +rel)
→ NORMAL
ARP (with filters)
→ NORMAL
(…)
SG rules in OF:
ip
Egress T1Input ...
SG OpenFlow Table structure
+trk(+est/+rel)
→ NORMAL
ARP (with filters)
→ NORMAL
(…)
SG rules in OF:
ip
Egress T1Input T0 ...
SG OpenFlow Table structure
+trk(+est/+rel)
→ NORMAL
ARP (with filters)
→ NORMAL
(…)
SG rules in OF:
ip
Egress T1Input T0 ...
SG OpenFlow Table structure
+trk(+est/+rel)
→ NORMAL
ARP (with filters)
→ NORMAL
(…)
SG rules in OF:
ip
Egress T1Input T0 ...
SG OpenFlow Table structure
+trk(+est/+rel)
→ NORMAL
ARP (with filters)
→ NORMAL
(…)
SG rules in OF:
ip
Egress T1Input T0 ...
SG OpenFlow Table structure
+trk(+est/+rel)
→ NORMAL
ARP (with filters)
→ NORMAL
(…)
SG rules in OF:
ip
Egress T1Input T0 ...
openvswitch_firewall.py
● update_security_group_{rules, members}
● prepare_port_filter
● update_port_filter
● remove_port_...
Performance Numbers
Test Setup Explained
System: 2 Socket, 24 core, IvyBridge
CPU: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz
Kernel: 3.10.0-22...
TCP Stream, Local, 1 netperf thread
64 128 512 1024 9000 64000
0
50
100
150
200
250
300
350
400
450
500
0
5,000
10,000
15,...
TCP Stream, Local, 16 netperf threads
64 128 512 1024 9000 64000
0
50
100
150
200
250
300
350
400
0
50,000
100,000
150,000...
TCP Stream, Multi Node, 8 netperf threads
64 128 512 1024 9000 64000
0
50
100
150
200
250
0
1,000
2,000
3,000
4,000
5,000
...
TCP Requests, Local, 1 netperf thread
64 128 512 1024 9000 64000
0
50
100
150
200
250
300
350
400
450
500
0
5,000
10,000
1...
TCP Requests, Local, 64K packets
1 4 8 16
0
10
20
30
40
50
60
70
0
20,000
40,000
60,000
80,000
100,000
120,000
140,000
160...
TCP Requests, Multi Node, 1 netperf thread
64 128 512 1024 9000 64000
0
2
4
6
8
10
12
14
16
18
20
0
5,000
10,000
15,000
20...
TCP Requests, Multi Node, 64K packets
1 4 8 16
0
5
10
15
20
25
30
35
40
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
16,...
Conclusion
● Both throughput and latency are considerably improve (Up to 6x in
some situations).
● If limited by wire spee...
Next Steps
● Convert ML2 PoC to a patch that can be merged
– Write functional tests
– Optimize OF rules/manipulation
● Com...
Q&A
● OVS w/ CT Neutron ML2 plugin
– https://github.com/mangelajo/vagrant-rdo-juno-ovs-ct
-firewall
● Open vSwitch
– http:...
Thank You!
Taking Security Groups to Ludicrous Speed with OVS (OpenStack Summit 2015)
Taking Security Groups to Ludicrous Speed with OVS (OpenStack Summit 2015)
Upcoming SlideShare
Loading in …5
×

Taking Security Groups to Ludicrous Speed with OVS (OpenStack Summit 2015)

3,574 views

Published on

Open vSwitch (OVS) has long been a critical component of the Neutron's reference implementation, offering reliable and flexible virtual switching for cloud environments.

Being an early adopter of the OVS technology, Neutron's reference implementation made some compromises to stay within the early, stable featureset OVS exposed. In particular, Security Groups (SG) have been so far implemented by leveraging hybrid Linux Bridging and IPTables, which come at a significant performance overhead. However, thanks to recent developments and ongoing improvements within the OVS community, we are now able to implement feature-complete security groups directly within OVS.

In this talk we will summarize the existing Security Groups implementation in Neutron and compare its performance with the Open vSwitch-only approach. We hope this analysis will form the foundation of future improvements to the Neutron Open vSwitch reference design.

Published in: Software
  • Be the first to comment

Taking Security Groups to Ludicrous Speed with OVS (OpenStack Summit 2015)

  1. 1. Taking Security Groups to Ludicrous Speed with Open vSwitch OpenStack Summit Vancouver, 2015 Miguel Angel Ajo @mangel_ajo Ivar Lazzaro @ivarlazzaro Thomas Graf @tgraf__ Justin Pettit @Justin_D_Pettit
  2. 2. Agenda Problem Statement – Status Quo – a.k.a “The Bridge Mess” Possible Solution – OVS + Stateful services (+ OVN) Results – Performance Numbers Q&A
  3. 3. Status Quo
  4. 4. Mess of Bridges. br-eth1 (Open vSwitch) OpenFlow table OVS bridge br-int (Open vSwitch)
  5. 5. Mess of Bridges. br-eth1 (Open vSwitch) veth OpenFlow table OVS bridge br-int (Open vSwitch)
  6. 6. Mess of Bridges. br-eth1 (Open vSwitch) qbr (Linux Bridge) qbr (Linux Bridge) qbr (Linux Bridge) veth iptables rules OpenFlow tableOVS bridge Linux bridge br-int (Open vSwitch)
  7. 7. Mess of Bridges. Why? VM br-eth1 (Open vSwitch) qbr (Linux Bridge) qbr (Linux Bridge) qbr (Linux Bridge) VM lxc tap veth iptables rules OpenFlow table OVS bridge Linux bridge 4-5 network devices per guest in host! br-int (Open vSwitch)
  8. 8. Possible Solution Stacking Things Properly (c) Karen Sagovac
  9. 9. Can we have a pure OVS Model? br-int (Open vSwitch) VM br-eth1 (Open vSwitch) VM lxc Tap, veth, or internal port OpenFlow table with security groups OVS bridge 1 network device per guest in host! Makes VMs and containers equally happy.
  10. 10. Some Background (OVS, OVN, Kernel CT)
  11. 11. ● Highly scaleable multi layer virtual switch for hypervisors – Apache License (User Space), GPL (Kernel) ● Extensive flow table programming capabilities – OpenFlow 1.0 – 1.5 (some partial) – Vendor Extensions ● Designed to manage overlay networks – VXLAN (+ extensions), GRE, Geneve, LISP, STT, VLAN, ... ● Remote management protocol (OVSDB) ● Monitoring capabilities Open vSwitch
  12. 12. ● Virtual Networking for OVS – Developed by same team that made OVS – Works on same platorms (Linux, Containers, Hyper-V) ● Provides L2/L3 virtual networking – Logical switches and routers – Conntrack-based security groups – L2/L3/L4 ACLs – Physical and DPDK-based logical-physical gateways ● Integrated with OpenStack and other CMSs OVN
  13. 13. Implementing a Firewall with OVS ● OVS has traditionally only supported stateless matches ● As an example, currently, two ways to implement a firewall in OVS – Match on TCP flags (Enforce policy on SYN, allow ACK|RST) ● Pro: Fast ● Con: Allows non-established flow through with ACK or RST set, only TCP – Use “learn” action to setup new flow in reverse direction ● Pro: More “correct” ● Con: Forces every new flow to OVS userspace, reducing flow setup by orders of magnitude – Neither approach supports “related” flows or TCP window enforcement
  14. 14. Connection Tracking ● We are adding the ability to use the conntrack module from Linux – Stateful tracking of flows – Supports ALGs to punch holes for related “data” channels ● FTP, TFTP, SIP ● Implement a distributed firewall with enforcement at the edge – Better performance – Better visibility ● Introduce new OpenFlow extensions: – Action to send to conntrack – Match fields on state of connection ● Have prototype working. Expect to ship as part of OVS in next release.
  15. 15. Netfilter Conntrack Integration OVS Flow Table Netfilter Connection Tracker CT Table Userspace Netlink API Create & Update CT entries Connection State (conn_state=) conntrack() Recirculation 1 2 3 4
  16. 16. Zone 1 Connection Tracking Zones OVS Flow Table CT Table Zone 2 CT Table Netfilter Connection Tracker
  17. 17. OVSFirewallDriver
  18. 18. OVSFirewallDriver ● Original proposal from Amir Sadoughi – https://review.openstack.org/#/c/89712 ● Stable/kilo (just a POC) – https://review.openstack.org/#/c/183725/
  19. 19. Example HTTP Request VM 1 VM 2 HTTP req response GLOSARY of OF actions NORMAL = “do like a normal switch” ct(commit) = “push this packet to CT” ct(recirc) = “grab any ct info we have, set +trk, and send to T0”
  20. 20. SG OpenFlow Table structure +trk(+est/+rel) → NORMAL ARP (with filters) → NORMAL (…) SG rules in OF: ip Egress T1Input T0 ct(commit, recirc) (…) SG rules: tp_dst=80 Ingress T2 From VM(n) (MAC+in_port -trk) -trk → ct(recirc) ToVM(n)(MAC) match ct(commit),NORMAL VM1 VM2
  21. 21. SG OpenFlow Table structure +trk(+est/+rel) → NORMAL ARP (with filters) → NORMAL (…) SG rules in OF: ip Egress T1Input T0 ct(commit, recirc) (…) SG rules: tp_dst=80 Ingress T2 From VM(n) (MAC+in_port -trk-trk) -trk → ct(recirc) ToVM(n)(MAC) match ct(commit),NORMAL VM1
  22. 22. SG OpenFlow Table structure +trk(+est/+rel) → NORMAL ARP (with filters) → NORMAL (…) SG rules in OF: ip Egress T1Input T0 ct(commit, recirc) (…) SG rules: tp_dst=80 Ingress T2 From VM(n) (MAC+in_port -trk) -trk → ct(recirc) ToVM(n)(MAC) match ct(commit),NORMAL
  23. 23. SG OpenFlow Table structure +trk(+est or +rel) → NORMAL ARP (with filters) → NORMAL (…) SG rules in OF: ip Egress T1Input T0 ct(commit, recirc) (…) SG rules: tp_dst=80 Ingress T2 From VM(n) (MAC+in_port -trk) -trk → ct(recirc) ToVM(n)(MAC) match ct(commit),NORMAL
  24. 24. SG OpenFlow Table structure +trk(+est/+rel) → NORMAL ARP (with filters) → NORMAL (…) SG rules in OF: ip Egress T1Input T0 ct(commit, recirc) (…) SG rules: tp_dst=80 Ingress T2 From VM(n) (MAC+in_port -trk) -trk → ct(recirc) ToVM(n)(MAC) match ct(commit),NORMAL VM2
  25. 25. SG OpenFlow Table structure +trk(+est/+rel) → NORMAL ARP (with filters) → NORMAL (…) SG rules in OF: ip Egress T1Input T0 ct(commit, recirc) (…) SG rules: tp_dst=80 Ingress T2 From VM(n) (MAC+in_port -trk) -trk → ct(recirc) ToVM(n)(MAC) match ct(commit),NORMAL VM2
  26. 26. SG OpenFlow Table structure +trk(+est/+rel) → NORMAL ARP (with filters) → NORMAL (…) SG rules in OF: ip Egress T1Input T0 ct(commit, recirc) (…) SG rules: tp_dst=80 Ingress T2 From VM(n) (MAC+in_port -trk) -trk → ct(recirc) ToVM(n)(MAC) match ct(commit),NORMAL
  27. 27. SG OpenFlow Table structure +trk(+est/+rel) → NORMAL ARP (with filters) → NORMAL (…) SG rules in OF: ip Egress T1Input T0 ct(commit, recirc) (…) SG rules: tp_dst=80 Ingress T2 From VM(n) (MAC+in_port -trk) -trk → ct(recirc) ToVM(n)(MAC) match ct(commit),NORMAL VM1
  28. 28. SG OpenFlow Table structure +trk(+est/+rel) → NORMAL ARP (with filters) → NORMAL (…) SG rules in OF: ip Egress T1Input T0 ct(commit, recirc) (…) SG rules: tp_dst=80 Ingress T2 From VM(n) (MAC+in_port -trk) -trk → ct(recirc) ToVM(n)(MAC) match ct(commit),NORMAL
  29. 29. openvswitch_firewall.py ● update_security_group_{rules, members} ● prepare_port_filter ● update_port_filter ● remove_port_filter ● filter_defer_apply_{on,off} neutron.agent.linux.firewall.FirewallDriver neutron.agent.linux.openvswitch_firewall.OVSFirewallDriver
  30. 30. Performance Numbers
  31. 31. Test Setup Explained System: 2 Socket, 24 core, IvyBridge CPU: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz Kernel: 3.10.0-229.1.2.el7.x86_64 Test: Netperf with TCP_STREAM and TCP_RR Notes: Virt overhead eliminated, netperf/netserver runs baremetal Compute 1 netperf Compute 2 10GiB Link netserver netserver local Multi node
  32. 32. TCP Stream, Local, 1 netperf thread 64 128 512 1024 9000 64000 0 50 100 150 200 250 300 350 400 450 500 0 5,000 10,000 15,000 20,000 25,000 TCP stream Local, 1 netperf threads sub-title iptables throughput OVS throughput iptables cycles OVS cycles Packet Size CPUMegacyclesperMbit Mbit
  33. 33. TCP Stream, Local, 16 netperf threads 64 128 512 1024 9000 64000 0 50 100 150 200 250 300 350 400 0 50,000 100,000 150,000 200,000 250,000 TCP stream Local, 16 netperf threads sub-title iptables throughput OVS throughput iptables cycles OVS cycles Packet Size CPUMegacyclesperMbit Mbit
  34. 34. TCP Stream, Multi Node, 8 netperf threads 64 128 512 1024 9000 64000 0 50 100 150 200 250 0 1,000 2,000 3,000 4,000 5,000 6,000 7,000 8,000 9,000 10,000 TCP stream node-to-node, 8 netperf threads iptables throughput OVS throughput iptables cycles OVS cycles Packet Size CPUMegacyclesperMbit Mbit
  35. 35. TCP Requests, Local, 1 netperf thread 64 128 512 1024 9000 64000 0 50 100 150 200 250 300 350 400 450 500 0 5,000 10,000 15,000 20,000 25,000 TCP stream Local, 1 netperf threads sub-title iptables throughput OVS throughput iptables cycles OVS cycles Packet Size CPUMegacyclesperMbit Mbit
  36. 36. TCP Requests, Local, 64K packets 1 4 8 16 0 10 20 30 40 50 60 70 0 20,000 40,000 60,000 80,000 100,000 120,000 140,000 160,000 TCP Requests/s Local, 64K packets sub-title iptables requests/s OVS requests/s iptables cycles OVS cycles Number netperf threads CPUMegacyclesperMbit Requests/s
  37. 37. TCP Requests, Multi Node, 1 netperf thread 64 128 512 1024 9000 64000 0 2 4 6 8 10 12 14 16 18 20 0 5,000 10,000 15,000 20,000 25,000 30,000 TCP Requests/s node-to-node, 1 netperf threads sub-title iptables requests/s OVS requests/s iptables cycles OVS cycles Packet Size CPUMegacyclesperMbit Requests/s
  38. 38. TCP Requests, Multi Node, 64K packets 1 4 8 16 0 5 10 15 20 25 30 35 40 0 2,000 4,000 6,000 8,000 10,000 12,000 14,000 16,000 18,000 20,000 TCP Requests/s node-to-node, 64K packets sub-title iptables requests/s OVS requests/s iptables cycles OVS cycles Number netperf threads CPUMegacyclesperMbit Requests/s
  39. 39. Conclusion ● Both throughput and latency are considerably improve (Up to 6x in some situations). ● If limited by wire speed, pure OVS approach generally consumes less CPU cycles for the same result, leaving more resources for actual workload. ● Issue for specific packet sizes to be investigated and resolved before merge.
  40. 40. Next Steps ● Convert ML2 PoC to a patch that can be merged – Write functional tests – Optimize OF rules/manipulation ● Complete upstream merge of connection tracking support in Open vSwitch in the Linux kernel ● Consider and realize OVN integration of this work ● Hopefully ready for Liberty
  41. 41. Q&A ● OVS w/ CT Neutron ML2 plugin – https://github.com/mangelajo/vagrant-rdo-juno-ovs-ct -firewall ● Open vSwitch – http://openvswitch.org/ ● Conntrack code on GitHub – https://github.com/justinpettit/ovs/tree/conntrack ● Stateful Connection Tracking & Stateful NAT (OVS conference) – http://www.openvswitch.org/support/ovscon2014/17 /1030-conntrack_nat.pdf
  42. 42. Thank You!

×