The document discusses QoS ingress rate limiting and OVS hardware offloads, outlining the differences in performance and functionality between OVS with and without TC offload. It describes test setups for rate limiting using OVS and provides commands for configuring interfaces and flow rules. Issues with TC offload and potential solutions are also addressed to optimize performance in these configurations.

1. Quality of Service (QoS) Ingress Rate Limiting
and OVS HW offloads

2. 2© 2019 NETRONOME SYSTEMS, INC.
Rate Limiting in OVS (1/2)
● QoS for OVS
○ ingress rate limiting
○ rate limiting
○ policing
● Future Work
○ egress rate limiting
○ flow based metering
○ set queue
○ shaping on egress
● Ingress Rate Limiting
○ from the hypervisor’s perspective
○ ingress traffic on tap0

3. 3© 2019 NETRONOME SYSTEMS, INC.
Rate Limiting in OVS (2/2)
OVS (TC offload disabled)
Userspace
OVS
TC
police
Kernel
OVS
● OVS without enabling TC offload
○ Makes use of basic filter for policing
○ Packet handled by TC, then OVS datapath
■ inherent priority
○ Disables rate limiting feature for TC offload
■ “Policing with offload isn't supported”
■ ignores rate limit
filter protocol all pref 49 basic chain 0
filter protocol all pref 49 basic chain 0 handle 0x1
police 0x2 rate 10Mbit burst 125Kb mtu 64Kb action drop overhead 0b
ref 1 bind 1 installed 8 sec used 0 sec
Sent 12133076 bytes 8241 pkts (dropped 885, overlimits 885)

4. 4© 2019 NETRONOME SYSTEMS, INC.
OVS (TC offload enabled)
Userspace
OVS
TC
police
TC
datapath
Rate Limiting in OVS with TC Offload
● OVS with TC offload
● Remove restriction in OVS
○ Insufficient
● Add a matchall type cls with police action
○ Could also be flower classifier
● Introduce reserved priorities
○ OVS should install TC filters with priority offset
OVS (TC offload enabled)
Userspace
OVS
TC
matchall
+ police
TC
datapath
filter protocol all pref 1 matchall chain 0
filter protocol all pref 1 matchall chain 0 handle 0x1
in_hw
action order 1: police 0x1 rate 1Mbit burst 1000Kb mtu 64Kb action
ref 1 bind 1 installed 23 sec used 0 sec
Action statistics:
Sent 4704024 bytes 3202 pkt (dropped 337, overlimits 337 requeues 0)
backlog 0b 0p requeues 0

5. 5© 2019 NETRONOME SYSTEMS, INC.
SmartNIC
OVS-TC
Userspace
OVS
TC
police
TC
filters
Issues with OVS-TC Offload
● Possible Issues:
○ Software -> Hardware
■ TC police is in software
■ filters are offloaded
■ lower performance
■ could break semantics
○ Hardware -> Software
■ TC police is offloaded
■ filters are in software
■ lower performance
■ still functionally correct
● Solutions:
○ Software -> Hardware
■ Revert to original semantics “policing with offload isn't supported”
■ OVS forces TC filters in software only
SmartNIC
OVS-TC
Userspace
OVS
TC
police
TC
filters

6. 6© 2019 NETRONOME SYSTEMS, INC.
OVS-TC Test/Demo (1/4)
● Original OVS Example
○ Source:
http://docs.openvswitch.org/en/latest/howto/qos/?highlight=r
ate%20limiting
● Setup:
○ Rate limit VM1 to 1 Mbps
○ Rate limit VM2 to 10 Mbps
○ Uses netperf
● Useful Tools:
○ ovs-vsctl show
○ ovs-vsctl list interface tap0
○ tc -s filter show dev tap0 parent ffff:

7. 7© 2019 NETRONOME SYSTEMS, INC.
OVS-TC Test/Demo (2/4)
● Enable TC offload
○ ovs-vsctl --no-wait set Open_vSwitch .
other_config:hw-offload=true'
○ ovs-vsctl --no-wait set Open_vSwitch .
other_config:tc-policy=none
● Add bridge and interfaces
○ ovs-vsctl add-br br0
○ ovs-vsctl add-port br0 eth11 -- set
Interface eth11 ofport_request=1
○ ovs-vsctl add-port br0 eth12 -- set
Interface eth12 ofport_request=2
○ ovs-vsctl add-port br0 enp3s0np0 -- set
Interface enp3s0np0 ofport_request=100

8. 8© 2019 NETRONOME SYSTEMS, INC.
OVS-TC Test/Demo (3/4)
● Configure rate limiters
○ ovs-vsctl set interface eth11 ingress_policing_rate=1000
○ ovs-vsctl set interface eth11 ingress_policing_burst=100
○ ovs-vsctl set interface eth12 ingress_policing_rate=10000
○ ovs-vsctl set interface eth12 ingress_policing_burst=1000
● Configure OVS flow rules
○ ovs-ofctl -O OpenFlow13 del-flows br0
○ ovs-ofctl -O OpenFlow13 add-flow br0 in_port=1,action=100
○ ovs-ofctl -O OpenFlow13 add-flow br0 in_port=2,action=100
○ ovs-ofctl -O OpenFlow13 add-flow br0 in_port=100,action=1
○ ovs-ofctl -O OpenFlow13 add-flow br0 in_port=100,action=2

9. 9© 2019 NETRONOME SYSTEMS, INC.
OVS-TC Test/Demo (4/4)
● Start netperf test on VM1
○ netserver (on measurement host)
○ netperf -H 10.0.0.200 -l 60 (on VM1)
● Repeat on VM2
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.0.0.200 () port 0 AF_INET : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
131072 16384 16384 63.25 1.11
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.0.0.200 () port 0 AF_INET : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
131072 16384 16384 60.19 10.00

10. 10© 2019 NETRONOME SYSTEMS, INC.
Examples of Useful Tools (1/2)
● ovs-vsctl show
dec65564-ce8c-4e48-974d-43f4df4acec4
Bridge "br0"
Port "eth11"
Interface "eth11"
Port "eth12"
Interface "eth12"
Port "enp3s0np0"
Interface "enp3s0np0"
Port "br0"
Interface "br0"
type: internal
ovs_version: "2.10.90"
● ovs-vsctl list interface eth11
_uuid : 4c6986e7-de1f-4559-96fb-9b0f5b9bb1d6
admin_state : up
...
ingress_policing_burst: 100
ingress_policing_rate: 1000
...

11. 11© 2019 NETRONOME SYSTEMS, INC.
Examples of Useful Tools (2/2)
● tc -s filter show dev eth11 parent ffff:
filter protocol ip pref 1 matchall chain 0
filter protocol ip pref 1 matchall chain 0 handle 0x1
not_in_hw
action order 1: police 0x1 rate 1Mbit burst 125Kb mtu 64Kb action drop/continue overhead 0b
ref 1 bind 1 installed 3 sec used 3 sec
Action statistics:
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
filter protocol ip pref 2 flower chain 0
filter protocol ip pref 2 flower chain 0 handle 0x1
eth_type ipv4
ip_flags nofrag
in_hw
action order 1: mirred (Egress Redirect to device enp3s0np0) stolen
index 4 ref 1 bind 1 installed 23 sec used 23 sec
Action statistics:
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
cookie 00f910dde846fe479f5d4eb0ae335692

12. © 2019 NETRONOME SYSTEMS, INC.
Thank you