This document summarizes test results from an HPE lab that compared network performance of VMs using Intel X710 and Qlogic FastLinQ QL41xxx NICs passed through to ESXi hosts. A variety of TCP, UDP, HTTP and HTTPS network throughput tests were run between VMs on the two hosts. In most tests, the Intel and Qlogic NICs performed comparably, with Qlogic outperforming Intel in some UDP and HTTP tests when using receive-side scaling. The tests helped validate that the Qlogic NIC meets expectations for VM network throughput.

1. Confidential │ ©2020 VMware, Inc.
Marvell – Qlogic NIC FastLinQ QL41xxx driver issue
David Pasek, dpasek@vmware.com
VMware TAM
2020, Nov 25

2. Confidential │ ©2020 VMware, Inc. 2
Problem description - symptoms
Network throughput issue in Qlogic FastLinQ QL41xxx
• Containerized application running in OpenShift on top of vSphere has in some circumstances 3x higher network throughput on Intel X710 (1.9.5) in
comparison to Qlogic FastLinQ QL41xxx (3.11.16.0) with reasonable latency
• The issue was observed in production environment – the problem was mitigated by replacing Qlogic by Intel NIC
• VM on ESXi with Intel NIC can achieve total throughput 300 MB/s (~3Gbps) with application latency 20 ms
• VM on ESXi with Qlogic NIC can achieve total throughput 100 MB/s (~1Gbps) with application latency 2000 ms during the issue
• The question is why the same VM running on the identical server hardware (except NIC), can easily handle 300 MB/s (~ 3 Gbps) on Intel NIC and only
100 MB/s (~ 1 GBps) on Qlogic NIC?

3. Confidential │ ©2020 VMware, Inc.
Test results from HPE Lab
Full test plan with test methodology and network throughput
results is available on demand

4. Confidential │ ©2020 VMware, Inc. 4
HPE Lab
Lab environment details
2x ESXi hosts
• Server Platform: HPE ProLiant DL560 Gen10
• BIOS: U34 | Date (ISO-8601): 2020-04-08
• NIC1: Intel X710, driver i40en version: 1.9.5, firmware 10.51.5
• NIC2: QLogic QL41xxx, driver qedentv version: 3.11.16.0, firmware mfw 8.52.9.0 storm
• OS/Hypervisor: VMware ESXi 6.7.0 build-16075168 (6.7 U3)
1x Physical Switch
• 10Gb switch ports << network bottleneck by purpose, because customer is using 10Gb switch ports as well

5. Confidential │ ©2020 VMware, Inc. 5
Test Plan and HPE Lab results – 4.x.1
SUCCESS CRITERIA (Expectation)
• At least 5 Gbps (~500 MB/s) throughput
RESULTS
• Intel:
• Bandwidth/Throughput: 5.91 [Gbps]
• Qlogic:
• Bandwidth/Throughput: 9.39 [Gbps]
COMPARISON
• Qlogic outperform Intel by 59%
VMKernel - TCP (iperf) communication between two ESXi host consoles
iperf -s iperf -c
ESXi Host ESXi Host
tcp
TEST METHOD
• ESXi (DCUI/shell):
• esxcli network firewall set --enabled false
• /usr/lib/vmware/vsan/bin/iperf3.copy -s -B [APP-SERVER-01 vMotion IP]
• ESXi (DCUI/shell):
• esxcli network firewall set --enabled false
• /usr/lib/vmware/vsan/bin/iperf3.copy -t 300 -c [APP-SERVER-01 vMotion IP]

6. Confidential │ ©2020 VMware, Inc. 6
Test Plan and HPE Lab results – 4.x.2
VM - TCP (nuttcp) communication of 2 VMs across two ESXi hosts
SUCCESS CRITERIA (Expectation)
• At least 8 Gbps (~800 MB/s) throughput
• Note: 20 Gbps should be achievable in pure software stack.
RESULTS
• Intel:
• Bandwidth/Throughput: 9.1 [Gbps]
• Qlogic:
• Bandwidth/Throughput: 8.35 [Gbps]
COMPARISON
• Intel outperform Qlogic by 9%
DRS Rule to keep VMs on ESXi Host
App
Server
nuttcp -S
App
Client
nuttcp -t
DRS Rule to keep VMs on ESXi Host
ESXi Host ESXi Host
tcp
TEST METHOD
• 1x App Server:
• nuttcp -S -P 5000 -N 20
• 1x App Client:
• nuttcp -t -N 4 -P 5000 -T 300 APP-SERVER-01

7. Confidential │ ©2020 VMware, Inc. 7
Test Plan and HPE Lab results – 4.x.3
VM - TCP (iperf) communication of 2 VMs across two ESXi hosts
SUCCESS CRITERIA (Expectation)
• At least 8 Gbps (~800 MB/s) throughput
• Note: 20 Gbps should be achievable in pure software stack.
RESULTS
• Intel:
• Bandwidth/Throughput: 9.39 [Gbps]
• Qlogic:
• Bandwidth/Throughput: 8.43 [Gbps]
COMPARISON
• Intel outperform Qlogic by 11%
DRS Rule to keep VMs on ESXi Host
App
Server
iperf -s
App
Client
iperf -c
DRS Rule to keep VMs on ESXi Host
ESXi Host ESXi Host
tcp
TEST METHOD
• 1x App Server:
• iperf3 –s
• 1x App Client:
• iperf3 -c [APP-SERVER-01] -t 300 -b 25g -P 4

8. Confidential │ ©2020 VMware, Inc. 8
Test Plan and HPE Lab results – 4.x.4
VM - UDP (nuttcp – 64 KB) communication of 2 VMs across two ESXi hosts
SUCCESS CRITERIA (Expectation)
• At least 8 Gbps (~800 MB/s) throughput
• Note: 8 Gbps should be achievable in pure software stack.
RESULTS
• Intel:
• Bandwidth/Throughput: 9.1 [Gbps]
• Qlogic:
• Bandwidth/Throughput: 8.68 [Gbps]
COMPARISON
• Intel outperform Qlogic by 5%
TEST METHOD
• 1x App Server:
• nuttcp -S -P 5000 -N 20
• 1x App Client:
• nuttcp -u -Ru -l65507 -N 4 -P 5000 -T 300 -i APP-SERVER-01
DRS Rule to keep VMs on ESXi Host
App
Server
nuttcp -S
App
Client
nuttcp -u
DRS Rule to keep VMs on ESXi Host
ESXi Host ESXi Host
udp

9. Confidential │ ©2020 VMware, Inc. 9
Test Plan and HPE Lab results – 4.x.5
VM - UDP (iperf 64KB) communication of 2 VMs across two ESXi hosts
SUCCESS CRITERIA (Expectation)
• At least 8 Gbps (~800 MB/s) throughput
• Note: 8 Gbps should be achievable in pure software stack.
RESULTS
• Intel:
• Bandwidth/Throughput: 7.91 [Gbps]
• Qlogic:
• Bandwidth/Throughput: 7.91 [Gbps]
COMPARISON
• Equal Intel and Qlogic results
TEST METHOD
• 1x App Server:
• iperf3 –s
• 1x App Client:
• iperf3 -c [APP-SERVER-01] -u -t 300 -b 25g -P 4 -l 65507
udp

10. Confidential │ ©2020 VMware, Inc. 10
Test Plan and HPE Lab results – 4.x.6
VM - HTTP (nginx) communication of 2 VMs across two ESXi hosts
SUCCESS CRITERIA (Expectation)
• At least 8 Gbps (~800 MB/s) throughput
• Note: 22 Gbps should be achievable on faster physical network
RESULTS
• Intel:
• Bandwidth/Throughput: 5.15 [Gbps]
• Qlogic:
• Bandwidth/Throughput: 5.93 [Gbps]
COMPARISON
• Qlogic outperform Intel by 15%
FINDINGS
• Expectation is not met. It is strange we cannot
achieve at least 8 Gbps for HTTPS traffic.
DRS Rule to keep VMs on ESXi Host
App
Server
NGINX
App
Client
WRK
DRS Rule to keep VMs on ESXi Host
ESXi Host ESXi Host
http
TEST METHOD
• 1x App Client:
• taskset -c 0-8 /root/wrk -t 8 -c 8 -d 300s http://[APP-LB-01]/1M.txt
• 1x App Server:
• NGINX http server serving 1MB file

11. Confidential │ ©2020 VMware, Inc. 11
Test Plan - HPE Lab results – 4.x.7
VM - HTTPS (nginx) communication of 2 VMs across two ESXi hosts
SUCCESS CRITERIA (Expectation)
• At least 8 Gbps (~800 MB/s) throughput
• Note: 22 Gbps should be achievable in pure software stack. Intel CPU
Instructions AES-NI accelerates SSL, thus encryption penalty should be
mitigated.
RESULTS
• Intel:
• Bandwidth/Throughput: 8.72 [Gbps]
• Qlogic:
• Bandwidth/Throughput: 8.64 [Gbps]
COMPARISON
• Equal Intel and Qlogic results
FINDINGS
• Expectation is met.
• HTTPS overachieved HTTP results on both NICs, why? Because
on Intel QuickAssist Technology & OpenSSL & NGINX. See. LINK
DRS Rule to keep VMs on ESXi Host
App
Server
NGINX
App
Client
WRK
DRS Rule to keep VMs on ESXi Host
ESXi Host ESXi Host
https
TEST METHOD
• 1x App Client:
• taskset -c 0-8 /root/wrk -t 8 -c 8 -d 300s https://[APP-LB-01]/1M.txt
• 1x App Server:
• NGINX http server serving 1MB file

12. Confidential │ ©2020 VMware, Inc. 12
Test Plan - HPE Lab results – 4.x.8
VM - HTTP communication across two ESXi hosts via LoadBalancer no RSS
SUCCESS CRITERIA (Expectation)
• At least 4 Gbps (~400 MB/s) throughput
• Note: 4 Gbps should be achievable in pure software stack
RESULTS
• Intel:
• Bandwidth/Throughput: 4.97 [Gbps]
• Qlogic:
• Bandwidth/Throughput: 4.73 [Gbps]
COMPARISON
• Intel outperform Qlogic by 5%
FINDINGS
• Expectation is met.
DRS Rule to keep VMs on ESXi Host
App
Client
WRK
App
Server
NGINX
App
Load Balancer
(Reverse Proxy)
NGINX
App
Server
NGINX
App
Server
NGINX
App
Server
NGINX
ESXi Host ESXi Host
DRS Rule to keep VMs on ESXi Host
http
TEST METHOD
• 1x App Client:
• taskset -c 0-8 /root/wrk -t 8 -c 8 -d 300s http://[APP-LB-01]/1M.txt
• 1x App Load balancer:
• NGINX L7 Load Balancer (reverse http proxy) to 4 members
• RSS not enabled in VM settings (default config)
• 4x App Server:
• NGINX http server serving 1MB file

13. Confidential │ ©2020 VMware, Inc. 13
Test Plan - HPE Lab results – 4.x.9
VM - HTTP communication across two ESXi hosts via LoadBalancer (RSS)
SUCCESS CRITERIA (Expectation)
• At least 4 Gbps (~400 MB/s) throughput
• Note: 4 Gbps should be achievable in pure software stack
RESULTS
• Intel:
• Bandwidth/Throughput: 5.18 [Gbps]
• Qlogic:
• Bandwidth/Throughput: 6.58 [Gbps]
COMPARISON
• Qlogic outperform Intel by 27%
FINDINGS
• Expectation is met.
• RSS has positive impact on performance in Qlogic
• RSS is not supported on Intel NIC driver, no impact
DRS Rule to keep VMs on ESXi Host
App
Client
WRK
App
Server
NGINX
App
Load Balancer
(Reverse Proxy)
NGINX
App
Server
NGINX
App
Server
NGINX
App
Server
NGINX
ESXi Host ESXi Host
DRS Rule to keep VMs on ESXi Host
http
TEST METHOD
• 1x App Client:
• taskset -c 0-8 /root/wrk -t 8 -c 8 -d 300s http://[APP-LB-01]/1M.txt
• 1x App Load balancer:
• NGINX L7 Load Balancer (reverse http proxy) to 4 members
• End-to-end RSS enabled
• 4x App Server:
• NGINX http server serving 1MB file

14. Confidential │ ©2020 VMware, Inc.
Thank You