I/O virtualization with InfiniBand and 40 Gigabit Ethernet
Ali Ayoub – Staff Architect
VMworld 2013 San Francisco
© 2013 Mellanox Technologies 2
Remote Direct Memory Access
What is RDMA?
RDMA
Engine
Ethernet
(RoCE)
InfiniBand
© 2013 Mellanox Technologies 3
Virtual Protocol Interconnect Technology
VPI
Eth
InfiniBand
What is VPI?
© 2013 Mellanox Technologies 4
IPoIB
iSER/SRP
iSER
• 10GigE
• 40GigE
VPI
Eth
InfiniBand
InfiniBand
Ethernet
Mellanox Software Solutions
© 2013 Mellanox Technologies 5
Mellanox Software Solutions
RDMA
VPI
TCP/IP
10GigE
40GigE
iSCSI
NFS
NFS
SRP
iSER
IPoIB
TCP/IP Storage
Solutions
RDMA Storage
Solutions
Network
Solutions
Ethernet
InfiniBand
© 2013 Mellanox Technologies 6
Mellanox Software Solutions
10GigE
40GigE
IPoIB
SRP
iSER
iSCSI
NFS
NFS
RDMA Storage
Solutions
TCP/IP Storage
Solutions
Network
Solutions
CPU
Utilization
Bandwidth
Latency
Consolidation
Performance
More Virtual
Machine
© 2013 Mellanox Technologies 7
Mellanox Software Solutions
10GigE
40GigE
IPoIB
SRP
iSER
RDMA Storage
Solutions
Network
Solutions
vSphere 5.5 In-box
Part of ESX Distribution
Downloadable from vmware.com
vSphere 5.5 PVSP
Partner Verified and Supported Products
Downloadable from mellanox.com
© 2013 Mellanox Technologies 8
40GigE Performance Results
36Gbps I/O
Bandwidth on
vSphere 5.x !
© 2013 Mellanox Technologies 9
Single VM Migration Acceleration
0.00
5.00
10.00
15.00
20.00
Time(sec)
Migration of Passive VM*
10GE 40GE
0.00
10.00
20.00
30.00
40.00
50.00
Time(sec)
Migration of Active VM*
10GE C 40GE C
>200%
2X to 3X Faster Execution Time over 40GigE
>350%
One VM Migration with VMware vMotion over
10/40GigE NIC I/O Controller
* 16GB Memory
© 2013 Mellanox Technologies 10
Multiple VM Migration Acceleration
Enabling Concurrent Migration of More VMs over 40GigE
vMotion Average Migration Duration
Duration[mm:ss]
# VM
Multiple VM Migration with VMware vMotion over
10/40GigE NIC I/O Controller
© 2013 Mellanox Technologies 11
iSER Performance Results
190 IOPS on
vSphere 5
174
176
178
180
182
184
186
188
190
192
2 4 6 8
#VMs/LUNs (LUN per VM)
IOPS
35.2
35.3
35.4
35.5
35.6
35.7
35.8
35.9
36
36.1
2 4 6 8
#VMs/LUNs (LUN per VM)
Bandwidth (Gbps)
© 2013 Mellanox Technologies 12
SRIOV
© 2013 Mellanox Technologies 13
IPoIB
© 2013 Mellanox Technologies 14
iSER/SRP
© 2013 Mellanox Technologies 15
Leading Supplier of End-to-End Interconnect Solutions
Virtual Protocol Interconnect
Storage
Front / Back-End
Server / Compute Switch / Gateway
56G IB & FCoIB 56G InfiniBand
10/40/56GbE & FCoE 10/40/56GbE
Virtual Protocol Interconnect
Host/Fabric SoftwareICs Switches/GatewaysAdapter Cards Cables
Comprehensive End-to-End InfiniBand and Ethernet Portfolio
© 2013 Mellanox Technologies
Mellanox 40GbE In-Box with VMware vSphere 5.5
© 2013 Mellanox Technologies

I/O virtualization with InfiniBand and 40 Gigabit Ethernet

  • 1.
    I/O virtualization withInfiniBand and 40 Gigabit Ethernet Ali Ayoub – Staff Architect VMworld 2013 San Francisco
  • 2.
    © 2013 MellanoxTechnologies 2 Remote Direct Memory Access What is RDMA? RDMA Engine Ethernet (RoCE) InfiniBand
  • 3.
    © 2013 MellanoxTechnologies 3 Virtual Protocol Interconnect Technology VPI Eth InfiniBand What is VPI?
  • 4.
    © 2013 MellanoxTechnologies 4 IPoIB iSER/SRP iSER • 10GigE • 40GigE VPI Eth InfiniBand InfiniBand Ethernet Mellanox Software Solutions
  • 5.
    © 2013 MellanoxTechnologies 5 Mellanox Software Solutions RDMA VPI TCP/IP 10GigE 40GigE iSCSI NFS NFS SRP iSER IPoIB TCP/IP Storage Solutions RDMA Storage Solutions Network Solutions Ethernet InfiniBand
  • 6.
    © 2013 MellanoxTechnologies 6 Mellanox Software Solutions 10GigE 40GigE IPoIB SRP iSER iSCSI NFS NFS RDMA Storage Solutions TCP/IP Storage Solutions Network Solutions CPU Utilization Bandwidth Latency Consolidation Performance More Virtual Machine
  • 7.
    © 2013 MellanoxTechnologies 7 Mellanox Software Solutions 10GigE 40GigE IPoIB SRP iSER RDMA Storage Solutions Network Solutions vSphere 5.5 In-box Part of ESX Distribution Downloadable from vmware.com vSphere 5.5 PVSP Partner Verified and Supported Products Downloadable from mellanox.com
  • 8.
    © 2013 MellanoxTechnologies 8 40GigE Performance Results 36Gbps I/O Bandwidth on vSphere 5.x !
  • 9.
    © 2013 MellanoxTechnologies 9 Single VM Migration Acceleration 0.00 5.00 10.00 15.00 20.00 Time(sec) Migration of Passive VM* 10GE 40GE 0.00 10.00 20.00 30.00 40.00 50.00 Time(sec) Migration of Active VM* 10GE C 40GE C >200% 2X to 3X Faster Execution Time over 40GigE >350% One VM Migration with VMware vMotion over 10/40GigE NIC I/O Controller * 16GB Memory
  • 10.
    © 2013 MellanoxTechnologies 10 Multiple VM Migration Acceleration Enabling Concurrent Migration of More VMs over 40GigE vMotion Average Migration Duration Duration[mm:ss] # VM Multiple VM Migration with VMware vMotion over 10/40GigE NIC I/O Controller
  • 11.
    © 2013 MellanoxTechnologies 11 iSER Performance Results 190 IOPS on vSphere 5 174 176 178 180 182 184 186 188 190 192 2 4 6 8 #VMs/LUNs (LUN per VM) IOPS 35.2 35.3 35.4 35.5 35.6 35.7 35.8 35.9 36 36.1 2 4 6 8 #VMs/LUNs (LUN per VM) Bandwidth (Gbps)
  • 12.
    © 2013 MellanoxTechnologies 12 SRIOV
  • 13.
    © 2013 MellanoxTechnologies 13 IPoIB
  • 14.
    © 2013 MellanoxTechnologies 14 iSER/SRP
  • 15.
    © 2013 MellanoxTechnologies 15 Leading Supplier of End-to-End Interconnect Solutions Virtual Protocol Interconnect Storage Front / Back-End Server / Compute Switch / Gateway 56G IB & FCoIB 56G InfiniBand 10/40/56GbE & FCoE 10/40/56GbE Virtual Protocol Interconnect Host/Fabric SoftwareICs Switches/GatewaysAdapter Cards Cables Comprehensive End-to-End InfiniBand and Ethernet Portfolio
  • 16.
    © 2013 MellanoxTechnologies Mellanox 40GbE In-Box with VMware vSphere 5.5
  • 17.
    © 2013 MellanoxTechnologies