Your SlideShare is downloading. ×
05 Feb, 2013     SAKURA Internet Research CenterSenior Researcher / Naoto MATSUMOTO
HighGig Market Overview 56Gbps-HCA $1,311-                      Over $16,000- 56Gbps-IBSW/40GbE-SW 40Gbps-HCA             ...
HighGig Network Benchmark   RAMDISK                           SERVER                                                      ...
InfiniBand-Ethernet Interconnect                                                                     Peak (RX) 8.23 Gbit/s...
Cost Effective IP networking model                                                                             Flexibility...
(C)Copyright 1996-2010 SAKURA Internet Inc.
Infiniband Device list                                     40/56Gbit/s Infiniband mode                                    ...
IB Networking Example               SERVER         OpenSM       *OpenSM                                            InfiniB...
Install OFED for Linux1) Download MLNX_OFED# uname -aLinux 2.6.32-220.el6.x86_64 #1 SMP Sat Dec 10 17:04:11 CST 2011# wget...
Install OFED for VMware ESXi 5.11) Enable ESXi Shell & SSHTroubleshooting Options > Enable ESXi Shell, Enable SSH and SSH ...
Install OFED for Windows  1) DownloadMLNX_OFED from Mellanox Web page.  2) Install MLNX_OFED Package.
IB Fabric Example                    VM      VM      VM      VM                     VMware          VMware  Windows       ...
IB Diagnostic Commands# ibhosts       *show InfiniBand host nodes in topologyCa    : 0x0002c9030010c36c ports 2 "hostname ...
Thanks for your interest.SAKURA Internet Research Center.
Upcoming SlideShare
Loading in...5
×

Mutating IP Network Model Ethernet-InfiniBand Interconnect

2,246

Published on

Mutating IP Network Model
Ethernet-InfiniBand Interconnect

05 Feb, 2013
SAKURA Internet Research Center
Senior Researcher / Naoto MATSUMOTO

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,246
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
1
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Transcript of "Mutating IP Network Model Ethernet-InfiniBand Interconnect"

  1. 1. 05 Feb, 2013 SAKURA Internet Research CenterSenior Researcher / Naoto MATSUMOTO
  2. 2. HighGig Market Overview 56Gbps-HCA $1,311- Over $16,000- 56Gbps-IBSW/40GbE-SW 40Gbps-HCA $814- $8,280 - $11,750 10GbE-SW(48-64P) 40GbE-NIC $779- $1,923 - $6,250 10GbE-NIC $299- 40Gbps-IBSW(8-18-36P)$Network Adapter $Network SwitchLowest Price (Dual-Port) Lowest Price (8-18-36-48-64-Port) SOURCE: © 2013 Colfax International.
  3. 3. HighGig Network Benchmark RAMDISK SERVER SERVER RAMDISK 5.58 Gbit/sec using 8.00 Gbit/sec THTTPD 13.68 Gbit/sec RCOPY 18.23 Gbit/secCorei7-3930K CPU @ 3.20GHz / 32GB DDR3-DIMM / PCI Express 3.0 / Mellanox Connect-X3 VPI Card (10/40/56Gbit/s) SOURCE: SAKURA Internet Research Center. 12/2012 rev2 Project THORN.using “wget + thttpd” and “rcopy libcm package” for 10GB DATA Chunk Transfer.
  4. 4. InfiniBand-Ethernet Interconnect Peak (RX) 8.23 Gbit/s 709,288 pps (MTU1500) Peak (RX) 308.99 Mbit/s 791,004 pps (MTU64) Packet Generator VyattaCore 6.5R1[VM] SERVER CLIENT VMware ESXi 5.1 SERVER Ethernet InfiniBand SERVER 10Gbit/s 40/56Gbit/s IP over Ethernet Network IP over Infiniband NetworkCorei7-3930K CPU @ 3.20GHz / 32GB DDR3-DIMM / PCI Express 3.0 / Mellanox Connect-X3 VPI Card (10/40/56Gbit/s) SOURCE: SAKURA Internet Research Center. 02/2013 Project THORN.using linux standard pktgen.
  5. 5. Cost Effective IP networking model Flexibility /Cost EffectiveExisting Infrastructure / Resource High Performance * 48/64 port 10GbE Switch $8,280 - $11,750 * 8/18/36 port 40G IBSwitch $1,923 - $6,250 Cost > Ethernet IPoIB Gateway InfiniBand (Software/Hardware) 10Gbit/s 40/56Gbit/s IP over Ethernet Network IP over Infiniband Network SOURCE: © 2013 Colfax International. SOURCE: SAKURA Internet Research Center. 02/2013 Project THORN.
  6. 6. (C)Copyright 1996-2010 SAKURA Internet Inc.
  7. 7. Infiniband Device list 40/56Gbit/s Infiniband mode 10/40Gbit/s Ethernet mode *Mellanox VPI (Virtual Protocol Interconnect) Only Infiniband HCA (Host Channel Adapter) Infiniband 40/56Gbit/s QDR Switch QSFP Cable (Copper/Fibre) + Infiniband OFED (OpenFabrics Enterprise Distribution) Software
  8. 8. IB Networking Example SERVER OpenSM *OpenSM InfiniBand subnet manager and administration Loop-Free IB-SWITCH OpenSM OpenSM IB-SWITCHIB-SWITCH IB-SWITCH IB-SWITCH IB-SWITCH
  9. 9. Install OFED for Linux1) Download MLNX_OFED# uname -aLinux 2.6.32-220.el6.x86_64 #1 SMP Sat Dec 10 17:04:11 CST 2011# wget http://mellanox.com/downloads/ofed/MLNX_OFED_LINUX-1.5.3-3.1.0-rhel6.2-x86_64.tgz2) Install MLNX_OFED Package# yum install tcl tk glibc-develglibc-devel.i686# tar xzvf ./MLNX_OFED_LINUX-1.5.3-3.1.0-rhel6.2-x86_64.tgz# ./MLNX_OFED_LINUX-1.5.3-3.1.0-rhel6.2-x86_64/mlnxofedinstall# chkcnfig opensmd on; opensm –c /etc/opensm/opensm.conf# sync; sync; sync; reboot
  10. 10. Install OFED for VMware ESXi 5.11) Enable ESXi Shell & SSHTroubleshooting Options > Enable ESXi Shell, Enable SSH and SSH login.# vmware -vVMware ESXi 5.1.0 build-7997332) Download/Install MLNX_OFED# cd /opt# wget http://mellanox.com/downloads/Drivers/MLNX-OFED-ESX-1.8.0.0.zip# esxcli software vib install -d /opt/MLNX-OFED-ESX-1.8.0.0.zip# sync; sync; sync; reboot –f# esxcfg-nics –lvmnic_ib0 0000:01:00.00 ib_ipoib Up 56252Mbps Full00:02:c9:34:1c:f1 1500 Mellanox Technologies MT27500 Family [ConnectX-3]
  11. 11. Install OFED for Windows 1) DownloadMLNX_OFED from Mellanox Web page. 2) Install MLNX_OFED Package.
  12. 12. IB Fabric Example VM VM VM VM VMware VMware Windows Linux Windows Linux Windows Linux 40/56Gbit/s Windows Linux IP over Infiniband NetworkHigh Speed Server Interconnect Fabric for Mixed PHYSICAL and VIRTUAL.
  13. 13. IB Diagnostic Commands# ibhosts *show InfiniBand host nodes in topologyCa : 0x0002c9030010c36c ports 2 "hostname HCA-2"Ca : 0x0002c9030010c360 ports 2 "hostname HCA-2"Ca : 0x0002c9030010c328 ports 2 "sl6 HCA-1# iblinkinfo *report link info for all links in the fabricSwitch 0x000b8cffff006f38 MT43132 Mellanox Technologies::2 6[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 8 1[ ] "sl6 HCA-1" ( )2 7[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 9 1[ ] "hostname HCA-2" ( )2 8[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 10 1[ ] "hostname HCA-2" ( )# ibtracert 9 10 *trace InfiniBand pathFrom ca {0x0002c9030010c360} portnum 1 lid 9-9 "hostname HCA-2"[1] -> switch port {0x000b8cffff006f38}[7] lid 2-2 "MT43132 Mellanox Technologies"[8] -> ca port {0x0002c9030010c36d}[1] lid 10-10 "hostname HCA-2"To ca {0x0002c9030010c36c} portnum 1 lid 10-10 "hostname HCA-2"
  14. 14. Thanks for your interest.SAKURA Internet Research Center.

×