SlideShare a Scribd company logo
1 of 18
Download to read offline
InfiniBand in the Lab

Erik Bussink

@ErikBussink

www.bussink.ch

www.VSAN.info

"1
InfiniBand

an alternative

to 10Gb

Fast

Cheap

"2
Price comparison on Adapters

"3
Connectors
• For SDR (10Gb/s) and DDR (20Gb/s) use the CX4 connectors





• For QDR (40Gb/s) and FDR (56Gb/s) use QSFP connectors

"4
Connectors

"5
And switches too...
Example... 24 ports DDR (20Gb/s) for £212

Latest firmware (4.2.5) has embedded Subnet Manager

"6
What InfiniBand Offers

• Lowest Layer for scalable IO interconnect

• High-Performance (SDR 10Gb, DDR 20Gb, QDR 40Gb)

• Low Latency

• Reliable Switch Fabric

• Offers higher layers of functionality

• Application Clustering

• Fast Inter Process Communications

• Storage Area Networks

"7
InfiniBand Layers
• InfiniBand Adapters (Mellanox ConnectX Family)

• Drivers (mlx4_en-mlnx-1.6.1.2-471530.zip)

• Mellanox OFED 1.8.2 Package for ESXi 5.x

• Provides ConnectX family low-level drivers

• Provides kernel modules for InfiniBand (ib_ipoib & ib_srp)

• OpenSM Package (ib-opensm-3.3.16-64.x86_64.vib)

• Config partitions.conf for HCA

• Configure vmnic_ib in vSphere 5.5.0

"8
OpenFabrics Enterprise Distribution (OFED)


"9
Installing InfiniBand on vSphere 5.5
• Mellanox drivers are in vSphere 5.5, and they support 40GbE inbox
when using ConnectX-3 and SwitchX products.


• If not same HCA you need to uninstall the Mellanox drivers

• esxcli software vib remove -n=net-mlx4-en -n=net-mlx4-core

• Reboot ESXi host

• Install the Mellanox drivers net-mlx4-1.6.1.2

• Install the Mellanox OFED 1.8.2

• Install the OpenSM 3.3.15-x86_64 or 3.3.16-x86_64

• Reboot ESXi host

• Stop OpenSM "/etc/init.d/opensmd stop"

• Disable the OpenSM with "chkconfig opensmd off" (only one needed
if no HW SM)

"10
"11
Configure MTU and OpenSM

Partitions.conf contains Protocol identifiers, like IPoIB.

"12
Physical adapters

"13
InfiniBand IPoIB backbone for VSAN

"14
My hosted lab
• Voltaire 9024D (DDR) 24x 20GBps (without SubnetManager)

• Silverstorm 9024-CU24-ST2 24x 10GBps with SubnetManager)

"15
My hosted lab
• Voltaire 9024D (DDR) 24x 20GBps (without SubnetManager)

• Silverstorm 9024-CU24-ST2 (with SubnetManager)

"16
My hosted lab (Compute & Storage)
• 3x Cisco UCS C200M2 VSAN Storage Nodes

• 2X Cisco UCS C210M2 VSAN Compute Nodes & FVP Nodes

"17
InfiniBand in the Lab
Fast & Cheap








Erik Bussink


Thanks to Raphael Schitz,William Lam,
Vladan Seget, Gregory Roche


@ErikBussink




www.bussink.ch

www.VSAN.info




"18

More Related Content

What's hot

What's hot (20)

Gluster as Block Store in Containers
Gluster as Block Store in ContainersGluster as Block Store in Containers
Gluster as Block Store in Containers
 
Cncf meetup kubespray
Cncf meetup kubesprayCncf meetup kubespray
Cncf meetup kubespray
 
CoreOS Overview
CoreOS OverviewCoreOS Overview
CoreOS Overview
 
The Open vSwitch and OVN Projects
The Open vSwitch and OVN ProjectsThe Open vSwitch and OVN Projects
The Open vSwitch and OVN Projects
 
XPDS14 - OSv - A Modern Semi-POSIX LibraryOS - Glauber Costa, Cloudius Systems
XPDS14 - OSv - A Modern Semi-POSIX LibraryOS - Glauber Costa, Cloudius SystemsXPDS14 - OSv - A Modern Semi-POSIX LibraryOS - Glauber Costa, Cloudius Systems
XPDS14 - OSv - A Modern Semi-POSIX LibraryOS - Glauber Costa, Cloudius Systems
 
RISC-V on Edge: Porting EVE and Alpine Linux to RISC-V
RISC-V on Edge: Porting EVE and Alpine Linux to RISC-VRISC-V on Edge: Porting EVE and Alpine Linux to RISC-V
RISC-V on Edge: Porting EVE and Alpine Linux to RISC-V
 
XPDDS17: Xen Test Lab: The Installation and Our Plans - Ian Jackson, Citrix
XPDDS17: Xen Test Lab: The Installation and Our Plans - Ian Jackson, CitrixXPDDS17: Xen Test Lab: The Installation and Our Plans - Ian Jackson, Citrix
XPDDS17: Xen Test Lab: The Installation and Our Plans - Ian Jackson, Citrix
 
Rethinking the OS
Rethinking the OSRethinking the OS
Rethinking the OS
 
Integrating gluster fs,_qemu_and_ovirt-vijay_bellur-linuxcon_eu_2013
Integrating gluster fs,_qemu_and_ovirt-vijay_bellur-linuxcon_eu_2013Integrating gluster fs,_qemu_and_ovirt-vijay_bellur-linuxcon_eu_2013
Integrating gluster fs,_qemu_and_ovirt-vijay_bellur-linuxcon_eu_2013
 
Disaster Recovery Strategies Using oVirt's new Storage Connection Management ...
Disaster Recovery Strategies Using oVirt's new Storage Connection Management ...Disaster Recovery Strategies Using oVirt's new Storage Connection Management ...
Disaster Recovery Strategies Using oVirt's new Storage Connection Management ...
 
Fundamental Virtualisasi di openSUSE
Fundamental Virtualisasi di openSUSEFundamental Virtualisasi di openSUSE
Fundamental Virtualisasi di openSUSE
 
LMAX Disruptor - High Performance Inter-Thread Messaging Library
LMAX Disruptor - High Performance Inter-Thread Messaging LibraryLMAX Disruptor - High Performance Inter-Thread Messaging Library
LMAX Disruptor - High Performance Inter-Thread Messaging Library
 
Open v switch20150410b
Open v switch20150410bOpen v switch20150410b
Open v switch20150410b
 
Kvm forum 2013 - future integration points for oVirt storage
Kvm forum 2013 - future integration points for oVirt storageKvm forum 2013 - future integration points for oVirt storage
Kvm forum 2013 - future integration points for oVirt storage
 
Persistent Storage with Containers with Kubernetes & OpenShift
Persistent Storage with Containers with Kubernetes & OpenShiftPersistent Storage with Containers with Kubernetes & OpenShift
Persistent Storage with Containers with Kubernetes & OpenShift
 
Ceph on Windows
Ceph on WindowsCeph on Windows
Ceph on Windows
 
Virtualization Management The oVirt Way (August Penguin 2015)
Virtualization Management The oVirt Way (August Penguin 2015)Virtualization Management The oVirt Way (August Penguin 2015)
Virtualization Management The oVirt Way (August Penguin 2015)
 
OpenNebula Conf 2014 | Lightning talk: OpenNebula at Etnetera by Jan Horacek
OpenNebula Conf 2014 | Lightning talk: OpenNebula at Etnetera by Jan HoracekOpenNebula Conf 2014 | Lightning talk: OpenNebula at Etnetera by Jan Horacek
OpenNebula Conf 2014 | Lightning talk: OpenNebula at Etnetera by Jan Horacek
 
Ovirt and gluster_hyperconvergence_devconf-2016
Ovirt and gluster_hyperconvergence_devconf-2016Ovirt and gluster_hyperconvergence_devconf-2016
Ovirt and gluster_hyperconvergence_devconf-2016
 
OVN - Basics and deep dive
OVN - Basics and deep diveOVN - Basics and deep dive
OVN - Basics and deep dive
 

Similar to InfiniBand in the Lab (London VMUG)

Similar to InfiniBand in the Lab (London VMUG) (20)

RISC V in Spacer
RISC V in SpacerRISC V in Spacer
RISC V in Spacer
 
RouterOS v6
RouterOS v6RouterOS v6
RouterOS v6
 
Corralling Big Data at TACC
Corralling Big Data at TACCCorralling Big Data at TACC
Corralling Big Data at TACC
 
Cumulus Linux 2.2 Overview
Cumulus Linux 2.2 OverviewCumulus Linux 2.2 Overview
Cumulus Linux 2.2 Overview
 
100G Networking Berlin.pdf
100G Networking Berlin.pdf100G Networking Berlin.pdf
100G Networking Berlin.pdf
 
Introduction to DPDK
Introduction to DPDKIntroduction to DPDK
Introduction to DPDK
 
Janus & docker: friends or foe
Janus & docker: friends or foe Janus & docker: friends or foe
Janus & docker: friends or foe
 
GIST AI-X Computing Cluster
GIST AI-X Computing ClusterGIST AI-X Computing Cluster
GIST AI-X Computing Cluster
 
Ippevent : openshift Introduction
Ippevent : openshift IntroductionIppevent : openshift Introduction
Ippevent : openshift Introduction
 
Secure lustre on openstack
Secure lustre on openstackSecure lustre on openstack
Secure lustre on openstack
 
Nexenta at VMworld Hands-on Lab
Nexenta at VMworld Hands-on LabNexenta at VMworld Hands-on Lab
Nexenta at VMworld Hands-on Lab
 
High performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User GroupHigh performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User Group
 
[OpenStack Days Korea 2016] Track3 - OpenStack on 64-bit ARM with X-Gene
[OpenStack Days Korea 2016] Track3 - OpenStack on 64-bit ARM with X-Gene[OpenStack Days Korea 2016] Track3 - OpenStack on 64-bit ARM with X-Gene
[OpenStack Days Korea 2016] Track3 - OpenStack on 64-bit ARM with X-Gene
 
D108636GC10_les01.pptx
D108636GC10_les01.pptxD108636GC10_les01.pptx
D108636GC10_les01.pptx
 
QCT Ceph Solution - Design Consideration and Reference Architecture
QCT Ceph Solution - Design Consideration and Reference ArchitectureQCT Ceph Solution - Design Consideration and Reference Architecture
QCT Ceph Solution - Design Consideration and Reference Architecture
 
QCT Ceph Solution - Design Consideration and Reference Architecture
QCT Ceph Solution - Design Consideration and Reference ArchitectureQCT Ceph Solution - Design Consideration and Reference Architecture
QCT Ceph Solution - Design Consideration and Reference Architecture
 
Fast datastacks - fast and flexible nfv solution stacks leveraging fd.io
Fast datastacks - fast and flexible nfv solution stacks leveraging fd.ioFast datastacks - fast and flexible nfv solution stacks leveraging fd.io
Fast datastacks - fast and flexible nfv solution stacks leveraging fd.io
 
Oow 2008 yahoo_pie-db
Oow 2008 yahoo_pie-dbOow 2008 yahoo_pie-db
Oow 2008 yahoo_pie-db
 
Running Applications on the NetBSD Rump Kernel by Justin Cormack
Running Applications on the NetBSD Rump Kernel by Justin Cormack Running Applications on the NetBSD Rump Kernel by Justin Cormack
Running Applications on the NetBSD Rump Kernel by Justin Cormack
 
pps Matters
pps Matterspps Matters
pps Matters
 

Recently uploaded

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
panagenda
 

Recently uploaded (20)

Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ..."I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
 
Ransomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdfRansomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdf
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 

InfiniBand in the Lab (London VMUG)

  • 1. InfiniBand in the Lab Erik Bussink @ErikBussink www.bussink.ch www.VSAN.info "1
  • 3. Price comparison on Adapters "3
  • 4. Connectors • For SDR (10Gb/s) and DDR (20Gb/s) use the CX4 connectors • For QDR (40Gb/s) and FDR (56Gb/s) use QSFP connectors "4
  • 6. And switches too... Example... 24 ports DDR (20Gb/s) for £212 Latest firmware (4.2.5) has embedded Subnet Manager "6
  • 7. What InfiniBand Offers • Lowest Layer for scalable IO interconnect • High-Performance (SDR 10Gb, DDR 20Gb, QDR 40Gb) • Low Latency • Reliable Switch Fabric • Offers higher layers of functionality • Application Clustering • Fast Inter Process Communications • Storage Area Networks "7
  • 8. InfiniBand Layers • InfiniBand Adapters (Mellanox ConnectX Family) • Drivers (mlx4_en-mlnx-1.6.1.2-471530.zip) • Mellanox OFED 1.8.2 Package for ESXi 5.x • Provides ConnectX family low-level drivers • Provides kernel modules for InfiniBand (ib_ipoib & ib_srp) • OpenSM Package (ib-opensm-3.3.16-64.x86_64.vib) • Config partitions.conf for HCA • Configure vmnic_ib in vSphere 5.5.0 "8
  • 10. Installing InfiniBand on vSphere 5.5 • Mellanox drivers are in vSphere 5.5, and they support 40GbE inbox when using ConnectX-3 and SwitchX products. • If not same HCA you need to uninstall the Mellanox drivers • esxcli software vib remove -n=net-mlx4-en -n=net-mlx4-core • Reboot ESXi host • Install the Mellanox drivers net-mlx4-1.6.1.2 • Install the Mellanox OFED 1.8.2 • Install the OpenSM 3.3.15-x86_64 or 3.3.16-x86_64 • Reboot ESXi host • Stop OpenSM "/etc/init.d/opensmd stop" • Disable the OpenSM with "chkconfig opensmd off" (only one needed if no HW SM) "10
  • 11. "11
  • 12. Configure MTU and OpenSM Partitions.conf contains Protocol identifiers, like IPoIB. "12
  • 15. My hosted lab • Voltaire 9024D (DDR) 24x 20GBps (without SubnetManager) • Silverstorm 9024-CU24-ST2 24x 10GBps with SubnetManager) "15
  • 16. My hosted lab • Voltaire 9024D (DDR) 24x 20GBps (without SubnetManager) • Silverstorm 9024-CU24-ST2 (with SubnetManager) "16
  • 17. My hosted lab (Compute & Storage) • 3x Cisco UCS C200M2 VSAN Storage Nodes • 2X Cisco UCS C210M2 VSAN Compute Nodes & FVP Nodes "17
  • 18. InfiniBand in the Lab Fast & Cheap Erik Bussink Thanks to Raphael Schitz,William Lam, Vladan Seget, Gregory Roche @ErikBussink www.bussink.ch www.VSAN.info "18