SlideShare a Scribd company logo
InfiniBand in the Lab

Erik Bussink
@ErikBussink
www.bussink.ch
www.VSAN.info

1
Executive
Summary

Fast
Cheap
2
Who uses
InfiniBand for
Storage ?
• EMC Isilon
• EMC xtremIO
• PureStorage
• Oracle Exadata
• Nexenta
• TeraData
• Gluster

3
What InfiniBand
Offers
• Lowest Layer for scalable IO interconnect
• High-Performance
• Low Latency
• Reliable Switch Fabric
• Offers higher layers of functionality
• Application Clustering
• Fast Inter Process Communications
• Storage Area Networks

4
InfiniBand physical layer
• Physical layer based on 802.3z specification operating at 2.5Gb/s
same standard as 10G ethernet (3.125Gb/s)

• InfiniBand layer 2 switching uses 16 bit local address (LID), so limited
to 2^16 nodes on a subnet.

5
6
Connectors
• For SDR (10Gb/s) and DDR (20Gb/s) use the CX4 connectors

• For QDR (40Gb/s) and FDR (56Gb/s) use QSFP connectors

7
Price comparison on Adapters

8
And switches too...
Example... 24 ports DDR (20Gb/s) for $499
Latest firmware (4.2.5) has embedded Subnet Manager

9
Subnet Manager
• The Subnet Manager assigns Local IDentifiers (LIDs) to each port in
the IB fabric, and develops a routing table based off the assigned
LIDs

• Hardware based Subnet Manager are located in the Switches
• Software based Subnet Manager is located on a host connected to
the IB fabric

• The OpenSM from the OpenFabrics Alliance works on Windows and
Linux

10
OpenFabrics Enterprise
Distribution (OFED)

11
Installing InfiniBand on vSphere
5.1

12
Configure MTU and OpenSM
• esxcli system module paramters set y-m=mlx4_core -p=mtu_4k=1
• On Old switches (No 4K support) MTU is 2044 (2048-4) for IPoIB
• vi partitions.conf
• Add single line "Default=ox7fff,ipoib,mtu=5:ALL=full;"
• copy partitions.conf /scratch/opensm/adapter_1_hca/
• copy partitions.conf /scratch/opensm/adapter_2_hca/

13
14
Installing InfiniBand on vSphere
5.5
• Mellanox drivers are in vSphere 5.5, and they support 40GbE inbox
when using ConnectX-3and SwitchX products.

• If not same HCA you need to uninstall the Mellanox drivers
• esxcli software vib remove -n=net-mlx4-en -n=net-mlx4-core
• And reboot
• Then install the Mellanox OFED 1.8.2
• esxcli software vib install -d /tmp/MLX
• And reboot

15
Installing InfiniBand on vSphere 5.5

16
Copy partitions.conf
• Set MTU for mlx4
• Copy partitions.conf to /scratch/opensm/<HCA_Port>/

17
InfiniBand IPoIB backbone for VSAN

18
Configure VMkernel MTU
InfiniBand in the Lab
Fast & Cheap
Erik Bussink
Thanks to Raphael Schitz,William Lam,
Vladan Seget, Gregory Roche

@ErikBussink
www.bussink.ch
www.VSAN.info

20

More Related Content

What's hot

Integrating GlusterFS with iSCSI Target
Integrating GlusterFS with iSCSI TargetIntegrating GlusterFS with iSCSI Target
Integrating GlusterFS with iSCSI Target
ijsrd.com
 
Gluster as Block Store in Containers
Gluster as Block Store in ContainersGluster as Block Store in Containers
Gluster as Block Store in Containers
Gluster.org
 
The Open vSwitch and OVN Projects
The Open vSwitch and OVN ProjectsThe Open vSwitch and OVN Projects
The Open vSwitch and OVN Projects
LinuxCon ContainerCon CloudOpen China
 
CoreOS Overview
CoreOS OverviewCoreOS Overview
CoreOS Overview
Nikolay Yurin
 
XPDDS17: Xen Test Lab: The Installation and Our Plans - Ian Jackson, Citrix
XPDDS17: Xen Test Lab: The Installation and Our Plans - Ian Jackson, CitrixXPDDS17: Xen Test Lab: The Installation and Our Plans - Ian Jackson, Citrix
XPDDS17: Xen Test Lab: The Installation and Our Plans - Ian Jackson, Citrix
The Linux Foundation
 
RISC-V on Edge: Porting EVE and Alpine Linux to RISC-V
RISC-V on Edge: Porting EVE and Alpine Linux to RISC-VRISC-V on Edge: Porting EVE and Alpine Linux to RISC-V
RISC-V on Edge: Porting EVE and Alpine Linux to RISC-V
ScyllaDB
 
XPDS14 - OSv - A Modern Semi-POSIX LibraryOS - Glauber Costa, Cloudius Systems
XPDS14 - OSv - A Modern Semi-POSIX LibraryOS - Glauber Costa, Cloudius SystemsXPDS14 - OSv - A Modern Semi-POSIX LibraryOS - Glauber Costa, Cloudius Systems
XPDS14 - OSv - A Modern Semi-POSIX LibraryOS - Glauber Costa, Cloudius Systems
The Linux Foundation
 
Rethinking the OS
Rethinking the OSRethinking the OS
Kvm forum 2013 - future integration points for oVirt storage
Kvm forum 2013 - future integration points for oVirt storageKvm forum 2013 - future integration points for oVirt storage
Kvm forum 2013 - future integration points for oVirt storage
Sean Cohen
 
Integrating gluster fs,_qemu_and_ovirt-vijay_bellur-linuxcon_eu_2013
Integrating gluster fs,_qemu_and_ovirt-vijay_bellur-linuxcon_eu_2013Integrating gluster fs,_qemu_and_ovirt-vijay_bellur-linuxcon_eu_2013
Integrating gluster fs,_qemu_and_ovirt-vijay_bellur-linuxcon_eu_2013
Gluster.org
 
Open v switch20150410b
Open v switch20150410bOpen v switch20150410b
Open v switch20150410b
Richard Kuo
 
Disaster Recovery Strategies Using oVirt's new Storage Connection Management ...
Disaster Recovery Strategies Using oVirt's new Storage Connection Management ...Disaster Recovery Strategies Using oVirt's new Storage Connection Management ...
Disaster Recovery Strategies Using oVirt's new Storage Connection Management ...
Allon Mureinik
 
Fundamental Virtualisasi di openSUSE
Fundamental Virtualisasi di openSUSEFundamental Virtualisasi di openSUSE
Fundamental Virtualisasi di openSUSE
utianayuba
 
Virtualization Management The oVirt Way (August Penguin 2015)
Virtualization Management The oVirt Way (August Penguin 2015)Virtualization Management The oVirt Way (August Penguin 2015)
Virtualization Management The oVirt Way (August Penguin 2015)
Allon Mureinik
 
OpenNebula Conf 2014 | Lightning talk: OpenNebula at Etnetera by Jan Horacek
OpenNebula Conf 2014 | Lightning talk: OpenNebula at Etnetera by Jan HoracekOpenNebula Conf 2014 | Lightning talk: OpenNebula at Etnetera by Jan Horacek
OpenNebula Conf 2014 | Lightning talk: OpenNebula at Etnetera by Jan Horacek
NETWAYS
 
Persistent Storage with Containers with Kubernetes & OpenShift
Persistent Storage with Containers with Kubernetes & OpenShiftPersistent Storage with Containers with Kubernetes & OpenShift
Persistent Storage with Containers with Kubernetes & OpenShift
Red Hat Events
 
Ceph on Windows
Ceph on WindowsCeph on Windows
Ceph on Windows
Ceph Community
 
Ovirt and gluster_hyperconvergence_devconf-2016
Ovirt and gluster_hyperconvergence_devconf-2016Ovirt and gluster_hyperconvergence_devconf-2016
Ovirt and gluster_hyperconvergence_devconf-2016
Ramesh Nachimuthu
 
OVN - Basics and deep dive
OVN - Basics and deep diveOVN - Basics and deep dive
OVN - Basics and deep dive
Trinath Somanchi
 
openSUSE12.2 Review
openSUSE12.2 ReviewopenSUSE12.2 Review
openSUSE12.2 Review
SUSE Labs Taipei
 

What's hot (20)

Integrating GlusterFS with iSCSI Target
Integrating GlusterFS with iSCSI TargetIntegrating GlusterFS with iSCSI Target
Integrating GlusterFS with iSCSI Target
 
Gluster as Block Store in Containers
Gluster as Block Store in ContainersGluster as Block Store in Containers
Gluster as Block Store in Containers
 
The Open vSwitch and OVN Projects
The Open vSwitch and OVN ProjectsThe Open vSwitch and OVN Projects
The Open vSwitch and OVN Projects
 
CoreOS Overview
CoreOS OverviewCoreOS Overview
CoreOS Overview
 
XPDDS17: Xen Test Lab: The Installation and Our Plans - Ian Jackson, Citrix
XPDDS17: Xen Test Lab: The Installation and Our Plans - Ian Jackson, CitrixXPDDS17: Xen Test Lab: The Installation and Our Plans - Ian Jackson, Citrix
XPDDS17: Xen Test Lab: The Installation and Our Plans - Ian Jackson, Citrix
 
RISC-V on Edge: Porting EVE and Alpine Linux to RISC-V
RISC-V on Edge: Porting EVE and Alpine Linux to RISC-VRISC-V on Edge: Porting EVE and Alpine Linux to RISC-V
RISC-V on Edge: Porting EVE and Alpine Linux to RISC-V
 
XPDS14 - OSv - A Modern Semi-POSIX LibraryOS - Glauber Costa, Cloudius Systems
XPDS14 - OSv - A Modern Semi-POSIX LibraryOS - Glauber Costa, Cloudius SystemsXPDS14 - OSv - A Modern Semi-POSIX LibraryOS - Glauber Costa, Cloudius Systems
XPDS14 - OSv - A Modern Semi-POSIX LibraryOS - Glauber Costa, Cloudius Systems
 
Rethinking the OS
Rethinking the OSRethinking the OS
Rethinking the OS
 
Kvm forum 2013 - future integration points for oVirt storage
Kvm forum 2013 - future integration points for oVirt storageKvm forum 2013 - future integration points for oVirt storage
Kvm forum 2013 - future integration points for oVirt storage
 
Integrating gluster fs,_qemu_and_ovirt-vijay_bellur-linuxcon_eu_2013
Integrating gluster fs,_qemu_and_ovirt-vijay_bellur-linuxcon_eu_2013Integrating gluster fs,_qemu_and_ovirt-vijay_bellur-linuxcon_eu_2013
Integrating gluster fs,_qemu_and_ovirt-vijay_bellur-linuxcon_eu_2013
 
Open v switch20150410b
Open v switch20150410bOpen v switch20150410b
Open v switch20150410b
 
Disaster Recovery Strategies Using oVirt's new Storage Connection Management ...
Disaster Recovery Strategies Using oVirt's new Storage Connection Management ...Disaster Recovery Strategies Using oVirt's new Storage Connection Management ...
Disaster Recovery Strategies Using oVirt's new Storage Connection Management ...
 
Fundamental Virtualisasi di openSUSE
Fundamental Virtualisasi di openSUSEFundamental Virtualisasi di openSUSE
Fundamental Virtualisasi di openSUSE
 
Virtualization Management The oVirt Way (August Penguin 2015)
Virtualization Management The oVirt Way (August Penguin 2015)Virtualization Management The oVirt Way (August Penguin 2015)
Virtualization Management The oVirt Way (August Penguin 2015)
 
OpenNebula Conf 2014 | Lightning talk: OpenNebula at Etnetera by Jan Horacek
OpenNebula Conf 2014 | Lightning talk: OpenNebula at Etnetera by Jan HoracekOpenNebula Conf 2014 | Lightning talk: OpenNebula at Etnetera by Jan Horacek
OpenNebula Conf 2014 | Lightning talk: OpenNebula at Etnetera by Jan Horacek
 
Persistent Storage with Containers with Kubernetes & OpenShift
Persistent Storage with Containers with Kubernetes & OpenShiftPersistent Storage with Containers with Kubernetes & OpenShift
Persistent Storage with Containers with Kubernetes & OpenShift
 
Ceph on Windows
Ceph on WindowsCeph on Windows
Ceph on Windows
 
Ovirt and gluster_hyperconvergence_devconf-2016
Ovirt and gluster_hyperconvergence_devconf-2016Ovirt and gluster_hyperconvergence_devconf-2016
Ovirt and gluster_hyperconvergence_devconf-2016
 
OVN - Basics and deep dive
OVN - Basics and deep diveOVN - Basics and deep dive
OVN - Basics and deep dive
 
openSUSE12.2 Review
openSUSE12.2 ReviewopenSUSE12.2 Review
openSUSE12.2 Review
 

Similar to InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk

100G Networking Berlin.pdf
100G Networking Berlin.pdf100G Networking Berlin.pdf
100G Networking Berlin.pdf
JunZhao68
 
PLNOG 8: Ivan Pepelnjak - Data Center Fabrics - What Really Matters
PLNOG 8: Ivan Pepelnjak - Data Center Fabrics - What Really Matters PLNOG 8: Ivan Pepelnjak - Data Center Fabrics - What Really Matters
PLNOG 8: Ivan Pepelnjak - Data Center Fabrics - What Really Matters
PROIDEA
 
Linux one vs x86
Linux one vs x86 Linux one vs x86
Linux one vs x86
Diego Rodriguez
 
Linux one vs x86 18 july
Linux one vs x86 18 julyLinux one vs x86 18 july
Linux one vs x86 18 july
Diego Rodriguez
 
Ceph Day Amsterdam 2015 - Deploying flash storage for Ceph without compromisi...
Ceph Day Amsterdam 2015 - Deploying flash storage for Ceph without compromisi...Ceph Day Amsterdam 2015 - Deploying flash storage for Ceph without compromisi...
Ceph Day Amsterdam 2015 - Deploying flash storage for Ceph without compromisi...
Ceph Community
 
Exalogic workshop overview__hardwarev4
Exalogic workshop overview__hardwarev4Exalogic workshop overview__hardwarev4
Exalogic workshop overview__hardwarev4
Fran Navarro
 
Devicemgmt
DevicemgmtDevicemgmt
Devicemgmt
xyxz
 
Hitachi Data Systems and Brocade Build the Optimal Mainframe Storage Architec...
Hitachi Data Systems and Brocade Build the Optimal Mainframe Storage Architec...Hitachi Data Systems and Brocade Build the Optimal Mainframe Storage Architec...
Hitachi Data Systems and Brocade Build the Optimal Mainframe Storage Architec...
Hitachi Vantara
 
Mellanox Approach to NFV & SDN
Mellanox Approach to NFV & SDNMellanox Approach to NFV & SDN
Mellanox Approach to NFV & SDN
Mellanox Technologies
 
RISC V in Spacer
RISC V in SpacerRISC V in Spacer
RISC V in Spacer
klepsydratechnologie
 
Ceph Day SF 2015 - Deploying flash storage for Ceph without compromising perf...
Ceph Day SF 2015 - Deploying flash storage for Ceph without compromising perf...Ceph Day SF 2015 - Deploying flash storage for Ceph without compromising perf...
Ceph Day SF 2015 - Deploying flash storage for Ceph without compromising perf...
Ceph Community
 
Cisco Live! :: Introduction to IOS XR for Enterprises and Service Providers
Cisco Live! :: Introduction to IOS XR for Enterprises and Service ProvidersCisco Live! :: Introduction to IOS XR for Enterprises and Service Providers
Cisco Live! :: Introduction to IOS XR for Enterprises and Service Providers
Bruno Teixeira
 
Build the Optimal Mainframe Storage Architecture
Build the Optimal Mainframe Storage ArchitectureBuild the Optimal Mainframe Storage Architecture
Build the Optimal Mainframe Storage Architecture
Hitachi Vantara
 
Ln family portfolio_customer_presentation
Ln family portfolio_customer_presentationLn family portfolio_customer_presentation
Ln family portfolio_customer_presentation
Lenovo Data Center
 
Connecting an IBM PureFlex System to the Network
Connecting an IBM PureFlex System to the NetworkConnecting an IBM PureFlex System to the Network
Connecting an IBM PureFlex System to the Network
IBM India Smarter Computing
 
LCNA14: Why Use Xen for Large Scale Enterprise Deployments? - Konrad Rzeszute...
LCNA14: Why Use Xen for Large Scale Enterprise Deployments? - Konrad Rzeszute...LCNA14: Why Use Xen for Large Scale Enterprise Deployments? - Konrad Rzeszute...
LCNA14: Why Use Xen for Large Scale Enterprise Deployments? - Konrad Rzeszute...
The Linux Foundation
 
InfiniBand Essentials Every HPC Expert Must Know
InfiniBand Essentials Every HPC Expert Must KnowInfiniBand Essentials Every HPC Expert Must Know
InfiniBand Essentials Every HPC Expert Must Know
Mellanox Technologies
 
Implementation of Soft-core Processor on FPGA
Implementation of Soft-core Processor on FPGAImplementation of Soft-core Processor on FPGA
Implementation of Soft-core Processor on FPGA
Deepak Kumar
 
InfiniBand for the enterprise
InfiniBand for the enterpriseInfiniBand for the enterprise
InfiniBand for the enterprise
Anas Kanzoua
 
CommScope RUCKUS ICX Switching Configuration
CommScope RUCKUS ICX Switching ConfigurationCommScope RUCKUS ICX Switching Configuration
CommScope RUCKUS ICX Switching Configuration
Carla Nadin
 

Similar to InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk (20)

100G Networking Berlin.pdf
100G Networking Berlin.pdf100G Networking Berlin.pdf
100G Networking Berlin.pdf
 
PLNOG 8: Ivan Pepelnjak - Data Center Fabrics - What Really Matters
PLNOG 8: Ivan Pepelnjak - Data Center Fabrics - What Really Matters PLNOG 8: Ivan Pepelnjak - Data Center Fabrics - What Really Matters
PLNOG 8: Ivan Pepelnjak - Data Center Fabrics - What Really Matters
 
Linux one vs x86
Linux one vs x86 Linux one vs x86
Linux one vs x86
 
Linux one vs x86 18 july
Linux one vs x86 18 julyLinux one vs x86 18 july
Linux one vs x86 18 july
 
Ceph Day Amsterdam 2015 - Deploying flash storage for Ceph without compromisi...
Ceph Day Amsterdam 2015 - Deploying flash storage for Ceph without compromisi...Ceph Day Amsterdam 2015 - Deploying flash storage for Ceph without compromisi...
Ceph Day Amsterdam 2015 - Deploying flash storage for Ceph without compromisi...
 
Exalogic workshop overview__hardwarev4
Exalogic workshop overview__hardwarev4Exalogic workshop overview__hardwarev4
Exalogic workshop overview__hardwarev4
 
Devicemgmt
DevicemgmtDevicemgmt
Devicemgmt
 
Hitachi Data Systems and Brocade Build the Optimal Mainframe Storage Architec...
Hitachi Data Systems and Brocade Build the Optimal Mainframe Storage Architec...Hitachi Data Systems and Brocade Build the Optimal Mainframe Storage Architec...
Hitachi Data Systems and Brocade Build the Optimal Mainframe Storage Architec...
 
Mellanox Approach to NFV & SDN
Mellanox Approach to NFV & SDNMellanox Approach to NFV & SDN
Mellanox Approach to NFV & SDN
 
RISC V in Spacer
RISC V in SpacerRISC V in Spacer
RISC V in Spacer
 
Ceph Day SF 2015 - Deploying flash storage for Ceph without compromising perf...
Ceph Day SF 2015 - Deploying flash storage for Ceph without compromising perf...Ceph Day SF 2015 - Deploying flash storage for Ceph without compromising perf...
Ceph Day SF 2015 - Deploying flash storage for Ceph without compromising perf...
 
Cisco Live! :: Introduction to IOS XR for Enterprises and Service Providers
Cisco Live! :: Introduction to IOS XR for Enterprises and Service ProvidersCisco Live! :: Introduction to IOS XR for Enterprises and Service Providers
Cisco Live! :: Introduction to IOS XR for Enterprises and Service Providers
 
Build the Optimal Mainframe Storage Architecture
Build the Optimal Mainframe Storage ArchitectureBuild the Optimal Mainframe Storage Architecture
Build the Optimal Mainframe Storage Architecture
 
Ln family portfolio_customer_presentation
Ln family portfolio_customer_presentationLn family portfolio_customer_presentation
Ln family portfolio_customer_presentation
 
Connecting an IBM PureFlex System to the Network
Connecting an IBM PureFlex System to the NetworkConnecting an IBM PureFlex System to the Network
Connecting an IBM PureFlex System to the Network
 
LCNA14: Why Use Xen for Large Scale Enterprise Deployments? - Konrad Rzeszute...
LCNA14: Why Use Xen for Large Scale Enterprise Deployments? - Konrad Rzeszute...LCNA14: Why Use Xen for Large Scale Enterprise Deployments? - Konrad Rzeszute...
LCNA14: Why Use Xen for Large Scale Enterprise Deployments? - Konrad Rzeszute...
 
InfiniBand Essentials Every HPC Expert Must Know
InfiniBand Essentials Every HPC Expert Must KnowInfiniBand Essentials Every HPC Expert Must Know
InfiniBand Essentials Every HPC Expert Must Know
 
Implementation of Soft-core Processor on FPGA
Implementation of Soft-core Processor on FPGAImplementation of Soft-core Processor on FPGA
Implementation of Soft-core Processor on FPGA
 
InfiniBand for the enterprise
InfiniBand for the enterpriseInfiniBand for the enterprise
InfiniBand for the enterprise
 
CommScope RUCKUS ICX Switching Configuration
CommScope RUCKUS ICX Switching ConfigurationCommScope RUCKUS ICX Switching Configuration
CommScope RUCKUS ICX Switching Configuration
 

Recently uploaded

“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
Claudio Di Ciccio
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Speck&Tech
 
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIEnchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Vladimir Iglovikov, Ph.D.
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
DianaGray10
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
KAMESHS29
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Neo4j
 
Introduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - CybersecurityIntroduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - Cybersecurity
mikeeftimakis1
 
Data structures and Algorithms in Python.pdf
Data structures and Algorithms in Python.pdfData structures and Algorithms in Python.pdf
Data structures and Algorithms in Python.pdf
TIPNGVN2
 
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalizationFull-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
Zilliz
 
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
SOFTTECHHUB
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
Rohit Gautam
 
Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...
Zilliz
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
SOFTTECHHUB
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems S.M.S.A.
 
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
sonjaschweigert1
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
Matthew Sinclair
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Kari Kakkonen
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Neo4j
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
Alpen-Adria-Universität
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 

Recently uploaded (20)

“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
 
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIEnchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
 
Introduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - CybersecurityIntroduction to CHERI technology - Cybersecurity
Introduction to CHERI technology - Cybersecurity
 
Data structures and Algorithms in Python.pdf
Data structures and Algorithms in Python.pdfData structures and Algorithms in Python.pdf
Data structures and Algorithms in Python.pdf
 
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalizationFull-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
 
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
 
Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...Building RAG with self-deployed Milvus vector database and Snowpark Container...
Building RAG with self-deployed Milvus vector database and Snowpark Container...
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
 
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdfUni Systems Copilot event_05062024_C.Vlachos.pdf
Uni Systems Copilot event_05062024_C.Vlachos.pdf
 
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 

InfiniBand in the Lab (vSphere 5.5) VMworld 2013 Barcelona vBrownbag Tech Talk

  • 1. InfiniBand in the Lab Erik Bussink @ErikBussink www.bussink.ch www.VSAN.info 1
  • 3. Who uses InfiniBand for Storage ? • EMC Isilon • EMC xtremIO • PureStorage • Oracle Exadata • Nexenta • TeraData • Gluster 3
  • 4. What InfiniBand Offers • Lowest Layer for scalable IO interconnect • High-Performance • Low Latency • Reliable Switch Fabric • Offers higher layers of functionality • Application Clustering • Fast Inter Process Communications • Storage Area Networks 4
  • 5. InfiniBand physical layer • Physical layer based on 802.3z specification operating at 2.5Gb/s same standard as 10G ethernet (3.125Gb/s) • InfiniBand layer 2 switching uses 16 bit local address (LID), so limited to 2^16 nodes on a subnet. 5
  • 6. 6
  • 7. Connectors • For SDR (10Gb/s) and DDR (20Gb/s) use the CX4 connectors • For QDR (40Gb/s) and FDR (56Gb/s) use QSFP connectors 7
  • 8. Price comparison on Adapters 8
  • 9. And switches too... Example... 24 ports DDR (20Gb/s) for $499 Latest firmware (4.2.5) has embedded Subnet Manager 9
  • 10. Subnet Manager • The Subnet Manager assigns Local IDentifiers (LIDs) to each port in the IB fabric, and develops a routing table based off the assigned LIDs • Hardware based Subnet Manager are located in the Switches • Software based Subnet Manager is located on a host connected to the IB fabric • The OpenSM from the OpenFabrics Alliance works on Windows and Linux 10
  • 12. Installing InfiniBand on vSphere 5.1 12
  • 13. Configure MTU and OpenSM • esxcli system module paramters set y-m=mlx4_core -p=mtu_4k=1 • On Old switches (No 4K support) MTU is 2044 (2048-4) for IPoIB • vi partitions.conf • Add single line "Default=ox7fff,ipoib,mtu=5:ALL=full;" • copy partitions.conf /scratch/opensm/adapter_1_hca/ • copy partitions.conf /scratch/opensm/adapter_2_hca/ 13
  • 14. 14
  • 15. Installing InfiniBand on vSphere 5.5 • Mellanox drivers are in vSphere 5.5, and they support 40GbE inbox when using ConnectX-3and SwitchX products. • If not same HCA you need to uninstall the Mellanox drivers • esxcli software vib remove -n=net-mlx4-en -n=net-mlx4-core • And reboot • Then install the Mellanox OFED 1.8.2 • esxcli software vib install -d /tmp/MLX • And reboot 15
  • 16. Installing InfiniBand on vSphere 5.5 16
  • 17. Copy partitions.conf • Set MTU for mlx4 • Copy partitions.conf to /scratch/opensm/<HCA_Port>/ 17
  • 20. InfiniBand in the Lab Fast & Cheap Erik Bussink Thanks to Raphael Schitz,William Lam, Vladan Seget, Gregory Roche @ErikBussink www.bussink.ch www.VSAN.info 20