In this video from VMworld 2014, Josh Simons from VMware presents: Virtualized High Performance Computing with Mellanox FDR and RoCE on VMware ESXi 5.5. "The HPC community can realize significant benefits from adopting enterprise-capable IT solutions grounded in proven virtualization and cloud technology. And conversely, as business IT environments become increasingly compute-intensive, lessons learned by the scientists and engineers working with HPC can be transferred to their counterparts in the enterprise. It’s a win-win situation." Watch the video presentation: http://insidehpc.com/2014/09/virtualized-high-performance-computing-mellanox-fdr-roce/ Learn more in the insideHPC Guide to Virtualization, the Cloud and HPC: http://bit.ly/1w8kMfu

1. © 2014 VMware Inc. All rights reserved.
Virtualized High Performance Computing
with Mellanox FDR InfiniBand and RoCE
on VMware ESXi 5.5
Initial Performance Results
Josh Simons
Office of the CTO
VMware, Inc.
August 26, 2014

2. High Performance Computing
2
“High Performance Computing refers to the practice of aggregating computing power
in a way that delivers much higher performance than one could get out of a typical desktop or
workstation in order to solve large problems in science, engineering, or business.” [1]
Commercial:
 Oil exploration
 Pharmaceutical design
 Financial and economic modeling
 Advanced data visualization
HPC Applications
Science and Engineering:
 Atmosphere, earth, environment
 Bioscience, biotechnology, genetics
 Physics - applied, nuclear, particle, condensed matter;
 Electrical engineering, circuit design, microelectronics
 Mechanical engineering - from prosthetics to spacecraft
[1] G. Sravanthi, B. Grace. A Review of High Performance Computing. IOSR Journal of
Computer Engineering, vol. 16, pp36-43, 2014

3. VMware vCAC API
Secure Private Cloud for HPC
Research Group 1 Research Group m
Users IT
Public/Hybrid
Clouds
Programmatic
Control and
Integrations
VMware vCloud Automation Center
User Portals Blueprints
Security
Research Cluster 1 Research Cluster n
NSX
VMware
vCenter Server
VMware
vCenter Server
VMware
vCenter Server
VMware vSphere VMware vSphere VMware vSphere
3

4. HPC Workloads
• Scientific or technical workloads
• Often floating-point intensive
• Often parallel
• Often storage intensive
• Run on server-class systems
4
Throughput Workloads MPI Oriented Workloads

5. FDR InfiniBand Read Latency
5
2048
1024
512
256
128
64
32
16
8
4
2
1
0.5
Half Roundtrip Latency (μs)
Message Sizes (Bytes)
Native
ESXi 5.5
ESXi 5.5 with
SR-IOV
3
2.5
2
1.5
1
0.5
Four-node HP cluster
DL380 G8 nodes
128 GB, 3.0 GHz
FDR IB/RoCE HCAs
12-port FDR switch
ESX 5.5u1

6. RoCE Read Latency
6
2048
1024
512
256
128
64
32
16
8
4
2
1
0.5
Half Roundtrip latency (us)
3
2.5
2
Native
ESXi 5.5 0.5
Message Size (Bytes)
1.5
1

7. HPC Challenge Benchmark (HPCC) with FDR IB
7
100.00%
80.00%
60.00%
40.00%
20.00%
0.00%
HPCC Virtual to Native Ratios
(higher is better)
n4np4 n4np8
n4np16 n4np32
n4np64

8. NAMD Molecular Dynamics with FDR IB
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
NAMD Benchmarks Native to Virtual Ratios
(Higher is Better)
Apoa1 f1atpase
n4np4 n4np8
n4np16 n4np32
n4np64
8

9. LAMMPS Molecular Dynamics with FDR IB
CONFIDENTIAL 9
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
LAMMPS Benchmarks Native to Virtual Ratios
(Higher is Better)
Atomic Fluid Bulk Copper Bead-Spring Polymer
n4np4
n4np8
n4np16
n4np32
n4np64

10. To Learn More:
• Office of the CTO Expo Booth (2-6pm)
• How to Engage with Your Engineering, Science,
and Research Groups About Virtualization and
Cloud Computing
Thursday 10:30-11:30 Moscone West 2003
Josh Simons, simons@vmware.com