inside-BigData.com, profile picture
Uploaded byinside-BigData.com
PDF, PPTX3,459 views

HPC Best Practices: Application Performance Optimization

This document provides an overview of best practices and performance optimization for high-performance computing (HPC) applications, focusing on specific software such as LS-DYNA and GROMACS. It covers performance benchmarking, profiling tools, and hardware configurations, highlighting the impact of network interconnects, CPU optimizations, and memory management on application scalability and efficiency. Key findings include significant performance gains with EDR InfiniBand and HPC-X MPI library optimizations, showcasing improvements over previous system generations and tuning parameters.

Embed presentation

Download as PDF, PPTX
Application Performance Optimizations Pak Lui
2 •  MILC •  OpenMX •  PARATEC •  PFA •  PFLOTRAN •  Quantum ESPRESSO •  RADIOSS •  SPECFEM3D •  WRF •  LS-DYNA •  miniFE •  MILC •  MSC Nastran •  MR Bayes •  MM5 •  MPQC •  NAMD •  Nekbone •  NEMO •  NWChem •  Octopus •  OpenAtom •  OpenFOAM 140 Applications Best Practices Published •  Abaqus •  AcuSolve •  Amber •  AMG •  AMR •  ABySS •  ANSYS CFX •  ANSYS FLUENT •  ANSYS Mechanics •  BQCD •  CCSM •  CESM •  COSMO •  CP2K •  CPMD •  Dacapo •  Desmond •  DL-POLY •  Eclipse •  FLOW-3D •  GADGET-2 •  GROMACS •  Himeno •  HOOMD-blue •  HYCOM •  ICON •  Lattice QCD •  LAMMPS For more information, visit: http://www.hpcadvisorycouncil.com/best_practices.php
3 HPC Advisory Council HPC Center Dell™ PowerEdge™ R730 32-node cluster HP Cluster Platform 3000SL 16-node cluster HP ProLiant SL230s Gen8 4-node cluster Dell™ PowerEdge™ R815 11-node cluster Dell™ PowerEdge™ C6145 6-node cluster Dell™ PowerEdge™ M610 38-node cluster Dell™ PowerEdge™ C6100 4-node cluster Dell™ PowerVault MD3420 / MD3460 InfiniBand-based Lustre Storage Dell™ PowerEdge™ R720/R720xd 32-node cluster HP Proliant XL230a Gen9 10-node cluster Colfax CX1350s-XK5 4-node cluster
4 Agenda •  Overview of HPC Applications Performance •  Way to Inspect, Profile, Optimize HPC Applications –  CPU, memory, file I/O, network •  System Configurations and Tuning •  Case Studies, Performance Comparisons, Optimizations and Highlights •  Conclusions
5 HPC Application Performance Overview •  To achieve scalability performance on HPC applications –  Involves understanding of the workload by performing profile analysis •  Tune for the most time spent (either CPU, Network, IO, etc) –  Underlying implicit requirement: Each node to perform similarly •  Run CPU/memory /network tests or cluster checker to identify bad node(s) –  Comparing behaviors of using different HW components •  Which pinpoint bottlenecks in different areas of the HPC cluster •  A selection of HPC applications will be shown –  To demonstrate method of profiling and analysis –  To determine the bottleneck in SW/HW –  To determine the effectiveness of tuning to improve on performance
6 Ways To Inspect and Profile Applications •  Computation (CPU/Accelerators) –  Tools: gprof, top, htop, perf top, pstack, Visual Profiler, etc –  Tests and Benchmarks: HPL, STREAM •  File I/O –  Bandwidth and Block Size: iostat, collectl, darshan, etc –  Characterization Tools and Benchmarks: iozone, ior, etc •  Network Interconnect and MPI communications –  Tools and Profilers: perfquery, MPI profilers (IPM, TAU, etc) –  Characterization Tools and Benchmarks: –  Latency and Bandwidth: OSU benchmarks, IMB
7 Case Study: LS-DYNA
8 Note •  The following research was performed under the HPC Advisory Council activities –  Participating vendors: Intel, Dell, Mellanox –  Compute resource - HPC Advisory Council Cluster Center •  The following was done to provide best practices –  LS-DYNA performance overview –  Understanding LS-DYNA communication patterns –  Ways to increase LS-DYNA productivity –  MPI libraries comparisons •  For more info please refer to –  http://www.dell.com –  http://www.intel.com –  http://www.mellanox.com –  http://www.lstc.com
9 LS-DYNA •  LS-DYNA –  A general purpose structural and fluid analysis simulation software package capable of simulating complex real world problems –  Developed by the Livermore Software Technology Corporation (LSTC) •  LS-DYNA used by –  Automobile –  Aerospace –  Construction –  Military –  Manufacturing –  Bioengineering
10 Objectives •  The presented research was done to provide best practices –  LS-DYNA performance benchmarking •  MPI Library performance comparison •  Interconnect performance comparison •  CPU cores/speed comparison •  Optimization tuning •  The presented results will demonstrate –  The scalability of the compute environment/application –  Considerations for higher productivity and efficiency
11 Test Cluster Configuration •  Dell PowerEdge R730 32-node (896-core) “Thor” cluster –  Dual-Socket 14-Core Intel E5-2697v3 @ 2.60 GHz CPUs (Power Management in BIOS sets to Maximum Performance) –  Memory: 64GB memory, DDR4 2133 MHz, Memory Snoop Mode in BIOS sets to Home Snoop –  OS: RHEL 6.5, MLNX_OFED_LINUX-2.4-1.0.5.1_20150408_1555 InfiniBand SW stack –  Hard Drives: 2x 1TB 7.2 RPM SATA 2.5” on RAID 1 •  Mellanox ConnectX-4 EDR 100Gb/s InfiniBand Adapters •  Mellanox Switch-IB SB7700 36-port EDR 100Gb/s InfiniBand Switch •  Mellanox ConnectX-3 FDR VPI InfiniBand and 40Gb/s Ethernet Adapters •  Mellanox SwitchX-2 SX6036 36-port 56Gb/s FDR InfiniBand / VPI Ethernet Switch •  MPI: Open MPI 1.8.4, Mellanox HPC-X v1.2.0-326, Intel MPI 5.0.2.044, IBM Platform MPI 9.1 •  Application: –  LS-DYNA 8.0.0 (builds 95359, 95610), Single Precision •  Benchmarks: 3 Vehicle Collision, Neon refined revised
12 PowerEdge R730 Massive flexibility for data intensive operations •  Performance and efficiency –  Intelligent hardware-driven systems management with extensive power management features –  Innovative tools including automation for parts replacement and lifecycle manageability –  Broad choice of networking technologies from GigE to IB –  Built in redundancy with hot plug and swappable PSU, HDDs and fans •  Benefits –  Designed for performance workloads •  from big data analytics, distributed storage or distributed computing where local storage is key to classic HPC and large scale hosting environments •  High performance scale-out compute and low cost dense storage in one package •  Hardware Capabilities –  Flexible compute platform with dense storage capacity •  2S/2U server, 6 PCIe slots –  Large memory footprint (Up to 768GB / 24 DIMMs) –  High I/O performance and optional storage configurations •  HDD options: 12 x 3.5” - or - 24 x 2.5 + 2x 2.5 HDDs in rear of server •  Up to 26 HDDs with 2 hot plug drives in rear of server for boot or scratch
13 LS-DYNA Performance – Network Interconnects •  EDR InfiniBand delivers superior scalability in application performance –  Provides higher performance by over 4-5 times than 1GbE, 10GbE and 40GbE –  1GbE stop scaling beyond 4 nodes, and 10GbE stops scaling beyond 8 nodes –  InfiniBand demonstrates continuous performance gain at scale 28 MPI Processes / NodeHigher is better 444% 505% 572%
14 •  Majority of the MPI time is spent on MPI_recv and MPI Collective Ops –  MPI_Recv(36%), MPI_Allreduce(27%), MPI_Bcast(24%) •  Similar communication characteristics seen on both input dataset –  Both exhibit similar communication patterns LS-DYNA Profiling – Time Spent in MPI 3 Vehicle Collision – 32 nodesNeon_refined_revised – 32 nodes
15 LS-DYNA Profiling – Time Spent in MPI •  Most of the MPI messages are in the medium sizes –  Most message sizes are between 0 to 64B •  For the most time consuming MPI calls –  MPI_Recv: Most messages are under 4KB –  MPI_Bcast: Majority are less than 16B, but larger messages exist –  MPI_Allreduce: Most messages are less than 256B neon_refined_revised
16 LS-DYNA Performance – EDR vs FDR InfiniBand •  EDR InfiniBand delivers superior scalability in application performance –  As the cluster scales, performance gap of EDR IB becomes widen •  Performance advantage of EDR InfiniBand increases for larger core counts –  EDR IB provides 15% versus FDR IB at 32 nodes (896 cores) Higher is better 15% 28 MPI Processes / Node
17 LS-DYNA Performance – Cores Per Node •  Better performance is seen at scale with less CPU cores per node –  At low node counts, higher performance can be achieved with more cores per node –  At high node counts, slightly better performance by using less cores per node –  Memory bandwidth might be limited by more CPU cores being used CPU @ 2.6GHzHigher is better 5%3%2% 5%
18 LS-DYNA Performance – AVX2/SSE2 CPU Instructions •  LS-DYNA provides executables with supports for different CPU instructions –  AVX2 is supported on “Haswell” while SSE2 is supported on previous generations –  Due to runtime issue, AVX2 executable build 95610 is used, instead of the public build 95359 –  Slight improvement of ~2-4% by using executable with AVX2 instructions –  The AVX2 instructions runs at a lower clock speed (2.2GHz) than normal CPU clock (2.6GHz) Higher is better 4% 24 MPI Processes / Node
19 LS-DYNA Performance – Turbo Mode •  Turbo Boost enables processors to run above its base frequency –  Capability to allow CPU cores to run dynamically above the CPU clock –  When thermal headroom allows the CPU to operate –  The 2.6GHz clock speed could boost to Max Turbo Frequency of 3.3GHz –  Running with Turbo Boost translates to a ~25% of performance boost 40% 25% Higher is better 28 MPI Processes / Node
20 LS-DYNA Performance – Memory Optimization •  Setting the environment variables for memory allocator improve on performance –  Modifying the memory allocator allows faster memory registration for communications •  Environment variables used: –  export MALLOC_MMAP_MAX_=0 –  export MALLOC_TRIM_THRESHOLD_=-1 176% Higher is better 19% 28 MPI Processes / Node
21 LS-DYNA Performance – MPI Optimization •  FCA and MXM enhance LS-DYNA performance at scale for HPC-X –  Open MPI and HPC-X are based on the Open MPI distribution –  The “yalla” PML, UD transport and memory optimization in HPC-X reduce overhead –  MXM provides a speedup of 38% over un-tuned baseline run at 32 nodes (768 cores) •  MCA parameters: •  -mca btl_sm_use_knem 1 -mca pml yalla -x MXM_TLS=ud,shm,self -x MXM_SHM_RNDV_THRESH=32768 •  -x HCOLL_CONTEXT_CACHE_ENABLE=1 Higher is better 38% 24 MPI Processes / Node
22 LS-DYNA Performance – Intel MPI Optimization •  The DAPL provider performs better than OFA provider for Intel MPI –  DAPL would provide better scalability performance for Intel MPI on LS-DYNA •  MCA parameters for MXM: –  Common for 2 tests: I_MPI_DAPL_SCALABLE_PROGRESS 1, I_MPI_RDMA_TRANSLATION_CACHE 1, I_MPI_FAIR_CONN_SPIN_COUNT 2147483647, I_MPI_FAIR_READ_SPIN_COUNT 2147483647, I_MPI_ADJUST_REDUCE 2, I_MPI_ADJUST_BCAST 0, I_MPI_RDMA_TRANSLATION_CACHE 1, I_MPI_RDMA_RNDV_BUF_ALIGN 65536, I_MPI_SPIN_COUNT 121 –  For OFA: -IB, MV2_USE_APM 0, I_MPI_OFA_USE_XRC 1 –  For DAPL: -DAPL, I_MPI_DAPL_DIRECT_COPY_THRESHOLD 65536, I_MPI_DAPL_UD enable, I_MPI_DAPL_PROVIDER ofa-v2-mlx5_0-1u Higher is better 20% 24 MPI Processes / Node
23 LS-DYNA Performance – MPI Libraries •  HPC-X outperforms Platform MPI, and Open MPI in scalability performance –  HPC-X delivers higher performance than Intel MPI (OFA) by 33%, (DAPL) by 11%, Platform MPI by 27% on neon_refined_revised –  Performance is 20% higher than Intel OFA, and % 8% better than Platform MPI in 3cars •  Tuning parameter used: –  For Open MPI: -bind-to-core and KNEM. For Platform MPI: -cpu_bind, -xrc. For Intel MPI: see previous slide 7%27% Higher is better 33% 20% 11% 24 MPI Processes / Node
24 •  Current Haswell system configuration outperforms prior system generations –  Current systems outperformed Ivy Bridge by 47%, Sandy Bridge by 75%, Westmere by 148%, Nehalem by 290% –  Scalability support from EDR InfiniBand and HPC-X provide huge boost in performance at scale for LS-DYNA •  System components used: –  Haswell: 2-socket 14-core E5-2697v3@2.6GHz, 2133MHz DIMMs, ConnectX-4 EDR InfiniBand –  Ivy Bridge: 2-socket 10-core E5-2680v2@2.8GHz, 1600MHz DIMMs, Connect-IB FDR InfiniBand –  Sandy Bridge: 2-socket 8-core E5-2680@2.7GHz, 1600MHz DIMMs, ConnectX-3 FDR InfiniBand –  Westmere: 2-socket 6-core x5670@2.93GHz, 1333MHz DIMMs, ConnectX-2 QDR InfiniBand –  Nehalem: 2-socket 4-core x5570@2.93GHz, 1333MHz DIMMs, ConnectX-2 QDR InfiniBand 290% 47% 75%148% Higher is better LS-DYNA Performance – System Generations
25 LS-DYNA Summary •  Performance –  Compute: Intel Haswell cluster outperforms system architecture of previous generations •  Outperforms Ivy Bridge by 47%, Sandy Bridge by 75%, Westmere by 148%, and Nehalem by 290% •  Using executable with AVX2 instructions provides slight advantage •  Slight improvement of ~2-4% by using executable with AVX2 instructions –  Turbo Mode: Running with Turbo Boost provides ~25% of performance boost in some cases •  Turbo Boost enables processors to run above its base frequency –  Network: EDR InfiniBand and HPC-X MPI library deliver superior scalability in application performance •  EDR IB provides higher performance by over 4-5 times vs 1GbE, 10GbE and 40GbE, 15% vs FDR IB at 32 nodes •  MPI Tuning –  HPC-X enhances LS-DYNA performance at scale for LS-DYNA •  MXM UD provides a speedup of 38% over un-tuned baseline run at 32 nodes –  HPC-X outperforms Platform MPI, and Open MPI in scalability performance •  Up to 27% better than Platform MPI on neon_refined_revised, and 8% better than Platform MPI in 3cars
26 Case Study: GROMACS
27 GROMACS •  GROMACS (GROningen MAchine for Chemical Simulation) –  A molecular dynamics simulation package –  Primarily designed for biochemical molecules like proteins, lipids and nucleic acids •  A lot of algorithmic optimizations have been introduced in the code •  Extremely fast at calculating the nonbonded interactions –  Ongoing development to extend GROMACS with interfaces both to Quantum Chemistry and Bioinformatics/databases –  An open source software released under the GPL
28 Objectives •  The presented research was done to provide best practices –  GROMACS performance benchmarking •  CPU performance comparison •  MPI library performance comparison •  Interconnect performance comparison •  System generations comparison •  The presented results will demonstrate –  The scalability of the compute environment/application –  Considerations for higher productivity and efficiency
29 Test Cluster Configuration •  Dell PowerEdge R730 32-node (896-core) “Thor” cluster –  Dual-Socket 14-Core Intel E5-2697v3 @ 2.60 GHz CPUs (Power Management in BIOS sets to Maximum Performance) –  Memory: 64GB memory, DDR4 2133 MHz, Memory Snoop Mode in BIOS sets to Home Snoop, Turbo Enabled –  OS: RHEL 6.5, MLNX_OFED_LINUX-3.0-1.0.1 InfiniBand SW stack –  Hard Drives: 2x 1TB 7.2 RPM SATA 2.5” on RAID 1 •  Mellanox ConnectX-4 EDR 100Gbps EDR InfiniBand Adapters •  Mellanox Switch-IB SB7700 36-port 100Gb/s EDR InfiniBand Switch •  Mellanox ConnectX-3 FDR InfiniBand, 10/40GbE Ethernet VPI Adapters •  Mellanox SwitchX-2 SX6036 36-port 56Gb/s FDR InfiniBand / VPI Ethernet Switch •  MPI: Mellanox HPC-X v1.2.0-326 •  Compiler and Libraries: Intel Composer XE 2015.3.187 and MKL •  Application: GROMACS 4.6.7 •  Benchmark datasets: DPPC in Water (d.dppc, 121856 atoms, 150000 steps, SP) unless stated otherwise
30 GROMACS Performance – Network Interconnects •  InfiniBand is the only interconnect that delivers superior scalability performance –  EDR InfiniBand provides higher performance and more scalable than 1GbE, 10GbE, or 40GbE –  Performance for Ethernet stays flat (or stops scaling) beyond 2 nodes –  EDR InfiniBand outperforms 10GbE-RoCE on scalability performance by 55% at 32 nodes / 896c –  EDR InfiniBand demonstrates continuous performance gain at scale 28 MPI Processes / NodeHigher is better 55%4.1x4.2x 4.6x
31 GROMACS Profiling – Time Spent by MPI Calls •  The most time consuming MPI call os MPI_Sendrecv –  MPI_Sendrecv: 66% (or 27% of runtime) at 32 nodes (896 cores) –  MPI_Waitall: 18% (or 7% of runtime), MPI_Bcast: 6% (or 2% of runtime) –  Point to point and non-blocking sends and receives consume most time in GROMACS 32 Nodes
32 GROMACS Profiling – MPI Message Sizes •  Majority of data transfer messages are medium sizes, except for: –  MPI_Sendrecv has a large concentration (from 8B to 8KB) –  MPI_Bcast shows some concentration 32 Nodes
33 GROMACS Profiling – MPI Data Transfer •  As the cluster grows, similar communication behavior is seen –  Majority of communications are between neighboring ranks –  Non-blocking (point to point) data, and point-to-point transfers are shown in the graph –  Collective data communications are small compared to point-to-point communications 2 Nodes / 56 Cores32 Nodes / 896 Cores
34 GROMACS Performance – EDR vs FDR InfiniBand •  EDR InfiniBand delivers superior scalability in application performance –  As the number of nodes scales, performance gap of EDR IB becomes widen •  Performance advantage of EDR InfiniBand increases for larger core counts –  EDR InfiniBand provides 29% versus FDR InfiniBand at 32 nodes (896 cores) Higher is better 29% 28 MPI Processes / Node
35 GROMACS Performance – System Generations •  Thor cluster (based on Intel E5-2697v3 - Haswell) outperforms prior generations –  1.1 to 3.5x higher performance than clusters based on previous generations of Intel architecture •  System components used: –  Janus: 2-socket 6-core Xeon X5670 @ 2.93GHz, 1333MHz DIMMs, ConnectX-2 QDR IB –  Jupiter: 2-socket 8-core Xeon E5-2680 @ 2.7GHz, 1600MHz DIMMs, ConnectX-3 FDR IB –  Thor: 2-socket 14-core Xeon E5-2680V3 @2.6GHz, 2133MHz DIMMs, ConnectX-4 EDR IB 356% 118% Higher is better
36 GROMACS Performance – Cores Per Node •  Running more CPU cores provides higher performance –  ~7-10% higher productivity with 28PPN compared to 24PPN –  Higher demand on memory bandwidth and network might limit performance as more cores are used CPU @ 2.6GHzHigher is better 13% 39% 7%
37 GROMACS Performance – Turbo Mode & CPU Clock •  Advantages are seen with running higher clock rate –  Either by enabling Turbo mode or higher CPU clock frequency •  Boosting CPU clock rate yields higher performance at lower cost –  Increasing to 2600MHz (from 2300MHz) run 11% faster CPU @ 2.6GHzHigher is better 16% 11%
38 GROMACS Performance – Floating Point Precision •  GROMACS allows running either SP and DP for floating point precision •  Running at SP is shown to be faster than running at DP –  Seen around 41%-47% faster running at SP (Single Precision) versus DP (Double Precision) –  All other slides are running using Single Precision CPU @ 2.6GHzHigher is better 47% 41%
39 GROMACS Summary •  Latest system generation improve GROMACS performance at scale –  Compute: Intel Haswell cluster outperforms system architecture of previous generations •  Haswell cluster outperforms Sandy Bridge cluster by 110%, and outperforms Westmere cluster by 350% at 32 node –  Compute: Running more CPU cores provides higher performance •  ~7-10% higher productivity with 28PPN compared to 24PPN –  Network: EDR InfiniBand delivers superior scalability in application performance •  EDR InfiniBand provides higher performance and more scalable than 1GbE, 10GbE, or 40GbE •  Performance for Ethernet (1GbE/10GbE/40GbE) stays flat (or stops scaling) beyond 2 nodes •  EDR InfiniBand outperforms 10GbE-RoCE on scalability performance by 55% at 32 nodes / 896c –  Running at Single Precision is approximately twice as fast as running at Double Precision •  Seen around 41%-47% faster running at SP (Single Precision) versus DP (Double Precision) –  MPI Profile shows majority of data transfer are point-to-point and non-blocking communications •  MPI_Sendrecv and MPI_Waitall are the most used MPI communication
4040 Thank You HPC Advisory Council All trademarks are property of their respective owners. All information is provided “As-Is” without any kind of warranty. The HPC Advisory Council makes no representation to the accuracy and completeness of the information contained herein. HPC Advisory Council undertakes no duty and assumes no obligation to update or correct any information presented herein

More Related Content

PDF
HPC Application Profiling and Analysis
byRishi Pathak
 
PDF
UnboundとNSDの紹介　BIND9との比較編
byhdais
 
PDF
QNX Sales Engineering Presentation
byRobert-Emmanuel Mayssat
 
PDF
엘라스틱서치 클러스터로 수십억 건의 데이터 운영하기
by흥래 김
 
ODP
Kvm and libvirt
byplarsen67
 
PPTX
Nfv
byAhmad Hijazi
 
PDF
Producer Performance Tuning for Apache Kafka
byJiangjie Qin
 
PPT
IP Address Management Best Practices
bySolarWinds
 
HPC Application Profiling and Analysis
byRishi Pathak
 
UnboundとNSDの紹介　BIND9との比較編
byhdais
 
QNX Sales Engineering Presentation
byRobert-Emmanuel Mayssat
 
엘라스틱서치 클러스터로 수십억 건의 데이터 운영하기
by흥래 김
 
Kvm and libvirt
byplarsen67
 
Nfv
byAhmad Hijazi
 
Producer Performance Tuning for Apache Kafka
byJiangjie Qin
 
IP Address Management Best Practices
bySolarWinds
 

What's hot

PDF
Performance Profiling in Rust
byInfluxData
 
PDF
Yahoo! JAPANにおけるApache Cassandraへの取り組み
byYahoo!デベロッパーネットワーク
 
PDF
L2 over L3 ecnaspsulations
byMotonori Shindo
 
PDF
임태현, 게임 서버 디자인 가이드, NDC2013
bydevCAT Studio, NEXON
 
PDF
QNX Software Systems
byRobert-Emmanuel Mayssat
 
PDF
Velocity 2015 linux perf tools
byBrendan Gregg
 
PPT
A verifiable random function with short proofs and keys
byAleksandr Yampolskiy
 
PPTX
Next-generation MMORPG service architecture
byJongwon Kim
 
PPTX
Kafka Quotas Talk at LinkedIn
byAditya Auradkar
 
PDF
LinuxのFull ticklessを試してみた
byHiraku Toyooka
 
PPTX
ネットワークの基礎①
bytatsuya yamaguchi
 
PDF
Multiplayer Game Sync Techniques through CAP theorem
bySeungmo Koo
 
PDF
CDNのトラフィックエンジニアリング:CDNの現状とSDNの可能性
byJ-Stream Inc.
 
PDF
KubeCon + CloudNativeCon Europe 2022 Recap / Kubernetes Meetup Tokyo #51 / #k...
byPreferred Networks
 
PPT
アメーバピグにおける自作サーバ運用
byAkihiro Kuwano
 
PDF
Brkarc 3454 - in-depth and personal with the cisco nexus 2000 fabric extender...
bykds850
 
PDF
[Cloud OnAir] Google Networking Deep Dive ! その技術と設計の紹介 2018年8月9日 放送
byGoogle Cloud Platform - Japan
 
PDF
네트워크 가상화 발표자료-SDN/NFV/Cloud
byseungdols
 
PPTX
Apache Kafka Best Practices
byDataWorks Summit/Hadoop Summit
 
PDF
Tuning Apache Spark for Large-Scale Workloads Gaoxiang Liu and Sital Kedia
byDatabricks
 
Performance Profiling in Rust
byInfluxData
 
Yahoo! JAPANにおけるApache Cassandraへの取り組み
byYahoo!デベロッパーネットワーク
 
L2 over L3 ecnaspsulations
byMotonori Shindo
 
임태현, 게임 서버 디자인 가이드, NDC2013
bydevCAT Studio, NEXON
 
QNX Software Systems
byRobert-Emmanuel Mayssat
 
Velocity 2015 linux perf tools
byBrendan Gregg
 
A verifiable random function with short proofs and keys
byAleksandr Yampolskiy
 
Next-generation MMORPG service architecture
byJongwon Kim
 
Kafka Quotas Talk at LinkedIn
byAditya Auradkar
 
LinuxのFull ticklessを試してみた
byHiraku Toyooka
 
ネットワークの基礎①
bytatsuya yamaguchi
 
Multiplayer Game Sync Techniques through CAP theorem
bySeungmo Koo
 
CDNのトラフィックエンジニアリング:CDNの現状とSDNの可能性
byJ-Stream Inc.
 
KubeCon + CloudNativeCon Europe 2022 Recap / Kubernetes Meetup Tokyo #51 / #k...
byPreferred Networks
 
アメーバピグにおける自作サーバ運用
byAkihiro Kuwano
 
Brkarc 3454 - in-depth and personal with the cisco nexus 2000 fabric extender...
bykds850
 
[Cloud OnAir] Google Networking Deep Dive ! その技術と設計の紹介 2018年8月9日 放送
byGoogle Cloud Platform - Japan
 
네트워크 가상화 발표자료-SDN/NFV/Cloud
byseungdols
 
Apache Kafka Best Practices
byDataWorks Summit/Hadoop Summit
 
Tuning Apache Spark for Large-Scale Workloads Gaoxiang Liu and Sital Kedia
byDatabricks
 

Viewers also liked

PPTX
GROMACS Molecular Dynamics on GPU
byDevang Sachdev
 
PDF
Intel® Xeon® Processor E5-2600 v4 Tech Computing Applications Showcase
byIntel IT Center
 
PPTX
Fun with Github webhooks: verifying Signed-off-by
byJeff Squyres
 
PDF
Dell High-Performance Computing solutions: Enable innovations, outperform exp...
byDell World
 
PDF
Deploying HPC Cluster with Mellanox InfiniBand Interconnect Solutions
byMellanox Technologies
 
PDF
Open MPI State of the Union X SC'16 BOF
byJeff Squyres
 
PDF
Intel® Xeon® Processor E5-2600 v4 Product Family EAMG
byIntel IT Center
 
GROMACS Molecular Dynamics on GPU
byDevang Sachdev
 
Intel® Xeon® Processor E5-2600 v4 Tech Computing Applications Showcase
byIntel IT Center
 
Fun with Github webhooks: verifying Signed-off-by
byJeff Squyres
 
Dell High-Performance Computing solutions: Enable innovations, outperform exp...
byDell World
 
Deploying HPC Cluster with Mellanox InfiniBand Interconnect Solutions
byMellanox Technologies
 
Open MPI State of the Union X SC'16 BOF
byJeff Squyres
 
Intel® Xeon® Processor E5-2600 v4 Product Family EAMG
byIntel IT Center
 

Similar to HPC Best Practices: Application Performance Optimization

PDF
Application Profiling at the HPCAC High Performance Center
byinside-BigData.com
 
PDF
Performance Optimization of HPC Applications: From Hardware to Source Code
byFisnik Kraja
 
PDF
Best Practices and Performance Studies for High-Performance Computing Clusters
byIntel® Software
 
PPTX
TechWiseTV Workshop: Cisco HyperFlex Systems
byRobb Boyd
 
PDF
Introducing Cisco HyperFlex Systems: The Next Generation in Complete Hypercon...
byCisco Canada
 
PDF
Infrastructure optimization for seismic processing (eng)
byVsevolod Shabad
 
PPTX
Virtualized high performance computing with mellanox fdr and ro ce
byinside-BigData.com
 
PDF
OpenPOWER Acceleration of HPCC Systems
byHPCC Systems
 
PDF
Performance Analysis and Optimizations of CAE Applications (Case Study: STAR_...
byFisnik Kraja
 
PDF
Ac922 cdac webinar
byGanesan Narayanasamy
 
PDF
Co-Design Architecture for Exascale
byinside-BigData.com
 
PPTX
CISCO presentation used during the SWITCHPOINT NV/SA Quarterly Experience Day...
bySWITCHPOINT NV/SA
 
PDF
power9_performance_best_practices_IBM_AIX.pdf
byVenu Singamaneni
 
PDF
Lessons Learned on Benchmarking Big Data Platforms
byt_ivanov
 
PDF
InfiniBand In-Network Computing Technology and Roadmap
byinside-BigData.com
 
PPTX
Mellnox Interconnect presentation in OpenPOWER Brazil workshop
byGanesan Narayanasamy
 
PDF
"Performance Evaluation, Scalability Analysis, and Optimization Tuning of A...
byAltair
 
PDF
InfiniBand In-Network Computing Technology and Roadmap
byinside-BigData.com
 
PDF
Overview of Intel® Omni-Path Architecture
byIntel® Software
 
PDF
Omni path-fabric-software-architecture-overview
byDESMOND YUEN
 
Application Profiling at the HPCAC High Performance Center
byinside-BigData.com
 
Performance Optimization of HPC Applications: From Hardware to Source Code
byFisnik Kraja
 
Best Practices and Performance Studies for High-Performance Computing Clusters
byIntel® Software
 
TechWiseTV Workshop: Cisco HyperFlex Systems
byRobb Boyd
 
Introducing Cisco HyperFlex Systems: The Next Generation in Complete Hypercon...
byCisco Canada
 
Infrastructure optimization for seismic processing (eng)
byVsevolod Shabad
 
Virtualized high performance computing with mellanox fdr and ro ce
byinside-BigData.com
 
OpenPOWER Acceleration of HPCC Systems
byHPCC Systems
 
Performance Analysis and Optimizations of CAE Applications (Case Study: STAR_...
byFisnik Kraja
 
Ac922 cdac webinar
byGanesan Narayanasamy
 
Co-Design Architecture for Exascale
byinside-BigData.com
 
CISCO presentation used during the SWITCHPOINT NV/SA Quarterly Experience Day...
bySWITCHPOINT NV/SA
 
power9_performance_best_practices_IBM_AIX.pdf
byVenu Singamaneni
 
Lessons Learned on Benchmarking Big Data Platforms
byt_ivanov
 
InfiniBand In-Network Computing Technology and Roadmap
byinside-BigData.com
 
Mellnox Interconnect presentation in OpenPOWER Brazil workshop
byGanesan Narayanasamy
 
"Performance Evaluation, Scalability Analysis, and Optimization Tuning of A...
byAltair
 
InfiniBand In-Network Computing Technology and Roadmap
byinside-BigData.com
 
Overview of Intel® Omni-Path Architecture
byIntel® Software
 
Omni path-fabric-software-architecture-overview
byDESMOND YUEN
 

More from inside-BigData.com

PDF
Major Market Shifts in IT
byinside-BigData.com
 
PDF
Preparing to program Aurora at Exascale - Early experiences and future direct...
byinside-BigData.com
 
PPTX
Transforming Private 5G Networks
byinside-BigData.com
 
PDF
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
byinside-BigData.com
 
PDF
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
byinside-BigData.com
 
PDF
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
byinside-BigData.com
 
PDF
HPC Impact: EDA Telemetry Neural Networks
byinside-BigData.com
 
PDF
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
byinside-BigData.com
 
PDF
Machine Learning for Weather Forecasts
byinside-BigData.com
 
PPTX
HPC AI Advisory Council Update
byinside-BigData.com
 
PDF
Fugaku Supercomputer joins fight against COVID-19
byinside-BigData.com
 
PDF
Energy Efficient Computing using Dynamic Tuning
byinside-BigData.com
 
PDF
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
byinside-BigData.com
 
PDF
State of ARM-based HPC
byinside-BigData.com
 
PDF
Versal Premium ACAP for Network and Cloud Acceleration
byinside-BigData.com
 
PDF
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
byinside-BigData.com
 
PDF
Scaling TCO in a Post Moore's Era
byinside-BigData.com
 
PDF
CUDA-Python and RAPIDS for blazing fast scientific computing
byinside-BigData.com
 
PDF
Introducing HPC with a Raspberry Pi Cluster
byinside-BigData.com
 
PDF
Overview of HPC Interconnects
byinside-BigData.com
 
Major Market Shifts in IT
byinside-BigData.com
 
Preparing to program Aurora at Exascale - Early experiences and future direct...
byinside-BigData.com
 
Transforming Private 5G Networks
byinside-BigData.com
 
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
byinside-BigData.com
 
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
byinside-BigData.com
 
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
byinside-BigData.com
 
HPC Impact: EDA Telemetry Neural Networks
byinside-BigData.com
 
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
byinside-BigData.com
 
Machine Learning for Weather Forecasts
byinside-BigData.com
 
HPC AI Advisory Council Update
byinside-BigData.com
 
Fugaku Supercomputer joins fight against COVID-19
byinside-BigData.com
 
Energy Efficient Computing using Dynamic Tuning
byinside-BigData.com
 
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
byinside-BigData.com
 
State of ARM-based HPC
byinside-BigData.com
 
Versal Premium ACAP for Network and Cloud Acceleration
byinside-BigData.com
 
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
byinside-BigData.com
 
Scaling TCO in a Post Moore's Era
byinside-BigData.com
 
CUDA-Python and RAPIDS for blazing fast scientific computing
byinside-BigData.com
 
Introducing HPC with a Raspberry Pi Cluster
byinside-BigData.com
 
Overview of HPC Interconnects
byinside-BigData.com
 

Recently uploaded

PDF
Preserve workload integrity during cross-architecture migration
byPrincipled Technologies
 
PDF
NTG -Company Profile_Software_Vendor_Company_in_MENA
byMustafa Kuğu
 
PPTX
Introducing VisualSim 2610 The Next Leap in System Level Modeling
byDeepak Shankar
 
PPTX
CTO Strategy OS 2026: The Tech, AI & Cloud Playbook Boards Want
byridwansassman
 
PDF
20260212 Security-JAWS activity results for 2025 and activity goals for 2026
byTyphon 666
 
PDF
shayk.online - Anonymous chat with Sinatra and WebSockets
byEleanor McHugh
 
PDF
Effortless Distributed Systems with Aspire.pdf
byParticular Software
 
PDF
Transcript: Escape from the Forbidden Zone: Smuggling green and inclusive tec...
byBookNet Canada
 
PDF
Oracle Cloud Infrastructure 2025 Architect Professional (1Z0-1127-25) Master ...
byExamCert App
 
PPTX
Automation Without Apprentices: How AI Challenges the Open Source Way
byIcinga
 
PDF
Odoo Implementation Checklist: A Strategic ERP Blueprint for Business-Ready D...
byBANIBRO IT SOLUTION
 
PDF
GenerationAI_Paris_2025_Architecting_Intelligence.pdf
byapidays
 
PDF
How AI Can Help Platform Engineers Build Better Platforms
byAll Things Open
 
PPTX
Do You Control the AI, or Does the AI Control You?
bymedhateltoukhy
 
PDF
Explaining specific examples of purpose-specific BOM
byakipii ogaoga
 
PPTX
apidays Paris 2025 | Building Agentic Workflows
byapidays
 
PDF
apidays New York 2025 | AI in Application Security
byapidays
 
PDF
HOW TO OVERCOME THE THREATS OF ARTIFICIAL INTELLIGENCE AGAINST HUMANITY.pdf
byFaga1939
 
PDF
final~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.pdf
byomarbishtawi04
 
PDF
UiPath Automation Developer Associate Training Series 2026 - Session 3
byDianaGray10
 
Preserve workload integrity during cross-architecture migration
byPrincipled Technologies
 
NTG -Company Profile_Software_Vendor_Company_in_MENA
byMustafa Kuğu
 
Introducing VisualSim 2610 The Next Leap in System Level Modeling
byDeepak Shankar
 
CTO Strategy OS 2026: The Tech, AI & Cloud Playbook Boards Want
byridwansassman
 
20260212 Security-JAWS activity results for 2025 and activity goals for 2026
byTyphon 666
 
shayk.online - Anonymous chat with Sinatra and WebSockets
byEleanor McHugh
 
Effortless Distributed Systems with Aspire.pdf
byParticular Software
 
Transcript: Escape from the Forbidden Zone: Smuggling green and inclusive tec...
byBookNet Canada
 
Oracle Cloud Infrastructure 2025 Architect Professional (1Z0-1127-25) Master ...
byExamCert App
 
Automation Without Apprentices: How AI Challenges the Open Source Way
byIcinga
 
Odoo Implementation Checklist: A Strategic ERP Blueprint for Business-Ready D...
byBANIBRO IT SOLUTION
 
GenerationAI_Paris_2025_Architecting_Intelligence.pdf
byapidays
 
How AI Can Help Platform Engineers Build Better Platforms
byAll Things Open
 
Do You Control the AI, or Does the AI Control You?
bymedhateltoukhy
 
Explaining specific examples of purpose-specific BOM
byakipii ogaoga
 
apidays Paris 2025 | Building Agentic Workflows
byapidays
 
apidays New York 2025 | AI in Application Security
byapidays
 
HOW TO OVERCOME THE THREATS OF ARTIFICIAL INTELLIGENCE AGAINST HUMANITY.pdf
byFaga1939
 
final~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.pdf
byomarbishtawi04
 
UiPath Automation Developer Associate Training Series 2026 - Session 3
byDianaGray10
 

HPC Best Practices: Application Performance Optimization

  • 1.
  • 2.
    2 •  MILC •  OpenMX • PARATEC •  PFA •  PFLOTRAN •  Quantum ESPRESSO •  RADIOSS •  SPECFEM3D •  WRF •  LS-DYNA •  miniFE •  MILC •  MSC Nastran •  MR Bayes •  MM5 •  MPQC •  NAMD •  Nekbone •  NEMO •  NWChem •  Octopus •  OpenAtom •  OpenFOAM 140 Applications Best Practices Published •  Abaqus •  AcuSolve •  Amber •  AMG •  AMR •  ABySS •  ANSYS CFX •  ANSYS FLUENT •  ANSYS Mechanics •  BQCD •  CCSM •  CESM •  COSMO •  CP2K •  CPMD •  Dacapo •  Desmond •  DL-POLY •  Eclipse •  FLOW-3D •  GADGET-2 •  GROMACS •  Himeno •  HOOMD-blue •  HYCOM •  ICON •  Lattice QCD •  LAMMPS For more information, visit: http://www.hpcadvisorycouncil.com/best_practices.php
  • 3.
    3 HPC Advisory CouncilHPC Center Dell™ PowerEdge™ R730 32-node cluster HP Cluster Platform 3000SL 16-node cluster HP ProLiant SL230s Gen8 4-node cluster Dell™ PowerEdge™ R815 11-node cluster Dell™ PowerEdge™ C6145 6-node cluster Dell™ PowerEdge™ M610 38-node cluster Dell™ PowerEdge™ C6100 4-node cluster Dell™ PowerVault MD3420 / MD3460 InfiniBand-based Lustre Storage Dell™ PowerEdge™ R720/R720xd 32-node cluster HP Proliant XL230a Gen9 10-node cluster Colfax CX1350s-XK5 4-node cluster
  • 4.
    4 Agenda •  Overview ofHPC Applications Performance •  Way to Inspect, Profile, Optimize HPC Applications –  CPU, memory, file I/O, network •  System Configurations and Tuning •  Case Studies, Performance Comparisons, Optimizations and Highlights •  Conclusions
  • 5.
    5 HPC Application PerformanceOverview •  To achieve scalability performance on HPC applications –  Involves understanding of the workload by performing profile analysis •  Tune for the most time spent (either CPU, Network, IO, etc) –  Underlying implicit requirement: Each node to perform similarly •  Run CPU/memory /network tests or cluster checker to identify bad node(s) –  Comparing behaviors of using different HW components •  Which pinpoint bottlenecks in different areas of the HPC cluster •  A selection of HPC applications will be shown –  To demonstrate method of profiling and analysis –  To determine the bottleneck in SW/HW –  To determine the effectiveness of tuning to improve on performance
  • 6.
    6 Ways To Inspectand Profile Applications •  Computation (CPU/Accelerators) –  Tools: gprof, top, htop, perf top, pstack, Visual Profiler, etc –  Tests and Benchmarks: HPL, STREAM •  File I/O –  Bandwidth and Block Size: iostat, collectl, darshan, etc –  Characterization Tools and Benchmarks: iozone, ior, etc •  Network Interconnect and MPI communications –  Tools and Profilers: perfquery, MPI profilers (IPM, TAU, etc) –  Characterization Tools and Benchmarks: –  Latency and Bandwidth: OSU benchmarks, IMB
  • 7.
  • 8.
    8 Note •  The followingresearch was performed under the HPC Advisory Council activities –  Participating vendors: Intel, Dell, Mellanox –  Compute resource - HPC Advisory Council Cluster Center •  The following was done to provide best practices –  LS-DYNA performance overview –  Understanding LS-DYNA communication patterns –  Ways to increase LS-DYNA productivity –  MPI libraries comparisons •  For more info please refer to –  http://www.dell.com –  http://www.intel.com –  http://www.mellanox.com –  http://www.lstc.com
  • 9.
    9 LS-DYNA •  LS-DYNA –  Ageneral purpose structural and fluid analysis simulation software package capable of simulating complex real world problems –  Developed by the Livermore Software Technology Corporation (LSTC) •  LS-DYNA used by –  Automobile –  Aerospace –  Construction –  Military –  Manufacturing –  Bioengineering
  • 10.
    10 Objectives •  The presentedresearch was done to provide best practices –  LS-DYNA performance benchmarking •  MPI Library performance comparison •  Interconnect performance comparison •  CPU cores/speed comparison •  Optimization tuning •  The presented results will demonstrate –  The scalability of the compute environment/application –  Considerations for higher productivity and efficiency
  • 11.
    11 Test Cluster Configuration • Dell PowerEdge R730 32-node (896-core) “Thor” cluster –  Dual-Socket 14-Core Intel E5-2697v3 @ 2.60 GHz CPUs (Power Management in BIOS sets to Maximum Performance) –  Memory: 64GB memory, DDR4 2133 MHz, Memory Snoop Mode in BIOS sets to Home Snoop –  OS: RHEL 6.5, MLNX_OFED_LINUX-2.4-1.0.5.1_20150408_1555 InfiniBand SW stack –  Hard Drives: 2x 1TB 7.2 RPM SATA 2.5” on RAID 1 •  Mellanox ConnectX-4 EDR 100Gb/s InfiniBand Adapters •  Mellanox Switch-IB SB7700 36-port EDR 100Gb/s InfiniBand Switch •  Mellanox ConnectX-3 FDR VPI InfiniBand and 40Gb/s Ethernet Adapters •  Mellanox SwitchX-2 SX6036 36-port 56Gb/s FDR InfiniBand / VPI Ethernet Switch •  MPI: Open MPI 1.8.4, Mellanox HPC-X v1.2.0-326, Intel MPI 5.0.2.044, IBM Platform MPI 9.1 •  Application: –  LS-DYNA 8.0.0 (builds 95359, 95610), Single Precision •  Benchmarks: 3 Vehicle Collision, Neon refined revised
  • 12.
    12 PowerEdge R730 Massive flexibilityfor data intensive operations •  Performance and efficiency –  Intelligent hardware-driven systems management with extensive power management features –  Innovative tools including automation for parts replacement and lifecycle manageability –  Broad choice of networking technologies from GigE to IB –  Built in redundancy with hot plug and swappable PSU, HDDs and fans •  Benefits –  Designed for performance workloads •  from big data analytics, distributed storage or distributed computing where local storage is key to classic HPC and large scale hosting environments •  High performance scale-out compute and low cost dense storage in one package •  Hardware Capabilities –  Flexible compute platform with dense storage capacity •  2S/2U server, 6 PCIe slots –  Large memory footprint (Up to 768GB / 24 DIMMs) –  High I/O performance and optional storage configurations •  HDD options: 12 x 3.5” - or - 24 x 2.5 + 2x 2.5 HDDs in rear of server •  Up to 26 HDDs with 2 hot plug drives in rear of server for boot or scratch
  • 13.
    13 LS-DYNA Performance –Network Interconnects •  EDR InfiniBand delivers superior scalability in application performance –  Provides higher performance by over 4-5 times than 1GbE, 10GbE and 40GbE –  1GbE stop scaling beyond 4 nodes, and 10GbE stops scaling beyond 8 nodes –  InfiniBand demonstrates continuous performance gain at scale 28 MPI Processes / NodeHigher is better 444% 505% 572%
  • 14.
    14 •  Majority ofthe MPI time is spent on MPI_recv and MPI Collective Ops –  MPI_Recv(36%), MPI_Allreduce(27%), MPI_Bcast(24%) •  Similar communication characteristics seen on both input dataset –  Both exhibit similar communication patterns LS-DYNA Profiling – Time Spent in MPI 3 Vehicle Collision – 32 nodesNeon_refined_revised – 32 nodes
  • 15.
    15 LS-DYNA Profiling –Time Spent in MPI •  Most of the MPI messages are in the medium sizes –  Most message sizes are between 0 to 64B •  For the most time consuming MPI calls –  MPI_Recv: Most messages are under 4KB –  MPI_Bcast: Majority are less than 16B, but larger messages exist –  MPI_Allreduce: Most messages are less than 256B neon_refined_revised
  • 16.
    16 LS-DYNA Performance –EDR vs FDR InfiniBand •  EDR InfiniBand delivers superior scalability in application performance –  As the cluster scales, performance gap of EDR IB becomes widen •  Performance advantage of EDR InfiniBand increases for larger core counts –  EDR IB provides 15% versus FDR IB at 32 nodes (896 cores) Higher is better 15% 28 MPI Processes / Node
  • 17.
    17 LS-DYNA Performance –Cores Per Node •  Better performance is seen at scale with less CPU cores per node –  At low node counts, higher performance can be achieved with more cores per node –  At high node counts, slightly better performance by using less cores per node –  Memory bandwidth might be limited by more CPU cores being used CPU @ 2.6GHzHigher is better 5%3%2% 5%
  • 18.
    18 LS-DYNA Performance –AVX2/SSE2 CPU Instructions •  LS-DYNA provides executables with supports for different CPU instructions –  AVX2 is supported on “Haswell” while SSE2 is supported on previous generations –  Due to runtime issue, AVX2 executable build 95610 is used, instead of the public build 95359 –  Slight improvement of ~2-4% by using executable with AVX2 instructions –  The AVX2 instructions runs at a lower clock speed (2.2GHz) than normal CPU clock (2.6GHz) Higher is better 4% 24 MPI Processes / Node
  • 19.
    19 LS-DYNA Performance –Turbo Mode •  Turbo Boost enables processors to run above its base frequency –  Capability to allow CPU cores to run dynamically above the CPU clock –  When thermal headroom allows the CPU to operate –  The 2.6GHz clock speed could boost to Max Turbo Frequency of 3.3GHz –  Running with Turbo Boost translates to a ~25% of performance boost 40% 25% Higher is better 28 MPI Processes / Node
  • 20.
    20 LS-DYNA Performance –Memory Optimization •  Setting the environment variables for memory allocator improve on performance –  Modifying the memory allocator allows faster memory registration for communications •  Environment variables used: –  export MALLOC_MMAP_MAX_=0 –  export MALLOC_TRIM_THRESHOLD_=-1 176% Higher is better 19% 28 MPI Processes / Node
  • 21.
    21 LS-DYNA Performance –MPI Optimization •  FCA and MXM enhance LS-DYNA performance at scale for HPC-X –  Open MPI and HPC-X are based on the Open MPI distribution –  The “yalla” PML, UD transport and memory optimization in HPC-X reduce overhead –  MXM provides a speedup of 38% over un-tuned baseline run at 32 nodes (768 cores) •  MCA parameters: •  -mca btl_sm_use_knem 1 -mca pml yalla -x MXM_TLS=ud,shm,self -x MXM_SHM_RNDV_THRESH=32768 •  -x HCOLL_CONTEXT_CACHE_ENABLE=1 Higher is better 38% 24 MPI Processes / Node
  • 22.
    22 LS-DYNA Performance –Intel MPI Optimization •  The DAPL provider performs better than OFA provider for Intel MPI –  DAPL would provide better scalability performance for Intel MPI on LS-DYNA •  MCA parameters for MXM: –  Common for 2 tests: I_MPI_DAPL_SCALABLE_PROGRESS 1, I_MPI_RDMA_TRANSLATION_CACHE 1, I_MPI_FAIR_CONN_SPIN_COUNT 2147483647, I_MPI_FAIR_READ_SPIN_COUNT 2147483647, I_MPI_ADJUST_REDUCE 2, I_MPI_ADJUST_BCAST 0, I_MPI_RDMA_TRANSLATION_CACHE 1, I_MPI_RDMA_RNDV_BUF_ALIGN 65536, I_MPI_SPIN_COUNT 121 –  For OFA: -IB, MV2_USE_APM 0, I_MPI_OFA_USE_XRC 1 –  For DAPL: -DAPL, I_MPI_DAPL_DIRECT_COPY_THRESHOLD 65536, I_MPI_DAPL_UD enable, I_MPI_DAPL_PROVIDER ofa-v2-mlx5_0-1u Higher is better 20% 24 MPI Processes / Node
  • 23.
    23 LS-DYNA Performance –MPI Libraries •  HPC-X outperforms Platform MPI, and Open MPI in scalability performance –  HPC-X delivers higher performance than Intel MPI (OFA) by 33%, (DAPL) by 11%, Platform MPI by 27% on neon_refined_revised –  Performance is 20% higher than Intel OFA, and % 8% better than Platform MPI in 3cars •  Tuning parameter used: –  For Open MPI: -bind-to-core and KNEM. For Platform MPI: -cpu_bind, -xrc. For Intel MPI: see previous slide 7%27% Higher is better 33% 20% 11% 24 MPI Processes / Node
  • 24.
    24 •  Current Haswellsystem configuration outperforms prior system generations –  Current systems outperformed Ivy Bridge by 47%, Sandy Bridge by 75%, Westmere by 148%, Nehalem by 290% –  Scalability support from EDR InfiniBand and HPC-X provide huge boost in performance at scale for LS-DYNA •  System components used: –  Haswell: 2-socket 14-core E5-2697v3@2.6GHz, 2133MHz DIMMs, ConnectX-4 EDR InfiniBand –  Ivy Bridge: 2-socket 10-core E5-2680v2@2.8GHz, 1600MHz DIMMs, Connect-IB FDR InfiniBand –  Sandy Bridge: 2-socket 8-core E5-2680@2.7GHz, 1600MHz DIMMs, ConnectX-3 FDR InfiniBand –  Westmere: 2-socket 6-core x5670@2.93GHz, 1333MHz DIMMs, ConnectX-2 QDR InfiniBand –  Nehalem: 2-socket 4-core x5570@2.93GHz, 1333MHz DIMMs, ConnectX-2 QDR InfiniBand 290% 47% 75%148% Higher is better LS-DYNA Performance – System Generations
  • 25.
    25 LS-DYNA Summary •  Performance – Compute: Intel Haswell cluster outperforms system architecture of previous generations •  Outperforms Ivy Bridge by 47%, Sandy Bridge by 75%, Westmere by 148%, and Nehalem by 290% •  Using executable with AVX2 instructions provides slight advantage •  Slight improvement of ~2-4% by using executable with AVX2 instructions –  Turbo Mode: Running with Turbo Boost provides ~25% of performance boost in some cases •  Turbo Boost enables processors to run above its base frequency –  Network: EDR InfiniBand and HPC-X MPI library deliver superior scalability in application performance •  EDR IB provides higher performance by over 4-5 times vs 1GbE, 10GbE and 40GbE, 15% vs FDR IB at 32 nodes •  MPI Tuning –  HPC-X enhances LS-DYNA performance at scale for LS-DYNA •  MXM UD provides a speedup of 38% over un-tuned baseline run at 32 nodes –  HPC-X outperforms Platform MPI, and Open MPI in scalability performance •  Up to 27% better than Platform MPI on neon_refined_revised, and 8% better than Platform MPI in 3cars
  • 26.
  • 27.
    27 GROMACS •  GROMACS (GROningenMAchine for Chemical Simulation) –  A molecular dynamics simulation package –  Primarily designed for biochemical molecules like proteins, lipids and nucleic acids •  A lot of algorithmic optimizations have been introduced in the code •  Extremely fast at calculating the nonbonded interactions –  Ongoing development to extend GROMACS with interfaces both to Quantum Chemistry and Bioinformatics/databases –  An open source software released under the GPL
  • 28.
    28 Objectives •  The presentedresearch was done to provide best practices –  GROMACS performance benchmarking •  CPU performance comparison •  MPI library performance comparison •  Interconnect performance comparison •  System generations comparison •  The presented results will demonstrate –  The scalability of the compute environment/application –  Considerations for higher productivity and efficiency
  • 29.
    29 Test Cluster Configuration • Dell PowerEdge R730 32-node (896-core) “Thor” cluster –  Dual-Socket 14-Core Intel E5-2697v3 @ 2.60 GHz CPUs (Power Management in BIOS sets to Maximum Performance) –  Memory: 64GB memory, DDR4 2133 MHz, Memory Snoop Mode in BIOS sets to Home Snoop, Turbo Enabled –  OS: RHEL 6.5, MLNX_OFED_LINUX-3.0-1.0.1 InfiniBand SW stack –  Hard Drives: 2x 1TB 7.2 RPM SATA 2.5” on RAID 1 •  Mellanox ConnectX-4 EDR 100Gbps EDR InfiniBand Adapters •  Mellanox Switch-IB SB7700 36-port 100Gb/s EDR InfiniBand Switch •  Mellanox ConnectX-3 FDR InfiniBand, 10/40GbE Ethernet VPI Adapters •  Mellanox SwitchX-2 SX6036 36-port 56Gb/s FDR InfiniBand / VPI Ethernet Switch •  MPI: Mellanox HPC-X v1.2.0-326 •  Compiler and Libraries: Intel Composer XE 2015.3.187 and MKL •  Application: GROMACS 4.6.7 •  Benchmark datasets: DPPC in Water (d.dppc, 121856 atoms, 150000 steps, SP) unless stated otherwise
  • 30.
    30 GROMACS Performance –Network Interconnects •  InfiniBand is the only interconnect that delivers superior scalability performance –  EDR InfiniBand provides higher performance and more scalable than 1GbE, 10GbE, or 40GbE –  Performance for Ethernet stays flat (or stops scaling) beyond 2 nodes –  EDR InfiniBand outperforms 10GbE-RoCE on scalability performance by 55% at 32 nodes / 896c –  EDR InfiniBand demonstrates continuous performance gain at scale 28 MPI Processes / NodeHigher is better 55%4.1x4.2x 4.6x
  • 31.
    31 GROMACS Profiling –Time Spent by MPI Calls •  The most time consuming MPI call os MPI_Sendrecv –  MPI_Sendrecv: 66% (or 27% of runtime) at 32 nodes (896 cores) –  MPI_Waitall: 18% (or 7% of runtime), MPI_Bcast: 6% (or 2% of runtime) –  Point to point and non-blocking sends and receives consume most time in GROMACS 32 Nodes
  • 32.
    32 GROMACS Profiling –MPI Message Sizes •  Majority of data transfer messages are medium sizes, except for: –  MPI_Sendrecv has a large concentration (from 8B to 8KB) –  MPI_Bcast shows some concentration 32 Nodes
  • 33.
    33 GROMACS Profiling –MPI Data Transfer •  As the cluster grows, similar communication behavior is seen –  Majority of communications are between neighboring ranks –  Non-blocking (point to point) data, and point-to-point transfers are shown in the graph –  Collective data communications are small compared to point-to-point communications 2 Nodes / 56 Cores32 Nodes / 896 Cores
  • 34.
    34 GROMACS Performance –EDR vs FDR InfiniBand •  EDR InfiniBand delivers superior scalability in application performance –  As the number of nodes scales, performance gap of EDR IB becomes widen •  Performance advantage of EDR InfiniBand increases for larger core counts –  EDR InfiniBand provides 29% versus FDR InfiniBand at 32 nodes (896 cores) Higher is better 29% 28 MPI Processes / Node
  • 35.
    35 GROMACS Performance –System Generations •  Thor cluster (based on Intel E5-2697v3 - Haswell) outperforms prior generations –  1.1 to 3.5x higher performance than clusters based on previous generations of Intel architecture •  System components used: –  Janus: 2-socket 6-core Xeon X5670 @ 2.93GHz, 1333MHz DIMMs, ConnectX-2 QDR IB –  Jupiter: 2-socket 8-core Xeon E5-2680 @ 2.7GHz, 1600MHz DIMMs, ConnectX-3 FDR IB –  Thor: 2-socket 14-core Xeon E5-2680V3 @2.6GHz, 2133MHz DIMMs, ConnectX-4 EDR IB 356% 118% Higher is better
  • 36.
    36 GROMACS Performance –Cores Per Node •  Running more CPU cores provides higher performance –  ~7-10% higher productivity with 28PPN compared to 24PPN –  Higher demand on memory bandwidth and network might limit performance as more cores are used CPU @ 2.6GHzHigher is better 13% 39% 7%
  • 37.
    37 GROMACS Performance –Turbo Mode & CPU Clock •  Advantages are seen with running higher clock rate –  Either by enabling Turbo mode or higher CPU clock frequency •  Boosting CPU clock rate yields higher performance at lower cost –  Increasing to 2600MHz (from 2300MHz) run 11% faster CPU @ 2.6GHzHigher is better 16% 11%
  • 38.
    38 GROMACS Performance –Floating Point Precision •  GROMACS allows running either SP and DP for floating point precision •  Running at SP is shown to be faster than running at DP –  Seen around 41%-47% faster running at SP (Single Precision) versus DP (Double Precision) –  All other slides are running using Single Precision CPU @ 2.6GHzHigher is better 47% 41%
  • 39.
    39 GROMACS Summary •  Latestsystem generation improve GROMACS performance at scale –  Compute: Intel Haswell cluster outperforms system architecture of previous generations •  Haswell cluster outperforms Sandy Bridge cluster by 110%, and outperforms Westmere cluster by 350% at 32 node –  Compute: Running more CPU cores provides higher performance •  ~7-10% higher productivity with 28PPN compared to 24PPN –  Network: EDR InfiniBand delivers superior scalability in application performance •  EDR InfiniBand provides higher performance and more scalable than 1GbE, 10GbE, or 40GbE •  Performance for Ethernet (1GbE/10GbE/40GbE) stays flat (or stops scaling) beyond 2 nodes •  EDR InfiniBand outperforms 10GbE-RoCE on scalability performance by 55% at 32 nodes / 896c –  Running at Single Precision is approximately twice as fast as running at Double Precision •  Seen around 41%-47% faster running at SP (Single Precision) versus DP (Double Precision) –  MPI Profile shows majority of data transfer are point-to-point and non-blocking communications •  MPI_Sendrecv and MPI_Waitall are the most used MPI communication
  • 40.
    4040 Thank You HPC AdvisoryCouncil All trademarks are property of their respective owners. All information is provided “As-Is” without any kind of warranty. The HPC Advisory Council makes no representation to the accuracy and completeness of the information contained herein. HPC Advisory Council undertakes no duty and assumes no obligation to update or correct any information presented herein