Prof. dr. sc. Zlatan Car, Center of advanced computing and modelling, university of Rijeka

1. 1
Prof. dr. sc. Zlatan Car
2016 ANSYS' Convergence Conference
Ljubljana

2. Rmax: 233.565TFlop/s
Rpeak: 287.539TFlop/s
Peak Power (kW): 108.48
Processor: Xeon E5-2690v3 12C 2.6GHz
Sockets per Node: 2
Cores per Socket: 12
Nodes: 288
Primary Interconnect: InfiniBand FDR
441 175 48
Prof. dr. sc. Zlatan Car
2

3. Schematic Architecture
3

4. Supercomputer BURA
• Hybrid computer architecture – multiprocessor and multicomputer system
• System with a large amount of shared memory and processors (SMP) is
comprised of 12TB of memory, 512 processor cores and 245TB of local data
storage.
• Multicomputer system - computer cluster: 288 compute nodes (6912
processor cores); each node comprises 64GB of memory and 320GB of data
storage. In total 18 terabytes of RAM and disk systems capacities of 850
terabytes.
• Four nodes with double accelerators.
• Central system for data storage has a capacity of 1PB, while data archive
has 2,5 PB.
4
Prof. dr. sc. Zlatan Car

5. Supercomputer BURA
Cooling system consists of 2 subsystems:
• WARMWATER subsystem for direct cooling of computing nodes
• COLDWATER subsystem for cooling of other components including 4 cooling
chambers from data center (Cooling power is 210kW with additional 210kW in
redundancy)
• Both subsystems have possibility to operate in a free-cooling mode when the
outside temperature is below 27 (24) °C, inlet water temperature is 35 (30)°C
• 550 meters of pipes and 10 pumps have been installed in the cooling system.As
for the automation purposes, 45 relays for digital outputs and 110 different types
of sensors for analog and digital inputs and outputs have been integrated.Only
for interconnection of control components, 5 km of cable is used.
5
Prof. dr. sc. Zlatan Car

6. Bura Supercomputer
6
bullx DLC B700 Rack
Solution
Plateform bullx DLC B700
Sockets Intel® Xeon EP E5-2690 v3
High Speed
Interconnect
Network
InfiniBand Mellanox 4x FDR (56 GB/s)
Management
Network
Ethernet 1 Gbit
Cisco 3750
Large memory
nodes
bullx S6130
Service & I/O
nodes
bullx R423-E4i
Scratch Storage Lustre File system
Home Storage NFS based*
Software Stack/OS bullx scs4 / RHEL 6
HSM Solution Graudata (GAM) software
▶ Mature Technology
▶ Second DLC Blade
Generation
▶ Standard Architecture

7. bullx Direct Liquid Cooling system
Cooperative Power
Chassis (CPC)
bullx DLC B700 chassis
Front view
Rear view

8. Compute Solution based on Conventionnal CPU
8
bullx DLC B720 Blade
Conventional Nodes
Racks DLC B700 4
B700 Châssis 16
B720 compute
blades
144
Compute Nodes 288
Intel®E5-2690 v3
(12c, 2.6 GHz, 30
Mo/135W)
576
bullx DLC B700
bullx DLC B700 chassis
287,5
Tflops Peak
18,5 TB
memory
78,7% HPL
efficiency

9. Compute Solution based on Accelerators node
9
Accelerators Nodes
Racks DLC B700 No addon
B700 Châssis 1
B715 compute blades 4
Compute Nodes 4
NVIDIA K40 8
▶ Per node
– 2 Ivy Bridge-EP E5-2650 v3
– 64 GB memory (8 x 8 GB DDR3 DIMMs
1866 MT/s)
– 2 NVIDIA K40 GPU
– 1 HDD 320 GB
– 1 IB 4x FDR C-IB NIC
– 1 Ethernet 1 Gb
– 1 BMC
8 GT/s
IBFDR
56Gb/s
DMI
IV
B
4x DDR3@1600
51.2GB/s
IV
B
bullx B715
CX3HDD/S
SD
PCH
PCIe-3 8x
HDD/S
SD
gbE
BHBMC
Accelerator
PCIe-3
16x
PCIe-3
16x
Accelerator

10. bullx S6130
10
▶ 2 bullx octo-modules S6130
▶ Second Generation of bullx Shared memory nodes
▶ Based on BCS2 chip & Intel Haswell EX sockets (Intel
Brickland)
▶ Local Storage attachment (3x NetApp E5500)
Per bullx S6130
Bi-sockets
modules
2
Sockets 16
SKU E7-8867 v3 cores
(16c/2.5 GHz/9,6
GT/45Mo, 165W)
Shared
memory
6 TB (384 x 16 GB
RDIMM)
Raid board
SAS
4
HDD 600Go
SAS 10k
8
10G ports
(RJ45)
2
BMC 1
Octo-modules
All-To-All Topology
XQPI link: 14 GT/s
unidirectional

11. Bura Supercomputer
11
Performances
System Benchmark Test case Labelling Commitmts Final
Thin Nodes
HPL
Performance 234.9 TFlops 233.56 Tflops
Power
consumption at
Rmax
123.6 kW 108.48 kW
Computational
efficiency
81.7 % 98.97 %
Power Efficiency
1.90
TFlops/kW
2.15
TFlops/kW
SPEC mpi 2007 x
SPECmpiL_2007_
base
90 90,8 (92,4)
OpenFoam Motorbike ClockTime 3.66 min 3.53 min
LS-Dyna Car2Car Elapsed time 25.41 min 22.00 min

12. Bura Supercomputer
12
Performances
System Benchmark Test case Labelling Commitmts Final
SMP Nodes
iozone 3.347
x Read throughput 12 GB/s 12.50 GB/s
x Write throughput 12 GB/s 13.21 GB/s
SPEC omp 2012 x
SPECompG_2012
_base
43 47.9
Gaussian G09
D.01
g09-big.com Wall-clock time 5.83 h 5.13 h
Abaqus s4e Elapsed time 1.08 h 0.93 h
Scratch iozone 3.347
x Read throughput 30 GB/s min 37.91 GB/s min
x Write throughput 30 GB/s min 32.85 GB/s min
FDR Throughput IMB Ping Pong x x 6,31 GB/s 6,31 GB/s

13. Running on BURA / What do we compute?
• Design of complex computational systems
(HPC Cloud)
• Design of low power computing
• Computational chemistry and biology
• Computer aided drug design
• Cyber security
• Structural mechanical engineering and
design
• Advanced scheduling and planning systems
• Advanced weather forecast
• Air pollution
• Fluid dynamics

14. Example of calculation
Prof. dr. sc. Zlatan Car
Prof. dr. sc. Željko Svedružić (Department of biotechnology, UNIRI)

15. Center for advanced computing and modelling
• Technical expertise:
• Design of High Performance Computing system (cluster, hybrid cluster, SMP)
• Design of data storage
• Design of complex green Data Center
• Procurement expertise:
• Preparation of procurement documentation (technical and capacity
requirements)
• Leading of procurement, open call and competitive negotiation dialogue
Consulting expertise

16. Collaboration / Joint projects

17. 17

18. Email: cnrm@cnrm.uniri.hr
URL: www.cnrm.uniri.hr