Parallel efficiency of GAMG solver in OpenFOAM is evaluated for EPYC server. Especially, in this study, the influence of coarsestLevelCorr on the calculation time is evaluated in lid driven cavity flow.

1. OpenFOAM benchmark
for EPYC server
- GAMG solver
coarsestLevelCorr -
Osaka Metropolitan University
Takuya YAMAMOTO
2023/8/23

2. GAMG solver control
p OpenFOAM v1906
https://www.openfoam.com/news/main-
news/openfoam-v1906/numerics#numerics-
adjoint
Improved GAMG solver controls
マルチグリッドで解く際に最も粗い格⼦レベルでの格⼦数が⾮常に少ない
(数⼗セル)
並列計算時にこの粗い格⼦レベルでの通信量がボトルネックになることで、
並列化効率が悪化
これを改善するために、最も粗い格⼦レベルでの解法を選択できるように
pFinal
{
solver GAMG;
tolerance 1e-6;
relTol 0;
smoother GaussSeidel;
coarsestLevelCorr
{
solver PCG;
preconditioner DIC;
relTol 0.05;
}
}

3. GAMG solver control
p OpenFOAM v2206
https://www.openfoam.com/news/main-
news/openfoam-v20-06/solvers-and-
physics#solver-and-physics-pipelined-cg-
solvers
New pipelined Conjugate Gradient solvers
並列計算時に共役勾配系(CG) solverでは全てのプロセッサーで同じ探索⽅向に
解く必要があり、並列計算時のボトルネックになることがある
パイプラインCG solverではオーバーヘッド、通信を減らすことができる
特に、GAMGの最も粗いレベルの格⼦にこの⼿法を取り⼊れると有効である
PPCR, PPCGが選択可能
pFinal
{
$p;
relTol 0;
coarsestLevelCorr
{
solver PPCR;
preconditioner DIC;
relTol 0.05;
}
}
これらの⼿法が果たしてどれだけEPYCサーバーで効くのか?

4. OpenFOAM benchmark
p Benchmark of High Performance Computing (HPC) Technical
Committee
• Small, S (1M)
• Medium, M (8M)
• Extra-Large, XL (64M)
Ø 3-D Lid Driven cavity flow
Ø HPC Motorbike
Ø Conical Diffuser
Ø …
Many benchmark has been prepared.
In this study, we use 3-D Lid Driven Cavity Flow, S and M.
The used OpenFOAM is ESI v2212 version.
https://develop.openfoam.com/committees/hpc/-/tree/develop/

5. Server resource
p Used server
• Server 1: EPYC 7352 Dual CPU (2.3 GHz x 48 cores)
RAM 128 GB (8GB x 16 channel)
BW 187.7 GB/s (2933 MT/s x 8 channel x 8)
L3 Cache 128 MB
• Server 2: EPYC 7513 Dual CPU (2.6 GHz x 64 cores)
RAM 128 GB (8GB x 16 channel)
BW 204.8 GB/s (3200 MT/s x 8 channel x 8)
L3 Cache 128 MB
EPYC 3rd Gen
EPYC 2nd Gen
• Server 3: EPYC 7542 Dual CPU (2.9 GHz x 64 cores)
RAM 128 GB (8GB x 16 channel)
BW 187.7 GB/s (2933 MT/s x 8 channel x 8)
L3 Cache 128 MB
EPYC 2nd Gen

6. Server resource
p Used server
• Server 4: EPYC 7713 Dual CPU (2.0 GHz x 128 cores)
RAM 256 GB (16GB x 16 channel)
BW 204.8 GB/s (3200 MT/s x 8 channel x 8)
L3 Cache 256 MB
EPYC 3rd Gen
• Server 5: EPYC 7763 Dual CPU (2.45 GHz x 128 cores)
RAM 128 GB (8GB x 16 channel)
BW 204.8 GB/s (3200 MT/s x 8 channel x 8)
L3 Cache 256 MB
EPYC 3rd Gen

7. Solver of algebraic matrix
p solver
• Solver 1: solver, p GAMG
GAMG preconditioner, p GaussSeidel
tolerance, p 1 x 10-4
solver, U smoothSolver
preconditioner, U GaussSeidel
tolerance, U 0
relTol, U 0
maxIter, U 5
• Solver 2: solver, p GAMG
GAMG-PPCR preconditioner, p GaussSeidel
tolerance, p 1 x 10-4
solver, U smoothSolver
preconditioner, U GaussSeidel
tolerance, U 0
relTol, U 0
maxIter, U 5
coarsestLevelCorr
{
solver PPCR;
preconditioner DIC;
relTol 0.05;
}

8. Server 1
EPYC-DualCPU-7352

9. 3D- Lid Driven cavity flow (S, 1M)

10. 3D- Lid Driven cavity flow (M, 8M)

11. Server 2
EPYC-DualCPU-7513

12. 3D- Lid Driven cavity flow (S, 1M)

13. 3D- Lid Driven cavity flow (M, 8M)

14. Server 3
EPYC-DualCPU-7542

15. 3D- Lid Driven cavity flow (S, 1M)

16. 3D- Lid Driven cavity flow (M, 8M)

17. Server 4
EPYC-DualCPU-7713

18. 3D- Lid Driven cavity flow (S, 1M)

19. 3D- Lid Driven cavity flow (M, 8M)

20. Server 5
EPYC-DualCPU-7763

21. 3D- Lid Driven cavity flow (S, 1M)

22. 3D- Lid Driven cavity flow (M, 8M)

23. Comparison between
servers

24. 3D- Lid Driven cavity flow (S, 1M)

25. 3D- Lid Driven cavity flow (M, 8M)