We present a fast transient solver suitable for the simulation of transient turbulent flows. The main characteristic of the solver is that it is based on the projection method and requires only one pressure and the momentum solve per time step. Furthermore, using the projection method has the additional advantage that the formulation of the pressure equation is particularly efficient because the Laplacian term depends only on geometric quantities. This improves the parallel scalability of the Algebraic Multigrid Method (AMG) because the coarse agglomeration levels can be cached if the grid is not changing and are consistent throughout each time step. The fractional step error near the boundaries is removed by utilising the incremental version of the algorithm. The solver is implemented for the freely available Caelus computational mechanics library. Accuracy and performance of the solver was investigated through several validation cases. The results indicate the solver is both accurate and has excellent computational efficiency.

1. logo.png
Applied CCM Motivation PISO SLIM Results Summary
A Projection Method Based Fast Transient
Solver for Incompressible Turbulent Flows
Chris Sideroff
08 June 2015
Applied CCM © 2012-2015 23rd
CFD Society of Canada Conference

2. logo.png
Applied CCM Motivation PISO SLIM Results Summary
Applied CCM
Specialize in the application, development and support of
OpenFOAM® - based software
Creators and maintainers of Caelus
Locations: Canada, Australia, USA
Applied CCM © 2012-2015 23rd
CFD Society of Canada Conference

3. logo.png
Applied CCM Motivation PISO SLIM Results Summary
Motivation
Why develop another transient solver?
DES and LES attractive because RANS tends to be
problem specific
Low cost hardware + open-source software ⇒ DES and
LES feasible
Traditional transient, incompressible algorithms (PISO and
SIMPLE) do not scale well for large HPC, GPU and Many
Integrated Core (MIC) environments
Let’s review PISO algorithm
Applied CCM © 2012-2015 23rd
CFD Society of Canada Conference

4. logo.png
Applied CCM Motivation PISO SLIM Results Summary
PISO Overview
Pressure Implicit with Splitting of Operators (PISO)1 method:
1. Solve momentum equation (predictor step)
2. Calculate intermediate velocity, u∗ (pressure dissipation
added)
3. Calculate momentum fluxes
4. Solve pressure equation:
· ( 1
Ap
p) = · u∗
5. Correct momentum fluxes
6. Correct velocity (corrector step)
Repeat steps 2 – 6 for PISO (1 – 6 for transient SIMPLE)
1Isaa, R.A. 1985, “Solution of the implicitly discretised fluid flow equations by
operator splitting” J. Comp. Phys., 61, 40.
Applied CCM © 2012-2015 23rd
CFD Society of Canada Conference

5. logo.png
Applied CCM Motivation PISO SLIM Results Summary
Fractional Step Error
Step 2 main issue with PISO
Predicted velocity used only to update matrix coefficients:
u∗ = 1
ap
Σ anb unb − ( p − p)
Pseudo-velocity, u∗, is used on the RHS of pressure
equation
Therefore requires at least two corrections to make velocity
and pressure consistent
Applied CCM © 2012-2015 23rd
CFD Society of Canada Conference

6. logo.png
Applied CCM Motivation PISO SLIM Results Summary
Pressure Matrix
Non-constant coefficients ( 1
ap
) in pressure matrix affects
multi-grid solver performance
Multi-grid agglomeration levels cached first time pressure
matrix assembled
Coefficients ( 1
ap
) only valid for the first time step
Turning off caching of agglomeration too expensive
Applied CCM © 2012-2015 23rd
CFD Society of Canada Conference

7. logo.png
Applied CCM Motivation PISO SLIM Results Summary
SLIM Overview
Semi Linear Implicit Method (SLIM), based on projection
method1: decompose velocity into vortical and irrotational
components.
1. Solve momentum equation (vortical velocity)
2. Calculate momentum fluxes (pressure dissipation added)
3. Solve pressure equation (irrotational velocity):
∆t 2(p) = · u
4. Correct momentum flux
5. Correct velocity (solenoidal)
Use incremental pressure approach to recover correct
boundary pressure
1Chorin, A.J. 1968, “Numerical Solution of the Navier-Stokes
Equations”,Mathematics of Computation 22: 745-762
Applied CCM © 2012-2015 23rd
CFD Society of Canada Conference

8. logo.png
Applied CCM Motivation PISO SLIM Results Summary
Fractional Step Error
Velocity split into vortical and potential components - much
smaller fractional step error
Pressure and velocity maintain stronger coupling
Continuity satisfied within one pressure solve because
predicted velocity used directly in pressure equation
Applied CCM © 2012-2015 23rd
CFD Society of Canada Conference

9. logo.png
Applied CCM Motivation PISO SLIM Results Summary
Pressure Matrix
Pressure matrix coefficients purely geometric
Multi-grid agglomeration levels assembled during first step
now consistent for all time steps
Significantly improves parallel scalability for multi-grid
solver
Applied CCM © 2012-2015 23rd
CFD Society of Canada Conference

10. logo.png
Applied CCM Motivation PISO SLIM Results Summary
2D Periodic Hills
Two dimensional, stream-wise, staggered hills of
polynomial shape
Reh = 10,595
Stream-wise and span-wise boundaries periodic. Hills and
top boundaries no slip.
Grid: ∼ 4.5 million hex cells; LES model: Smagorinsky
Applied CCM © 2012-2015 23rd
CFD Society of Canada Conference

11. logo.png
Applied CCM Motivation PISO SLIM Results Summary
Validation
Experimental data of Rapp (2009)
Mean and second moment components at 10 vertical rakes
Applied CCM © 2012-2015 23rd
CFD Society of Canada Conference

12. logo.png
Applied CCM Motivation PISO SLIM Results Summary
x/h = 2
Both compare favorably
SLIM slightly closer than PISO
Applied CCM © 2012-2015 23rd
CFD Society of Canada Conference

13. logo.png
Applied CCM Motivation PISO SLIM Results Summary
x/h = 4
SLIM consistently closer than PISO at all locations
Likely due to lower fractional step error
Applied CCM © 2012-2015 23rd
CFD Society of Canada Conference

14. logo.png
Applied CCM Motivation PISO SLIM Results Summary
Simulation Time
SLIM on average about 30% faster on modest HPC system
Fewer total iterations of pressure equation (SLIM: 10;
PISO: 14)
# cores PISO SLIM % diff.
1 2095 1550 26
5 988 711 28
10 419 302 28
20 330 231 30
40 219 147 33
60 216 138 36
Applied CCM © 2012-2015 23rd
CFD Society of Canada Conference

15. logo.png
Applied CCM Motivation PISO SLIM Results Summary
Precursor Simulation
Establish turbulent conditions to use as initial condition for
wind park simulation
Start from quiescent condition. Run until fully turbulent.
Steam-wise and span-wise periodic
Grid size: 50 million hex cells
Results courtesy of Greg Oxley at Vestas using Firestorm
super computer
Applied CCM © 2012-2015 23rd
CFD Society of Canada Conference

16. logo.png
Applied CCM Motivation PISO SLIM Results Summary
Mean Wind Profile
SLIM slightly more accurate than PISO
Fully turbulent condition reached sooner than PISO
Applied CCM © 2012-2015 23rd
CFD Society of Canada Conference

17. logo.png
Applied CCM Motivation PISO SLIM Results Summary
Scaling
Consistent multi-grid agglomeration levels give SLIM
significant advantage
Applied CCM © 2012-2015 23rd
CFD Society of Canada Conference

18. logo.png
Applied CCM Motivation PISO SLIM Results Summary
MPI Profiling
Profiled MPI calls on 125 million cell mesh up to 4096
cores
Applied CCM © 2012-2015 23rd
CFD Society of Canada Conference

19. logo.png
Applied CCM Motivation PISO SLIM Results Summary
Future Work
For static grids, pressure matrix construction may be pulled
entirely from time loop to save assembly of pressure matrix
every time step
Advantageous for GPU and MIC computing. Compute
pressure matrix once. Only need to transfer RHS vector
For peta-scale core counts, solve momentum equations
explicitly (Runga-Kutta). Combined with above, could
perform close to fully explicit codes
Solvers have been developed and are undergoing testing
Applied CCM © 2012-2015 23rd
CFD Society of Canada Conference

20. logo.png
Applied CCM Motivation PISO SLIM Results Summary
Summary
SLIM significantly faster than PISO. Problem dependent
but 30-100% is typical improvement and even more for
very large HPC calculations.
Exact velocity splitting improves both convergence and
accuracy
Geometric pressure matrix coefficients advantageous for
parallel efficiency, particularly for multi-grid solvers
Additional modifications enable scaling to very large
number of cores (HPC, GPU, MIC)
Applied CCM © 2012-2015 23rd
CFD Society of Canada Conference

21. logo.png
Applied CCM Motivation PISO SLIM Results Summary
Strategic Perspective
Select research and development projects that are unique and
help transfer knowledge to industrial applications.
Solvers: transient, compressible, multi-phase, combustion,
acoustics
Turbulence: RANS, DES and LES, VLES, wall models
Sensitivity, design optimisation, and uncertainty
propagation: adjoint, tangent
Numerical acceleration and stabilisation
Platforms and architectures: HPC, GPU, MIC
Applied CCM © 2012-2015 23rd
CFD Society of Canada Conference