The document summarizes a new Parallel Krylov Solver (PKS) package that allows for parallel computing of large hydrological models. PKS divides the model grid and linear system over multiple processor cores using domain decomposition techniques. It has been incorporated into several Deltares hydrological models including MODFLOW, iMODFLOW, and MODFLOW-USG. Results show speedups of up to 5x for large national models of the Netherlands and good agreement with serial solutions. PKS makes it possible to efficiently solve high-resolution hydrological problems for decision support.

1. The new Parallel Krylov Solver package
Jarno Verkaik (Deltares)
Joseph Hughes (USGS)
Edwin Sutanudjaja (UU)
Oliver Schmitz (UU)
Paul van Walsum (Alterra WUR)
Raju Ram (TUD)

2. 1 November, 2016
Contents
• Problem statement and solution
• Short overview of (related) developments
• Concept of domain decomposition
• (Preliminary) results
• Practical usage with iMOD
Cartesius: the Dutch supercomputer

3. Problem statement and solution
Problem statement:
• In order to support decision makers in solving hydrological problems,
detailed high resolution models are often needed.
• These models typically consist of a large number of computational cells
and have large memory requirements and long run times.
Solution:
• An efficient technique for obtaining realistic run times and memory
requirements is parallel computing, where the problem is divided over
multiple processor cores.
1 November, 2016

4. Short overview of developments
2010 ----- Deltares develops parallel MT3DMS using Message Passing Interface.
2012 ----- USGS develops parallel U(nstructured)PCG-solver using OpenMP.
2013 ----- Deltares & USGS start working on new Parallel Krylov Solver package
for MODFLOW-2005 based on UPCG (hybrid, combined MPI/OpenMP).
2013 ----- USGS releases MODFLOW-USG (UnStructured Grid) that includes the
PCGU-solver, a derivative solver of UPCG.
2015 ----- Deltares incorporates PKS in MODFLOW-USG. Cases: Indonesia and global.
2016 ----- Deltares incorporates PKS in iMODFLOW, together with Alterra for
MetaSWAP. Main case: Netherlands Hydrological Model.
2017 ----- Deltares releases iMODFLOW with PKS.
2017 ----- Deltares incorporates PKS in iMOD-SEAWAT. Cases: fresh-salt global Deltas.
(2017+---- Deltares & USGS incorporate PKS in new MODFLOW-6.)
1 November, 2016

5. • Distribute the memory over multiple
(connected) processor cores.
• For this, partitionize the MODFLOW grid:
• iMODFLOW: uniform blocks,
Recursive Coordinate Bisection
• MODFLOW-USG: METIS library
Concept of domain decomposition (1/3)
1 November, 2016
Example
MODFLOW-USG
METIS
Example
iMODFLOW
“uniform”
Example
iMODFLOW
RCB

6. • Distribute the linear system Ah = b over
the partitions, where h is the groundwater
head to be solved.
• Connect the partitions tightly through MPI,
using an overlap for exchanging data.
• Solve this system in parallel with exactly
the same accuracy as for the serial case.
• Krylov-Schwarz domain decomposition:
• Restricted Additive Schwarz
parallel preconditioner:
• Applied to CG/GCR Krylov methods
• Inaccurate subdomain solve: ILU only
• Dirichlet transmission condition
Concept of domain decomposition (2/3)
1 November, 2016
www.ddm.org
Example structure additive
Schwarz preconditioner M

7. Concept of domain decomposition (3/3)
1 November, 2016

8. Results iMODFLOW: NHM (1)
1 November, 2016
• Netherlands Hydrological Model for drought simulation
• iMODFLOW-MetaSWAP-TRANSOL-MOZART-DM
parallel PKS serial
• Simulation period: 2006, daily time-step
• MODFLOW: confined, 7 layers, 7x1300x1200 (~6.5M cells)
• MetaSWAP: ~0.5M cells

9. Results iMODFLOW: NHM (2)
1 November, 2016
• Maximum measured speedup ~5.
• Maximum theoretical speedup is
limited by surface water (< 1/0.06  16.7).
• Exactly the same heads are computed
with PKS as for the serial case.
Amdahl’s law

10. Results MODFLOW-USG: global model
1 November, 2016
• PCRGLOB-WB
• Period 1996-jan, transient with daily time-step,
confined, 2 layers, ~4.5M cells, 5 arc-min.
• Run 1: watershed-based input/output
(SIO, 53 watersheds)
• Run 2: Input/output clipped on METIS
partitions (NO_SIO)

11. Results MODFLOW-USG: Indonesia & synthetic
1 November, 2016
• Indonesia model:
steady-state, confined, 1 layer, ~4M cells,
30 arc-sec.
• Synthetic:
steady-state, confined,
heterogeneous conductivity, 2 layers,
10 km x 10 km, ~112M cells (2x7500x7500)
Synthetic Indonesia

12. Practical usage with iMOD (Windows)
Easy-to-use in three steps:
1. Install Microsoft MPI:
https://msdn.microsoft.com/en-us/library/bb524831(v=vs.85).aspx
2. Modify your run-file, Dataset 5 (Solver Configuration)
3. Start your parallel job. E.g. starting from the DOS prompt using 4 cores:
mpiexec -n 4 iMODFLOW.exe imodflow.run
1 November, 2016
Enable PKS
Same options as PCG
Partition method, flag for merging IDF output

13. !!! THANK YOU FOR YOUR ATTENTION !!!
1 November, 2016
…
…
42