Presentation by Jarno Verkaik, Deltares, at the iMOD - User Day, during Delft Software Days - Edition 2019. Tuesday, 12 November 2019, Delft.

1. i M O D U s e r D a y 2 0 1 9 – D S D - I N T 2 0 1 9
Parallelization project for the USGS
Jarno Verkaik (Deltares, groundwater management department)
SURFsara Cartesius supercomputer
(47,776 cores, 130TB RAM)

2. Why (distributed memory) parallel computing?
MODFLOW grid
Serial computing
Parallel computing
256 GB RAM
64 GB RAM
64 GB RAM
64 GB RAM
64 GB RAM MPI
iMODUserDay2019–DSD-INT2019
1 day computing
@ 256 GB machine
6 hours computing
@ 64 GB machines
MPI = Message Passing Interface
2

3. Contents
• Organization
• Project results and plans
• Global scale application
iMODUserDay2019–DSD-INT2019
3

4. How it started…
• 2010: Email correspondence on parallel MT3DMS
• 2013: Visit to USGS, start of joined code development (in kind)
• 2015: Start development of Parallel Krylov Solver for
MODFLOW-2005 and MODFLOW-USG
→ poster @ AGU Fall Meeting 2015, San Francisco
• 2016: First application of PKS at national and global scale
→ poster @ AGU Fall Meeting 2016, San Francisco
• Jul.2017: PKS as main feature for iMOD 4.0
& applied as default solver in National Water Model
• Oct.2017: Start parallelization of MODFLOW 6
→ funded by USGS through USGS-Deltares co-op
iMODUserDay2019–DSD-INT2019
4
AGU Fall Meeting 2015
AGU Fall Meeting 2016

5. Organization through (coastal morphology) USGS-Deltares co-op
iMODUserDay2019–DSD-INT2019
5
Robert McCall
Applied Morphodynamics,
Delft
Kees Nederhoff
Deltares USA,
Silver Spring
Martijn Russcher
Numerical Simulation Software,
Delft
Jarno Verkaik
Groundwater management,
Utrecht
Joseph D. Hughes
Integrated Modeling and Prediction,
Reston
Christian D. Langevin
Integrated Modeling and Prediction,
Mounds View
Li Erikson
Pacific Coastal and Marine
Science Center, Santa Cruz

6. USGS project FY2018 (Oct.2017 – Sep.2018)
• Start parallelization of MODFLOW 6
• Such that it can be part of a future release
• Target application: CONUS model by
Wesley Zell and Ward Sanford (USGS)
• USGS requirements:
- Proof of concept applicable to CONUS model
- Low code footprint
- Version controlled code at GitHub
- Easy to use
- Not depending on 3rd party libraries
iMODUserDay2019–DSD-INT2019
6

7. USGS project FY2018 (Oct.2017 – Sep.2018)
• Proof of concept was developed, applicable to CONUS
• Parallelization of IMS linear solver using Schwarz domain decomposition
(similar to Parallel Krylov Solver in iMOD)
• Repos: https://github.com/verkaik/modflow6-parallel.git
→MODFLOW 6 framework refactoring required for
exchanges between models (subdomains):
- That is generic for both serial and parallel computing
- Such that numerical schemes can be evaluated more easily at model interfaces
- Such that XT3D option can be used with multiple models (serial and parallel)
iMODUserDay2019–DSD-INT2019
7
Halo v2. concept

8. USGS project FY2019 & FY2020
• FY2019 (Oct.2018 – Sep.2019)
• Support XT3D option with multi-models (serial only)
• Development of interface model concept (revised halo v2)
• FY2020 (Oct.2019 – Sep.2020)
(To be determined)
• Continue working on parallel MODFLOW
• Development of Basic Model Interface
iMODUserDay2019–DSD-INT2019
8
M1
M2

9. USGS project FY2018 results: circle test 1250M cells
iMODUserDay2019–DSD-INT2019
9

10. USGS project FY2018 results: circle test 12.5M cells
iMODUserDay2019–DSD-INT2019
10

11. Related work to USGS project
iMODUserDay2019–DSD-INT2019
• PhD project (start 2018)
“Towards Exascale Computing for Large Scale Groundwater Simulation”
Goal: development of distributed parallel methods applying to large real-life groundwater models of
O(106)–O(109) cells.
• Mainly funded by Deltares research
• Directly relates to MODFLOW 6 kernel development for new iMOD 6
(see next presentation by Otto de Keizer)
Prof. Marc Bierkens
(Utrecht University)
Prof. Hai Xiang Lin
(Delft University of Technology)
Gualbert Oude Essink, PhD
(Deltares)
11

12. Contributions from PhD project
iMODUserDay2019–DSD-INT2019
Short term coding:
• Improve linear solver convergence when using many subdomains:
→ add coarse grid parallel preconditioner (implementation largely done)
• Option to check parallel implementation
→ add serial block Jacobi preconditioner (first implementation done)
• Code profiling & optimizing parallel performance (ongoing)
Longer term coding:
• Robustness option when using many subdomains:
→ add recovery mechanism for failing hardware
• Add physics-based parallel preconditioner
Short term modeling:
• Run USGS CONUS model in parallel @ 250 m 12

13. • Development of PCR-GLOBWB global groundwater model
having 1km x 1km resolution, O(108) cells
• First experience with parallel MODFLOW 6 with this scale:
• Physics based subdomain partitioning
• Model generation (pre-processing)
• Parallel computing
• Visualization of model results
→ Big data!
Typical raster: 43200 columns x 21600 rows, 3 GB binary
Global groundwater model @ 1km and 428M cells
Ref: Verkaik, J., Sutanudjaja, E.H., Oude Essink, G.H.P., Lin, H.X., and Bierkens, M.F.P., 2019. Parallel global hydrology and water resources
PCR-GLOBWB-MODFLOW model at hyper-resolution scale (1 km): first results, in: EGU General Assembly Conference Abstracts. p. 13397.
iMODUserDay2019–DSD-INT2019
13

14. MODFLOW 6 model characteristics:
• Steady-state, 2 layers, subsurface down-scaled from 10km
• Unstructured DISU grid met only “land cells”, total 428M
• CHD for sea, RIV in layer 1 + DRN in layer 1 & 2 (HydroSHEDS)
Global groundwater model @ 1km and 428M cells
iMODUserDay2019–DSD-INT2019
Parallel pre-processing using 128 subdomains
14

15. Global groundwater model @ 1km and 428M cells
Can we come up with predefined subdomain boundaries (e.g. hydrologically / administrative
boundary) such that they are useful for both modeler and parallel computing?
→ How to partition the world into 1024 subdomains using 1.8M catchments?
→ How to solve a sub-optimal optimization problem (load + edge cuts)?
1. Determine independent regions for groundwater
flow (continents, islands)
→ ~20k regions
2. Further divide large regions/catchments using
a lumped graph method → define parallel models
3. Cluster small regions → define serial models
iMODUserDay2019–DSD-INT2019
15

16. Global groundwater model @ 1km and 428M cells
• Partitioning results in 52 separate MODFLOW 6 models:
• 38 serial, small islands
• 13 parallel, 3 largest on super computer
428M
(2 layers)
Small parallel+
serial models
2. America
120M cells
286 cores
1min 36sec
112GB memory
1. Africa+EurAsia
256M cells
612 cores
3min 31sec
390 GB memory
3. Australia
20M cells
48 cores
33 sec
13 GB memory
5%
28%
60%
iMODUserDay2019–DSD-INT2019
16

17. Global groundwater model @ 1km and 428M cells
iMODUserDay2019–DSD-INT2019
Simulated
Groundwater Table
subdomain
boundary
(total: 1024)
17

18. Global groundwater model @ 1km and 428M cells
iMODUserDay2019–DSD-INT2019
Simulated
Groundwater Table
subdomain
boundary
(total: 1024)
Take home message:
USGS and Deltares are making progress on MPI parallelization
of the MODFLOW 6 multi-model capability
for reducing computing times & memory usage
THANK YOU! 18