Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DSD-NL 2017 Parallel Krylov Solver Package for iMODFLOW-MetaSWAP - Verkaik

495 views

Published on

Presentatie door Jarna Verkaik (Deltares) voor de iMOD NL Gebruikersdag, tijdens de Deltares Software Dagen- Editie 2017. Woensdag 14 juni 2017, Delft.

Published in: Software
  • Be the first to comment

  • Be the first to like this

DSD-NL 2017 Parallel Krylov Solver Package for iMODFLOW-MetaSWAP - Verkaik

  1. 1. The new Parallel Krylov Solver package for iMODFLOW-MetaSWAP Jarno Verkaik (Deltares) Paul van Walsum (WUR/Alterra) and Joseph Hughes (USGS) Edwin Sutanudjaja (UU) Raju Ram (TUD)
  2. 2. 29 June, 2017 Contents • Problem statement and solution • Context • Mathematical model • Concept of domain decomposition • (Preliminary) results • Practical usage with iMOD SURFSARA-Cartesius: the Dutch supercomputer
  3. 3. Problem statement and solution Problem statement: • In order to support decision makers in solving hydrological problems, detailed high resolution models are often needed. • These models typically consist of a large number of computational cells and have large memory requirements and long run times. Solution: • An efficient technique for obtaining realistic run times and memory requirements is parallel computing, where the problem is divided over multiple processor cores. 29 juni 2017
  4. 4. Context • 2010: start with parallelization of MT3DMS • 2012: first applied to the National Nutrient Model (H2O, vol18) • 2016: start parallelization for the Netherlands Hydrological Model (NHM)  entirely funded by Deltares and Alterra from own investments • 2017: to be released with iMOD • 2017 other: PKS added to iMOD-SEAWAT fresh-salt Measured speedup for Sand Engine Model: ~40 • 2017 other: PKS planned to be released by USGS 29 juni 2017
  5. 5. Netherlands Hydrological Model (NHM) Components: - MODFLOW: 3D Groundwater flow - MetaSWAP: 1D Soil Vegetation Atmosphere Transfer (“SVAT”) - TRANSOL : 1 D model of salinity - MOZART : Surface water Coupling: - ID-based N:1 coupling tables - Seamless MODFLOW-MetaSWAP coupling with groundwater level as shared state variable 29 juni 2017
  6. 6. Mathematical model 29 juni 2017 • Solving the groundwater flow equation requires solving the linear system for heads ℎ: 𝐴ℎ = 𝑞 • Parallel Krylov Solver solves: 𝑀−1 𝐴ℎ = 𝑀−1 𝑞, where 𝑴 is the block-Jacobi preconditioner • E.g. for 2 subdomains: 𝐴11 𝐴12 𝐴21 𝐴22 ℎ1 ℎ2 = 𝑞1 𝑞2 • This linear system is solved by CG • Parallel solution = serial solution partitioning MODFLOW grid 1 3 7 8 92 4 6 10 11 125 3 7 8 9 6 10 11 12 1 3 72 4 6 105 processor 1 processor 2 Message Passing Interface A11 A12 A21 A22
  7. 7. • Distribute the memory over multiple (connected) processor cores. • For this, partition the MODFLOW domain (and hence the MetaSWAP SVATs), using • Uniform blocks, or • Load weighted blocks (Recursive Coordinate Bisection algorithm) Concept of domain decomposition 29 juni 2017 Example “uniform” 128 subdomains Example “RCB” 128 subdomains
  8. 8. Results: NHM (1) 29 juni 2017 • iMODFLOW-MetaSWAP + surface water • Maximum measured speedup ~5 on NHM Deltares server (Windows) • Maximum theoretical speedup is limited by surface water (< 1/0.06  16.7) • Exactly the same heads are computed with PKS as for the serial case Amdahl’s law
  9. 9. Results: NHM (2) 29 juni 2017 • iMODFLOW-MetaSWAP only • SURFSARA-Cartesius Dutch National supercomputer (Linux) • Maximum measured speedup ~24. Further reading: Deltares R&D Highlights 2016 https://www.deltares.nl/app/uploads/2015/02/RD-Highlights-2016_lowres.pdf Page. 64
  10. 10. Results: California model 29 juni 2017 * Simulated on the SURFSARA-Cartesius Dutch National Super Computer # On a single code estimated to consume 12 hours MODFLOW only; 128 core (245 Gb RAM)*; 50 x 50 meter; 23,897 x 27,974 ≈ 335 million active nodes 2,6 Gb file size; 16:34 minutes# Further reading: Vermeulen et. al., Large scale high resolution modeling, MODFLOW and More conference 2017, Golden, USA.
  11. 11. Practical usage with iMOD (1) Easy-to-use in three steps: 1. Install MPICH: www.mpich.org/static/downloads/1.4.1p1/mpich2-1.4.1p1-win-x86-64.msi 2. Modify your run-file*, Dataset 5 (Solver Configuration) 3. Start your parallel job. E.g. starting from the DOS prompt using 4 cores: mpiexec -localonly 4 iMODFLOW.exe imodflow.run 29 juni 2017 Enable PKS with a “minus” Same options as PCG Partitioning method, flag for merging IDF output * The new iMOD project file (.PRJ) will be supported for PKS.
  12. 12. 29 juni 2017 Practical usage with iMOD (2)
  13. 13. Practical usage with iMOD (3) To-do regarding iMODFLOW: • Support packages: MNW, SFR, LAK, and iPEST • Improve line segment input (HFB, ISG) which may slow down speed up • Develop post-processing tools (e.g. IDF merging) To-do regarding MetaSWAP: • Develop post-processing tool for merging bda-files into a single file • Rename output csv-files (leave out p* extensions) • Combine end-state files of partitions into single files for initialization and restart (MetaSWAP, TRANSOL, soil temperature) And: • Overall speed up strongly depends on hardware • To get maximum speed-up for a hi-res model tuning is necessary (e.g. load balancing) • When you want new iMOD functionality, check impact on PKS 29 juni 2017

×