This document discusses progress towards developing kilometer-scale Earth system simulations. It outlines a goal of achieving globally uniform 1 km weather and climate modeling by defining fundamental algorithmic building blocks ("Weather and Climate Dwarfs") and exploring hardware adaptation. It also reviews requirements for precision and verification as resolution increases. Achieving 1 km simulations would represent a major advance but require orders of magnitude increases in computational resources.

1. From Weather Dwarfs to Kilometre-Scale
Earth System Simulations
Nils P. Wedi, Peter Bauer, ECMWF colleagues, ESCAPE project partners
European Centre for Medium-Range Weather Forecasts (ECMWF)
PASC2018, Basel

2. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014
Outline
• Numerical weather prediction & climate, a brief (HPC) history
• Ensemble-based assimilation and forecasts of Weather & Climate
• Earth-System complexity and the notion to resolve rather than parametrise
• An intermediate goal: globally uniform weather & climate modelling at 1 km horizontal resolution
– Defining and encapsulating the fundamental algorithmic building blocks ("Weather and Climate Dwarfs")
– Pioneering algorithm development with hardware adaptation using DSL toolchains
– Reviewing the need for precision
– A HPCW benchmark and cross-disciplinary Verification, Validation, and Uncertainty Quantification (VVUQ)
2

3. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014 3
ECMWF’s progress in degrees of freedom
(levels x grid columns x prognostic variables)
(Schulthess et al, 2018)

4. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014
Sustained HPC performance
4

5. October 29, 2014
I/O - Impact of NVRAM on Data Access
Byte Addressable Hypercubes
• Longitude (3600)
• Latitude (1800)
• Atmospheric levels, Physical parameters (~200)
• Time steps (~100)
• Probabilistic pertubations (50)
@ double precision
• 9km 48 TiB
• 5km 192 TiB
• 1.25km 1.82 PiB
Not included: historical observations, multiple models, etc...
5EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
Clients want to do different analytics
across multiple axis
Tiago Quintino
ECMWF archives ~150TB / Day
Growing exponentially …

6. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014
Ensemble of assimilations and forecasts
6

7. Ocean – Land – Atmosphere – Sea ice
7

8. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014 8
Copernicus climate change monitoring service C3S

9. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014 9
Observation impact on reducing the short-range forecast error
Future:
Mobile phones,
cars, homes, ….

10. Atmosphere
Monitoring
CO2 SURFACE FLUXES
Vegetation (CTESSEL)
Fires (GFAS)
Ocean (inventory)
Anthropogenic (inventory)
TRANSPORT (ECMWF)
PBL mixing
Advection
Convection
CHEMISTRY
Oxidation of CO
(not yet represented in model)
CO2 OBSERVATIONS
GOSAT CO2 (IUP- Uni Bremen)
Graphic: A Agusti-Panareda, S Massart (ECMWF)
C h a l l e n g e : M o n i t o r i n g a t m o s p h e r i c C O 2

11. Extreme events: Hurricane IRMA
11
NASA GOES-16
10 day forecast
6 day forecast
High impact disruptive events: Use of high-resolution ensembles
to estimate uncertainty in large-scale (steering) flow.

12. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014
Hurricane IRMA 18km vs 5km ensemble
12
ECMWF Strategy 2025
a 5km ensemble …

13. October 29, 2014
Global KE - Spectra ~500hPa
~9km ~2.5km ~1.25km
Resolve rather than parametrize much of the
crucial vertical transport of momentum and heat

14. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014
TCo1279 9km
NARVAL region

15. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014
TCo7999 1.25km
NARVAL region

16. 16
48h forecast ~9km
48h forecast ~1km
http://gigapan.com/gigapans/206287
Take the “Turing test” of climate & weather modelling (T. Palmer)

17. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014
(Daniel Klocke, DWD)

18. October 29, 2014 PETER BAUER 2018
• 1 km simulation on reduced planet
• No convection parameterization!
• Same microphysics scheme
• Different dynamical cores
• Different physics-dynamics coupling
• Different time stepping
DCMIP16

19. 26%
24%
30%
9%
9%
GP_DYNAMICS SI_SOLVER SP_TRANSFORMS PHYSICS+RAD WAVEMODEL OCEANMODEL
coupled TCo1279 L137 (~9km operational) run
Where do we spent the time ?
Where do we spend the time ?
Single electrical group:
~47minutes wallclock time
(single electrical group==384 nodes)
1408 MPI tasks x 18 threads
306 FC/day

20. October 29, 2014
Ocean resolution requirements
Hallberg 2013
From the presentation of H. Hewitt, at ECMWF (2016)

21. 21EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
AWI FESOM2 team
Animation: Nikolay Koldunov (AWI)
Arctiv Ocean
circulation at ~1km
4 months/day, time step 2 min, 1700 Broadwell
Cores on Cray CS400

22. October 29, 2014 22EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS
AWI FESOM2 team
Animation: Nikolay Koldunov (AWI)
Arctic sea-ice evolution at
~1km

23. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014
Cut-off where power is 50% of k-5/3
IFS KE spectra compared to altimeter observations
Resolved 1 km scales need at least x6 more resolution and x10000s
in compute => intermediate goal 1 km grid
[Courtesy S. Abdalla]

25. October 29, 2014
(hpc-
escape.eu)
Weather & Climate Dwarfs
Extract model dwarfs…
… explore alternative
numerical algorithms
…
… hardware
adaptation …
… reassemble
model and
benchmark

26. Atlas: a library for NWP and climate modelling
26
Deconinck et al. 2017
https://github.com/ecmwf

27. Domain-specific language toolchain
Advection (MPDATA)
Complementary skills of CLAW, GridTools (MeteoSwiss) and Atlas (ECMWF)

28. October 29, 2014
28
Weather & Climate Dwarfs
Comparison of software optimized for GPU
and Xeon processors
Poulsen & Berg (2017)

29. October 29, 2014
Advection

30. 30
Christian Kuehnlein
Choice of dynamical core on the
same grid with the same physics!

31. Bespoke Krylov solvers
31
Parallel restriction, prolongation and Atlas mesh generation
interpolation
restriction
Coarse mesh Coarse mesh Fine mesh
Distributed & Atlas

32. October 29, 2014
0
20
40
60
80
100
120
799 1279 2047 3999 7999
dgemm FLT
Number of floating point operations for direct or inverse spectral
transforms of a single field, scaled by N2log3N
(Wedi et al, 2013)
Fast Legendre Transform
Reduces number of
floating point operations

33. October 29, 2014
33
Alan Gray, Peter Messmer, NVIDIA
Schematic description of the spectral transform method
in the ECMWF IFS model

34. October 29, 2014
34
Hardware co-design
GPU speed-up
Advection
Spectral transforms

35. 16%
45%
34%
GP_DYNAMICS SI_SOLVER SP_TRANSFORMS PHYSICS+RAD
Example: TCo7999 L62 (~1.25km)
Where do we spent the time ?
The cost profile of a 1.25km IFS atmosphere simulation on Piz Daint (CPU only)
4880 MPI tasks x 12 threads
69 FC/day ~ 0.19 SYPD
single precision / FLT
75% comms;
25% compute
~85.21 MWh / SY
Based on the Piz Daint, Swiss
Cray XC50 Haswell, Aries
interconnect, ~5000 nodes total

36. October 29, 2014 October 29, 2014
Simulating performance and scalability of
MPI communications
36
Zheng and Marginaud (GMD, 2018)
Communication time as a function
of halo size, topology & routing algorithm,
compared to spectral transpositions
MPI collectives at least across a subset
of nodes are required in NWP & climate!

37. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014
Communication is bad – small time steps are worse
Same time to solution!
Energy efficiency?
Spectral element model
Time step = 4s
Spectral transform model
Time step = 240s
Spectral element model
Time step = 4s
Data movement x100 (x1000)
more expensive than
computations in time (energy)!
[Shalf et al. 2011]

38. October 29, 2014
Will Deep Learning influence algorithmic choices for weather & climate ?
https://news.developer.nvidia.com/nvswitch-leveraging-nvlink-to-maximum-effect/

39. Co-models (model + radiation)
- Overlapping of compute and communication
- co-execution on different threads
(G. Mozdzynski)

40. 52
96
154
197
70
133
239
380
295
536
10
21
170
323
0
100
200
300
400
500
600
120 240 480 960 1920
FORECASTDAYS/DAY
XC-40 NODES (X36 == CORES)
IFS single precision performance – Atmosphere only (no I/O)
TCo3999 (2.5km) TCo1999 (5km) TCo1279 (9km) TCo7999 (1km) TCo1279_DP (9km)
Singe vs double precision
10 days in 1 hour
137 levels
62 levels
(Vana, Dueben et al 2017)

41. A scale-selective approach: Track of Hurricane Irma
• Spectral models allow to treat different scales at different
precision.
• We can reduce precision when calculating the small scales.
• This is intuitive due to the high inherent uncertainty in small scale
dynamics (parametrisation, viscosity, data-assimilation,...).
Chantry, Düben and Palmer in preparation

42. EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS October 29, 2014
IFS 1km: strong scaling on PizDaint
42
Goal ~1 year / Day
Many thanks to
Thomas Schulthess &
Maria Grazia Giuffreda !

45. starting October 2018

46. October 29, 2014 PETER BAUER 2018
| 2016 | 2017 | 2018 | 2019 | 2020 | 2021 | 2022 |
ESiWACE-2?
EuroHPC
Roadmap for weather & climate computing

47. ExtremeEarth

48. October 29, 2014
ECMWF’s mission strategy
• In collaboration with its member and associated states
– Address the challenges in sustainable computing and data handling
– Move towards Earth-System complexity with rigorous quantification of forecast uncertainty
• High-resolution global ensemble of assimilation and forecasts coupling the ocean, sea-ice, land and
atmosphere
– Supporting Copernicus Services
• Climate monitoring services for the atmosphere, hydrological and carbon cycle
• European Reanalysis (currently producing ERA-5)
• Atmospheric composition monitoring
• Emergency alert system (Floods, Fires, …)
48EUROPEAN CENTRE FOR MEDIUM-RANGE WEATHER FORECASTS

49. Additional slides

50. FETFLAG-01-2018: ‘Energy, Environment and Climate change’:
“Earth, Climate Change and Natural Resources: New technologies and ambitious approaches for high-
precision modelling and simulation, including the necessary data integration, that enable an in-depth
understanding of the earth and climate change, helping in the long run to manage natural resources in
a sustainable way, ensure food security and sustainable farming, and protect natural ecosystems
In 2016, the European Commission (EC) issued a call for ideas for future Flagships, to be funded by the
Future and Emerging Technology (FET) programme.
FET Flagships are:
“… science- and technology-driven, large-scale, multidisciplinary research initiatives built around a
visionary unifying goal ... tackle grand science and technology challenges … strong and broad basis for
future innovation and economic exploitation … novel benefits for society of a potential high impact …
long-term and sustained effort.”
European Commission Flagships

51. 18%
60%
17%
GP_DYNAMICS SI_SOLVER SP_TRANSFORMS PHYSICS+RAD
Example: TCo7999 L62 (~1.25km)
Where do we spent the time ?
The cost profile of a 1.25km (non-hydrostatic) IFS atmosphere simulation Piz Daint
4880 MPI tasks x 12 threads
32 FC/day ~ 0.088 SYPD
single precision / FLT
63% comms;
37% compute
Based on the Piz Daint, Swiss
Cray XC50 Haswell, Aries
interconnect, ~5000 nodes total
~191.74 MWh / SY

52. The IFS model grid
52
N24 reduced Gaussian grid N24 octahedral Gaussian grid
A further ~20% reduction in gridpoints
=> ~50% less points compared to full grid
(Wedi et al, 2014, 2015; Malardel et al, 2015; Deconinck et al, 2017)
Integrated Forecasting System (IFS)