This document discusses advances in scalable adaptive mesh refinement (AMR) for extreme scale simulations using Enzo-P/Cello. Enzo-P uses numerical solvers from the Enzo cosmology code within the Cello AMR framework. Cello implements a forest-of-octrees AMR approach and uses Charm++ for parallelization. Weak scaling tests on Blue Waters showed good scaling to 1.7 trillion cells on 262k cores. Future work includes improving gravity solvers and local timestepping to enable trillion particle hydrodynamic cosmology simulations.

1. Advances in Scalable Adaptive
Mesh Refinement (AMR) at
Extreme Scales
Prof. Michael L Norman
Director, San Diego Supercomputer Center
University of California, San Diego
Developer: Dr. James Bordner
Supported by NSF grants SI2-SSE-1440709, PHY-1104819 and AST-0808184.
10/19/2017 M. L. Norman - HPC China 2017, Hefei 1

2. My intrepid partner in this journey
• James Bordner
• PhD CS UIUC, 1999
• C++ programmer
extraordinaire
• Enzo-P/Cello is entirely his
design and implementation
10/19/2017 M. L. Norman - Charm++ Workshop 2017 2

3. Motivation : Extreme Scale Numerical Cosmology
• Dark matter only N-body
simulations have crossed the
1012 particle threshold on the
world’s largest
supercomputers
• Hydrodynamic cosmology
applications are lagging behind
N-body simulations
• This is due to the lack of
extreme scale AMR
frameworks
1 trillion particle dark matter simulation on
IBM BG/Q, Habib et al. (2013)
10/19/2017 M. L. Norman - HPC China 2017, Hefei 3

4. Motivation: Hydrodynamic Cosmological
Simulations of the First Galaxies
10/19/2017 M. L. Norman - HPC China 2017, Hefei 4
Norman & O’Shea, NCSA Blue Waters

5. Enzo: AMR Hydrodynamic Cosmology Code
http://enzo-project.org
• Enzo code under
continuous development
since 1994
– First hydrodynamic
cosmological AMR code
– Hundreds of users
• Rich set of physics solvers
(hydro, N-body, radiation
transport, chemistry,…)
• Have done simulations with
1012 dynamic range and 42
levels
First Stars
First Galaxies Reionization
10/19/2017 M. L. Norman - HPC China 2017, Hefei 5

6. Extreme AMR: 1012 dynamic range from 42
levels of refinement
• Gordon Bell prize
finalist 2001
• Formation of the first
star in the universe
10/19/2017 M. L. Norman - HPC China 2017, Hefei 6
Bryan, Norman & Abel 2001

7. Enzo uses structured AMR (Berger & Colella 1989)
10/19/2017 M. L. Norman - HPC China 2arrier017, Hefei 7
• Unequal patch sizes
• Arbitrary patch locations
• Complex parent-child-sibling relationships
 barriers to scalability
Structured AMR produces
Norman & Bryan 1998

8. time
∆t
∆t/2 ∆t/2
∆t/4 ∆t/4 ∆t/4 ∆t/4
“W cycle”
Serialization over level
updates also limits
scalability and performance
10/19/2017 M. L. Norman - HPC China 2017, Hefei 8

9. Adopted Strategy
• Keep the best part of Enzo (numerical solvers) and
replace the AMR infrastructure
• Implement using modern OOP best practices for
modularity and extensibility
• Use the best available scalable AMR algorithm
• Move from bulk synchronous to data-driven
asynchronous execution model to support patch
adaptive timestepping
• Leverage parallel runtimes that support this execution
model, and have a path to exascale
• Make AMR software library application-independent so
others can use it
10/19/2017 M. L. Norman - HPC China 2017, Hefei 10

10. Software Architecture
10/19/2017 M. L. Norman - HPC China 2017, Hefei 11
Enzo numerical solvers (Enzo-P)
Forest-of-octrees AMR (Cello)
Charm++
Hardware (heterogeneous, hierarchical)

11. (C++)
10/19/2017 M. L. Norman - HPC China 2017, Hefei 13
Object-oriented design implements “separation of concerns”, enhancing extensibility,
maintainability, understandability

12. Forest (=Array) of Octrees
Burstedde, Wilcox, Gattas 2011
10/19/2017 M. L. Norman - HPC China 2017, Hefei 14
2 x 2 x 2 trees 6 x 2 x 2 trees
unrefined
tree
refined
tree

13. p4est weak scaling: mantle convection
Burstedde et al. (2010), Gordon Bell prize finalist paper
10/19/2017 M. L. Norman - HPC China 2017, Hefei 15

14. What makes it so scalable?
Fully distributed data structure; no parent-child
10/19/2017 M. L. Norman - HPC China 2017, Hefei 16
Burstedde, Wilcox, Gattas 2011

15. Charm++
10/19/2017 M. L. Norman - HPC China 2017, Hefei 17
(Laxmikant Kale et al. PPL/UIUC)

16. 10/19/2017 M. L. Norman - HPC China 2017, Hefei 18

17. How does Cello use Charm++ to
implement FOT?
• A forest is array of octrees
of arbitrary size K x L x M
• An octree has leaf nodes
which are blocks (N x N x N)
• Each block is a chare (unit
of sequential work)
• The entire FOT is stored as a
chare array using a bit index
scheme
• Chare arrays are fully
distributed data structures
in Charm++
10/19/2017 M. L. Norman - HPC China 2017, Hefei 19
2 x 2 x 2 tree
N x N x N block

18. 10/19/2017 M. L. Norman - HPC China 2017, Hefei
20
• Each leaf node of the tree is a block
• Each block is a chare
• The forest of trees is represented as a chare array

19. Demonstration of Enzo-P/Cello
10/19/2017 M. L. Norman - HPC China 2017, Hefei 22
PPM gas dynamics

20. Demonstration of Enzo-P/Cello
10/19/2017 M. L. Norman - HPC China 2017, Hefei 24
Quadtree AMR mesh – 5 levels
Effective resolution: 4096 x 2048 mesh
Every square is a 162 block, communicating and executing asynchronously with neighbors

21. WEAK SCALING TEST –
HOW BIG AN AMR MESH CAN WE DO?
10/19/2017 M. L. Norman - HPC China 2017, Hefei 29

22. A 3x5 array of blast waves
10/19/2017 M. L. Norman - HPC China 2017, Hefei 30
Total energy

23. Unit cell: 1 tree per core
201 blocks/tree, 323 cells/block
10/19/2017 M. L. Norman - Charm++ Workshop 2017 32

24. Weak scaling test: Alphabet Soup
array of supersonic blast waves mesh
N trees Np =
cores
Blocks/
Chares
Cells
13 1 201 6.6 M
23 8 1,608
33 27 5,427
43 64 12,864
53 125
63 216
83 512
103 1000 201,000
163 4096
243 13824
323 32768
403 64000 12.9M
483 110592 22.2M 0.7T
543 157464 31.6M 1.0T
643 262144 52.7M 1.7T
10/19/2017 M. L. Norman - Charm++ Workshop 2017 33

25. Largest
AMR
Simulation
in the
world
1.7 trillion
cells
262K cores
on NCSA
Blue Waters
M. L. Norman - Charm++ Workshop 2017 34
html10/19/2017

26. Cello fcns
Enzo-P solver
10/19/2017 M. L. Norman - Charm++ Workshop 2017 35
262k cores

27. Charm++
messaging
bottleneck
10/19/2017 M. L. Norman - Charm++ Workshop 2017 36
p4est

28. Toward Extreme Scale AMR
Hydrodynamic Cosmology
10/19/2017 M. L. Norman - HPC China 2017, Hefei 37
Scalable
hydrodynamics
Scalable N-body
dynamics
Scalable
gravity

29. 10/19/2017 M. L. Norman - HPC China 2017, Hefei 38

30. Demonstration of Enzo-P/Cello
10/19/2017 M. L. Norman - HPC China 2017, Hefei 39
Tracer particles

31. 10/19/2017 M. L. Norman - HPC China 2017, Hefei 40
110k cores

32. 10/19/2017 M. L. Norman - SJTU Intel Briefing 41

33. 10/19/2017 M. L. Norman - HPC China 2017, Hefei 42

34. 10/19/2017 M. L. Norman - SJTU Intel Briefing 43
64k cores

35. 10/19/2017 M. L. Norman - SJTU Intel Briefing 44
64k cores

36. Hydrodynamic Cosmology Scaling Tests
• Uniform grid only
• Weak and strong scaling
• 323 blocks/chare
• 1, 8, 64 chares/core
• 643 to 20483 meshes
• p=8 to 128k cores
10/19/2017 M. L. Norman - HPC China 2017, Hefei 45
Density projection in 5123 simulation

37. Scalable Hydrodynamic Cosmology
10/19/2017 M. L. Norman - SJTU Intel Briefing 46
64k cores

38. Scalable Hydrodynamic Cosmology
10/19/2017 M. L. Norman - SJTU Intel Briefing 47
64k cores
8k cores

39. Toward Extreme Scale AMR
Hydrodynamic Cosmology
10/19/2017 M. L. Norman - HPC China 2017, Hefei 48
Scalable
hydrodynamics
Scalable N-body
dynamics
Scalable
gravity
uniform
grid only

40. Path Forward
• Finish debugging scalable AMR gravity solver
• Implement block-local timestepping
• Experiment with Charm’s built-in DLB schemes
and dynamic execution
• Do a 1 trillion cell/particle hydro cosmology
simulation to demonstrate capability
• Publish!
10/19/2017 M. L. Norman - HPC China 2017, Hefei 57

41. Resources
• Project site: http://www.cello-project.org
• Source code: https://bitbucket.org/cello-project
• Tutorials: on project site
10/19/2017 M. L. Norman - HPC China 2017, Hefei 58

42. 10/19/2017 M. L. Norman - HPC China 2017, Hefei 59