Developing with Cello

Using and Developing Enzo-P/Cello
Scalable Adaptive Mesh Renement
for Astrophysics and Cosmology
PART II: Developing with Cello
NSF SI2-SSE-1440709 PHY-1104819 AST-0808184
James Bordner Michael L. Norman
University of California, San Diego
San Diego Supercomputer Center
Enzo Workshop
28 September1 October 2015
James Bordner, Michael L. Norman Using and Developing Enzo-P/Cello 2015-09-28/10-01 1 / 216

What is the purpose of this tutorial?
Three main goals:
1 Tell you what Enzo-P is (and why it exists)
2 Help you learn to use Enzo-P
download Enzo-P / Cello
congure / compile / run
write parameter les
3 Show you how to develop Enzo-P
Charm++ parallel programming system
Cello software design
add parameters
add methods, renement criteria
add initial and boundary conditions

Modules
1 Charm++ System
2 Software Design
3 Control Flow
4 Developing Enzo-P

Charm++ System
(7) Charm++ System
1 What is Charm++?
2 How is Charm++ used in Cello?
3 Summary

Charm++ System What is Charm++?
(7.1) What is Charm++?
MPI parallel programs: passing messages between processes
P=0 P=1
main()
MPI_Allreduce()MPI_Allreduce()
main()
MPI_Send() MPI_Recv()
data
data
An MPI parallel program
MPI program
decomposed by processes
calls MPI library
communicate / synchronize
MPI runtime system
sends data between processes
synchronizes between processes
Additional features
MPI-2: Parallel I/O, remote
DMA, one-sided communication
MPI-3: fault-tolerance, hybrid
programming, persistence

Charm++ parallel programs: collections of asynchronously-interacting objects
pX()
Main
ChareC
pW()
Main()
ChareA
pZ()
pY()
ChareB
pV()
A Charm++ parallel program
P=0 P=1
main()
MPI_Allreduce()MPI_Allreduce()
main()
MPI_Send() MPI_Recv()
data
data
An MPI parallel program
Charm++ program
Decomposed by objects
Charm++ objects called chares
invoke entry methods
asynchronous
communicate via messages
Charm++ runtime system
maps chares to processors
schedules entry methods
migrates chares to load balance
Additional features
checkpoint/restart
dynamic load balancing
fault-tolerance

C++ program with extra features
Dened using .ci control les:
which classes are chares
which methods are entry methods
declare message objects
declare global readonly variables
compiled using charmc
generates .decl.h and .def.h from .ci
hooks program to Charm++ runtime
.decl.h declarations
.def.h: include denitions
Run executable using charmrun

Chares are concurrent objects
Chares are C++ objects
Inherit from Charm++ base class
#include foo.decl.h
class Foo : public CBase_Foo {
Foo (int n);
void p_receive_data (int n, double * a);
void print_results ();
};
#include foo.def.h
Chares and entry methods declared in .ci control le
module foo {
chare Foo {
entry Foo (int n);
entry void p_receive_data (int n, double a[n]);
};
}

Charm++ collections of chares
Chare Arrays
distributed array of chares
migratable elements
exible indexing
Chare Groups
one chare per processor (non-migratable)
Chare Nodegroups
one chare per node (non-migratable)

Charm++ System How is Charm++ used in Cello?
(7.2) How is Charm++ used in Cello?
Enzo-P begins in the mainchare
single chare on root process
creates a Simulation chare group
Simulation stores global data
one Simulation object per physical process
creates Block chare array
Blocks associated with octree nodes
many Blocks per process
indexed using bit-coding of location

Charm++ System How is Charm++ used in Cello?
(7.2) How is Charm++ used in Cello?
Octree
Forest
Block
x y z
0
31
indices
encoding
level User-dened chare array indices supported
Cello indices for Block arrays:
3 × 10 bits for forest indices
3 × 20 bits for the octree encoding
6 bits for the block level
Up to 1024
3
array of octrees
Up to 20 octree levels
−31 level 31
Block id's use index: e.g. B100:11_1:01

Charm++ System Summary
(7.3) Summary

Software Design
(8) Software Design
1 Design overview
2 Cello software components
3 Enzo-P / Cello classes
4 Simulation classes
5 Problem classes
6 Block and Data classes
7 Field data classes
8 Method classes
9 Initial conditions classes
10 Boundary condition classes
11 Mesh renement criteria classes
12 Summary

Software Design Design overview
(8.1) Design overview
Enzo-P/Cello's design is object-oriented
Package-level design
Enzo-P application implemented using Cello AMR framework
Cello implemented using Charm++ parallel programming system
Component-level design
Cello is decomposed into high-level components
Components are comprised of one or more classes
Source code is organized by package, component, and class
Enzo-P: cello-src/src/Enzo/enzo_class.[ch]pp
Cello: cello-src/src/Cello/component_class.[ch]pp

Software Design Cello software components
(8.2) Design overview
Cello software components
MemoryDisk Parallel
Performance
Error
Monitor
MethodSimulation Problem
IoParametersControl
Mesh Data ParticleField
Portal
CHARM++
Enzo−P

(8.2) Cello software components
Cello software components
Cello's components are loosely grouped into categories
High-level Simulation, Problem, Method
Data structures Mesh, Data, Field, (Particle)
Middle-level Control, Parameters, Io
Hardware-interface Disk, Memory, Parallel
Interface Monitor, (Portal)
Cross-cutting Performance, Error

High-level components
High-level components dene the core classes in Enzo-P/Cello
Simulations dene and manage computational Problems
(Simulation, EnzoSimulation)
A Problem denes the problem to be solved:
numerical methods (EnzoMethodPpm, etc.)
initial conditions (Initial)
boundary conditions (Boundary)
stopping criteria (Stopping)
output (Output)
etc.
A Method implements a numerical method
operates on block Data

Data structure components
Data structure classes dene the AMR hierarchy and its data
Mesh classes represent and operate on an AMR mesh (Hierarchy)
Data classes store data on an AMR block (Data, EnzoData)
Field classes store eld data on a block
FieldData: eld data on a block
FieldDescr: global description of eld data
Particle classes will represent particle data on a block
ParticleData: particle data on a block
ParticleDescr: global description of particle data

Middle-level components
Middle-level component interface high- and low-level classes
Control handles problem evolution
analagous to Enzo's EvolveHierarchy, EvolveLevel
Parameter classes
read parameters from a le (Parameters)
provide access to parameters (Config, EnzoConfig)
Io classes perform parallel IO
OutputData for writing data les
OutputImage for writing image les
Schedule for dening when to write les
IoHierarchy, IoBlock, IoFieldData interface data structure
classes with Disk classes

Hardware-interface components
Hardware-interface components interface to system libraries
Disk classes write meta-data and data to les (FileHdf5)
Memory tracks dynamic memory allocation
Parallel provides helper classes for parallelization
ArrayMap: how to map chare array elements to processes
Sync: simple counter to ease Charm++ synchronization

Interface and Cross-cutting components
Interface components link Enzo-P / Cello with the outside world
Monitor keeps users informed of application status (Monitor)
(Portal) will interface with external users / applications
Cross-cutting components are for globally-accessed classes
Performance measures and reports on application performance
Error provides basic error-handling support

Software Design Enzo-P / Cello classes
(8.3) Enzo-P / Cello classes

Software Design Simulation classes
(8.4) Simulation classes
Simulation classes represents a numerical simulation
Problem A numerical problem
Config Parameter values
Hierarchy Mesh hierarchy
Performance Performance measurements
Monitor Monitor output
state data cycle, time, dt, etc.
Implemented as a chare group
One EnzoSimulation object per process
Stores simulation global data

Software Design Problem classes
(8.5) Problem classes
Problem classes represents a numerical problem
Method Numerical methods
Initial Initial conditions
Boundary Boundary conditions
Refine Mesh renement criteria
Stopping Stopping criteria
Output Disk Output
Prolong Interpolation scheme
Restrict Coarsening scheme
Base class Problem initializes Cello Problem objects
Subclass EnzoProblem initializes Enzo Problem objects

Software Design Block and Data classes
(8.6) Block and Data classes
Block classes represent blocks in the octree forest
Data Block data
Sync Synchronization counters
state data cycle, time, dt, etc.
face data neighbor renement levels
etc.
Implemented as a chare array
One EnzoBlock object per octree forest node
The Data class stores numerical data associated with a Block
FieldBlock Field (array) block data
ParticleBlock Particle block data

Software Design Field data classes
(8.7) Field data classes
Field classes represent eld data on Blocks
FieldBlock
represents state-independent (intrinsic) data
stored in Block Data objects
raw arrays
FieldDescr
represents state-dependent (extrinsic) data
stored in Simulation object
precision, ghost depth, padding, alignment, centering
Field used to simplify using FieldBlock and FieldDescr

Field class API: basic functions
# Return the number of elds
int field_count ();
# Return the integer handle for the named eld.
int field_id (name);
# Return name of the ith eld.
string field_name (id);

Field class API: accessing eld data
# Return the array associated with the specied eld
char * values (id);
char * values (name);
Return dimensions of elds on the data, assuming centered.
void dimensions (id, *mx, *my=0, *mz=0)
# Return size of elds on the data, assuming centered.
void size (*nx, *ny=0, *nz=0)
Accessing eld data is relatively easy

Field class API: eld characteristics
# Ghost depth of each eld
void set_ghost_depth (id, gx, gy=0, gz=0);
void ghost_depth (id, *gx, *gy=0, *gz=0);
# Precision of each eld: single, double, long double
void set_precision (id, precision);
int precision (id);
# Cell centering of each eld: -1, 0, or 1
void set_centering (id, cx, cy=0, cz=0);
void centering (id, *cx, *cy=0, *cz=0);

Field class API: performance
# Byte alignment of eld arrays in memory
void set_alignment (alignment);
int alignment ();
# Byte padding between eld arrays in memory
void set_padding (padding);
int padding ();

Field class API: elds can be associated with groups
# Return the Grouping object for the Fields
Grouping * Field::groups ();
# Add an item to a group. (Cello)
void Grouping::add (item, group);
# Return whether the item is in the given Grouping.
bool Grouping::is_in (item, group)
# Return the number of items in the Grouping.
int Grouping::size (item)
# Return the ith Field in the Grouping.
string Grouping::item (group, i);

Software Design Method classes
(8.8) Method classes
Method classes implement numerical methods on Blocks
compute() Apply the method to a block
name() Return the method name (e.g. ppm)
timestep() Return maximum allowed timestep
Current methods include
EnzoMethodPpm PPM hydrodynamics (Greg Bryan et al)
EnzoMethodPpml PPML ideal MHD (Sergey Ustyugov)
EnzoMethodHeat Forward Euler heat equation
EnzoMethodGravityBiCGStab BiCG-STAB gravity solver (Dan Reynolds)
EnzoMethodGravityCg CG gravity solver
EnzoMethodGravityMg0 MG root-grid gravity solver (incomplete)
EnzoMethodGravityMlat MG gravity solver (incomplete)
EnzoMethodGrackle Grackle chemistry (Britton Smith) (incomplete)
EnzoMethodTurbulence Turbulent mixing (Alexei Kritsuk)

Software Design Initial conditions classes
(8.9) Initial conditions classes
Initial classes implement initial conditions
enforce_block() Apply the initial conditions on a block
Current initial conditions include
InitialValue Initialize from parameter le
EnzoInitialImplosion2 Implosion test problem
EnzoInitialSedovArray2 2D array of Sedov blasts
EnzoInitialSedovArray3 3D array of Sedov blasts
EnzoInitialTurbulence Turbulence initial conditions (Kritusk)
EnzoInitialGrackleTest Grackle test problem (Britton) (incomplete)
InitialFile Initialize from a le (incomplete)

Software Design Boundary condition classes
(8.10) Boundary condition classes
Boundary classes implement boundary conditions
enforce() Apply the boundary conditions on a block
Current boundary conditions include
EnzoBoundary Reecting and outow (Greg Bryan)
BoundaryPeriodic Periodic
BoundaryValue Inow

Software Design Mesh renement criteria classes
(8.11) Mesh renement criteria classes
Rene classes implement renement criteria
apply() Apply the renement criteria on a block
name() Return the name of the criteria (e.g. shock) block
Current rene conditions include
RefineSlope rene on relative slope
RefineDensity rene on density threshold
RefineMask rene according to level array
RefineShear rene on shear
EnzoRefineShock rene on shocks
RefineMass rene on minimum mass (not implemented)

Software Design Summary
(8.12) Summary
Enzo-P / Cello has an object-oriented design
Related classes grouped into components
Classes are kept relatively simple
Field and (soon) Particle classes simplify implementing Methods
Enzo-P is implemented by inheriting from Cello classes
Method Numerical methods
Initial Initial conditions
Boundary Boundary conditions
Refine Renement criteria
.
.
.
.
.
.

Control Flow
(9) Control Flow
1 How are phases of the computation controlled?
2 Adaptive Mesh Renement
3 Refresh ghost zones
4 Summary

Control Flow How are phases of the computation controlled?
(9.1) How are phases of the computation controlled?
Simulation evolution is controlled in control_charm.cpp
Initialize
Adapt Output
Stopping
criteria
Compute
Refresh

Control Flow Adaptive Mesh Renement
(9.2) Adaptive Mesh Renement
Mesh renement proceeds in several steps
1 Apply renement criteria (Refine)
2 Tell neighbor Blocks your desired level
Blocks form a chare array
remote entry method call to neighbor blocks
3 Receive neighbor level
entry method
called by neighbors
4 Update own level if needed (goto 2)
5 Exit after quiescence
no processor is executing an entry point
no messages are awaiting processing
and no messages in-ight

next level
actual
li
k
lj
k
li
k
li
k+1
li
k+1
level
current
actual
next level
desired
Neighbor Blocks
level
current
next level
desired
next level
This Block
lj
k+1
lj
k+1
1
1
2
4
4
temporal level jump criterion
spacial level jump criterion

Control Flow Refresh ghost zones
(9.3) Refresh ghost zones
Neighbor in same renement level
1. copy 2. send
3. copy 1 Face data copied to array
FieldFace object
2 Array sent to neighbor
chare entry method
array sent as message
3 Array copied to ghost zones
Refresh ends when arrays from all neighbors have been received.

Neighbor in coarser renement level
2. send1. restrict
3. copy
1 Face data coarsened to
array
Restrict object
FieldFace array
3 Array copied to ghost zones

Neighbor in ner renement level
1. copy
3. prolong
2. send
1 Face data copied to array
3 Data interpolated to ghost
zones
Prolong object

Control Flow Summary
(9.4) Summary

Developing Enzo-P
(10) Developing Enzo-P
1 Enzo-P develper coding guidelines and suggestions
2 How do I add a new input parameter?
3 How do I add a new method?
4 How do I add initial conditions?
5 How do I add new boundary conditions?
6 How do I add a new renement criterium?
7 How do I add a new unit test program?
8 Summary

Developing Enzo-P Enzo-P develper coding guidelines and suggestions
(10.1) Enzo-P develper coding guidelines and suggestions
Naming things
Naming classes
methods EnzoMethodName
initial conditions EnzoInitialName
boundary conditions EnzoBoundaryName
Naming les
methods enzo_EnzoMethodName.[hc]pp
initial conditions enzo_EnzoInitialName.[hc]pp
boundary conditions enzo_EnzoBoundaryName.[hc]pp

Naming things
Naming class methods
public methods thing_1()
private methods thing_2_()
entry methods p_blah()
reduction entry methods r_reduce()
Naming variables
Array dimensions mx,my,mz
Active region size nx,ny,nz
Ghost zone depth gx,gy,gz
Loop variables ix,iy,iz

Accessing eld data
Accessing eld data is relatively easy
Field field = block-data()-field();
id = field.field_id(density);
field.dimensions(id, mx, my, mz);
field.size (nx, ny, nz);
field.ghost_depth(id gx, gy, gz);
double * d = (double *) field.values(id);
for (int iz=gz; izgz+nz; iz++) {
for (int iy=gy; iygy+ny; iy++) {
for (int ix=gx; ixgx+nx; ix++) {
int i = ix + mx*(iy + my*iz);
d[i] = tiny ;
}
}
}

Developing Enzo-P How do I add a new input parameter?
(10.2) How do I add a new input parameter?
1 Add parameter declaration to [Enzo]Config
Cello parameter: src/Cello/parameters_Config.hpp
Enzo parameter: src/Enzo/enzo_EnzoConfig.hpp
2 Read parameter value in [Enzo]Config
Cello parameter: src/Cello/parameters_Config.cpp
Enzo parameter: src/Enzo/enzo_EnzoConfig.cpp
3 Add Charm++ pack/unpack call to [Enzo]Config::pup()
4 Update documentation
cello-doc/source/parameters-list.rst
5 Access parameter via [Enzo]Config

1. Add parameter declaration to enzo_EnzoCong.hpp
Name parameter according to owning class, e.g. for MethodGravityMg
int method_gravity_mg_iter_max;
2. Read parameter value in enzo_EnzoConfig.cpp
method_gravity_mg_iter_max = p-value_integer
(Method:gravity_mg:iter_max,10);
3. Add Charm++ pack/unpack call to enzo_EnzoConfig.cpp
void EnzoConfig::pup (PUP::er p) {
...
p | method_gravity_cg_iter_max;
...
}

4. Update documentation in enzo/-doc/source/parameters-list.rst
:Parameter: :p:`Method` : :p:`gravity_mg` : :p:`iter_max`
:Summary: :s: `Maximum number of multigrid cycles.`
:Type: :t:`int`
:Default: :d:`10`
:Scope: Enzo
:e:`Maximum number of cycles of the multigrid solver.`

5. Access parameter via EnzoConfig class enzo_config
Method * EnzoProblem::create_method_ ()
{
if (name == ppm) {
...
} else if (name == gravity_mg) {
method = new EnzoMethodGravityMg0
(field_descr, rank,
...
enzo_config-method_gravity_mg_iter_max,
...
);
} ...
}

Developing Enzo-P How do I add a new method?
(10.3) How do I add a new method?
Suppose we wish to add a FE heat equation solver to Enzo-P.
ut − α 2
u = 0
1. Create EnzoMethodHeat class
2. Include enzo_EnzoMethodHeat.hpp le
3. Call EnzoMethodHeat constructor
4. Declare EnzoMethodHeat parameters
5. Read in EnzoMethodHeat parameters
6. Update Charm++ control le enzo.ci
7. Create test_heat.in test problem
8. Run the test and verify test results

1. Create an EnzoMethodHeat class
Create header and source code les
src/Enzo/enzo_EnzoMethodHeat.hpp
src/Enzo/enzo_EnzoMethodHeat.cpp
Implement virtual functions
EnzoMethodHeat::EnzoMethodHeat ()
EnzoMethodHeat::compute ()
EnzoMethodHeat::timestep ()
EnzoMethodHeat::pup ()

Create the src/Enzo/EnzoMethodHeat.hpp header le
class EnzoMethodHeat : public Method {
public: // interface
EnzoMethodHeat ( double alpha , double courant );
virtual void compute ( Block * block ) throw ();
virtual double timestep ( Block * block ) throw ();
EnzoMethodHeat () {};
EnzoMethodHeat ( CkMigrateMessage * m ) {}
PUPable_decl(EnzoMethodHeat);
void pup ( PUP::er p )
{ Method::pup( p ); p | alpha_ ; p | courant_ ; };
protected : // methods
template class T void compute_ ( Block * block , T * Unew );
protected : // attributes
double alpha_ ;
double courant_ ;
};

Implement the EnzoMethodHeat::compute() method
void EnzoMethodHeat::compute ( Block * block) throw()
{
if (block-is_leaf()) {
const int id_temp = field.field_id (temperature);
void * t = field.values (id_temp);
const int p = field.precision (id_temp);
if (p == precision_single) compute_ (block,(float *)t);
else if (p == precision_double) compute_ (block,(double*)t);
else if (p == precision_quadruple) compute_ (block,(long double*) t);
else
ERROR1(EnzoMethodHeat(), precision %d not recognized, p);
}
block-compute_done();
}

Implement the EnzoMethodHeat::compute_() method
template class T
void EnzoMethodHeat::compute_ (T * Unew) const throw()
{
const int id_temp = field.field_id (temperature);
field.dimensions (id_temp,mx,my,mz);
field.size (nx,ny,nz);
field.ghost_depth (id_temp,gx,gy,gz);
...

Implement the EnzoMethodHeat::compute_() method
...
for (int iz=gz; izgz+nz; iz++) {
for (int iy=gy; iygy+ny; iy++) {
for (int ix=gx; ixgx+nx; ix++) {
int i = ix + mx*(iy + my*iz);
double Uxx = dxi*(U[i-idx] - 2*U[i] + U[i+idx]);
double Uyy = dyi*(U[i-idy] - 2*U[i] + U[i+idy]);
double Uzz = dzi*(U[i-idz] - 2*U[i] + U[i+idz]);
Unew[i] = U[i] + alpha_*dt*(Uxx + Uyy + Uzz);
}
}
}
}

2. Include the enzo_EnzoMethodHeat.hpp le
Update src/Enzo/_enzo.hpp
...
#include enzo_EnzoMethodPpm.hpp
#include enzo_EnzoMethodPpml.hpp
#include enzo_EnzoMethodHeat.hpp
#include enzo_EnzoProlong.hpp
...

3. Call the EnzoMethodHeat constructor
Update src/Enzo/enzo_EnzoProblem.cpp
Method * EnzoProblem::create_method_ (...)
{
...
if (type == ppm) {
method = new EnzoMethodPpm;
} else if (type == ppml) {
method = new EnzoMethodPpml;
} else if (type == heat) {
method = new EnzoMethodHeat
(enzo_config-method_heat_alpha,
enzo_config-field_courant);
} else if
...
}

4. Declare any EnzoMethodHeat parameters
Update src/Enzo/enzo_EnzoConfig.hpp
class EnzoConfig : public Config {
...
public: // attributes
...
std::string interpolation_method;
// EnzoMethodHeat
double method_heat_alpha;
...
};

5. Read in the EnzoMethodHeat parameters
Update src/Enzo/enzo_EnzoConfig.cpp
void EnzoConfig::read(...)
{
...
interpolation_method = p-value_string
(Field:interpolation_method,SecondOrderA);
method_heat_alpha = p-value_float
(Method:heat:alpha,1.0);
method_null_dt = p-value_float
(Method:null:dt,std::numeric_limitsdouble::max());
...
}

6. Update the Charm++ control le enzo.ci
Update src/Enzo/enzo.ci
module enzo {
...
PUPable EnzoInitialImplosion2;
PUPable EnzoInitialSedovArray2;
PUPable EnzoMethodPpm;
PUPable EnzoMethodPpml;
PUPable EnzoMethodHeat;
PUPable EnzoProblem;
...
};

7. Create a test_heat.in test problem
Create input/test_heat.in
...
Method {
list = [heat];
heat { alpha = 1.0; }
}
Field {
list = [temperature];
courant = 0.5;
}
Adapt {
list = [slope];
slope {
type = slope;
field_list = [temperature];
}
}

Lets make it unstable and see what happens! (bwahaha)
Field { courant = 1.1; }

Developing Enzo-P How do I add initial conditions?
(10.4) How do I add initial conditions?
Suppose we wish to add an implosion test problem to Enzo-P.
1. Create EnzoInitialImplosion class
2. Include enzo_EnzoInitialImplosion.hpp le
3. Call EnzoInitialImplosion constructor
4. Declare any EnzoInitialImplosion parameters
5. Read in any EnzoInitialImplosion parameters
7. Create test_implosion.in test problem

1. Create an EnzoInitialImplosion class
src/Enzo/enzo_EnzoInitialImplosion.hpp
src/Enzo/enzo_EnzoInitialImplosion.cpp
EnzoInitialImplosion::EnzoInitialImplosion ()
EnzoInitialImplosion::enforce_block ()
EnzoInitialImplosion::pup ()

Create the src/Enzo/EnzoInitialImplosion.hpp header le
class EnzoInitialImplosion : public Initial {
EnzoInitialImplosion ( int * cycle , double * time );
virtual void enforce_block
( Block * block ,
const FieldDescr * field_descr ,
const Hierarchy * hierarchy ) throw ();
EnzoInitialImplosion () {};
EnzoInitialImplosion ( CkMigrateMessage * m ) {}
PUPable_decl(EnzoInitialImplosion);
{ Initial::pup( p ); };
}

Implement the EnzoInitialImplosion::enforce_block() method
void EnzoInitialImplosion::enforce_block
(
Block * block,
const FieldDescr * field_descr,
const Hierarchy * hierarchy
) throw()
{
enzo_float * d = (enzo_float *) field.values(density);
enzo_float * vx = (enzo_float *) field.values(velocity_x);
enzo_float * vy = (enzo_float *) field.values(velocity_y);
enzo_float * te = (enzo_float *) field.values(total_energy);
...

...
// Field attributes
int mx,my,nx,ny,gx,gy;
field.dimensions (0,mx,my);
field.size (nx,ny);
field.ghost_depth (0,gx,gy);
// Cell widths
double xm,ym,xp,yp,hx,hy;
block-data()-lower(xm,ym);
block-data()-upper(xp,yp);
field.cell_width(xm,xp,hx,ym,yp,hy);
...

...
for (int iy=gy; iyny+gy; iy++) {
double y = ym + (iy - gy + 0.5)*hy;
for (int ix=gx; ixnx+gx; ix++) {
double x = xm + (ix - gx + 0.5)*hx;
int i = ix + mx*iy;
if (x + y 0.1517) {
d[i] = 0.125;
te[i] = 0.14 / ((EnzoBlock::Gamma - 1.0) * d[i]);
} else {
d[i] = 1.0;
te[i] = 1.0 / ((EnzoBlock::Gamma - 1.0) * d[i]);
}
vx[i] = vy[i] = 0.0;
}
}
}

2. Include the enzo_EnzoInitialImplosion.hpp le
...
#include enzo_EnzoInitialGrackleTest.hpp
#include enzo_EnzoInitialImplosion.hpp
#include enzo_EnzoInitialSedovArray2.hpp
#include enzo_EnzoInitialSedovArray3.hpp
#include enzo_EnzoInitialTurbulence.hpp
...

3. Call the EnzoInitialImplosion constructor
Initial * EnzoProblem::create_initial_ (...)
{
...
if (type == implosion) {
initial = new EnzoInitialImplosion(cycle,time);
} else if (type == sedov_array_2d) {
initial = new EnzoInitialSedovArray2(enzo_config);
} else if (type == sedov_array_3d) {
initial = new EnzoInitialSedovArray3(enzo_config);
...
}

45. Handle any parameters
(No parameters: see section on adding parameters to Methods)

module enzo {
...
PUPable EnzoInitialImplosion;
PUPable EnzoInitialTurbulence;
...
};

7. Create a test_implosion.in test problem
Create input/test_implosion.in
...
Initial {
type = [implosion];
}
...

Developing Enzo-P How do I add new boundary conditions?
(10.5) How do I add new boundary conditions?
Suppose we wish to add outow boundary conditions to Enzo-P.
1. Create EnzoBoundaryOutflow class
2. Include enzo_EnzoBoundaryOutflow.hpp le
3. Call EnzoBoundaryOutflow constructor
4. Declare any EnzoBoundaryOutflow parameters
5. Read in any EnzoBoundaryOutflow parameters
7. Create test_outflow.in test problem
(Currently implemented in EnzoBoundary)

1. Create an EnzoBoundaryOutflow class
src/Enzo/enzo_EnzoBoundaryOutflow.hpp
src/Enzo/enzo_EnzoBoundaryOutflow.cpp
EnzoBoundaryOutflow::EnzoBoundaryOutflow ()
EnzoBoundaryOutflow::enforce ()
EnzoBoundaryOutflow::pup ()

Create the src/Enzo/EnzoBoundaryOutflow.hpp header le
class EnzoBoundaryOutflow : public Boundary {
EnzoBoundaryOutflow
( axis_enum axis , face_enum face , Mask * mask );
virtual void enforce
( Block * block ,
face_enum face , axis_enum axis ) throw ();
EnzoBoundaryOutflow () {};
EnzoBoundaryOutflow ( CkMigrateMessage * m ) {}
PUPable_decl(EnzoBoundaryOutflow);
{ Boundary::pup ( p ); };
}

Implement the EnzoBoundaryOutflow constructor
EnzoBoundary::EnzoBoundary
( axis_enum axis,
face_enum face,
Mask * mask ) throw()
: Boundary(axis,face,mask)
{ }

Implement the EnzoBoundaryOutflow::enforce() method (1/4)
void EnzoBoundaryOutflow::enforce
( Block * block, face_enum face, axis_enum axis ) throw()
{
// Skip if not applicable
if ( ! applies_(axis, face)) return;
if (face == face_all) {
enforce(block, face_lower,axis);
enforce(block, face_upper,axis);
} else if (axis == axis_all) {
enforce(block, face,axis_x);
enforce(block, face,axis_y);
enforce(block, face,axis_z);
} else {
...

...
Data * data = block-data();
Field field = data-field();
// Field attributes
int nx,ny,nz;
field.size(nx,ny,nz);
// Cell centers if needed for Mask
double *x=0, *y=0, *z=0;
if (mask_) {
x = new double [nx];
y = new double [ny];
z = new double [nz];
data-field_cells(x,y,z);
}
...

...
// Coordinates of Block edges
double xm,ym,zm;
double xp,yp,zp;
data - lower(xm,ym,zm);
data - upper(xp,yp,zp);
double t = block-time();
// @@@ BUG: loops through all elds!
for (int index = 0; index field.field_count(); index++) {
field.dimensions (index,mx,my,mz);
field.ghost_depth (index,gx,gy,gz);
enzo_float * array = (enzo_float *) field.values(index);
...

...
if (face == face_lower axis == axis_x) {
if (nx 1) {
for (iz=0; izmz; iz++) {
for (iy=0; iymy; iy++) {
for (ix=0; ixgx; ix++) {
int i_internal = gx + mx*(iy + my*iz);
int i_external = gx-ix-1 + mx*(iy + my*iz);
if (! mask_ || mask_-evaluate(t,xm,y[iy],z[iz]))
array[i_external] = array[i_internal];
}
}
}
}
...

2. Include the enzo_EnzoBoundaryOutflow.hpp le
...
#include enzo_EnzoBoundaryOutflow.hpp
...

3. Call the EnzoBoundaryOutflow constructor
Boundary * EnzoProblem::create_boundary_ (...)
{
...
if ( type == reflecting) {
boundary = new EnzoBoundary
(axis,face,mask,boundary_type_reflecting);
} else if (type == outflow) {
boundary = new EnzoBoundaryOutflow (axis,face,mask);
} else {
boundary = Problem::create_boundary_
(type,index,config,parameters);
}
...
}

(No parameters: see section on adding parameters to Methods)

module enzo {
...
PUPable EnzoBoundary;
...
};

7-8. Create a test_outflow.in test problem
Create input/test_outflow.in
...
Boundary {
type = [outflow];
}
...

Developing Enzo-P How do I add a new renement criterium?
(10.6) How do I add a new renement criterium?
Suppose we wish to add Jeans length renement criterion to
Enzo-P.
1. Create EnzoRefineJeansLength class
2. Include enzo_EnzoRefineJeansLength.hpp le
3. Call EnzoRefineJeansLength constructor
4. Declare any EnzoRefineJeansLength parameters
5. Read in any EnzoRefineJeansLength parameters
7. Create test_outflow.in test problem

1. Create an EnzoRefineJeansLength class
src/Enzo/enzo_EnzoRefineJeansLength.hpp
src/Enzo/enzo_EnzoRefineJeansLength.cpp
EnzoRefineJeansLength::EnzoRefineJeansLength ()
EnzoRefineJeansLength::enforce ()
EnzoRefineJeansLength::pup ()

Create the src/Enzo/EnzoRefineJeansLength.hpp header le
class EnzoRefineJeansLength : public Refine {
EnzoRefineJeansLength
( axis_enum axis , face_enum face , Mask * mask );
virtual void enforce
( Block * block ,
face_enum face , axis_enum axis ) throw ();
EnzoRefineJeansLength () {};
EnzoRefineJeansLength ( CkMigrateMessage * m ) {}
PUPable_decl(EnzoRefineJeansLength);
{ Refine::pup ( p ); };
}

Implement the EnzoRefineJeansLength constructor
EnzoRefine::EnzoRefine
( axis_enum axis,
face_enum face,
Mask * mask ) throw()
: Refine(axis,face,mask)
{ }

Implement the EnzoRefineJeansLength::enforce() method (1/4)
void EnzoRefineJeansLength::enforce
( Block * block, face_enum face, axis_enum axis ) throw()
{
// Skip if not applicable
if ( ! applies_(axis, face)) return;
if (face == face_all) {
enforce(block, face_lower,axis);
enforce(block, face_upper,axis);
} else if (axis == axis_all) {
enforce(block, face,axis_x);
enforce(block, face,axis_y);
enforce(block, face,axis_z);
} else {
...

...
Data * data = block-data();
Field field = data-field();
// Field attributes
int nx,ny,nz;
field.size(nx,ny,nz);
// Cell centers if needed for Mask
double *x=0, *y=0, *z=0;
if (mask_) {
x = new double [nx];
y = new double [ny];
z = new double [nz];
data-field_cells(x,y,z);
}
...

...
// Coordinates of Block edges
double xm,ym,zm;
double xp,yp,zp;
data - lower(xm,ym,zm);
data - upper(xp,yp,zp);
double t = block-time();
// @@@ BUG: loops through all elds!
for (int index = 0; index field.field_count(); index++) {
field.dimensions (index,mx,my,mz);
field.ghost_depth (index,gx,gy,gz);
enzo_float * array = (enzo_float *) field.values(index);
...

...
if (face == face_lower axis == axis_x) {
if (nx 1) {
for (iz=0; izmz; iz++) {
for (iy=0; iymy; iy++) {
for (ix=0; ixgx; ix++) {
int i_internal = gx + mx*(iy + my*iz);
int i_external = gx-ix-1 + mx*(iy + my*iz);
if (! mask_ || mask_-evaluate(t,xm,y[iy],z[iz]))
array[i_external] = array[i_internal];
}
}
}
}
...

2. Include the enzo_EnzoRefineJeansLength.hpp le
...
#include enzo_EnzoRefineJeansLength.hpp
...

3. Call the EnzoRefineJeansLength constructor
Refine * EnzoProblem::create_refine_ (...)
{
...
if ( type == reflecting) {
refine = new EnzoRefine
(axis,face,mask,refine_type_reflecting);
} else if (type == outflow) {
refine = new EnzoRefineJeansLength (axis,face,mask);
} else {
refine = Problem::create_refine_
(type,index,config,parameters);
}
...
}

Update src/Enzo/enzo_EnzoConfig.hpp
class EnzoConfig : public Config {
...
public: // attributes
...
# This parameter species the number of cells which
# must cover one Jeans length.
int refine_jeans_length_safety_factor;
# If this parameter is greater than zero, it will be
# used in place of the temperature in all cells.
double refine_jeans_length_cold_temperature;
...
};

module enzo {
...
PUPable EnzoRefine;
...
};

7-8. Create a test_outflow.in test problem
Create input/test_outflow.in
...
Refine {
type = [outflow];
}
...

Developing Enzo-P How do I add a new unit test program?
(10.7) How do I add a new unit test program?
Writing Enzo-P/Cello Tests
Test drivers: src/Cello/test_*.cpp
unit_init() Initialize unit testing
unit_class() Declare current class
unit_func() Declare current method
unit_assert() Test a result
unit_finalize() Finalize unit testing

Developing Enzo-P Summary
(10.8) Summary

Developing with Cello

Recommended

Recommended

More Related Content

Similar to Developing with Cello

Similar to Developing with Cello (20)

Recently uploaded

Recently uploaded (20)

Developing with Cello