web base testing

1. Cactus and SolvingCactus and Solving
Einstein’s EquationsEinstein’s Equations
Edward WangEdward Wang
2002.04.092002.04.09
Yfwang@mail.ustc.edu.cnYfwang@mail.ustc.edu.cn

2. 2
Episode one:
What is cactus?
http://www.cactus.org

3. 3
Description
Cactus is an open source problem solving environment
designed for scientists and engineers. Its modular
structure easily enables parallel computation across
different architectures and collaborative code
development between different groups. Cactus
originated in the academic research community, where
it was developed and used over many years by a large
international collaboration of physicists and
computational scientists.

4. 4
The name Cactus comes from the design of a central
core (or "flesh") which connects to application modules
(or "thorns") through an extensible interface. Thorns
can implement custom developed scientific or
engineering applications, such as computational fluid
dynamics. Other thorns from a standard
computational toolkit provide a range of computational
capabilities, such as parallel I/O, data distribution, or
checkpointing.

5. 5
Cactus runs on many architectures. Applications,
developed on standard workstations or laptops, can be
seamlessly run on clusters or supercomputers. Cactus
provides easy access to many cutting edge software
technologies being developed in the academic research
community, including the Globus Metacomputing
Toolkit, HDF5 parallel file I/O, the PETSc scientific
library, adaptive mesh refinement, web interfaces, and
advanced visualization tools.

6. 6
PETSc
http://www.globus.org/default.asp
The Globus Project is developing
fundamental technologies needed to
build computational grids.
http://hdf.ncsa.uiuc.edu/HDF5/
HDF5 is a library and file format for storing
scientific data.
http://www-fp.mcs.anl.gov/petsc/
PETSc is a suite of data structures and routines for
the scalable (parallel) solution of scientific
applications modeled by partial differential
equations.

7. 7
Features
Highly Portable
• Supported on most architectures
• Sophisticated make system
Powerful Application Programming Interface
• User modules (thorns) plug-into compact core (flesh)
• Configurable interfaces, schedules and parameters

8. 8
Advanced Computational Toolkit
• Accessible MPI-based parallelism for finite difference grids
• Access to a variety of supercomputing architectures and
clusters
• Several parallel I/O layers
• Fixed and Adaptive mesh refinement under development
• Elliptic solvers ?
• Parallel interpolators and reductions ?
• Metacomputing and distributed computing

9. 9
Collaborative Development
• Interactive monitoring, steering and visualization
• Enables sharing code base.
• TestSuite checking technology
• Visualization tools
Exhaustive Numerical Relativity and
Astrophysical Applications
• Black Hole coalescence
• Neutron star collisions
• Other cataclysms

10. 10
Applications
If your only goal is to implement a simulation of a 1D
wave equation, you probably don't want to use Cactus.
You should consider using Cactus if you need to
perform any of the following tasks - now or in future:
• perform parallel programming in an easy but powerful
manner using the language of your choice: F77, F90, C,
C++.
• run on a wide range of architectures and operating
systems.
• develop your code on the most convenient machine
available, for example your workstation or laptop ...

11. 11
• and simply move your code to a supercomputer (like a Cray
T3E or Origin) to go into production right away!
• engage in high performance cluster computing or
• get first parallel experience by turning your networked PC
pool into a computing cluster.
• work with collaborators on the same code and avoid having
your programs fragmented.
• make use of the latest software technology: e.g. take
advantage of research groups that spend their time
thinking about the fastest way to bring simulation data to
disk or how to visualize it while the code is running.

12. 12
Supported Architectures
Intel IA32 Linux
IBM SP
SGI 32 bit
Intel IA32 Windows NT
Cray T3E
SGI 64 bit
Intel IA32 Windows 2000
Intel IA64 Linux
Compaq Alpha
Sun Sparc
Hitachi SR8000-F1
MacOS X
Fujitsu(Beta 11)

13. 13
Cactus uses the cygwin package to provide a
unix-like environment in windows os.
cvgwin

14. 14
Associated Packages
MPI
LAM Version 6.5.4 and earlier
MPICH Version 1.2.2.1 and earlier
MPICH-G2
HDF5
Version 1.4 and 1.2

15. 15
The Computational Toolkit
Cactus is architectured to consist of modules (we call them
thorns) which plug into a core code (we call it the flesh)
which contains the APIs and infrastructure to glue the
thorns together.
The flesh on it's own doesn't actually do anything. The
thorns contain all the real activity, and can usually be
divided into application thorns (typically written by
scientists, solving problems in physics, astrophysics,
engineering, etc) and infrastructure thorns (typically written
by computational scientists, providing IO, interpolations,
drivers, elliptic solvers etc). We group thorns together into
arrangements, depending on their functionality and
applicability.

16. 16
• Drivers
• Boundary Conditions
• Coordinate Systems
• Output Methods
• Interpolators and Reduction Operators
• Elliptic Solvers
• Utilities

17. 17

18. 18
Live Demos
• WaveToy Demo
Our standard demonstration, see the description page for
an explanation and how to run this yourself.
• Cactus Worm
Our prototype dynamic Grid computing example ... this is
being developed to add new features and fault tolerance ...
please be understanding if it is down!

19. 19
E-Grid and Cactus
The Cactus Team are closely involved with the
activities of the European Grid Forum, in particular
with the Working Group for Testbeds, which is headed
by Ed Seidel. Cactus is being used by members of the
E-Grid, along with the distributed computing toolkit
Globus as an initial test application in a number of
institutions.

20. 20
Through work with the EGrid Consortium, Cactus will benefit
from the development of
• Access to a Europe wide Grid-TestBed, joining computing
resources at over ten institutions.
• More stable and secure methods of streaming data
between simulation and various visualisation clients
• Development of procedures and thorns for automatic
migrating Cactus jobs
• Ability to read streamed data into Cactus, e.g. checkpoint
files
• Ability to stream checkpoint files from Cactus
• Increasing expertise in performing distributed runs

21. 21
E-Grid TestBed
The E-Grid TestBed comprises a varied set of computational
resources which was set up for demonstrations at SC2000
in Dallas, and provides a common and supported grid
infrastructure with each site implementing
• Globus 1.1.4
• gsiftpd
• gsisshd (using port 2222)
• MPICH-G2 (MPICH 1.2.1 with
Globus device)
• HDF5 1.3.30 or above
• GRIS reporting to TestBed GIIS
• Cactus 4.0 Beta 9

22. 22
Episode Two:
Solving Einstein’s Equations
http://www.computer.org/computer
/articles/einstein_1299_1.htm

23. 23
Einstein’s Equations
Globally distributed scientific teams, linked to the most
powerful supercomputers, are running visual
simulations of Einstein’s equations on the gravitational
effects of colliding black holes.

24. 24
In 1916, Albert Einstein published his famous general
theory of relativity, which contains the rules of gravity
and provides the basis for modern theories of
astrophysics and cosmology. It describes phenomena
on all scales in the universe, from compact objects such
as black holes, neutron stars, and supernovae to large-
scale structure formation such as that involved in
creating the distribution of clusters of galaxies. For
many years, physicists, astrophysicists, and
mathematicians have striven to develop techniques for
unlocking the secrets contained in Einstein’s theory of
gravity; more recently, computational-science research
groups have added their expertise to the endeavor.

25. 25
Those who study these objects face a daunting
challenge: The equations are among the most
complicated seen in mathematical physics.
Together, they form a set of 10 coupled,
nonlinear, hyperbolic-elliptic partial differential
equations( 一组十对非线性双曲线－椭圆偏微分方
程） that contain many thousands of terms.
Despite more than 80 years of intense analytical
study, these equations have yielded only a
handful of special solutions relevant for
astrophysics and cosmology, giving only
tantalizing snapshots of the dynamics that occur
in our universe.

26. 26
Scientists have gradually realized that numerical
studies of Einstein’s equations will play an essential
role in uncovering the full picture. Realizing that this
work requires new tools, developers have created
computer programs to investigate gravity, giving birth
to the field of numerical relativity. Progress here has
been initially slow, due to the complexity of the
equations, the lack of computer resources, and the wide
variety of numerical algorithms, computational
techniques, and underlying physics needed to solve
these equations.

27. 27
A realistic 3D simulation based on the full Einstein
equations is an enormous task: A single simulation of
coalescing neutron stars or black holes would require
more than a teraflop per second for reasonable
performance, and a terabyte of memory. Even if the
computers needed to perform such work existed, we
must still increase our understanding of the underlying
physics, mathematics, numerical algorithms, high-
speed networking, and computational science. These
cross-disciplinary requirements ensure that no single
group of researchers has the expertise to solve the full
problem.

28. 28
Further, many traditional relativists trained in more
mathematical aspects of the problem lack expertise in
computational science, as do many astrophysicists
trained in the techniques of numerical relativity. We
need an effective community framework for bringing
together experts from disparate disciplines and
geographically distributed locations that will enable
mathematical relativists, astrophysicists, and computer
scientists to work together on this immensely difficult
problem.

29. 29
Some examples
Figure 1. Gravitational waves from
a full 3D grazing merger of two
black holes. The image shows the
objects immediately after the
coalescence, when the two holes
(seen just inside) have merged to
form a single, larger hole at the
center. The distortions of the
horizon (the Gaussian curvature of
the surface, 高斯曲面 ) appear as a
color map, while the resulting
burst of gravitational waves (the
even-parity polarization or real part
of Ψ4
) appears in red and yellow.

30. 30
Figure 2. Additional
polarization of gravitational
waves (imaginary part of Ψ4
)
from a 3D merging collision of
two black holes. The merged
single black-hole horizon can
just be seen through the cloud
of radiation emitted in the
process.

31. 31
Figure 3. Close-up of the
horizon—with Gaussian
curvature to show the
distorted nature of the
surface—of a black hole
formed by the collision of
two black holes. The two
individual horizon surfaces
can just be seen inside the
larger horizon formed
during the collision
process.

32. 32
Figure 4. Gravitational waves and
horizon of a newly formed black
hole, caused by massive collapse
of a strong gravitational wave on
itself. The dotted surface shows
the horizon, where green colors
indicate high curvature and
yellow means zero curvature. The
highest curvature indicates the
largest gravitational radiation.
The “distortion” (the non-
sphericity) of the black hole
radiates away over time, in
accordance with mathematical
theorems about black holes.

33. 33
Figure 5. Horizon of a
gravitational wave that
implodes to form a black
hole, with leftover waves
escaping, shown at a later
time in the same evolution
presented in Figure 4. We
see that the horizon
curvature is affected by,
and correlated with, the
evolving gravitational
wave.

34. 34
Episode Three:
WaveToy Demo

35. 35
including
• Remote monitoring and steering of an application
from any web browser
• Streaming of isosurfaces( 等值面） from a
simulation, which can then be viewed on a local
machine
• Remote visualization of 2D slices from any grid
function in a simulation as jpegs in a web
browser

36. 36
The application we are using is the simulation of the 3D
scalar field produced by two orbiting sources. The solution
is found by finite differencing a hyperbolic partial
differential equation （双曲线偏微分方程） for the scalar
field. This is a very simple application, however it is
representative of a large class of more complex systems,
including Einstein's Equations, Maxwell's Equations, or the
Navier-Stokes Equations. We use it for demonstrations
since the simulation is not computationally intensive, is
very robust, has simple parameter choices, and has
reasonable graphics.

37. 37
Before you start, make sure you have
the following items
• GetCactus the perl script for easily checking a Cactus
application out from our CVS server.
• WaveDemo.th the ThornList for the demo, this is used to
tell GetCactus which thorns to get.
• WaveDemo.par a parameter file for running the
demonstration.
• IsoView the isosurface visualization client.
• A web browser.
• Note that you'll need a live network connection to checkout
the code, but you can run the demo on a single machine,
the remote tools will look more impressive though if you
use two networked machines, preferably a long way apart.

38. 38
Check out and compile
Checkout the source code using something like
perl GetCactus WaveDemo.th
You should be able to use the default answers for all
the questions
Move into the Cactus directory (cd Cactus), and
compile the application using either
gmake WaveDemo-config <configuration
options>
gmake WaveDemo

39. 39
or if you have a configuration file (this is easiest,
you don't need to remember the options you use)
gmake WaveDemo-config
options=<configuration filename>
gmake WaveDemo
Hopefully that went OK, and you now have an
executable, exe/cactus_WaveDemo. Check it
really worked by running the testsuites, just type
gmake WaveDemo-testsuite
and use the default answers to each question.

40. 40
Run the demo
Move the downloaded demo parameter file into the
Cactus directory. To start the simulation, run
your new executable with the demo parameter
file, if you have a single processor executable
./exe/cactus_WaveDemo WaveDemo.par
If you compiled with MPI and have a multiprocessor
version, you will need to use the appropriate mpi
command for running.

41. 41
• When the simulation starts, you will see output
describing for example the activated thorns and
the scheduling tree.
• If you have the simple visualization client xgraph
installed, you can look at the 1D output

42. 42
Connecting with a web browser
To connect to the simulation, move to another machine if you
have one, and start up a web browser. Connect to
http://<machine name>:5555
where <machine name>:5555 is the name of the machine
where the simulation is running. Note that this information
was part of the standard output when the simulation
started for example
Server started on http://gullveig.aei-
potsdam.mpg.de:5555/
Now you should see a screen with information about the
simulation.

43. 43

44. 44
Viewing IsoSurfaces
Start up the IsoView client, using
IsoView -h <machine name> -dp 5557 -cp 5558
Again, this information can be found in the standard
output, for example
Isosurfacer listening for control connections on
gullveig.aei-potsdam.mpg.de port 5558/
Isosurfacer listening for data connections on
gullveig.aei-potsdam.mpg.de port 5557/

45. 45
You should now see rotating blobs appearing in
the client, looking something like this

46. 46
Steering the Simulation

47. 47
If you are watching the isosurfaces you should
see the blobs move together.

48. 48
Episode Four:
Some Other Sites
NCSA Datalink Article about Cactus from the NCSA Datalink
news letter.
http://archive.ncsa.uiuc.edu/datalink/9911/Cactus1.html
TASC Review Review of Cactus Framework by NASA/TASC
http://sdcd.gsfc.nasa.gov/ESS/esmf_tasc/Files/Cactus_b.ht
ml
IEEE Computing Cover story article about Cactus in IEEE
Computing.
http://www.computer.org/computer/articles/einstein_1299_
1.htm
Albert Einstein Institut: The birth place of the Cactus Code.
http://www.aei-potsdam.mpg.de/

49. 49
Thank you!