Invited talk at workshop "Exascale Computing in Astrophysics" held in Ascona, Switzerland, 8-13 September 2013. http://www.itp.uzh.ch/exastro2013/Home.html

1. The Einstein Toolkit:
A Community
Computational
Infrastructure for
Relativistic Astrophysics
Gabrielle Allen
Skolkovo Institute of Science & Technology
Moscow, Russia
http://www.einsteintoolkit.org

2. Gravitational Wave Physics
Observations
Models & Simulation
Theory
Scientific Discovery!
Gµν = 8π Tµν
Compact binaries, supernovae
collapse, gamma-ray bursts,
oscillating NSs, gravitational
waves, …

3. Basic Formalism: ADM
1. Choose initial spacelike surface
and provide initial data (3-metric,
extrinsic curvature)
2. Choose coordinates:
 Construct timelike unit normal to
surface, choose lapse function
 Choose time axis at each point on
next surface (shift vector)
1. Evolve 3-metric, extrinsic curvature
Usual numerical methods:
Structured meshes (including multi-patch), finite differences
(finite volumes for matter), adaptive mesh refinement (since
~2003). High order methods.
Some groups use high accuracy spectral methods for vacuum
space times

4. Black Holes & Vacuum Spacetimes
 40+ year effort to model … big
advance in 2005
 Now: Accurate waveforms for arbitrary
masses, spins, momentum
 Numerically generated waveforms
now used with gravitational wave data
analysis, analytic GR (NINJA, NR-AR)
B.Aylott et al 2009 Class
. Quantum Grav. 26 16500
• Opens the door to general
relativistic hydrodynamics

5. New Frontiers: Relativistic Matter
 General relativity
 Nuclear equations of state
 Relativistic
magnetohydrodynamics (GRMHD)
 Radiation Transport
(neutrinos/photons)
 Expensive and complicated!
 Requires opacities/emissivities
 Chemical reactions
(thermonuclear, chemical)
 Computation:
 Multiphysics!!
 GRMHD: petascale problem
 Radiation transport beyond this
 Resolve 102
m to 1010
m, 500 grid
variablesSchnetter et al, PetaScale Computing:
Algorithms and Applications, 2007

6. Software

7. Software: Component Framework
Einstein Toolkit
Cactus Computational Toolkit
Cactus Flesh
(APIs and Definitions)
MPI, Threads, New Programming Models
Driver Thorns (Parallelisation)
Group A
Thorns
Group B
Thorns
CS
CDSE
Computational
Relativists
Domain
Scientists

8. Cactus Framework
Software
Platform

9. Cactus: www.CactusCode.org
 Open source component framework for HPC
 Modular system with high level abstractions
 Components (“thorns”) defined by parameters, variables,
methods
 Cactus “flesh” binds together
 Cactus Computational Toolkit: general thorns
 Different application areas
 Numerical relativity, CFD, coastal science, petroleum,
quantum gravity, cosmology, …

10. Key Features
 Driver thorn provides scheduling, load balancing,
parallelization
 Application thorns deal only with local part of parallel
mesh
 Different thorns can be used to provide the same
functionality, easily swapped.

11. Cactus: 1997-today
 History:
 Black Hole Grand Challenge (‘94-’98): multiple codes,
groups trying to collaborate, tech/social challenges,
NCSA (USA) group moves to AEI (Germany).
 New software needed! Vision …
 Modular for easy code reuse, community sharing and
development of code
 Highly portable and flexible to take advantage of new
architectures and technologies (grid computing,
networks)
 Higher level programming than “MPI”: abstractions
 Emerging: general to support other applications, better
general code, shared infrastructure

12. AMR: Carpet
 Set of Cactus thorns
 Developed by Erik Schnetter
 Berger-Oliger style adaptive
mesh refinement with sub-
cycling in time
 High order differencing (4,6,8)
 Domain decomposition
 Hybrid MPI-OpenMP
 2002-03: Design of Cactus
means many groups, even
competing ones, suddenly
had AMR with little code
change
AEI (Rezzolla,
Kaehler)
E. Schnetter

13. Carpet Scaling
Schnetter, 2013, arXiv:1308.1343

14. Numerical Relativity with Cactus
 1997: 1st
version of Cactus just for relativity (Funding
from MPG/NCSA)
 1999: Cactus 4.0: “Cactus Einstein” thorns
 1999-2002: EU Network “Sources of Gravitational Waves”
 Led to Whisky Code for GR Hydro in Cactus
 Groups develop codes based on Cactus Einstein
 2007: LSU/RIT/PennState/GeorgiaTech: NSF XiRel
 Improve scaling for multiple codes using Cactus
 2009-: LSU/RIT/GeorgiaTech/Caltech/AEI: NSF CIGR
 Shared cyberinfrastructure including matter
 Einstein Toolkit from community contributions
 Sustainable, community supported model
So far …
Over 200
science
papers
Over 30
student
thesis
Around 50
contributors
14

15. Coastal Science: CaFUNWAVE
Ported from a mature community code:
FUNWAVE (phase-resolving, time-stepping
Boussinesq model for ocean surface
wave propagation in the nearshore)
Now uses total variation diminishing method and
shock capturing scheme
J. Chen (Civil Eng), J. Tao (CCT/LSU), S.Brandt
(CCT/LSU), F. Shi (Delaware)
General applications for Cactus

Career implications for staff

Increased funding opportunities

Improved code base and community

More support apparatus needed!

16. Einstein Toolkit
Community

17. Einstein Toolkit
“The Einstein Toolkit Consortium is developing and supporting
open software for relativistic astrophysics. Our aim is to
provide the core computational tools that can enable new
science, broaden our community, facilitate interdisciplinary
research and take advantage of emerging petascale
computers and advanced cyberinfrastructure.”
 Consortium: 94 members, 49 sites, 14 countries
 Sustainable community model:
 9 Maintainers from 6 sites: oversee technical developments,
quality control, verification and validation, distributions
and releases
 Whole consortium engaged in directions, support,
development
 Open development meetings
 Governance model: still being discussed (looking at CIG, iPlant)
http://www.einsteintoolkit.org

18. Einstein Toolkit Members

19. Components
 Currently
 150 Cactus thorns: Initial data, evolution, analysis, AMR, …
 Tools, viz, etc
 Provide extensible standard interface for general relativity
variables (e.g. variables, parameters, data model for output)
 Software contributions from other groups
 More than Cactus
 Examples and tutorials
 Complete production codes for black holes, neutron stars
 New users: Test account on LONI supercomputer (Louisiana)
 Community support
http://www.einsteintoolkit.org

20. Groups Concentrate on New Science
Einstein Toolkit
Cactus Computational Toolkit
Cactus Flesh
(APIs and Definitions)
MPI, Threads, New Programming Models
Driver Thorns (Parallelisation)
Group A
Thorns
Group B
Thorns
CS
CDSE
Computational
Relativists
Domain
Scientists

21. Example: Codes in Analytical Relativity
Collaboration (NRAR)
6 out of 9 codes using
Einstein Toolkit
Hinder et al (2013), arXiv:1307.5307

22. Issues
 Incentives for software sharing and reuse
 Credit for software (and data) as part of e.g.
promotion and tenure
 How to cite use of Einstein Toolkit in publications
and ensure that contributions are cited?
http://www.einsteintoolkit.org 22

23. Education
 Undergraduate Research (LSU
REU in Computational
Science)
 Undergraduates can do do
sophisticated AMR simulations
on parallel machines with
Einstein Toolkit
 Graduate Scientific
Computing Course at LSU
 Einstein Toolkit used in
teaching real world simulation
related scientific computing
concepts
 CS, ECE, Civil Eng,
Geoscience, Mech Eng

24. 3D Full GR Simulations of
Core-Collapse Supernovae
with the Einstein Toolkit
• Multi-Physics, Multi-Scale Simulations on
generalized cubed-sphere grids with AMR.
• Scaling: time-evolving GR, no elliptic PDEs ->
full physics sims scale strongly to O(10000)
cores with hybrid OpenMP/MPI.
• Monte-Carlo Boltzmann solver &
and efficient M1 radiation-hydro scheme
under development.
• MHD implementation (Moesta et al. 2013)
-> study GRB central engines.
Ott et al. 2013, ApJ

25. Jet-Driven Supernovae
with the Einstein Toolkit
• Full GR -> get special relativity
for free.
• GRMHD with full microphysics
(EOS, neutrinos) from
http://stellarcollapse.org
• Full adaptivity; shock/jet tracking
framework.
• Fully reproducible thanks to
open source.
Moesta et al. 2013 in prep.

26. Supermassive Stars and Black Holes
with the Einstein Toolkit
• Making massive seed black
holes from supermassive stars:
Simulation of fragmentation of
differentially rotating polytrope.
• Implementation is full GR:
simulate black hole formation
self-consistently.
• Get gravitational-waves for
“free” from spacetime metric ->
probe formation of the first
black holes.
Reisswig et al. 2013, PRL

27. New: GRHydro ET Thorn
 Base: GRHD public version of Whisky code (EU 5th
Framework)
 Much development plus new MHD
 Caltech, LSU, AEI, GATECH, Perimeter, RIT (NSF CIGR Award)
 Full 3D and dynamic general relativity
 Valencia formalism of GRMHD:
Relativistic magnetized fluids in
ideal MHD limit
 Published text results, convergence
 arXiv: 1304.5544 (Moesta et al, 2013)
 All code, input files etc part of
Einstein Toolkit
 User support

28. Strategy
Exascale

29. Automatic Code Generation
 Einstein equations very complex
 Coding cumbersome, error prone
 Deters experimentation
 Kranc: Mathematica tool to
generate Cactus thorns from PDEs,
specify differencing methods
 Vision: Generate entire codes from
underlying equations/problem
specification, optimize codes for
target architectures
 Revolutionize HPC
 Opportunity to integrate
verification/validation/data
description
Governing
Equations
Numerics
Kranc
Engine
Custom
Code
Resource

30. Chimora
 Use large scale CPU/GPU systems efficiently for complex
applications
 Reduce code rewrite, new programming paradigms
 Strategy uses:
 High level code transformations
 Loop traversal strategies
 Dynamically selected data/instruction cache
 JIT compiler tailored to application
Blazewicz et al (2013),
arXiv:1307.6488
1265449

31. Chimora

32. Software Partnerships
 NSF S2I2 Software Institutes
 Conceptualization awards
for institute level software
development and support
 Software Institute for
Abstractions and
Methodologies for HPC
Simulation Codes on
Future Architectures
(Anshu Dubey)
 High-Performance
Computational Science
with Structured Meshes
and Particles (Phil Colella)
Katz, NSF

33. Some Challenges for
Petascale and Beyond
 New physics: neutrino transport, photon radiation transport
 Massive scalability
 Local metadata, remove global operations
 Extend Cactus abstractions for new programming models
 Robust automatically generated code
 Multithreading, accelerators
 Tools: real time debuggers, profilers, more intelligent
application-specific tools
 Data, visualization, profiling tools, debugging tools, tools to run
codes, archive results, …
 Growing complexity of application, programming models,
architectures.
 Social: how to develop sustainable software for astrophysics?
CDSE and supporting career paths? Edcuation?

34. Conclusions
 Numerical relativity community generally now comfortable with
sharing software
 Didn’t happen overnight
 Some fundamental issues resolved first (BH-BH evolutions)
 Some trade-offs, flexibility/support
 Einstein Toolkit approach
 Mechanism for injecting new science (e.g. GRHydro) and taking full
benefit of new CS opportunities
 Need to focus on implications for young researchers, motivation to
contribute, scientific aims
 Focus on modularity/abstractions reduces dependence on Cactus
 Funding
 Need lightweight governance model to better target funding, help
funding agencies make decisions, enable leveraging international
funding
 Target limited science funding where it will make a difference,
leverage CS funding
 Cactus: broader application base has potential to coordinate with
other disciplines

35. Einstein Toolkit Credits
 Frank Loeffler (Louisiana State University)
 Erik Schnetter (Perimeter Institute)
 Christian Ott (Caltech)
 Ian Hinder (Albert Einstein Institute)
 Roland Haas (Caltech)
 Tanja Bode (Tuebingen)
 Bruno Mundim (Albert Einstein Institute)
 Peter Diener (Louisiana State University)
 Christian Reisswig (Caltech)
 Joshua Faber (RIT)
 Philipp Moesta (Caltech)
 And many others

36. Advertisement
Sunday @ SC13, Denver
http://wssspe.researchcomputing.org.uk