Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
The Einstein Toolkit:
A Community
Computational
Infrastructure for
Relativistic Astrophysics
Gabrielle Allen
Skolkovo Inst...
Gravitational Wave Physics
Observations
Models & Simulation
Theory
Scientific Discovery!
Gµν = 8π Tµν
Compact binaries, su...
Basic Formalism: ADM
1. Choose initial spacelike surface
and provide initial data (3-metric,
extrinsic curvature)
2. Choos...
Black Holes & Vacuum Spacetimes
 40+ year effort to model … big
advance in 2005
 Now: Accurate waveforms for arbitrary
m...
New Frontiers: Relativistic Matter
 General relativity
 Nuclear equations of state
 Relativistic
magnetohydrodynamics (...
Software
Software: Component Framework
Einstein Toolkit
Cactus Computational Toolkit
Cactus Flesh
(APIs and Definitions)
MPI, Threa...
Cactus Framework
Software
Platform
Cactus: www.CactusCode.org
 Open source component framework for HPC
 Modular system with high level abstractions
 Compo...
Key Features
 Driver thorn provides scheduling, load balancing,
parallelization
 Application thorns deal only with local...
Cactus: 1997-today
 History:
 Black Hole Grand Challenge (‘94-’98): multiple codes,
groups trying to collaborate, tech/s...
AMR: Carpet
 Set of Cactus thorns
 Developed by Erik Schnetter
 Berger-Oliger style adaptive
mesh refinement with sub-
...
Carpet Scaling
Schnetter, 2013, arXiv:1308.1343
Numerical Relativity with Cactus
 1997: 1st
version of Cactus just for relativity (Funding
from MPG/NCSA)
 1999: Cactus ...
Coastal Science: CaFUNWAVE
Ported from a mature community code:
FUNWAVE (phase-resolving, time-stepping
Boussinesq model f...
Einstein Toolkit
Community
Einstein Toolkit
“The Einstein Toolkit Consortium is developing and supporting
open software for relativistic astrophysics...
Einstein Toolkit Members
Components
 Currently
 150 Cactus thorns: Initial data, evolution, analysis, AMR, …
 Tools, viz, etc
 Provide extensib...
Groups Concentrate on New Science
Einstein Toolkit
Cactus Computational Toolkit
Cactus Flesh
(APIs and Definitions)
MPI, T...
Example: Codes in Analytical Relativity
Collaboration (NRAR)
6 out of 9 codes using
Einstein Toolkit
Hinder et al (2013), ...
Issues
 Incentives for software sharing and reuse
 Credit for software (and data) as part of e.g.
promotion and tenure
...
Education
 Undergraduate Research (LSU
REU in Computational
Science)
 Undergraduates can do do
sophisticated AMR simulat...
3D Full GR Simulations of
Core-Collapse Supernovae
with the Einstein Toolkit
• Multi-Physics, Multi-Scale Simulations on
g...
Jet-Driven Supernovae
with the Einstein Toolkit
• Full GR -> get special relativity
for free.
• GRMHD with full microphysi...
Supermassive Stars and Black Holes
with the Einstein Toolkit
• Making massive seed black
holes from supermassive stars:
Si...
New: GRHydro ET Thorn
 Base: GRHD public version of Whisky code (EU 5th
Framework)
 Much development plus new MHD
 Calt...
Strategy
Exascale
Automatic Code Generation
 Einstein equations very complex
 Coding cumbersome, error prone
 Deters experimentation
 Kr...
Chimora
 Use large scale CPU/GPU systems efficiently for complex
applications
 Reduce code rewrite, new programming para...
Chimora
Software Partnerships
 NSF S2I2 Software Institutes
 Conceptualization awards
for institute level software
development a...
Some Challenges for
Petascale and Beyond
 New physics: neutrino transport, photon radiation transport
 Massive scalabili...
Conclusions
 Numerical relativity community generally now comfortable with
sharing software
 Didn’t happen overnight
 S...
Einstein Toolkit Credits
 Frank Loeffler (Louisiana State University)
 Erik Schnetter (Perimeter Institute)
 Christian ...
Advertisement
Sunday @ SC13, Denver
http://wssspe.researchcomputing.org.uk
Upcoming SlideShare
Loading in …5
×

The Einstein Toolkit: A Community Computational Infrastructure for Relativistic Astrophysics

894 views

Published on

Invited talk at workshop "Exascale Computing in Astrophysics" held in Ascona, Switzerland, 8-13 September 2013.
http://www.itp.uzh.ch/exastro2013/Home.html

Published in: Technology
  • Be the first to comment

The Einstein Toolkit: A Community Computational Infrastructure for Relativistic Astrophysics

  1. 1. The Einstein Toolkit: A Community Computational Infrastructure for Relativistic Astrophysics Gabrielle Allen Skolkovo Institute of Science & Technology Moscow, Russia http://www.einsteintoolkit.org
  2. 2. Gravitational Wave Physics Observations Models & Simulation Theory Scientific Discovery! Gµν = 8π Tµν Compact binaries, supernovae collapse, gamma-ray bursts, oscillating NSs, gravitational waves, …
  3. 3. Basic Formalism: ADM 1. Choose initial spacelike surface and provide initial data (3-metric, extrinsic curvature) 2. Choose coordinates:  Construct timelike unit normal to surface, choose lapse function  Choose time axis at each point on next surface (shift vector) 1. Evolve 3-metric, extrinsic curvature Usual numerical methods: Structured meshes (including multi-patch), finite differences (finite volumes for matter), adaptive mesh refinement (since ~2003). High order methods. Some groups use high accuracy spectral methods for vacuum space times
  4. 4. Black Holes & Vacuum Spacetimes  40+ year effort to model … big advance in 2005  Now: Accurate waveforms for arbitrary masses, spins, momentum  Numerically generated waveforms now used with gravitational wave data analysis, analytic GR (NINJA, NR-AR) B.Aylott et al 2009 Class . Quantum Grav. 26 16500 • Opens the door to general relativistic hydrodynamics
  5. 5. New Frontiers: Relativistic Matter  General relativity  Nuclear equations of state  Relativistic magnetohydrodynamics (GRMHD)  Radiation Transport (neutrinos/photons)  Expensive and complicated!  Requires opacities/emissivities  Chemical reactions (thermonuclear, chemical)  Computation:  Multiphysics!!  GRMHD: petascale problem  Radiation transport beyond this  Resolve 102 m to 1010 m, 500 grid variablesSchnetter et al, PetaScale Computing: Algorithms and Applications, 2007
  6. 6. Software
  7. 7. Software: Component Framework Einstein Toolkit Cactus Computational Toolkit Cactus Flesh (APIs and Definitions) MPI, Threads, New Programming Models Driver Thorns (Parallelisation) Group A Thorns Group B Thorns CS CDSE Computational Relativists Domain Scientists
  8. 8. Cactus Framework Software Platform
  9. 9. Cactus: www.CactusCode.org  Open source component framework for HPC  Modular system with high level abstractions  Components (“thorns”) defined by parameters, variables, methods  Cactus “flesh” binds together  Cactus Computational Toolkit: general thorns  Different application areas  Numerical relativity, CFD, coastal science, petroleum, quantum gravity, cosmology, …
  10. 10. Key Features  Driver thorn provides scheduling, load balancing, parallelization  Application thorns deal only with local part of parallel mesh  Different thorns can be used to provide the same functionality, easily swapped.
  11. 11. Cactus: 1997-today  History:  Black Hole Grand Challenge (‘94-’98): multiple codes, groups trying to collaborate, tech/social challenges, NCSA (USA) group moves to AEI (Germany).  New software needed! Vision …  Modular for easy code reuse, community sharing and development of code  Highly portable and flexible to take advantage of new architectures and technologies (grid computing, networks)  Higher level programming than “MPI”: abstractions  Emerging: general to support other applications, better general code, shared infrastructure
  12. 12. AMR: Carpet  Set of Cactus thorns  Developed by Erik Schnetter  Berger-Oliger style adaptive mesh refinement with sub- cycling in time  High order differencing (4,6,8)  Domain decomposition  Hybrid MPI-OpenMP  2002-03: Design of Cactus means many groups, even competing ones, suddenly had AMR with little code change AEI (Rezzolla, Kaehler) E. Schnetter
  13. 13. Carpet Scaling Schnetter, 2013, arXiv:1308.1343
  14. 14. Numerical Relativity with Cactus  1997: 1st version of Cactus just for relativity (Funding from MPG/NCSA)  1999: Cactus 4.0: “Cactus Einstein” thorns  1999-2002: EU Network “Sources of Gravitational Waves”  Led to Whisky Code for GR Hydro in Cactus  Groups develop codes based on Cactus Einstein  2007: LSU/RIT/PennState/GeorgiaTech: NSF XiRel  Improve scaling for multiple codes using Cactus  2009-: LSU/RIT/GeorgiaTech/Caltech/AEI: NSF CIGR  Shared cyberinfrastructure including matter  Einstein Toolkit from community contributions  Sustainable, community supported model So far … Over 200 science papers Over 30 student thesis Around 50 contributors 14
  15. 15. Coastal Science: CaFUNWAVE Ported from a mature community code: FUNWAVE (phase-resolving, time-stepping Boussinesq model for ocean surface wave propagation in the nearshore) Now uses total variation diminishing method and shock capturing scheme J. Chen (Civil Eng), J. Tao (CCT/LSU), S.Brandt (CCT/LSU), F. Shi (Delaware) General applications for Cactus  Career implications for staff  Increased funding opportunities  Improved code base and community  More support apparatus needed!
  16. 16. Einstein Toolkit Community
  17. 17. Einstein Toolkit “The Einstein Toolkit Consortium is developing and supporting open software for relativistic astrophysics. Our aim is to provide the core computational tools that can enable new science, broaden our community, facilitate interdisciplinary research and take advantage of emerging petascale computers and advanced cyberinfrastructure.”  Consortium: 94 members, 49 sites, 14 countries  Sustainable community model:  9 Maintainers from 6 sites: oversee technical developments, quality control, verification and validation, distributions and releases  Whole consortium engaged in directions, support, development  Open development meetings  Governance model: still being discussed (looking at CIG, iPlant) http://www.einsteintoolkit.org
  18. 18. Einstein Toolkit Members
  19. 19. Components  Currently  150 Cactus thorns: Initial data, evolution, analysis, AMR, …  Tools, viz, etc  Provide extensible standard interface for general relativity variables (e.g. variables, parameters, data model for output)  Software contributions from other groups  More than Cactus  Examples and tutorials  Complete production codes for black holes, neutron stars  New users: Test account on LONI supercomputer (Louisiana)  Community support http://www.einsteintoolkit.org
  20. 20. Groups Concentrate on New Science Einstein Toolkit Cactus Computational Toolkit Cactus Flesh (APIs and Definitions) MPI, Threads, New Programming Models Driver Thorns (Parallelisation) Group A Thorns Group B Thorns CS CDSE Computational Relativists Domain Scientists
  21. 21. Example: Codes in Analytical Relativity Collaboration (NRAR) 6 out of 9 codes using Einstein Toolkit Hinder et al (2013), arXiv:1307.5307
  22. 22. Issues  Incentives for software sharing and reuse  Credit for software (and data) as part of e.g. promotion and tenure  How to cite use of Einstein Toolkit in publications and ensure that contributions are cited? http://www.einsteintoolkit.org 22
  23. 23. Education  Undergraduate Research (LSU REU in Computational Science)  Undergraduates can do do sophisticated AMR simulations on parallel machines with Einstein Toolkit  Graduate Scientific Computing Course at LSU  Einstein Toolkit used in teaching real world simulation related scientific computing concepts  CS, ECE, Civil Eng, Geoscience, Mech Eng
  24. 24. 3D Full GR Simulations of Core-Collapse Supernovae with the Einstein Toolkit • Multi-Physics, Multi-Scale Simulations on generalized cubed-sphere grids with AMR. • Scaling: time-evolving GR, no elliptic PDEs -> full physics sims scale strongly to O(10000) cores with hybrid OpenMP/MPI. • Monte-Carlo Boltzmann solver & and efficient M1 radiation-hydro scheme under development. • MHD implementation (Moesta et al. 2013) -> study GRB central engines. Ott et al. 2013, ApJ
  25. 25. Jet-Driven Supernovae with the Einstein Toolkit • Full GR -> get special relativity for free. • GRMHD with full microphysics (EOS, neutrinos) from http://stellarcollapse.org • Full adaptivity; shock/jet tracking framework. • Fully reproducible thanks to open source. Moesta et al. 2013 in prep.
  26. 26. Supermassive Stars and Black Holes with the Einstein Toolkit • Making massive seed black holes from supermassive stars: Simulation of fragmentation of differentially rotating polytrope. • Implementation is full GR: simulate black hole formation self-consistently. • Get gravitational-waves for “free” from spacetime metric -> probe formation of the first black holes. Reisswig et al. 2013, PRL
  27. 27. New: GRHydro ET Thorn  Base: GRHD public version of Whisky code (EU 5th Framework)  Much development plus new MHD  Caltech, LSU, AEI, GATECH, Perimeter, RIT (NSF CIGR Award)  Full 3D and dynamic general relativity  Valencia formalism of GRMHD: Relativistic magnetized fluids in ideal MHD limit  Published text results, convergence  arXiv: 1304.5544 (Moesta et al, 2013)  All code, input files etc part of Einstein Toolkit  User support
  28. 28. Strategy Exascale
  29. 29. Automatic Code Generation  Einstein equations very complex  Coding cumbersome, error prone  Deters experimentation  Kranc: Mathematica tool to generate Cactus thorns from PDEs, specify differencing methods  Vision: Generate entire codes from underlying equations/problem specification, optimize codes for target architectures  Revolutionize HPC  Opportunity to integrate verification/validation/data description Governing Equations Numerics Kranc Engine Custom Code Resource
  30. 30. Chimora  Use large scale CPU/GPU systems efficiently for complex applications  Reduce code rewrite, new programming paradigms  Strategy uses:  High level code transformations  Loop traversal strategies  Dynamically selected data/instruction cache  JIT compiler tailored to application Blazewicz et al (2013), arXiv:1307.6488 1265449
  31. 31. Chimora
  32. 32. Software Partnerships  NSF S2I2 Software Institutes  Conceptualization awards for institute level software development and support  Software Institute for Abstractions and Methodologies for HPC Simulation Codes on Future Architectures (Anshu Dubey)  High-Performance Computational Science with Structured Meshes and Particles (Phil Colella) Katz, NSF
  33. 33. Some Challenges for Petascale and Beyond  New physics: neutrino transport, photon radiation transport  Massive scalability  Local metadata, remove global operations  Extend Cactus abstractions for new programming models  Robust automatically generated code  Multithreading, accelerators  Tools: real time debuggers, profilers, more intelligent application-specific tools  Data, visualization, profiling tools, debugging tools, tools to run codes, archive results, …  Growing complexity of application, programming models, architectures.  Social: how to develop sustainable software for astrophysics? CDSE and supporting career paths? Edcuation?
  34. 34. Conclusions  Numerical relativity community generally now comfortable with sharing software  Didn’t happen overnight  Some fundamental issues resolved first (BH-BH evolutions)  Some trade-offs, flexibility/support  Einstein Toolkit approach  Mechanism for injecting new science (e.g. GRHydro) and taking full benefit of new CS opportunities  Need to focus on implications for young researchers, motivation to contribute, scientific aims  Focus on modularity/abstractions reduces dependence on Cactus  Funding  Need lightweight governance model to better target funding, help funding agencies make decisions, enable leveraging international funding  Target limited science funding where it will make a difference, leverage CS funding  Cactus: broader application base has potential to coordinate with other disciplines
  35. 35. Einstein Toolkit Credits  Frank Loeffler (Louisiana State University)  Erik Schnetter (Perimeter Institute)  Christian Ott (Caltech)  Ian Hinder (Albert Einstein Institute)  Roland Haas (Caltech)  Tanja Bode (Tuebingen)  Bruno Mundim (Albert Einstein Institute)  Peter Diener (Louisiana State University)  Christian Reisswig (Caltech)  Joshua Faber (RIT)  Philipp Moesta (Caltech)  And many others
  36. 36. Advertisement Sunday @ SC13, Denver http://wssspe.researchcomputing.org.uk

×