The Exascale Additive Manufacturing (ExaAM) project aims to accelerate the adoption of additive manufacturing by enabling the fabrication of qualifiable metal parts with minimal trial and error. ExaAM will couple high-fidelity sub-grid simulations within a continuum process simulation to determine microstructure and properties at each time-step using local conditions. ExaAM involves multiple computational codes, including ALE3D, Diablo, Truchas, MEUMAPPS, and AMPE, which model different additive manufacturing physics across continuum, meso, and micro scales. The goal is to utilize exascale concurrency and locality to dynamically bridge scales through an adaptive, task-based approach.

1. Overview of the Exascale Additive Manufacturing
Project (ExaAM)
One of 15 Applications in the US DOE Exascale Computing Project
John A. Turner
Oak Ridge National Laboratory
Group Leader: Computational Engineering and Energy Sciences
Chief Computational Scientist: Consortium for Advanced Simulation of Light Water Reactors (CASL)
Principle Investigator: Transforming Additive Manufacturing Through Exascale Simulation (ExaAM)
Numerous others on the ExaAM team (incomplete list):
Jim Belak (co-PI, LLNL), Andy Anderson (LLNL), Suresh Babu (UTK), Mark Berrill (ORNL), Curt Bronkhorst
(LANL), Neil Carlson (LANL), Ondrej Certik (LLNL), Jean-Luc Fattebert (LLNL), Neil Hodge (LLNL), Wayne King
(LLNL), Lyle Levine (NIST), Chris Newman (LANL), B. Radhakrishnan (ORNL), Adrian Sabau (ORNL), Srdjan
Simunovic (ORNL)
HPC User Forum
Santa Fe, NM
17-19 Apr 2017
www.ExascaleProject.org

2. 2 Exascale Computing Project
Outline
• Additive Manufacturing
• Exascale Computing Program
• ExaAM Project

3. 3 Exascale Computing Project
Slide from my presentation at the April 2014 HPC User Forum
Meeting
(also in Santa Fe)

4. 4 Exascale Computing Project
Slide from my presentation at the April 2014 HPC User Forum
Meeting
(also in Santa Fe)
• Computer-Aided Engineering for Batteries
Program (DOE / EERE / VTO)
• Battery Crashworthiness (DOT / NHTSA)
A lot has
happened in
the last three
years.

5. 5 Exascale Computing Project
I assume most are aware of additive manufacturing, a.k.a. 3D
printing, and that it is being used for metal as well as polymers
21.1 g 12.1 g 14.4 g

6. 6 Exascale Computing Project
Test Stand at NASA Marshall Space Flight Center (Huntsville, AL)

7. 7 Exascale Computing Project
Powder Bed Technologies
Design
Material
Feedstock
In-situ
Process
Control
Material
µm-nm
Structure
Static and
Dynamic
Mechanical
Properties
Plasma
(wire)
E-beam
(wire)
Laser
(wire)
Large Melt Pool Technologies
Laser
(powder)
Direct Metal Deposition
Laser
(powder)
E-beam
(powder)
There are multiple metal additive manufacturing technologies
Physical processes are similar
• Energy Deposition
• Melting & Powder Addition
• Evaporation & Condensation
• Heat & Mass Transfer
• Solidification
• Solid-State Phase Transformation
• Repeated Heating and Cooling
• Complex Geometries

8. 8 Exascale Computing Project
Multiple computational challenges must be addressed for AM
• 1 m3 ~ 1012 particles ~ 109 m of “weld” line (assuming 50µm particles) and
build times of hours
• Large temperature gradients, rapid heating and cooling
– necessary / sufficient coupling between thermomechanics and melt/solidification
• Heterogeneous and multi-scale
– resolution of energy sources and effective properties of powder for continuum simulations
• Path optimization
• Large number of parameters and incomplete understanding
– key uncertainties and propagation of those uncertainties
• Validation is difficult as characterization is limited

9. 9 Exascale Computing Project
Overview of electron beam Additive Manufacturing (Arcam®)
http://www.arcam.com/technology/electron-beam-melting/hardware/
3D CAD
Model
Thin 2D
Layers
To
Machine
Nth Layer
PreheatingMelting(N+1)th
layer
Final Part
Conventional raster
melt sequence
Microstructure manipulation of
IN718 via additive
manufacturing is not well
understood. Always results in
columnar grains oriented along
the build direction (001)
• Microstructure plays significant role in determining mechanical
properties of final part
• Directional vs. Isotropic properties
• Feasibility of site specific microstructure control?

10. 10 Exascale Computing Project
Mechanical anisotropy and poor properties are observed in
z-direction Kobryn and Semiatin (2001)
• Anisotropy is a function of material thermal path.
• Thermal path of deposit material is non-uniform
• HIP is not feasible for all additive deposits
• This poses a challenge in part qualification
lacking fundamental understanding of
process-structure-property-performance
relationships
trial and error optimization is incredibly inefficient
P. A. Kobryn and S. L. Semiatin, “The laser additive manufacture of Ti-6Al-4V,”
JOM, vol. 53, no. 9, pp. 40–42, Sep. 2001. doi:10.1007/s11837-001-0068-x.

11. 11 Exascale Computing Project
A more complete understanding of the linkage between
process, structure, properties, and performance is needed
Courtesy of Wayne King, Director of the Accelerated Certification of
Additively Manufactured Metals Initiative at LLNL

12. 12 Exascale Computing Project
What is the Exascale Computing Project (ECP)?
• Created in support of President Obama’s National Strategic
Computing initiative (NSCI)
• A collaborative effort of two US Dept of Energy (DOE) offices:
– Office of Science (DOE-SC)
– National Nuclear Security Administration (NNSA)
• A 10-year project to accelerate the development of a capable
exascale ecosystem
– 50x the performance of today’s 20 PF/s systems
– Operates in a power envelope of 20–30 MW
– Is sufficiently resilient (average fault rate: ≤1/week)
– Includes a software stack that meets the needs of a broad
spectrum of applications and workloads
– Led by DOE laboratories
– Executed in collaboration with academia and industry
A capable exascale
computing system will
have a well-balanced
ecosystem (software,
hardware, applications)

13. 13 Exascale Computing Project
Application Development
Software
Technology
Hardware
Technology
Exascale
Systems
Scalable software
stack
Science and mission
applications
Hardware technology
elements
Integrated exascale
supercomputers
ECP has formulated a holistic approach that uses
co-design and integration to achieve capable exascale
Correctness Visualization Data Analysis
Applications Co-Design
Programming models,
development environment, and
runtimes
Tools
Math libraries and
Frameworks
System Software, resource
management threading,
scheduling, monitoring, and
control
Memory and
Burst buffer
Data
management
I/O and file
system
Node OS, runtimes
Resilience
Workflows
Hardware interface

14. 14 Exascale Computing Project
ExaAM is one of 15 initial ECP application
development projects
Advanced Manufacturing
Gaps and Opportunities
• Improve quality, reliability, and application breadth of
additive manufacturing (AM)
• Accelerate innovation in clean energy manufacturing
institutes (NNMIs)
• Capture emerging manufacturing markets
Simulation Challenge Problems
• Continuum level predictions of non-uniform
microstructure and its relationship to process
parameters
• Predictive mesoscale models for dendritic solidification
scale-bridged to continuum
Prospective Outcomes and Impact
• Routine qualification of AM parts via process-aware
design specs and reproducibility through process control
• Fabrication of metal parts with unique properties such
as light weight strength and failure-proof joints and
welds

15. 15 Exascale Computing Project
Models and Code(s)
• Physical Models: fluid flow, heat transfer,
phase change (melting/solidification and solid-
solid), nucleation, microstructure formation and
evolution, residual stress
• Codes:
• Continuum: ALE3D, Diablo, Truchas
• Mesoscale: AMPE, MEUMAPPS, Tusas
• Motifs: Sparse Linear Algebra, Dense Linear
Algebra, Spectral Methods, Unstructured
Grids, Dynamical Programs, Particles
Transforming Additive Manufacturing through Exascale Simulation
(ExaAM)
PI: John Turner (ORNL), co-PI: Jim Belak (LLNL)
Goal and Approach
• Accelerate the widespread adoption of
additive manufacturing (AM) by enabling
fabrication of qualifiable metal parts with
minimal trial-and-error iteration and
realization of location-specific properties
• Coupling of high-fidelity sub-grid simulations
within a continuum process simulation to
determine microstructure and properties at
each time-step using local conditions
Software and Numerical
Library Dependencies
• C++, Fortran
• MPI, OpenMP, OpenACC, CUDA
• Kokkos, Raja, Charm++
• Hypre, Trilinos, P3DFFT,
SAMRAI, Sundials, Boost
• DTK, netCDF, HDF5, ADIOS,
Metis, Silo
• GitHub, GitLab, CMake, CDash,
Jira, Eclipse ICE
Critical Needs Currently Outside the
Scope of ExaAM
• modeling of powder properties and spreading
• shape and topology optimization
• post-build processing, e.g. hot isostatic
pressing (HIP)
• data analytics and machine learning of
process / build data
• reduced-order models

16. 16 Exascale Computing Project
Additive Manufacturing Physics / Process Workflow

17. 17 Exascale Computing Project
Quick survey of selected ExaAM application codes
(components)

18. 18 Exascale Computing Project
ExaAM codes and attributes
Code(s) Area(s) Physical Models Computational Motifs
Prog.
Lang.
(Model)
Numerical
Library
Depenencies
Proxy
App
Diablo
Process (part
scale),
Performance
solid mechanics, heat &
mass trans, contact,
implicit time integration
Lagrangian FEM, nonlinear
physics, staggered & monolithic
solvers, adaptive h-refinement
Fortran
(MPI)
Hypre, HDF,
Metis, Silo
TBD
Truchas
Process (melt
pool to part
scale)
free-surface flow, heat
transfer, phase change,
species diffusion
FVM, unstructured mesh,
implicit mimetic finite difference,
linear & nonlinear solvers
Fortran
(MPI)
Hypre, HDF5,
netCDF
Pececillo
ALE3D
Process (melt
pool scale),
Properties
implicit and explicit
hydro, heat trans,
phase change
FEM, unstructured mesh,
advection, linear & nonlinear
solvers
C++ (MPI) Hypre LULESH
MEUMAPPS
Microstructure phase-field Fourier spectral method Fortran
(MPI)
P3DFFT N/A
AMPE
Microstructure phase-field implicit FVM, linear & nonlinear
solvers, AMR C++ (MPI)
Hypre,
SAMRAI,
Sundials
AMG2013
Tusas
Microstructure phase-field implicit FEM, preconditioned
JFNK, unstructured 2D and 3D
C++, (MPI,
OpenMP)
Trilinos,
netCDF,
HDF5, Boost
N/A
ContinuumscaleMesoscale

19. 19 Exascale Computing Project
ALE3D (LLNL) has been used to study details of the beam-
powder interaction and melt pool dynamics
Khairallah, S.A., Anderson, A., 2014. Mesoscopic Simulation Model of Selective
Laser Melting of Stainless Steel Powder. Journal of Materials Processing
Technology 214, 2627-2636 DOI:10.1016/j.jmatprotec.2014.06.001.
Laser
Thin Powder Layer
Thick
Powder Layer
Bridge area
a
Yadroitsev, I., Gusarov, A., Yadroitsava, I., Smurov, I., 2010. Single
track formation in selective laser melting of metal powders. Journal
of Materials Processing Technology 210, 1624-1631.
Movie replaced by still
image for distribution

20. 20 Exascale Computing Project
Diablo (LLNL) simulation of residual stress during build
Hodge, N.E., Ferencz, R.M., Vignes, R.M., 2016. Experimental Comparison of Residual Stresses for a Thermomechanical Model for the Simulation of Selective
Laser Melting. Additive Manufacturing DOI. http://dx.doi.org/10.1016/j.addma.2016.05.011.
Movie replaced by still
image for distribution

21. 21 Exascale Computing Project
• J. D. Hunt, “Steady state columnar and equiaxed growth of dendrites and eutectic,” Mater. Sci. Eng., vol. 65, no. 1, pp. 75–83, 1984.
• M. Gäumann, C. Bezençon, P. Canalis, and W. Kurz, “Single-crystal laser deposition of superalloys: Processing-microstructure maps,” Acta
Mater., vol. 49, no. 6, pp. 1051–1062, 2001.
Simulation helped enable local control
of grain structure in AM parts
• Given G and R, can calculate volume fraction of equiaxed grains at any location
• G is temperature gradient,
• R is velocity of liquid-solid interface,
• No is nucleation density,
• Φ is volume fraction of equiaxed grains
(probability of stray grain formation)
• n and a are alloy constants
Lee, Y., Nordin, M., Babu, S. S., & Farson, D. F. (2014). Effect of Fluid
Convection on Dendrite Arm Spacing in Laser Deposition. Metallurgical
and Materials Transactions B, 45(4), 1520-1529.
• Columnar-to-Equiaxed Transition (CET) in rapid
solidification processes primarily controlled by:
– Thermal gradient at the liquid solid interface (G)
– Velocity or growth rate of liquid-solid interface (R)
• Difficult to measure experimentally
– Spatial resolution (microns)
– Temporal resolution required (milliseconds)
– Thermal imaging camera cannot capture 3D data

22. 22 Exascale Computing Project
Truchas provides:
• Thermal gradient at the liquid solid interface
• Velocity of liquid-solid interface
Truchas metal casting code (LANL) can determine G and R
Conventional
Raster Pattern
Spot Melt Pattern along
the contour “DOE”
Temperature gradient and melt pool isotherm
• Dehoff, R. R., Kirka, M. M., Sames, W. J., Bilheux, H., Tremsin, A. S.,
Lowe, L. E., & Babu, S. S. (2015). Site specific control of
crystallographic grain orientation through electron beam additive
manufacturing. Materials Science and Technology, 31(8), 931-938.
• N. Raghavan, R. Dehoff, S. Pannala, S. Simunovic, M. Kirka, J. Turner,
N. Carlson, and S. S. Babu, “Numerical modeling of heat-transfer and
the influence of process parameters on tailoring the grain morphology of
IN718 in electron beam additive manufacturing,” Acta Materialia, vol.
112, pp. 303–314, Jun. 2016. doi:10.1016/j.actamat.2016.03.063.
Movie replaced by still image for distribution

23. 23 Exascale Computing Project
Using MEUMAPPS (ORNL phase field code), nucleation rate has been
identified as the main factor in formation of colony grain structure
Crucial Findings
• Low nucleation rate promotes colony when a new nucleus
sees well developed strain field from a nearby variant
• High nucleation rate promotes basket weave when all nuclei
see complex strain field due to multiple, evolving nuclei
N=0.5 s-1
Colony
structure
N=5.0 s-1
Basket weave
structure
950K
1000K
B. Radhakrishnan, S. Gorti, and S. S. Babu, “Phase Field Simulations of Autocatalytic
Formation of Alpha Lamellar Colonies in Ti-6Al-4V,” Metallurgical and Materials Transactions
A, vol. 47, no. 12, pp. 6577–6592, Dec. 2016. doi:10.1007/s11661-016-3746-6.
Parametric studies performed using
phase field simulations
• Two levels of thermodynamic driving force: low
(1000K) and high: 950K
• Two levels of nucleation rate: low (0.5 s-1) and
high (5 s-1)
Auto-catalytic colony nucleation
Movie replaced by still image for
distribution

24. 24 Exascale Computing Project
Objective: Utilize exascale concurrency and locality to
dynamically bridge continuum and mesoscale physics
• Task-based embedded Scale-Bridging
escapes the traditional synchronous SPMD
paradigm and exploits the heterogeneity
expected in exascale hardware.
• To achieve this, we are developing a UQ-
driven adaptive physics refinement
approach.
• Coarse-scale simulations dynamically
spawn tightly coupled and self-consistent
fine-scale simulations as needed.
• This task-based approach naturally maps to
exascale heterogeneity, concurrency, and
resiliency issues.
24

25. 25 Exascale Computing Project
Ultimately, ExaAM will deliver and deploy a new integrated
simulation environment for AM

26. 26 Exascale Computing Project
Questions?
e-mail: turnerja@ornl.gov
The research and activities described in this presentation were
performed using the resources at Oak Ridge National Laboratory,
which is supported by the Office of Science of the U.S.
Department of Energy under Contract No. DE-AC0500OR22725.
This research was supported by the Exascale Computing
Project (http://www.exascaleproject.org), a joint U.S.
Department of Energy and National Nuclear Security
Administration project responsible for delivering a capable
exascale ecosystem, including software, applications,
hardware, and early testbed platforms, to support the
nation’s exascale computing imperative.