This document summarizes a presentation given at the 7th Annual International Conference on Numerical Modeling of Space Plasma Flows titled "A Solution Accurate, Efficient and Stable 3D Unsplit Staggered Mesh (USM) MHD Solver in FLASH". The presentation introduces a 3D USM MHD solver developed for the FLASH code that uses an unsplit Corner Transport Upwind (CTU) algorithm. This algorithm solves 6 or 12 Riemann problems per cell per time step, allowing for CFL numbers up to 1 while maintaining accuracy and stability. The solver uses high-order spatial reconstruction and temporal evolution to calculate Riemann states at interfaces before solving the Riemann problems.

1. Astronum
2012
The
7th
Annual
Interna4onal
Conference
On
Numerical
Modeling
of
Space
Plasma
Flows,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
A Solution Accurate, Efficient and Stable
3D Unsplit Staggered Mesh (USM) MHD
Solver in FLASH
Dongwook Lee
University of Chicago
The
Flash
Center
for
Computa2onal
Science

2. Two
USM
papers
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
2D
paper,
JCP,
2009
3D
paper,
submi.ed
to
JCP,
2012

3. Astrophysical
Applica2ons

4. HEDP
Applica2ons

5. Outline
q Part
1:
q Reduced/Full
corner-­‐transport-­‐upwind
(CTU)
for
3D
q 6
Riemann
solves
for
USM;
3
for
UHD
in
3D
q CFL
stability
limit
reaches
1
in
the
full
CTU
algorithm
q Part
2:
q Third
order
electric
fields
constrained-­‐transport
(CT)
scheme
for
USM-­‐
MHD
(Lee
and
Deane,
JCP,
2009)
q A
new
upwind
biased
CT
(Lee,
JCP,
2012,
under
review)
q Summary
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012

6. Part
1
3D
Unsplit
Algorithms
for
USM
&
UHD:
Reduced
&
Full
CTU
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012

7. MHD
Governing
Equa2ons
q MHD
system
of
equa4ons:
q This
can
be
wri[en
in
a
simple
matrix
form:
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
∂U
∂t
+
∂F
∂x
+
∂G
∂y
+
∂H
∂z
= 0

8. q A
primi4ve
form:
where
the
coefficient
matrix
is
Linearized
System
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012

9. High-­‐Order
Corner
Transport
Upwind
(CTU)
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
q Use
a
linearized
system
in
primi4ve
form
q High-­‐order
spa4al
reconstruc4on
&
temporal
evolu4on
to
obtain
Riemann
states
at
interfaces
(e.g.,
MH)
q Solve
Riemann
problems
using
the
Riemann
states
q x-­‐direc4on:
q y-­‐direc4on:
q z-­‐direc4on:

10. High-­‐Order
Corner
Transport
Upwind
(CTU)
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
q Use
a
linearized
system
in
primi4ve
form
q High-­‐order
spa4al
reconstruc4on
&
temporal
evolu4on
to
obtain
Riemann
states
at
interfaces
(e.g.,
MH)
q Solve
Riemann
problems
using
the
Riemann
states
q x-­‐direc4on:
q y-­‐direc4on:
q z-­‐direc4on:
Normal
predictor
Transverse
corrector

11. Usual
CTU
Algorithms
take…
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012

12. Usual
CTU
Algorithms
take…
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012

13. Usual
CTU
Algorithms
take…
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
q Solve
1D
high-­‐order
reconstruc4ons
using
characteris4c
tracing
in
each
normal
direc4on
to
get
Riemann
states

14. Usual
CTU
Algorithms
take…
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012

15. Usual
CTU
Algorithms
take…
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
F∗
Vi, j,E
n+1/2
,Vi+1, j,W
n+1/2
( )G∗
Vi, j,N
n+1/2
,Vi, j+1,S
n+1/2
( )

16. Usual
CTU
Algorithms
take…
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
F∗
Vi, j,E
n+1/2
,Vi+1, j,W
n+1/2
( )G∗
Vi, j,N
n+1/2
,Vi, j+1,S
n+1/2
( )
+
Trans
Fluxes
in
y
&
z
+
Trans
Fluxes
in
x
&
y
+
Trans
Fluxes
in
x
&
z

17. This
Leads
To
6-­‐CTU
&
12-­‐CTU
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
q Depends
on
the
number
of
Riemann
problems
per
cell
per
4me
step:
q 6-­‐CTU
(6
Riemann
problems)
q Colella,
JCP,
1990;
Gardiner
&
Stone,
JCP,
2008
q A
simple
direct
extension
of
2D
CTU
to
3D
accoun4ng
for
a
single
intermediate
state
n+1/2
q CFL
<
½
q 12-­‐CTU
(12
Riemann
problems)
q Saltzman,
JCP,
1992;
Gardiner
&
Stone,
JCP,
2008;
Minia4
&
Mar4n,
ApJS,
2011
q More
expensive
approach
q Accoun4ng
for
two
intermediate
states
of
n+1/3
&
n+1/2
q CFL
<
1
q
(Gardiner
&
Stone,
JCP,
2008)
CPU12−ctu
CPU6−ctu
≈1

18. Different
Treatment
for
Transverse
Fluxes
in
USM
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
Normal
predictor
Transverse
corrector
q Use
characteris4c
tracing
in
BOTH
normal
and
transverse
flux
calcula4ons
(Lee
&
Deane,
JCP,
2009)

19. Different
Treatment
for
Transverse
Fluxes
in
USM
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
Normal
predictor
Transverse
corrector
q Use
characteris4c
tracing
in
BOTH
normal
and
transverse
flux
calcula4ons
(Lee
&
Deane,
JCP,
2009)
!
Normal
predictor

20. Different
Treatment
for
Transverse
Fluxes
in
USM
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
Normal
predictor
Transverse
corrector
q Use
characteris4c
tracing
in
BOTH
normal
and
transverse
flux
calcula4ons
(Lee
&
Deane,
JCP,
2009)
!
Transverse
corrector
!
Normal
predictor

21. Different
Treatment
for
Transverse
Fluxes
in
USM
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
Normal
predictor
Transverse
corrector
q Use
characteris4c
tracing
in
BOTH
normal
and
transverse
flux
calcula4ons
(Lee
&
Deane,
JCP,
2009)
!
Transverse
corrector
!
Normal
predictor

22. Different
Treatment
for
Transverse
Fluxes
in
USM
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
Normal
predictor
Transverse
corrector
q Use
characteris4c
tracing
in
BOTH
normal
and
transverse
flux
calcula4ons
(Lee
&
Deane,
JCP,
2009)
!
Transverse
corrector
!
Normal
predictor

23. Different
Treatment
for
Transverse
Fluxes
in
USM
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
Normal
predictor
Transverse
corrector
q Use
characteris4c
tracing
in
BOTH
normal
and
transverse
flux
calcula4ons
(Lee
&
Deane,
JCP,
2009)
!
Transverse
corrector
!
Normal
predictor
!
Monotonicity
!
Stability

24. Characteris2c
tracing
for
Transverse
corrector
q A
jump
rela4onship:
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
AyVl +
m=1
m0−1
Σλm
rm
Δ
~
α = AyVr −
m=m0
7
Σλm
rm
Δ
~
α

25. Characteris2c
tracing
for
Transverse
corrector
q A
jump
rela4onship:
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
m=1
7
Σλm
rm
Δ
~
α = AyVr − AyVl

26. m=1
7
Σλm
rm
Δ
~
α = AyVr − AyVl
= AyΔ
= Gr −Gl
= Gi+1/2, j −Gi−1/2, j
Characteris2c
tracing
for
Transverse
corrector
q The
summa4on
of
all
waves
becomes
an
upwind
transverse
flux
gradient:
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012

27. m=1
7
Σλm
rm
Δ
~
α = AyVr − AyVl
= AyΔ
= Gr −Gl
= Gi+1/2, j −Gi−1/2, j
Characteris2c
tracing
for
Transverse
corrector
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
q The
summa4on
of
all
waves
becomes
an
upwind
transverse
flux
gradient:

28. Different
Treatment
for
Transverse
Fluxes
in
USM
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
!
Transverse
corrector
!
Normal
predictor

29. Single-­‐Step
Data
Reconstruc2on-­‐Evolu2on
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012

30. Reduced
&
Full
CTU
in
UHD
q New
approach
of
using
characteris4c
tracing
for
BOTH
normal
predictor
and
transverse
corrector
q Reduced
3D
CTU
q A
direct
extension
of
2D
CTU
to
3D
q Requires
3
Riemann
solves
for
3D
(6-­‐CTU
needs
6
Riemann
solves)
q Only
including
second
cross
deriva4ves
q CFL
limit
~
0.5
q Full
3D
CTU
q Full
considera4ons
of
accoun4ng
for
third
cross
deriva4ves
q Requires
3
Riemann
solves
for
3D
(12-­‐CTU
needs
12
Riemann
solves)
q CFL
limit
~
1.0
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
∂3
/∂x∂y∂z

31. Reduced
&
Full
CTU
in
USM
q New
approach
of
using
characteris4c
tracing
for
BOTH
normal
predictor
and
transverse
corrector
q Reduced
3D
CTU
q A
direct
extension
of
2D
CTU
to
3D
q Requires
6
Riemann
solves
for
3D
(6-­‐CTU
needs
6
Riemann
solves)
q Only
including
second
cross
deriva4ves
q CFL
limit
~
0.5
q Full
3D
CTU
q Full
considera4ons
of
accoun4ng
for
third
cross
deriva4ves
q Requires
6
Riemann
solves
for
3D
(12-­‐CTU
needs
12
Riemann
solves)
q CFL
limit
~
1.0
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
∂3
/∂x∂y∂z

32. Performance
of
Full-­‐CTU
q Verifica4on
tests
for
the
reduced/full
3D
CTU
schemes:
q CFL=0.95
for
all
3D
simula4ons
using
the
full
CTU
scheme
q CFL=0.475
for
the
reduced
CTU
scheme
q They
both
converge
in
2nd
order
q 20%
performance
gain
in
using
the
full
CTU
scheme:
CPUF−ctu
CPUR−ctu
≈ 0.8
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
cf.
CPU12−ctu
CPU6−ctu
≈1
#
$
%
&
'
(

33. 2nd
order
Convergence
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012

34. 3D
Orszag-­‐Tang
(PPM+MC+Roe+cfl
0.95)
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012

35. Cloud-­‐Shock
(WENO5+VL+Roe+cfl
0.95)
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012

36. Part
2
Divergence-­‐Free
fields:
Constrained
Transport
(CT)
MHD
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012

37. Constrained-­‐Transport
by
Balsara
&
Spicer
q CT
scheme
by
Balsara
and
Spicer,
JCP,
1999:
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012

38. Lack
of
Upwind
Considera2on
in
CT
q CT
scheme
by
Balsara
and
Spicer,
1998:
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
q Consider
u>0,
v"0:

39. Lack
of
Upwind
Considera2on
in
CT
q CT
scheme
by
Balsara
and
Spicer,
1998:
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
q Consider
u>0,
v"0:
q
is
the
only
E
term
in
the
upwind
direc4on!

40. Lack
of
Upwind
Considera2on
in
CT
q CT
scheme
by
Balsara
and
Spicer,
1998:
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
q Consider
u>0,
v"0:
q
is
the
only
E
term
in
the
upwind
direc4on!
q Numerical
Oscilla4ons!
q This
is
true
in
most
CT
schemes!

41. Standard-­‐MEC
q 3rd
order
modified
electric
field
construc4on
(standard-­‐MEC),
Lee
&
Deane,
JCP,
2009:
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012

42. New
Upwind-­‐MEC
q New
upwind
biased
modified
electric
field
construc4on(upwind-­‐MEC),
Lee,
JCP,
submi[ed,
2012
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012

43. New
Upwind-­‐MEC
q New
upwind
biased
modified
electric
field
construc4on(upwind-­‐MEC),
Lee,
JCP,
submi[ed,
2012.
Considering
u>0,
v"0:
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012

44. New
Upwind-­‐MEC
q New
upwind
biased
modified
electric
field
construc4on(upwind-­‐MEC),
Lee,
JCP,
submi[ed,
2012.
Considering
u>0,
v"0:
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012

45. New
Upwind-­‐MEC
q New
upwind
biased
modified
electric
field
construc4on(upwind-­‐MEC),
Lee,
JCP,
submi[ed,
2012.
Considering
u>0,
v"0:
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012

46. New
Upwind-­‐MEC
q New
upwind
biased
modified
electric
field
construc4on(upwind-­‐MEC),
Lee
2012:
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012

47. CT
vs.
Upwind-­‐MEC
q Small
angle
advec4on
of
the
2D
field
loop:
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
θ = tan−1
(0.01) = 0.573

48. CT,
Standard
&
Upwind
MECs
q Small
angle
advec4on
of
the
3D
field
loop:
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
θ = tan−1
(0.01) = 0.573

49. Conclusion
q Direc4onally
unsplit
formula4ons
for
the
USM-­‐MHD
(also
valid
for
the
unsplit
hydro
solver)
in
FLASH4
q Two
new
features:
q The
reduced
and
full
3D
CTU
algorithms
q Upwind-­‐MEC
scheme
for
MHD
q Efficiency
and
accuracy
in
the
full
CTU
scheme:
q Stable
solu4ons
with
2nd
order
convergence
with
CFL=0.95
q 20%
performance
gain
in
the
full
CTU
scheme
over
the
reduced
CTU
scheme
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012
CPUF−ctu
CPUR−ctu
≈ 0.8

50. Thank
You
Ques2ons?
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012

51. Standard
&
Upwind
MEC
q Small
angle
advec4on
of
the
3D
field
loop:
The
7th
Astronum,
The
Big
Island,
HI,
U.S.A,
Jun
25-­‐29,
2012