In this deck from PASC 2019, Thierry Poinsot from Toulouse Fluid Mechanics Institute presents: High Performance Computing for Instabilities in Aerospace Propulsion Systems. "Combustion produces more than 80 percent of the world's energy. This will continue for a long time as the global energy growth remains much larger than what new renewable energies can provide. Our civilization must allow the growth of combustion sources but, at the same time, keep global warming as well as pollution under control. Science has a key role in this scenario: it must optimize combustion systems far beyond the present state of the art. To do this, one promising path is to use High Performance Computation to compute and optimize combustors before they are built. This talk focuses on aerospace propulsion where optimization often leads to the occurrence of instabilities where combustion couples with acoustics, leading to unacceptable oscillations (the most famous example is the Apollo engine which required 1330 full scale tests to reach acceptable oscillation levels). The talk will show how simulation is used to control these problems, in real gas turbine engines and in rocket engines." Watch the video: https://wp.me/p3RLHQ-kq4 Learn more: https://pasc19.pasc-conference.org/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

1. High Performance Computing for instabilities
in aerospace propulsion systems
T. Poinsot
• Institut de Mécanique des Fluides
de Toulouse, CNRS
• CERFACS, Toulouse
• Stanford Center for Turbulence
Research
Support: ERC advanced grant INTECOCIS
(intecocis.inp-toulouse.fr).
SNECMA, TURBOMECA, ANSALDO, SIEMENS,
AIRBUS SAFRAN LAUNCHERS

2. BEFORE DISCUSSING HPC OR
INSTABILITIES, LET US TALK
ABOUT COMBUSTION (this is the political
moment of the talk)
2

3. What is combustion ?
!3
FUEL
AIR
PRODUCTST1
T2
Fuel + Air -> Products +Heat
COMBUSTION
CHAMBER
Heat

4. 4
(1): ENERGY ON EARTH TODAY
=
COMBUSTION

5. ENERGY ON EARTH = COMBUSTION
COMBUSTION IS PRODUCING MORE THAN 90 PERCENT OF THE
ENERGY TODAY. THIS WILL DECREASE... BUT NOT TOMORROW
5

6. Beware of ‘electric’ …
6
Electricity is NOT an energy source.
It is only an energy vector.
In Germany (when there is no wind and no sun) or in
China, electric cars run on… coal:
In Germany, a TESLA can have a CO2 signature which is
worse than a Diesel BMW…
TESLA RENAULT ZOE

7. COMBUSTION OVERVIEW
7
(1): ENERGY ON EARTH TODAY
=
COMBUSTION
(2): ENERGY ON EARTH TOMORROW
=
COMBUSTION

8. 8
1970 1980 1990 2000 2010
Years
0
20000
40000
60000
80000
100000
120000
140000
160000Energyconsumption[TWh]
Energy source
Other renewables
Hydropower
Nuclear
Coal
Gas
Oil

9. !9
CO, NO, CO2, soot
POLLUTION ON EARTH TODAY
=
COMBUSTION

10. !10
Chemical species are not the only
‘pollution’ produced by combustion:
NOISE is also a big issue

11. !11

12. 12
SO, COMBUSTION IS A PROBLEM
INTERESTINGLY, IT IS ALSO THE SOLUTION !

13. 13
WE ALL WANT TO GO TO A MASSIVE
USE OF RENEWABLE ENERGIES
THIS IS IMPOSSIBLE:
IF WE CAN NOT HAVE OTHER
PRODUCTION METHODS WHEN
RENEWABLE DO NOT PRODUCE
IF WE CAN NOT FIND METHODS TO
STORE ENERGY

14. 14
COMBUSTION IS A KEY ELEMENT TO
COMPLEMENT AND STORE RENEWABLE
ENERGIES
PRODUCTION
CONSUMPTION
=
20
15
10
5
0
MW
403020100
Days
TOTAL CONSUMPTION
PRODUCTION BY RENEWABLES
COMBUSTION

15. H2
O2
15
COMBUSTION CAN ALSO BE USED
TO STORE ENERGY:
TOO MUCH WIND
TOO MUCH SUN
NO WIND
NO SUN
Fossil
fuel
Air
Electrolysis
Combustor
Renewable fuel

16. AND COMBUSTION CAN STORE MUCH
MORE AND FOR MUCH LONGER TIMES
16
TESLA

17. 17
Whatever the scenario for energy production is,
we need the best combustion systems.
Optimisation of combustors must:
- maximize efficiency -> FUEL RESOURCES
- minimize pollutants -> AIR POLLUTION
- minimize CO2 emission -> GLOBAL WARMING

18. AEROSPACE: Aircraft, helicopters,
rockets cannot fly without combustion
18
No other way to reach the required power/weight
ratios needed to fly efficiently (1 kg of fuel contains
as much energy as 30 to 50 kgs of batteries).
« Optimization » of engines is badly needed because
regulations and technologies evolve rapidly

19. WHAT IS A GAS TURBINE ?
19
COMPRESSOR - COMBUSTION CHAMBER - TURBINE

20. 20
Sulzer
GAS TURBINES

21. OPTIMIZING SOMETHING WHICH HAS
BEEN HERE FOR 70 YEARS IS DIFFICULT
• Compromises (efficiency,
pollution, noise, stability, cost)
are difficult to find and
experimental costs too large to
test all possible designs
21
Sir F. WhittleW2 engine
One of the main problems encountered when
combustion chambers are « optimized » is
COMBUSTION INSTABILITIES (also called
THERMOACOUSTIC INSTABILITIES)

22. Thermoacoustic instabilities
• The most dangerous instabilities: they break the chamber !
• Coupling between unsteady combustion and acoustics
22
REVIEW PAPERS AND BOOKS:
Culick, Agardograph 2006
Candel, plenary lecture, Proc. Comb. Inst. 29 (1), 2002
Lieuwen and Yang, Prog. Ast. Aero. AIAA 210, 2005
Poinsot, Hottel plenary lecture. Proc. Comb. Inst. 36, 2016.

23. Why thermoacoustic
instabilities in engines ?
First, unsteady flames create acoustic waves: unsteady
heat release acts like an acoustic monopole
23
Far field
unsteady
pressure
Price et al, 12th Symp. (Int.) Comb. 1969
Schuller et al, Comb. Flame 128, 2002
p(r, t)
dQ
dt
Unsteady
heat release

24. 24
FLAME POSITION
DILATATION FIELD
WHEN A FLAME MOVES, IT MAKES NOISE

25. Viceversa, noise modify flames
25
Loudspeaker

26. 26
Forcing frequency
of the loudspeaker
Flames are sensitive to sound:

27. Flames like music:
• Credits: Shakira and D. Mejia, columbian PhD student
27
WHEN A FLAME HEARS NOISE, IT MOVES

28. 28
ACOUSTICS
FLAMES
Flames
create
acoustics
Acoustics
influence
flames
Karlsruhe Institute of Technology (C. Kraus)

29. 29
UNSTEADY HEAT
RELEASE
ACOUSTIC
WAVES
UNSTEADY
FLOW RATE
LARGE
VORTICES
Mechanisms of resonance:
2000 frames/s
Premixed
gas (air +
propane)
Keller et al, AIAA J. 1982, 20.
The inlet velocity becomes negative: the flow exits
the combustor by the inlet. Bad idea !

30. This is combustion instability in a little sturdy
burner designed to survive. In a real engine the
outcome of combustion instabilities is more
dramatic: engines can die in a few milliseconds

31. REAL ENGINES :
31
ROCKETS GAS TURBINES

32. A famous example of combustion
instability in a big engine: F1 (Saturn V)
32
Flow rate: 2.6 t/s. Thrust: 7000 kN
Review of F1 engine instabilities: Oefelein and Yang, J. Prop. Power, 9, 5, 1993.

33. 33
Initial engine was unstable
Cant be studied in lab-scale burners
Solutions: modify fuel injectors and
add acoustic baffles... but how ?
Small scale tests not sufficient
1332 full engine tests
Billions of dollars spent
Incompatible with present programs.
Today: we want to replace this
procedure by simulations
Source: Pr Vigor Yang
(Georgia Tech)

34. WHAT IS AT STAKE HERE ? :
34
When an engine is unstable, what
should we do ?
1/ Fix the problem by trial and error (as
for F1) - > got 2 bn $ ?
2/ Use simulation/theory to analyze WHY
the engine is unstable to fix the problem
and avoid that it happens again.

35. A THIRD SOLUTION ?
35
« The cure for this problem is better injector
design, baffles on the injector plate, and a
pinch of black magic and the Lord’s Prayer »

36. Let us start discussing HPC
• We use ‘LES (Large Eddy Simulation)’ of the reacting
flow within the combustion chamber, performed with
AVBP: legacy code, 800 000 lines in Fortran and C
• AVBP solves the filtered reacting Navier Stokes
equations for the flow and the chemical species
(typically N= 20 chemical species)
• On massively parallel machines (1000 to 200 000
processors)
• Used today by many labs and by industry: 200 engine
configurations per year
36

37. WHICH EQUATIONS DO WE SOLVE ?
The Navier Stokes equations:
• 5 unknowns (density, 3 velocities and energy)
to describe the flow at each grid point + N
chemical species.
• fully coupled, embarrassingly not parallel
(message passing needed for first and
second spatial derivatives)
• for an engine, grids = O(1 billion grid points).
37
Time advancement is explicit to avoid having to
solve a large linear system at each time iteration
(as needed when implicit codes are used).
Two consequences:
1/ Parallelisation remains very good
2/ Time steps become very small. 10 ns is usual
‘Legacy’ code but typically
rewritten every three years.

38. The Navier Stokes PDEs:
38

39. SO WHY IS THIS DIFFICULT ?
39
Fields of density in a H2-O2 engine
1/ MULTI SCALE PROBLEM
Combustion chamber:
meters and hours

40. SO WHY IS THIS DIFFICULT ?
40
Fields of density in a H2-O2 engine
Five hundred
injectors of
cryogenic H2
and O2

41. 1 cm
Within the combustion chamber:
nanoseconds and nanometers
SO WHY IS THIS DIFFICULT ?
41
Fields of density in a H2-O2 engine

42. SO WHY IS THIS DIFFICULT ?
42
Fields of density in a H2-O2 engine
2/ ALL FLOWS OF INTEREST ARE
TURBULENT

43. 43
WHAT IS TURBULENCE ?

44. Turbulence and combustion
44
Simulation in this field is (1) mandatory and (2) one of the
first users of HPC. Many ASCI projects in the US in the
past 20 years focused on turbulent reacting flows:
- gas turbines
- astrophysics
- fires
- rockets
- weapons
These topics are good and tough candidates to develop
HPC tools (more relevant for us than LAPACK)

45. Most simulations shown today:
• Compressible Navier-Stokes equations with
combustion solved with high order finite
volume/ finite element on hybrid meshes
• Massively parallel runs (10 to 64 000 in
production mode)
• Domain decomposition + MPI/OPENMP
• Large team (20 developers, 200 users)
• Used daily to design engines in industry
Jaravel, T., Riber, E., Cuenot, B. and Bulat, G. (2016) Proc. Comb. Inst. 36, doi:10.1016/j.proci.2016.07.027
Bauerheim et al Comb. Flame 2014, 162, 60-67.
Bauerheim et al. Phys. Fluids, 28, 021303, 2016 -> REVIEW PAPER
Bauerheim et al. Comb. Flame, 161, 5, 2013, 1374-1389.
Urbano et al. Proc. Comb. Inst. 36, Seoul -> Distinguished paper in 2016
Urbano et al Comb. Flame 169:129-140,2016.
Poinsot T. Proc. Comb. Inst. 36, - > Hottel invited plenary lecture.
45

47. Speed up of AVBP (strong scaling, production
mode for a real engine, not a toy model):
47

48. 48
Let us close the parenthesis

49. Instabilities
Gas turbines
Engines
Rockets

50. AZIMUTHAL INSTABILITIES IN
GAS TURBINE CHAMBERS
P. Wolf, M. Bauerheim, G. Staffelbach
(CERFACS)
JF Bourgouin, S. Ducruix, S. Candel
(EM2C, Ecole Centrale Paris and CNRS)
N. Worth and J. Dawson
(Cambridge and NTNU)
50

51. Full combustion chamber
(10 to 24 ‘sectors’ or ‘burners’)
!
AIR
FR
O
M
C
O
M
PR
ESSO
R
TO
TU
R
BIN
E KEROSENE

52. LPP injector!
Swirler (one per sector)
AIR FROM
COMPRESSOR
TO TURBINE
Complex geometries: we need
unstructured meshes. This does
not simplify domain decomposition.
Typically: PAR METIS

53. ANNULAR CHAMBER:
53
An annular shape allows ‘new’ modes where waves
propagate along the chamber azimuthal direction.

54. Frequency of the first azimuthal mode (easy):
54
f =
cc
2Lc
cc
4⇡Lc
⌃
Sound speed
Chamber perimeter
This mode is ‘degenerate’: there are two
acoustic eigenmodes at the same frequency
(clockwise and counter clockwise)
Instantaneous pressure

55. 55
‘SLIM’ (SIMPLE LINEAR MODEL)
FOR IN AN ANNULAR CHAMBER:
θ
A+ A-
p0
(✓, t)

56. SLIM FOR PURE ROTATING MODE (A-=0)
56
Pressure field

57. LAB EXPERIMENTS: Cambridge
57
Dawson and Worth, Comb. Flame 161 (2014)
Worth and Dawson, Comb. Flame 160 (2013)

58. Fluctuating heat release
58
BLUE: burns
less than
average
RED: burns
more than
average
Dawson and Worth, Comb. Flame 161 (2014)
Worth and Dawson, Comb. Flame 160 (2013)

59. 59
Bourgouin et al, Proc. Comb. Inst. 35, 3237.
LAB EXPERIMENTS: EM2C Paris

60. 60

61. LES were done first:
61
Air inlet
s
Choked nozzle
Dilution holes
Casing
Fuel inlet
Co-annular
contra-rotating swirlers
LPP injector!
Staffelbach et al. Proc. Comb. Inst., 32: 2909-2916, 2009
Wolf et al. Comb. Flame, 159: 3398-3413, 2012
Gicquel et al. Prog. En. Comb. Sci. 38, 782-817, 2012 (Review paper)

62. LES of the full chamber
62
Computed one sector first.
Then copied it 14 times and let LES run

63. HPC resources
63
PRACE (EU)+ INCITE (USA) allocations:
approximately 2000 CPU years spent over 12
months.
The most expensive part was to prove that the
results were grid independent

64. Pressure oscillations grow
in the LES
64
A strong azimuthal mode appears (same
frequency and amplitude as real engine data)
-4
-3
-2
-1
0
1
2
3
W_local[m/s]
40x10
-3
3020100
Time [s]
AzimuthalVelocity[m/s]
Time [s]
-4
0.01 0.02 0.03 100.040
-3
-2
-1
0
1
2
3
0.040
-4
Azimuthalvelocity[m/s]
-3
-2
-1
0
1
2
3
Time [s]0.02
Pressure(au)
Time (s)

65. 65
TemperaturePressure
And the mode captured by LES
is indeed an azimuthal one:

66. Intermittent
flashback
Temperature
This tool is used now by industry in France
to simulate instabilities in real helicopter
and aircraft engines

67. Instabilities
Gas turbines
Engines
Rockets

68. INSTABILITIES IN ROCKET
ENGINES
A. Urbano, G. Staffelbach, B. Cuenot, L. Selle
(CERFACS and IMFT, CNRS, Toulouse)
T. Schmitt, S. Ducruix, P. Scoufflaire, S. Candel
(EM2C, Ecole Centrale Paris and CNRS)
S. Gröning, D. Suslov, J. Hardi, M. Oschwald
(DLR Lampoldshausen)
L. Vingert, G. Ordonneau
(ONERA)
T. Sattelmayer
(TU Munich)
68
LES of the temperature field in a rocket
engine (Urbano et al 36th Symp)

69. ROCKETS: HIGH-FREQUENCY
INSTABILITIES ARE THE MOST
DANGEROUS
69
Extreme temperatures and pressures:
the power of a nuclear power plant in the volume
of a Renault Twingo engine

70. STUDYING INSTABILITIES IN
ROCKET ENGINES:
70
Lab-scale experiments are difficult
Simulations are difficult too !
MAIN ISSUE: can we rely on Large
Eddy simulations to predict
instabilities in a rocket engine ?

71. The BKD experiment (DLR)
71
Gröning et al. J. Prop. Power, 2016
Gröning et al. 5th EUCASS 2013
42 coaxial injectors
Cryogenic O2/H2 propellants
Pressure range: 50-80 bar
Not a real engine (only
42 injectors instead of
500) but close enough…

72. BKD GEOMETRY
3.6$
Sizes$in$mm$
O2$
H2$
2$
0.25
0.2
H
O
H
sizes in mm
72
O2
H2
LOx
dome
Injectors
H2 dome
Nozzle

73. 73
THE BKD EXPERIMENT

74. 74
EVEN THIS LAB EXPERIMENT IS A MONSTER:
= THE
POW
ER
OF
200
FERRARIS
POWER = 80 MW
TH
E
VO
LU
M
E
O
F
A
C
A
N

75. TWO CASES EXTRACTED FROM
DLR EXPERIMENTS:
Case: Case 1 Case II
Regime: Low power High power
75
EXPERIMENT: stable unstable
Gröning et al. 5th EUCASS 2013.
Here a real engine would
explode. This one survives
because it is built for this

76. 76
PRACE (EM2C, CERFACS, IMFT)
80 M hours on FERMI (cineca IT)
BlueGeneQ: production on 16000 cores
LOx
dome
H2 dome

77. 77
SELF-SUSTAINED CASE:
p’ [bar]
Temperature
Pressure
Pressure

78. 78
f [kHz]
PSD[dB/Hz]
0 5 10 15 20 25
-110
-100
-90
-80
-70
-60
-50
-40
2
1
LES
X
Experiment
MODE 1
MODE 2
COMPARING LES
AND EXPERIMENTS
Urbano et al. 36th Symp. (Int.) Comb.
Urbano et al Comb. Flame 169:129-140,2016.
Gröning et al. Space Prop. Conf. 2014
Gröning et al. J. Prop. Power 2016

79. 79
POST PROCESSING
(Big data… ah ah !):
Each simulation generates 20 instantaneous 3D fields
(u,v,w,P,T and 15 chemical species) over 1 million time
steps on a 1 billion point grid. Typically 10^16 real
numbers to store: -> Big data
Of course we dont store them: we need co post
processing on the fly.
DMD (Dynamic Mode Decomposition) is one typical
post processing example: it provides a frequency
resolved decomposition of the 3D unsteady fields.

80. 80
THERE ARE ACTUALLY TWO MODES
WHICH INTERFERE IN THIS LES:

81. 81
DMD: IDENTIFICATION OF
MODES 1 AND 2(X)
f1=10700Hz 3/2 L 1 T
MODE 1: Transverse mode in the chamber coupled to a
3/2L longitudinal mode in the oxygen ducts
MODE 2: Radial mode in the chamber coupled to a 3L
longitudinal mode in the oxygen ducts
CONCLUSIONS:
LES can be done for these engines
They are expensive but on a Tier0
system they take only a few weeks.

82. Conclusions
Combustion will remain our main energy source for a
long time. It will also allow us to store energy
Combustion instabilities are a major risk in most
engines: Fascinating field of investigation, combining
kinetics, unsteady flows, acoustics, heat transfer
82

83. The LES revolution associated to massively
parallel computations allows now to ‘look inside
real engines’ at low cost… (well, lower than
running/breaking the real engine)
83

84. Thanks to: S. Candel4, D. Veynante4, F. Nicoud18, B. Cuenot2, G. Staffelbach2, L. Selle1,
G. Lartigue5, L. Gicquel2, V. Moureau5, D. Durox4,S. Ducruix4,, C. Silva6, J. Oefelein19,
W. Polifke6, D. Durox4, M. Juniper7, J. Dawson7, N. Worth7, P. Moin3, P. Clavin8,
W. Krebs9, M. Bauerheim1,10, N. Noiray10, E. Riber2, A. Urbano11, C. Lapeyre1, A. Ghani1,2,
B. Emerson12, T. Lieuwen12, A. Ghoniem13, V. Yang12, J. Hardi14, D. Suslov14, D. Mejia1,
S. Groning14. M. Oschwald14, R. Koch15, C. Berat16, P. Wolf2, F. Duchaine2, O. Vermorel2,
P. De Goey17, T. Schmitt4, P. Scoufflaire4, T. Schuller4, G. Lacaze19, T. Sattelmayer6,
JC. Larroya20, D. Saucereau21, F. Lacas4, G. Mungal3, A. Trouvé22, M.Brebion1 and more !
1 IMFT, CNRS, Toulouse France
2 CERFACS, Toulouse France
3Center for Turbulence Research, Stanford
4 EM2C, Paris
5CORIA, Rouen
6TU Munich
7Cambridge University
8 IRPHE, Marseille
9 Siemens, Mullheim
10 ETH Zurich
11 SAFRAN Tech.
Thanks to the
thermoacoustics community !
12 GEORGIA Tech.
13 MIT
14 DLR Lampoldshausen
15 Karlsruhe Institute of Technology
16 SAFRAN HELICOPTER ENGINE
17 TU Eindhoven
18 University of Montpellier
19 SANDIA National Lab.
20 SAFRAN AIRCRAFT ENGINE
21 SNECMA VERNON
22 Univ. Maryland