Your SlideShare is downloading. ×
Parallel CFD Simulation
   “Fast Reactor Assembly”


Kurt D. Hamman

Multi-Physics Methods Group
Nuclear Science & Energy
...
Presentation Overview
• Project Objectives
   – Problem Size
• M&S Process
   – “Pre-processing”
   – “Solver Settings”
  ...
Project Objectives
• Primary
  – Develop a CFD Modeling Process using Commercial
    Software for “large” problems
     ( ...
Project Objectives – “Problem Size”
  65 – 85 million elements
                                Meshing ~ 1 to 2 days
 1 mi...
Modeling Process
                               19-Pin Fast Reactor Assembly
                Geometry
                    ...
“Preprocessing” – CAD Modeling
•     Geometry “similar” to Advanced Burner Test Reactor (ABTR)




                       ...
“Preprocessing” – Mesh
                         Boundary Layer
                          (Prism Cells)




               ...
Modeling Process “Solver”
 •   3 Dimensional
 •   “Steady-State”
 •   Sodium Properties (700 K)
 •   Constant Density
 •  ...
Modeling Process “Post-processing”
Modeling Process “Post-processing”
Modeling Process “Post-processing”
Validation Process
          Examine Iterative Convergence

               ● 3 to 4 orders of magnitude
          Examine...
Validation – Iterative Convergence
                                     Energy Equation Residual Response
                ...
Validation – Examine Consistency
                                      Polyhedral Mesh
                                   ...
Validation-Spatial Grid Convergence
             65 – 85 million elements
             1 million ≈ 1 Gb memory
           ...
Validation-Temporal Convergence
• Steady-State Simulation
   – Not performed
• General Comments
   – File size ~27.5 GB
  ...
Sensitivity Analyses Overview

•   Analysis #1
     – Assembly Geometry Changes
     – Several Turbulence Models
     – Em...
HPC “Lessons Learned”
• Understand Capabilities
   – Network
   – HPC Machines
   – Software
   – Productivity
      • Opt...
Understand Capabilities – “Network”
                                         PRIMARY FIRE WALL




                       ...
Understand Capabilities – “Network”
Understand Capabilities – “Machines”
 65 – 85 million elements
                               Meshing ~ 1 to 2 days
 1 mil...
Understand Capabilities - “Software”

 • SolidWorks
   – Compatibility with CFD Software
   – Functionality
      • Design...
Understand Capabilities – “Productivity”
                  Optimization
           (i.e. Visualization & Speedup)
Understand Capabilities – “Productivity”

                         “GREY SCREEN”




 Causes                            Re...
Understand Capabilities - Productivity
                                                        Icestorm/Altix ICE 8200 - S...
Understand Capabilities - Architecture




                                 e

      Business
                          Ic...
Understand Capabilities - Architecture


 • Productivity  ~30%

    – core count ↔
    – double nodes

                  ...
Understand Capabilities – “File Size”

 27 GB File Size
 • Storage
    – Parallel I/O
    – Optimal Topology
    – Save Ti...
Conclusions
•   M&S “Large Problems” is Challenging
     – CAD/Meshing
         • parallel mesher needed
     – Simulation...
References
1.   K.D. Hamman and R.A. Berry, “A CFD M&S Process for Fast Reactor Fuel
     Assemblies,” Experiments and CFD...
Upcoming SlideShare
Loading in...5
×

Modeling & Simulation

1,172

Published on

Modeling & Simulation of a fast reactor fuel assembly using commercial CFD software and high performance computing hardware.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,172
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Modeling & Simulation"

  1. 1. Parallel CFD Simulation “Fast Reactor Assembly” Kurt D. Hamman Multi-Physics Methods Group Nuclear Science & Energy July 21, 2008
  2. 2. Presentation Overview • Project Objectives – Problem Size • M&S Process – “Pre-processing” – “Solver Settings” – “Post-processing” • Validation Process • HPC “Lesson’s Learned” • Conclusions
  3. 3. Project Objectives • Primary – Develop a CFD Modeling Process using Commercial Software for “large” problems ( CAD → Meshing → Simulation → Visualization ) • Secondary – Evaluate Commercial Software • CAD and CFD – Evaluate HPC and Network Infrastructure • Problem Size (“production-type” simulations)
  4. 4. Project Objectives – “Problem Size” 65 – 85 million elements Meshing ~ 1 to 2 days 1 million ≈ 1 GB memory File Size → ~ 25 GB HPC (Meshing) Aurora (SGI Altix – 4700) • Shared Memory - 256 GB • 96 - 1.5 GHz Itanium 2 Processors • SUSE Linux Enterprise Server 10 • 5.72 TB Disk Capacity Simulation ~ 2 to 10 days HPC (Simulation) • Icestorm (Altix ICE 8200) • Distributed Memory(16 GB/Node) • 256 Nodes/8 PPN • 2.66 GHz clock speed • Linux OS
  5. 5. Modeling Process 19-Pin Fast Reactor Assembly Geometry & Results “Surface Mesh” Mesh Meshing Solver + CAD (HPC) (HPC) Validation 19 Pin Fuel Assembly 0.18 0.03500 0.16 0.03000 0.14 0.02500 0.12 0.02000 0.10 ( kpa/cm ) Empirical ( cm ) Numerical Pin Gap 0.08 0.01500 0.06 0.01000 0.04 Pin Gap 0.00500 0.02 0.00 0.00000 C(1) C(2) C(3) Model Validation Results Polyhedral Mesh (65.5 million elements) Solid Model
  6. 6. “Preprocessing” – CAD Modeling • Geometry “similar” to Advanced Burner Test Reactor (ABTR) 19 Pin Assembly Overlap (0.0065 cm) Todreas, N.E. & Kazimi, M.S., Nuclear Systems II Elements of Thermal Hydraulic Design, Taylor and Francis, 2001. Ds (3) Dft Dl Model Pins D P/D P ΔP Δg Length T (2) F (2) ABTR 217 0.800 0.103 1.130 0.904 0.001 0.033 13.598 - 260 0.0797 0.174 M(1) 19 0.800 0.103 1.169 0.936 0.039 0.032 4.299 2.482 20.0 0.2001 0.675 M(2) 19 0.800 0.103 1.149 0.919 0.023 0.016 4.210 2.431 20.0 0.1113 0.708 M(3) 19 0.800 0.103 1.135 0.908 0.012 0.005 4.148 2.395 20.0 0.0498 0.799 M(4) 19 0.800 0.103 1.127 0.902 0.005 0.005 4.126 2.382 20.3 0.0273 0.634 Notes: 1. All dimensions in centimeters. 2. Assembly geometry based on Todreas and Kazimi, where quot;Tquot; represents the flat-to-flat tolerance and quot;Fquot; represents the fraction of assembly tolerance distributed within the rods.
  7. 7. “Preprocessing” – Mesh Boundary Layer (Prism Cells) Interior Volume (Polyhedral Cells)
  8. 8. Modeling Process “Solver” • 3 Dimensional • “Steady-State” • Sodium Properties (700 K) • Constant Density • Segregated Solver • Second Order (convective) • Turbulent Flow • Turbulence Models: k-ε k-ω • Boundary Conditions – Inlet velocity 2 m/s – Outlet pressure 0 psi – Walls “no slip” – Heat Flux ~ 1MW/m2
  9. 9. Modeling Process “Post-processing”
  10. 10. Modeling Process “Post-processing”
  11. 11. Modeling Process “Post-processing”
  12. 12. Validation Process Examine Iterative Convergence  ● 3 to 4 orders of magnitude Examine Conservation  ● conservation of mass/energy X Examine Spatial (Grid) Convergence ● ordered discretization error X Examine Temporal Convergence ● N/A → Steady State Simulation Compare CFD Results to Data  ● “Average” Pressure Drop ● “Average” Mass Flowrate Examine Model Uncertainties  ● Turbulence Models/Solvers  Performed X Not Performed Reference: NPARC Alliance CFD Verification and Validation Web Site (www.grc.nasa.gov/WWW/wind/valid/validation.html)
  13. 13. Validation – Iterative Convergence Energy Equation Residual Response 1. Activating heat transfer (q” ~ 1 MW/m2 ) 2. Sdr and Tke “noise 85 Million Element Model (3.02.003)
  14. 14. Validation – Examine Consistency Polyhedral Mesh Numerical Interior Pin Wall-Pin Inlet Outlet Delta Difference Clearance Gap mass flowrate mass flowrate Δm (wrt inlet) Model (cm) (cm) (kg/s) (kg/s) (kg/s) C(1) 0.039 0.032 1.083 1.083 6.000E-06 0.00% C(2) 0.023 0.016 0.971 0.971 1.500E-06 0.00% C(3) 0.012 0.005 0.896 0.896 6.900E-06 0.00% Numerical Interior Pin Wall-Pin Heat Transfer Heat Transfer Delta Difference Clearance Gap quot;inquot; quot;outquot; ΔQ (wrt inlet) Model (cm) (cm) (W) (W) (W) C(1) 0.039 0.032 1.621E+06 1.620E+06 -1.636E+03 -0.10% C(2) 0.023 0.016 1.453E+06 1.451E+06 -1.534E+03 -0.11% C(3) 0.012 0.005 1.340E+06 1.338E+06 -1.666E+03 -0.12%
  15. 15. Validation-Spatial Grid Convergence 65 – 85 million elements 1 million ≈ 1 Gb memory File Size → ~ 25 GB (HPC Lessons Learned)
  16. 16. Validation-Temporal Convergence • Steady-State Simulation – Not performed • General Comments – File size ~27.5 GB – 1000 second transient – 27.5 TB storage • Suggests a Need – Parallel I/O Reference 5: “Advanced Burner Test Reactor Pre-Conceptual Design Report” – File Storage
  17. 17. Sensitivity Analyses Overview • Analysis #1 – Assembly Geometry Changes – Several Turbulence Models – Empirical Correlations • Analysis #2 – Wirewrap Geometry Changes – One Turbulence Model – Empirical Correlations • Analysis #3 – Mesh element types (polyhedral, trimmer)
  18. 18. HPC “Lessons Learned” • Understand Capabilities – Network – HPC Machines – Software – Productivity • Optimization – Memory Access • Mesh Refinement • Post processing • Unknowns – File Size
  19. 19. Understand Capabilities – “Network” PRIMARY FIRE WALL Subnet #1 Subnet #2 Subnet #3 e Business Enclave HPC Research Enclave Enclave Note: For illustration purposes only.
  20. 20. Understand Capabilities – “Network”
  21. 21. Understand Capabilities – “Machines” 65 – 85 million elements Meshing ~ 1 to 2 days 1 million ≈ 1 Gb memory File Size → ~ 25 Gb HPC (Meshing) Aurora (SGI Altix – 4700) • Shared Memory - 256 GB • 96 - 1.5 GHz Itanium 2 Processors • SUSE Linux Enterprise Server 10 • 5.72 TB Disk Capacity Simulation ~ 2 to 10 days HPC (Simulation) • Icestorm (Altix ICE 8200) • Distributed Memory(16 GB/Node) • 256 Nodes/8 PPN • 2.66 GHz clock speed • Linux OS
  22. 22. Understand Capabilities - “Software” • SolidWorks – Compatibility with CFD Software – Functionality • Design Changes • Equations • CD-adapco STAR-CCM+ – Physics – Compatibility • CAD Software • HPC Hardware • 3rd Party Post-Processors
  23. 23. Understand Capabilities – “Productivity” Optimization (i.e. Visualization & Speedup)
  24. 24. Understand Capabilities – “Productivity” “GREY SCREEN” Causes Results Dedicated Master Node → I/O Loss of Data Problems Size Problem Restart Network/Software Limitations Visualization/Analysis Limitations
  25. 25. Understand Capabilities - Productivity Icestorm/Altix ICE 8200 - Speedup (19-Pin 65 Million Elements, CCM+ 3.02.003) 18.0 16.0 14.0 12.0 Speed-up 10.0 Star-CCM+ Linear 8.0 6.0 4.0 2.0 0.0 0 64 128 192 256 320 384 448 512 576 640 704 768 832 896 960 1024 Processes
  26. 26. Understand Capabilities - Architecture e Business Icestorm (Altix ICE 8200) Enclave HPC Enclave • Distributed Memory(16 GB/Node) • 256 Nodes/8 PPN • 2.66 GHz clock speed • Linux OS
  27. 27. Understand Capabilities - Architecture • Productivity  ~30% – core count ↔ – double nodes e • Operate Machine 50% Business Rated Capacity Enclave HPC  – Productivity Enclave
  28. 28. Understand Capabilities – “File Size” 27 GB File Size • Storage – Parallel I/O – Optimal Topology – Save Times • Collaboration – File transfer – File sharing
  29. 29. Conclusions • M&S “Large Problems” is Challenging – CAD/Meshing • parallel mesher needed – Simulation/Visualization • Validation is Challenging – experimental data availability • Understand Capabilities and Limitations – software – HPC machines – network infrastructure – human infrastructure • Key Success Factor → “SYNERGY”
  30. 30. References 1. K.D. Hamman and R.A. Berry, “A CFD M&S Process for Fast Reactor Fuel Assemblies,” Experiments and CFD Code Applications to Nuclear Reactor Safety (XCFD4NRS 2008), Grenoble, France, accepted, 2008. 2. CD-adapco, “Star-CCM+ (2.08.004) User Guide”, 2007. 3. N. E. Todreas and M. S. Kazimi, “Nuclear Systems I, Thermal Hydraulic Fundamentals,” Hemisphere Publishing Corporation, 1990. 4. N. E. Todreas and M. S. Kazimi, “Nuclear Systems II, Elements of Thermal Hydraulic Design,” Taylor and Francis, 2001. 5. Y. I. Chang, P. J. Finck, and C. Grandy, “Advanced Burner Test Reactor Preconceptual Design Report,” ANL-ABR-1, September 2006. 6. E. H. Novendstern, Turbulent Flow Pressure Drop Model for Fuel Rod Assemblies Utilizing a Helical Wire-Wrap Spacer System, Nuclear Engineering Design, Vol. 22, pp. 19-27, 1972. 7. K. Rehme, Pressure Drop Correlations for Fuel Element Spacers, Nuclear Technology, Vol. 17, pp. 15-23, 1972. 8. R. Gajapathy, et al., CFD investigation of helical wire-wrapped 7-pin fuel bundle and the challenges in modeling full scale 217 pin bundle, Nuclear Engineering and Design (2007), doi:10.1016/j.nucengdes.2007.05.003. 9. NPARC Alliance CFD Verification and Validation Web Site (www.grc.nasa.gov/WWW/wind/valid/validation.html), accessed 11/1/07.

×