Advances in NS solver on GPU por M.Storti et.al.
Advances in the Solution of NS Eqs. in GPGPU
Hardware
by Mario Stortiab
,...
Advances in NS solver on GPU por M.Storti et.al.
Scientific computing on GPUs
• Graphics Processing Units (GPU’s) are speci...
Advances in NS solver on GPU por M.Storti et.al.
Scientific computing on GPUs (cont.)
• The difference between the GPUs
arc...
Advances in NS solver on GPU por M.Storti et.al.
GPUs, MIC, and multi-core processors
• Computing power is limited by the ...
Advances in NS solver on GPU por M.Storti et.al.
Solution of incompressible Navier-Stokes flows on GPU
• GPU’s are less effi...
Advances in NS solver on GPU por M.Storti et.al.
Fractional Step Method on structured grids with QUICK
Proposed by Molemak...
Advances in NS solver on GPU por M.Storti et.al.
Solution of the Poisson with FFT
• Solution of the Poisson equation is, f...
Advances in NS solver on GPU por M.Storti et.al.
Solution of the Poisson with FFT (cont.)
• So the following algorithm com...
Advances in NS solver on GPU por M.Storti et.al.
FFT computing rates in GPGPU. GTX-580 and Titan Black
1e+5 1e+6 1e+7 1e+8...
Advances in NS solver on GPU por M.Storti et.al.
Upper bound on computing rate for the NS solver w/FFT
• For instance for ...
Advances in NS solver on GPU por M.Storti et.al.
Solution of NS on embedded geometries: FVM/IBM1
• When solid bodies are
i...
Advances in NS solver on GPU por M.Storti et.al.
Solving Poisson with embedded bodies
• FFT solver and GMG are very fast b...
Advances in NS solver on GPU por M.Storti et.al.
The IOP (Iterated Orthogonal Projection) method
The method is based on su...
Advances in NS solver on GPU por M.Storti et.al.
The IOP (Iterated Orthogonal Projection) method (cont.)
• Fixed point ite...
Advances in NS solver on GPU por M.Storti et.al.
Accelerated Global Preconditioning (AGP) algorithm
• Storti et.al. ”A FFT...
Advances in NS solver on GPU por M.Storti et.al.
MOC+BFECC
• The Method of Characteristics (MOC) is a Lagrangian method. T...
Advances in NS solver on GPU por M.Storti et.al.
MOC+BFECC (cont.)
exact w/dissipation error
• If L introduces some dissip...
Advances in NS solver on GPU por M.Storti et.al.
MOC+BFECC (cont.)
Nbr of Cells QUICK-SP BFECC-SP QUICK-DP BFECC-DP
64 × 6...
Advances in NS solver on GPU por M.Storti et.al.
Analysis of performance
• Regarding the performance results shown above, ...
Advances in NS solver on GPU por M.Storti et.al.
Validation. A buoyant fully submerged sphere
• A fluid structure interacti...
Advances in NS solver on GPU por M.Storti et.al.
Spring-mass-damper oscillator
• The acceleration of the box creates a dri...
Advances in NS solver on GPU por M.Storti et.al.
Experimental results
• Experiments carried out atUniv Santiago de
Chile (...
Advances in NS solver on GPU por M.Storti et.al.
Experimental results (cont.)
0
20
40
60
80
100
120
140
160
180
0.5 0.6 0....
Advances in NS solver on GPU por M.Storti et.al.
abox
driving force
box desplacement
buoy displacement (f<<f0)
buoy displa...
Advances in NS solver on GPU por M.Storti et.al.
Computational FSI model
• The fluid structure interaction is performed wit...
Advances in NS solver on GPU por M.Storti et.al.
Computational FSI model (cont.)
• The simulation is performed in a non-in...
Advances in NS solver on GPU por M.Storti et.al.
Why use a positive buoyant body?
• Gov. equations for the body:
mbdy ¨x +...
Advances in NS solver on GPU por M.Storti et.al.
Comparison of experimental and numerical results
0
0.02
0.04
0.06
0.08
0....
Advances in NS solver on GPU por M.Storti et.al.
Comparison of experimental and numerical results
0
0.02
0.04
0.06
0.08
0....
Advances in NS solver on GPU por M.Storti et.al.
Convergence of FVM/IBM1
• Plot shows amplitude vs mesh size at resonance ...
Advances in NS solver on GPU por M.Storti et.al.
Immersed boundary second order scheme (FVM/IBM2)
• Fadlun, et al. ”Combin...
Advances in NS solver on GPU por M.Storti et.al.
Immersed boundary second order scheme (FVM/IBM2) (cont.)
1: Data initiali...
Advances in NS solver on GPU por M.Storti et.al.
The boundary projection operator
• Given a solid cell node S we have to d...
Advances in NS solver on GPU por M.Storti et.al.
Numerical results with FVM/IBM2
0.4 0.6 0.8 1 1.2 1.4 1.6
0
0.02
0.04
0.0...
Advances in NS solver on GPU por M.Storti et.al.
Convergence of FVM/IBM2
• FVM/IBM2 exhibits O(h2
) convergence.
0 0.001 0...
Advances in NS solver on GPU por M.Storti et.al.
Convergence of FVM/IBM2 (cont.)
• Results for FVM/IBM2 with 1923
are much...
Advances in NS solver on GPU por M.Storti et.al.
Computational efficiency of FVM/IBM2 on GPGPUs
• Efficiency of the above al...
Advances in NS solver on GPU por M.Storti et.al.
Tesla K40c vs GTX Titan Black
GTX Titan Black
2880 cores @ 980MHz2880 cor...
Advances in NS solver on GPU por M.Storti et.al.
Computational efficiency of FVM/IBM2 on GPGPUs
2e+06 4e+06 6e+06 8e+06 1e+...
Advances in NS solver on GPU por M.Storti et.al.
Computing rates for FVM/IBM2
• Maximum attained rate is
1.3 Gcell/s for G...
Advances in NS solver on GPU por M.Storti et.al.
Computing rates for FVM/IBM2 (cont.)
• For 1923
(the refinement used for t...
Advances in NS solver on GPU por M.Storti et.al.
Time consumption breakdown
CIMEC-CONICET-UNL 42
((version texstuff-1.2.8-...
Advances in NS solver on GPU por M.Storti et.al.
Numerical results
Vorticity isosurface
velocity at sym plane
tadpoles
col...
Advances in NS solver on GPU por M.Storti et.al.
FVM/IBM vs. LBM on GPGPU
• In LBM the Poisson equation is not solved; it ...
Advances in NS solver on GPU por M.Storti et.al.
FVM/IBM vs. LBM on GPGPU (cont.)
• So in fact not only Mcell/s must be re...
Advances in NS solver on GPU por M.Storti et.al.
Future work
• We are in the process of buying a HPC cluster (≈ 20 Tflops, ...
Advances in NS solver on GPU por M.Storti et.al.
Acknowledgments
• Chilean Council for Scientific and Technological Researc...
Upcoming SlideShare
Loading in …5
×

Advances in the Solution of NS Eqs. in GPGPU Hardware. Second order scheme and experimental validation

542 views

Published on

A Navier-Stokes solver based on Cartesian structured finite volume
discretization with embedded bodies is presented. Fluid structure
interaction with solid bodies is performed with an explicit
partitioned strategy. The Navier-Stokes equations are solved in the
whole domain via a Semi-Implicit Method for Pressure Linked Equations
(SIMPLE) using a colocated finite volume scheme, stabilized
via the Rhie-Chow discretization. As uniform Cartesian grids are used,
the solid interface usually do not coincide with the mesh, and then a
second order Immersed Boundary Method is proposed, in order to avoid
the loss of precision due to the staircase representation of the
surface. This fact also affects the computation of fluid forces on the
solid wall and, accordingly, the results in the fluid-structure
analysis. In the present work, first and second order approximations
for computing the fluid forces at the interface are studied and
compared. The solver is specially oriented to General Purpose Graphic
Processing Units (GPGPU) hardware and the efficiency is
discussed. Moreover, a novel submerged buoy experiment is also
reported and serves to validate the presented fluid-structure
algorithm. The experiment consists of a sphere with positive buoyancy
fully submerged in a cubic tank, subject to harmonic displacements
imposed by a shake table. The sphere is attached to the bottom of
the tank with a string. Position of the buoy is determined from video
records with a Motion Capture algorithm. The obtained amplitude and
phase curves allow a precise determination of the added mass and drag
forces. Due to this aspect the experimental data can be of interest
for the validation of fluid-structure interaction codes. Finally, the
numerical results are compared with the experiments, and allows the
validation of the numerically predicted drag and added mass of the
body.

Published in: Science
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
542
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Advances in the Solution of NS Eqs. in GPGPU Hardware. Second order scheme and experimental validation

  1. 1. Advances in NS solver on GPU por M.Storti et.al. Advances in the Solution of NS Eqs. in GPGPU Hardware by Mario Stortiab , Santiago Costarelliab , Luciano Garellia , Marcela Cruchagac , Ronald Ausensic , S. Idelsohnabde , Gilles Perrotf , Rapha¨el Couturierf a Centro de Investigaci´on de M´etodos Computacionales, CIMEC (CONICET-UNL) Santa Fe, Argentina, mario.storti@gmail.com, www.cimec.org.ar/mstorti b Facultad de Ingenier´ıa y Ciencias H´ıdricas. UN Litoral. Santa Fe, Argentina, fich.unl.edu.ar c Depto Ing. Mec´anica, Univ Santiago de Chile d International Center for Numerical Methods in Engineering (CIMNE) Technical University of Catalonia (UPC), www.cimne.com e Instituci´o Catalana de Recerca i Estudis Avanc¸ats (ICREA), Barcelona, Spain, www.icrea.cat f FEMTO-ST Institute, Franche-Comt´e Meso-Centre CIMEC-CONICET-UNL 1 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  2. 2. Advances in NS solver on GPU por M.Storti et.al. Scientific computing on GPUs • Graphics Processing Units (GPU’s) are specialized hardware designed to discharge computation from the CPU for intensive graphics applications. • They have many cores (thread processors), currently the Tesla K40 (Kepler GK110) has 2880 cores at 745 Mhz (boost 875 Mhz). • The raw computing power is in the order of Teraflops (4.3 Tflops in SP and 1.43 Tflops in DP). • Memory Bandwidth (GDDR5) 288 GiB/s. Memory size 12 GiB. • Cost: USD 3,100. Low end version Geforce GTX Titan: USD 1,500. CIMEC-CONICET-UNL 2 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  3. 3. Advances in NS solver on GPU por M.Storti et.al. Scientific computing on GPUs (cont.) • The difference between the GPUs architecture and standard multicore processors is that GPUs have much more computing units (ALU’s (Arithmetic-Logic Unit) and SFU’s (Special Function Unit), but few control units. • The programming model is SIMD (Single Instruction Multiple Data). • Branch-divergence happens when a branch condition gives true in some in some threads an false in others. Due to the SIMD model all threads can’t be executed simultaneously. • GPUs compete with many-core processors (e.g. Intel’s Xeon Phi) Knights-Corner, Xeon-Phi 60 cores). if (COND) { BODY-TRUE; } else { BODY-FALSE; } CIMEC-CONICET-UNL 3 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  4. 4. Advances in NS solver on GPU por M.Storti et.al. GPUs, MIC, and multi-core processors • Computing power is limited by the number of transistors we can put in a chip. This is limited in turn to the largest silicon chip you can grow (O(400 mm2 )), and the minimum size of the transistors you can make (O(22 nm)). So actually the limit is in the order of 5 × 109 transistors per chip. • So with that number of transistors you can do one of the following Have a few (6 to 12) very powerful cores: Xeon E5-26XX processors. Have 60 or more less powerful cores, each one with 40 million transistors, similar to a Pentium processor: Intel Many Integrated Core (MIC), Xeon Phi Have a very large (O(3000)) simple (without control) cores: Nvidia GPUs. Die shot of the Nvidia ”Kepler1” GK104 GPU CIMEC-CONICET-UNL 4 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  5. 5. Advances in NS solver on GPU por M.Storti et.al. Solution of incompressible Navier-Stokes flows on GPU • GPU’s are less efficient for algorithms that require access to the card’s (device) global memory. In older cards shared memory is much faster but usually scarce (16K per thread block in the Tesla C1060) . • The best algorithms are those that make computations for one cell requiring only information on that cell and their neighbors. These algorithms are classified as cellular automata (CA). • In newer cards there is a much larger memory cache. Anyway data locality is fundamental. • Lattice-Boltzmann and explicit F M (FDM/FVM/FEM) fall in this category. • Structured meshes require less data to exchange between cells (e.g. neighbor indices are computed, no stored), and so, they require less shared memory. Also, very fast solvers like FFT-based (Fast Fourier Transform) or Geometric Multigrid are available . CIMEC-CONICET-UNL 5 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  6. 6. Advances in NS solver on GPU por M.Storti et.al. Fractional Step Method on structured grids with QUICK Proposed by Molemaker et.al. SCA’08: 2008 ACM SIGGRAPH, Low viscosity flow simulations for animation. • Fractional Step Method (a.k.a. pressure segregation) • u, v, w and continuity cells are staggered (MAC=Marker And Cell). • QUICK advection scheme is used in the predictor stage. • Poisson system is solved with IOP (Iterated Orthogonal Projection) (to be described later), on top of Geometric MultiGrid j j+1 j+2 i+1 i+2(ih,jh) Vij PijUij x y i y-momentum cell x-momentum nodes y-momentum nodes continuity nodes x-momentum cell continuity cell CIMEC-CONICET-UNL 6 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  7. 7. Advances in NS solver on GPU por M.Storti et.al. Solution of the Poisson with FFT • Solution of the Poisson equation is, for large meshes, the more CPU consuming time stage in Fractional-Step like Navier-Stokes solvers. • We have to solve a linear system Ax = b • The Discrete Fourier Transform (DFT) is an orthogonal transformation ˆx = Ox = fft(x). • The inverse transformation O−1 = OT is the inverse Fourier Transform x = OT ˆx = ifft(ˆx). • If the operator matrix A is spatially invariant (i.e. the stencil is the same at all grid points) and the b.c.’s are periodic, then it can be shown that O diagonalizes A, i.e. OAO−1 = D. • So in the transformed basis the system of equations is diagonal (OAO−1 ) (Ox) = (Ob), Dˆx = ˆb, (1) • For N = 2p the Fast Fourier Transform (FFT) is an algorithm that computes the DFT (and its inverse) in O(N log(N)) operations. CIMEC-CONICET-UNL 7 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  8. 8. Advances in NS solver on GPU por M.Storti et.al. Solution of the Poisson with FFT (cont.) • So the following algorithm computes the solution of the system to machine precision in O(N log(N)) ops. ˆb = fft(b), (transform r.h.s) ˆx = D−1 ˆb, (solve diagonal system O(N)) x = ifft(ˆx), (anti-transform to get the sol. vector) • Total cost: 2 FFT’s, plus one element-by-element vector multiply (the reciprocals of the values of the diagonal of D are precomputed) • In order to precompute the diagonal values of D, We take any vector z and compute y = Az, then transform ˆz = fft(z), ˆy = fft(y), Djj = ˆyj/ˆzj. CIMEC-CONICET-UNL 8 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  9. 9. Advances in NS solver on GPU por M.Storti et.al. FFT computing rates in GPGPU. GTX-580 and Titan Black 1e+5 1e+6 1e+7 1e+8 0 100 200 300 400 500 600 580/DP 580/SP TB/DP TB/SP GTX 580 GTX Titan Black Launch Feb 2014, Arch: Kepler, USD 1,500, 2880 cores, 5.12 Tflops SP Launch June 2011, Arch: Fermi, USD 1,000, 772 cores, 1.5 Tflops SP GTX 580 DP FFTproc.rate[Gflops/sec] Ncell*t 64x64x64 128x64x64 128x128x64 128x128x128 256x128x128 256x256x128 256x256x256 512x256x256 GTX 580 SP GTX TB DP [*] Ncell=2*Nx*Ny*Nz because FFT is complex GTX TB SP CIMEC-CONICET-UNL 9 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  10. 10. Advances in NS solver on GPU por M.Storti et.al. Upper bound on computing rate for the NS solver w/FFT • For instance for a mesh of N = 2563 = 16Mcell the rate is 4.66 Gcell/s (SP). • So that one could perform one Poisson solution (two FFTs) in 4.2ms (doesn’t take into account the computation of the RHS, which is negligible). • Poisson eq. is a large component of the computing time in the Fractional Step Solver for Navier-Stokes (and the only one that is > O(N)) so that this puts an upper limit on the computing rate achievable with our algorithm (based on FFT). CIMEC-CONICET-UNL 10 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  11. 11. Advances in NS solver on GPU por M.Storti et.al. Solution of NS on embedded geometries: FVM/IBM1 • When solid bodies are included in the fluid, we have to deal with curved surfaces that in general do not fit with node planes. • The first approach is to fit the body surface by a stair-case one (a.k.a. Lego surface). • Convergence is degraded to 1st order. • Stair-case acts like an artificial roughness so it tends to give higher drag. solid body fluid CIMEC-CONICET-UNL 11 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  12. 12. Advances in NS solver on GPU por M.Storti et.al. Solving Poisson with embedded bodies • FFT solver and GMG are very fast but have several restrictions: invariance of translation, periodic boundary conditions. • No holes (solid bodies) in the fluid domain. • So they are not well suited for direct use on embedded geometries. • One approach for the solution is the IOP (Iterated Orthogonal Projection) algorithm. • It is based on solving iteratively the Poisson eq. on the whole domain (fluid+solid). Solving in the whole domain is fast, because algorithms like Geometric Multigrid or FFT can be used. Also, they are very efficient running on GPU’s . CIMEC-CONICET-UNL 12 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  13. 13. Advances in NS solver on GPU por M.Storti et.al. The IOP (Iterated Orthogonal Projection) method The method is based on succesively solve for the incompressibility condition (on the whole domain: solid+fluid), and impose the boundary condition. on the whole domain (fluid+solid) violates impenetrability b.c. satisfies impenetrability b.c. CIMEC-CONICET-UNL 13 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  14. 14. Advances in NS solver on GPU por M.Storti et.al. The IOP (Iterated Orthogonal Projection) method (cont.) • Fixed point iteration wk+1 = ΠbdyΠdivwk . • Projection on the space of divergence-free velocity fields: u = Πdiv(u) u = u − P, ∆P = · u, • Projection on the space of velocity fields that satisfy the impenetrability boundary condition u = Πbdy(u ) u = ubdy, in Ωbdy, u = u , in Ωfluid. CIMEC-CONICET-UNL 14 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  15. 15. Advances in NS solver on GPU por M.Storti et.al. Accelerated Global Preconditioning (AGP) algorithm • Storti et.al. ”A FFT preconditioning technique for the solution of incompressible flow on GPUs.” Computers & Fluids 74 (2013): 44-57. • It shares some features with IOP scheme. • Both solvers are based on the fact that an efficient preconditioning exists. The preco. consists in solving the Poisson problem on the global domain (fluid+solid, i.e. no “holes” in the fluid domain). • IOP is a stationary method and its limit rate of convergence doesn’t depend on refinement. • However id does depend on (O(AR)) where AR is the maximum aspect ratio of bodies inside the fluid. I.e. the method is bad when very slender bodies are present. • Our proposed method AGP is a preconditioned Krylov space method. • The condition number of the preconditioned system doesn’t depend on refinement but deteriorates with the AR of immersed bodies. • IOP iterates over both the velocity and pressure fields, whereas AGP iterates only on the pressure vector (which is better for implementation on GPU’s). CIMEC-CONICET-UNL 15 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  16. 16. Advances in NS solver on GPU por M.Storti et.al. MOC+BFECC • The Method of Characteristics (MOC) is a Lagrangian method. The position of the node (“particle”) is tracked backwards in time and the values at the previous time step are interpolated into that position. These projections introduce dissipation. • BFECC Back and Forth Error Compensation and Correction can fix this. • Assume we have a low order (dissipative) operator (may be MOC, full upwind, or any other) Φt+∆t = L(Φt , u). • BFECC allows to eliminate the dissipation error. Advance forward the state Φt+∆t,∗ = L(Φt , u). Advance backwards the state Φt,∗ = L(Φt+∆t,∗ , −u). CIMEC-CONICET-UNL 16 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  17. 17. Advances in NS solver on GPU por M.Storti et.al. MOC+BFECC (cont.) exact w/dissipation error • If L introduces some dissipative error , then Φt,∗ = Φt , in fact Φt,∗ = Φt + 2 . • So that we can compensate for the error: Φt+∆t = L(Φt , ∆t) − , = Φt+∆t,∗ − 1/2(Φt,∗ − Φt ) (2) CIMEC-CONICET-UNL 17 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  18. 18. Advances in NS solver on GPU por M.Storti et.al. MOC+BFECC (cont.) Nbr of Cells QUICK-SP BFECC-SP QUICK-DP BFECC-DP 64 × 64 × 64 29.09 12.38 15.9 5.23 128 × 128 × 128 75.74 18.00 28.6 7.29 192 × 192 × 192 78.32 17.81 30.3 7.52 Cubic cavity. Computing rates for the whole NS solver (one step) in [Mcell/sec] obtained with the BFECC and QUICK algorithms on a NVIDIA GTX 580. 3 Poisson iterations were used. Costarelli, et.al. ”GPGPU implementation of the BFECC algorithm for pure advection equations.” Cluster Computing 17(2) (2014): 243-254. CIMEC-CONICET-UNL 18 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  19. 19. Advances in NS solver on GPU por M.Storti et.al. Analysis of performance • Regarding the performance results shown above, it can be seen that the computing rate of QUICK is at most 4x faster than that of BFECC. So BFECC is more efficient than QUICK whenever used with CFL > 2, being the critical CFL for QUICK 0.5. The CFL used in our simulations is typically CFL ≈ 5 and, thus, at this CFL the BFECC version runs 2.5 times faster than the QUICK version. • Effective computing rate is more representative than the plain computing rate effective computing rate = CFLcrit · (number of cells) (elapsed time) • The speedup of MOC+BFECC versus QUICK increases with the number of Poisson iterations. In the limit of very large number of iters (very low tolerance in the tolerance for Poisson) we expect a speedup 10x (equal to the CFL ratio). CIMEC-CONICET-UNL 19 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  20. 20. Advances in NS solver on GPU por M.Storti et.al. Validation. A buoyant fully submerged sphere • A fluid structure interaction problem has been used for validation. • Fluid is water ρfluid = 1000 [kg/m3 ]. • The body is a smooth sphere D =10cm, fully submerged in a box of L =39cm. Solid density is ρbdy = 366 [kg/m3 ] ≈ 0.4 ρfluid so the body has positive buoyancy. The buoy (sphere) is tied by a string to an anchor point at the center of the bottom side of the box. • The box is subject to horizontal harmonic oscillations xbox = Abox sin(ωt), CIMEC-CONICET-UNL 20 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  21. 21. Advances in NS solver on GPU por M.Storti et.al. Spring-mass-damper oscillator • The acceleration of the box creates a driving force on the sphere. Note that as the sphere is positively buoyant the driving force is in phase with the displacement. That means that if the box accelerates to the right, then the sphere tends to move in the same direction. (launch video p2-f09) • The buoyancy of the sphere produces a restoring force that tends to keep aligned the string with the vertical direction. • The mass of the sphere plus the added mass of the liquid gives inertia. • The drag of the fluid gives a damping effect. • So the system can be approximated as a (nonlinear) spring-mass-damper oscillator inertia mtot ¨x + damp 0.5ρflCdA|v| ˙x + spring (mflotg/L)x = driving frc mflotabox, mtot = mbdy + madd, mflot = md − mbdy, madd = Cmaddmd, Cmadd ≈ 0.5, md = ρflVbdy CIMEC-CONICET-UNL 21 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  22. 22. Advances in NS solver on GPU por M.Storti et.al. Experimental results • Experiments carried out atUniv Santiago de Chile (Dra. Marcela Cruchaga). • Plexiglas box mounted on a shake table, subject to oscillations of 2cm amplitude in freqs 0.5-1.5 Hz. • Sphere position is obtained using Motion Capture (MoCap) from high speed (HS) camera (800x800 pixels @ 120fps). HS cam is AOS Q-PRI, can do 3 MPixel @500fps, max frame rate is 100’000 fps. • MoCap is performed with a home-made code. Finds current position by minimization of a functional. Can also detect rotations. In this exp setup 1 pixel=0.5mm (D=200 px). Camera distortion and refraction have been evaluated to be negligible. MoCap can obtain subpixel resolution. (launch video mocap2) CIMEC-CONICET-UNL 22 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  23. 23. Advances in NS solver on GPU por M.Storti et.al. Experimental results (cont.) 0 20 40 60 80 100 120 140 160 180 0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 phase[deg] freq[Hz] anly(CD=0.3,Cmadd=0.55) exp 0 0.02 0.04 0.06 0.08 0.1 0.4 0.6 0.8 1 1.2 1.4 1.6 Amplitude[m] freq[Hz] anly(CD=0.3,Cmadd=0.55) exp Best fit of the experimental results with the simple spring-mass-damper oscillator are obtained for Cmadd = 0.55 and Cd = 0.25. Reference values are: • Cmadd = 0.5 (potential flow), Cmadd = 0.58 (Neill etal Amer J Phys 2013), • Cd = 0.47 (reference value). (launch video description-buoy-resoviz) CIMEC-CONICET-UNL 23 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  24. 24. Advances in NS solver on GPU por M.Storti et.al. abox driving force box desplacement buoy displacement (f<<f0) buoy displacement (f=f0) buoy displacement (f>>f0) f buoy (positive buoyancy) pendulum (negative buoyancy) abox driving force box desplacement buoy displacement (f<<f0) buoy displacement (f=f0) buoy displacement (f>>f0) f driving force = (md − mbdy)abox (3) CIMEC-CONICET-UNL 24 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  25. 25. Advances in NS solver on GPU por M.Storti et.al. Computational FSI model • The fluid structure interaction is performed with a partitioned Fluid Structure Interaction (FSI) technique. • Fluid is solved in step [tn , tn+1 = tn + ∆t] between the known position xn of the body at tn and an extrapolated (predicted) position xn+1,P at tn+1 . • Then fluid forces (pressure and viscous tractions) are computed and the rigid body equations are solved to compute the true position of the solid xn+1 . • The scheme is second order precision. It can be put as a fixed point iteration and iterated to convergence (enhancing stability). CIMEC-CONICET-UNL 25 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  26. 26. Advances in NS solver on GPU por M.Storti et.al. Computational FSI model (cont.) • The simulation is performed in a non-inertial frame attached to the box. So, according to the Galilean equivalence principle we have just to add an inertial force due to the acceleration of the box. • Due to the incompressibility this is absorbed in an horizontal flotation force on the sphere, which is the driving force for the system. • The rigid body eqs. are now: mbdy ¨x + (mflotg/L)x = Ffluid + mflotabox where Ffluid = Fdrag + Fadd are now computed with the CFD FVM/IBM1 code. CIMEC-CONICET-UNL 26 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  27. 27. Advances in NS solver on GPU por M.Storti et.al. Why use a positive buoyant body? • Gov. equations for the body: mbdy ¨x + (mflotg/L)x = comp w/CFD Ffluid({x}) +mflotabox where Ffluid = Fdrag + Fadd are fluid forces computed with the FVM/IBM code. • The most important and harder from the point of view of the numerical method is to compute the the fluid forces (drag and added mass). • Buoyancy is almost trivial and can be easily well captured of factored out from the simulation. • Then, if the body is too heavy it’s not affected by the fluid forces, so we prefer a body as lighter as possible. • In the present example mbdy = 0.192[kg], md = 0.523[kg], madd = 0.288[kg] CIMEC-CONICET-UNL 27 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  28. 28. Advances in NS solver on GPU por M.Storti et.al. Comparison of experimental and numerical results 0 0.02 0.04 0.06 0.08 0.1 0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 Amplitude[m] freq[Hz] experimental FVM/IBM1 0 50 100 150 200 0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3Phase[deg] freq[Hz] experimental FVM/IBM1 CIMEC-CONICET-UNL 28 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  29. 29. Advances in NS solver on GPU por M.Storti et.al. Comparison of experimental and numerical results 0 0.02 0.04 0.06 0.08 0.1 0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 Amplitude[m] freq[Hz] experimental FVM/IBM1 0 50 100 150 200 0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 Phase[deg] freq[Hz] experimental FVM/IBM1 • Results obtained with mesh 2563 = 16.7 Mcell. • Maximum error 22% at resonance (f = 0.9Hz). • Added mass an buoyancy cancel out at resonance and amplitude is almost controlled by drag. Hence drag is not well computed. • Numerical amplitude is lower than experimental =⇒ numerical drag is higher than experimental. • Probable cause: staircase representation of surface acts as artificial roughness and increases drag. • Resonance frequency seems correct =⇒ added mass and buoyancy forces are correct. CIMEC-CONICET-UNL 29 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  30. 30. Advances in NS solver on GPU por M.Storti et.al. Convergence of FVM/IBM1 • Plot shows amplitude vs mesh size at resonance (f = 0.9Hz). • The scheme is O(h) convergent. • Need for a O(h2 ) scheme with computational efficiency comparable to the 1st order one (FVM/IBM1). 0 0.001 0.002 0.003 0.004 0.005 4 5 6 7 8 9 10 mesh size h [m] Abuoy[m] FVM/IBM1 experimental 256^3 = 16.7 Mcell 192^3 = 7.1 Mcell 128^3 = 2.1 Mcell 96^3 = 0.9 Mcell actual Lego sphere with O(40^3) blocks CIMEC-CONICET-UNL 30 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  31. 31. Advances in NS solver on GPU por M.Storti et.al. Immersed boundary second order scheme (FVM/IBM2) • Fadlun, et al. ”Combined immersed-boundary...” JCP (2000) • Hard to combine with staggered grids and Fractional Step schemes. • So switched to Rhie-Chow interpolation for using co-located grids (i.e. avoiding staggered grids) • Switch to SIMPLE algorithm for iterating the pressure-velocity coupling. • Bodies are represented by level set functions. This allows for easy computation of boundary operators without branch-divergence. CIMEC-CONICET-UNL 31 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  32. 32. Advances in NS solver on GPU por M.Storti et.al. Immersed boundary second order scheme (FVM/IBM2) (cont.) 1: Data initialization: u0 , p0 , q0 , ˙q0 2: for n = 0, Nstep do 3: u∗ ← un , p∗ ← pn 4: for k = 0, Nsimple do 5: w ← u∗ , f ← 0 6: for l = 0, NIBM do 7: w = Gp∗ + R(w) + f; ¯w = ΠBC(w, qn S , ˙qn S ) 8: f = ¯w − Gp∗ − R(w), f = f − f / fl=0 9: f = f , w = ¯w 10: If f < tolf break 11: end for 12: u ← ¯w 13: Solve A(p − p∗ ) = A p∗ + Du for p 14: u = u − u∗ / u − un 15: u∗ ← αuu + (1 − αu)u∗ ; p∗ ← p∗ + αpp 16: If u < tolu break 17: end for 18: un+1 ← u∗ , pn+1 ← p∗ 19: Compute the tractions of the fluid onto the solid tfl 20: Compute the new solid position qn+1 S and velocity ˙qn+1 S 21: end for CIMEC-CONICET-UNL 32 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  33. 33. Advances in NS solver on GPU por M.Storti et.al. The boundary projection operator • Given a solid cell node S we have to determine the fictitious (a.k.a. parasitic) velocity uf S such that it guarantees the satisfaction of the boundary condition at B • The closest point xB to S is determined as xS = xB − dnB. • d and nB can be determined from the level set function ψ nB = ( ψ/ ψ )B ≈ ( ψ/ ψ )S , d = ψS, • Mirror fluid point to S is xF = xB + dnS • Velocity at mirror point is determined from extrapolation uf S = 2uB − uF (*) • Forcing at S is iteratively found such that (*) is satisfied. Solid cell center Fluid cell center ψ<0 (solid) ψ>0 (fluid) ψ=0 S B nB ^ F d d CIMEC-CONICET-UNL 33 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  34. 34. Advances in NS solver on GPU por M.Storti et.al. Numerical results with FVM/IBM2 0.4 0.6 0.8 1 1.2 1.4 1.6 0 0.02 0.04 0.06 0.08 0.1 f [Hz] Ab[m] experimental [m] FVM/IBM1[m] FVM/IBM2[m] experimental FVM/IBM1 256^3 = 16.7 Mcell FVM/IBM2 192^3 = 7.1 Mcell 0.4 0.6 0.8 1 1.2 1.4 1.6 0 50 100 150 f [Hz] phi[deg] experimental [m] FVM/IBM1[m] FVM/IBM2[m] experimental FVM/IBM1 256^3 = 16.7 Mcell FVM/IBM2 192^3 = 7.1 Mcell CIMEC-CONICET-UNL 34 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  35. 35. Advances in NS solver on GPU por M.Storti et.al. Convergence of FVM/IBM2 • FVM/IBM2 exhibits O(h2 ) convergence. 0 0.001 0.002 0.003 0.004 0.005 4 5 6 7 8 9 10 mesh size h [m] FVM/IBM1 FVM/IBM2 experimental Abuoy[m] CIMEC-CONICET-UNL 35 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  36. 36. Advances in NS solver on GPU por M.Storti et.al. Convergence of FVM/IBM2 (cont.) • Results for FVM/IBM2 with 1923 are much better than FVM/IBM1 with a finer mesh 2563 . • Time step for FVM/IBM1 was chosen so that CFL = 0.2 (CFLcrit = 0.5, QUICK version). • Tolerance for Poisson 10−6 . • Results are almost converged for NIBM = 3, Nsimple = 5. • For FVM/IBM2 we used tolf = 1 × 10−3 , tolu = 2 × 10−3 (typically NIBM = 1, Nsimple = 3). CFL = 0.1 (∆t = 2 × 10−4 s). (Remember CFLcrit = 0.5 − 0.6). • Results have been fitted with the analytic model giving Cd,num = 0.23, experimental= 0.25 (reference value for sphere drag is Cd = 0.47). • Added mass coefficient: FVM/IBM2: Cmadd = 0.58, experimental = 0.58, reference: 0.55, potential theory: 0.5. CIMEC-CONICET-UNL 36 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  37. 37. Advances in NS solver on GPU por M.Storti et.al. Computational efficiency of FVM/IBM2 on GPGPUs • Efficiency of the above algorithm has been measured on Nvidia Tesla K40c, and GTX Titan Black. • K40c has 2880 CUDA cores @876 MHz, 12 GiB ECC RAM. GK110 model (Kepler arch). (It’s equivalent with K40 but has builtin fansink). Cost: O(USD 1500). • GTX Titan Black has 2880 cores @980 GHz with 6 GiB RAM (no ECC). Cost: O(USD 3100) CIMEC-CONICET-UNL 37 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  38. 38. Advances in NS solver on GPU por M.Storti et.al. Tesla K40c vs GTX Titan Black GTX Titan Black 2880 cores @ 980MHz2880 cores @ 875MHz 235W 250W 12 GiB ECC 6 GiB (no ECC) 7.10e9 7.08e9 4.29 Tflop/s 5.12 Tflop/s 336 GB/s288 GB/s cores /Turbo speed dissipation device RAM # transistors memory bandwidth SP performance TESLA K40 +12% +7% 0.5x +19% = +17% cost (aprox.) USD 3,100 USD 1,500 0.5x CIMEC-CONICET-UNL 38 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  39. 39. Advances in NS solver on GPU por M.Storti et.al. Computational efficiency of FVM/IBM2 on GPGPUs 2e+06 4e+06 6e+06 8e+06 1e+07 1.2e+07 200 400 600 800 1000 1200 1400 Ncell computingrate[Mcell/s] GTX,SP,NSIMPLE=1 K40c,SP,NSIMPLE=1 GTX,DP,NSIMPLE=1 K40c,DP,NSIMPLE=1 K40c,SP,NSIMPLE=3 GTX,DP,NSIMPLE=3 K40c,DP,NSIMPLE=3 GTX,SP,NSIMPLE=3 CIMEC-CONICET-UNL 39 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  40. 40. Advances in NS solver on GPU por M.Storti et.al. Computing rates for FVM/IBM2 • Maximum attained rate is 1.3 Gcell/s for GTX Titan Black in SP, Nsimple = 1, mesh of 2243 = 11.2 Mcell. • Computing rates are monotonically increasing with refinement in all cases. • Comp. rates are roughly ∝ (NsimpleNIBM)−1 • Titan Black outperforms K40c for both SP and DP. • Remarkable good performance of GTX Titan Black specially in DP. 2e+06 4e+06 6e+06 8e+06 1e+07 1.2e+07 0 200 400 600 800 1000 1200 Ncell computingrate[Mcell/s] GTX,SP,NSIMPLE=1 K40c,SP,NSIMPLE=1 GTX,DP,NSIMPLE=1 K40c,DP,NSIMPLE=1 GTX,SP,NSIMPLE=3 K40c,SP,NSIMPLE=3 GTX,DP,NSIMPLE=3 K40c,DP,NSIMPLE=3 CIMEC-CONICET-UNL 40 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  41. 41. Advances in NS solver on GPU por M.Storti et.al. Computing rates for FVM/IBM2 (cont.) • For 1923 (the refinement used for the results presented above) we can compute 1s simulation in 30s of computation (ST:RT=1:30). • We can estimate un upper bound for the computing rate counting the transactions to the device RAM. • We have 66 transactions (floats/doubles) per cell. So the theoretical limit is limit rate (SP) = 336GB/s 66 · 4 Bytes/cell = 1.27 Gcell/s, limit rate (DP) = 336GB/s 66 · 8 Bytes/cell = 636 Mcell/s, (4) CIMEC-CONICET-UNL 41 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  42. 42. Advances in NS solver on GPU por M.Storti et.al. Time consumption breakdown CIMEC-CONICET-UNL 42 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  43. 43. Advances in NS solver on GPU por M.Storti et.al. Numerical results Vorticity isosurface velocity at sym plane tadpoles colored vort isosurfaces (launch video fsi-boya-gpu-vorticity), (launch video gpu-corte) (launch video fsi-vort-3D), (launch video particles-buoyp5) CIMEC-CONICET-UNL 43 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  44. 44. Advances in NS solver on GPU por M.Storti et.al. FVM/IBM vs. LBM on GPGPU • In LBM the Poisson equation is not solved; it uses pseudo-compressibility, hence LBM is a true Cellular Automata (CA) algorithm. (FVM/IBM could so too, of course ) • So comparatively it can reach higher computing rates, but suffers from the artificial Mach penalization effect. • Suppose you have a maximum velocity umax in the flow field. • If you use pseudo-compressibility then you must choose an artificial speed of sound cart > umax. • The compressibility effects are O(M2 art) (Mart = umax/cart) so you choose, let’s say, Mart = 0.1 so that the error is O(10−2 ). ∆t < CFLcrit · h/(umax + cart) = Mart/(1 + Mart) · h/umax ≈ Mart · h/umax, CFL < Mart · CFLcrit, • So your CFL is penalized by a factor Mart (0.1 in the example). CIMEC-CONICET-UNL 44 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  45. 45. Advances in NS solver on GPU por M.Storti et.al. FVM/IBM vs. LBM on GPGPU (cont.) • So in fact not only Mcell/s must be reported, but also the critical CFL for the algorithm. • So if the computing rate in Mcell/s is doubled but have a penalization of 0.5x in the CFL the algorithm breaks even. • Or rather the effective computing rate should be reported effective computing rate = CFLcrit · (number of cells) (elapsed time) • The presented algorithm has (experimentally determined) CFLcrit = 0.5 − 0.6. CIMEC-CONICET-UNL 45 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  46. 46. Advances in NS solver on GPU por M.Storti et.al. Future work • We are in the process of buying a HPC cluster (≈ 20 Tflops, w/o GPGPUs) with funds from Provincia de Santa Fe Science Agency ASACTEI (Thanks! ). Includes nodes w/GTX Titan Z (2x2880=5760 cores @876 MHz (boost), 12 GiB RAM, equiv to 2 Titan (not Black! ) on the same card). We estimate reaching 2 Gcell/s on one card. • Replace momentum eqs. solver with MOC+BFECC. • Multi-GPU: (hybrid MPI+CUDA+OpenMP) Some experiments already done with Poisson eq. • Refinement with multi-block strategy. • LES Smagorinsky turbulence model. CIMEC-CONICET-UNL 46 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))
  47. 47. Advances in NS solver on GPU por M.Storti et.al. Acknowledgments • Chilean Council for Scientific and Technological Research (CONICYT-FONDECYT 1130278); • The Scientific Research Projects Management Department of the Vice Presidency of Research, Development and Innovation (DICYT-VRID) at Universidad de Santiago de Chile; • Association of Universities: Montevideo Group (AUGM); • Argentinean Council for Scientific Research (CONICET project PIP 112-20111-00978); • Argentinean National Agency for Technological and Scientific Promotion, ANPCyT, (grants PICT-E-2014-0191, PICT-2014-0191); • Universidad Nacional del Litoral, Argentina (grant CAI+D-501-201101-00233-LI); • Santa Fe Science Technology and Innovation Agency (grant ASACTEI-010-18-2014); and • European Research Council (ERC) Advanced Grant Real Time Computational Mechanics Techniques for Multi-Fluid Problems (REALTIME, Reference: ERC-2009-AdG). • We made extensive use of Free Software as GNU/Linux OS, GCC/G++ compilers, Octave, and Open Source software as VTK, ParaView, among many others. CIMEC-CONICET-UNL 47 ((version texstuff-1.2.8-4-g7cfd85f Mon Apr 27 23:01:25 2015 -0300) (date Thu Apr 30 12:04:10 2015 -0300))

×