Acceleration of OpenFOAM with SpeedIT 2.1

Acceleration of OpenFOAM with SpeedIT 2.1
Comparison to GAMG and DIC preconditioners

Vratis Ltd., Wroclaw, Poland
April 5, 2012

1. Objective

OpenFOAM® simulations take a significant amount of time leading to higher costs of simulations. GPGPU technology has a potential to overcome this problem. As a solution of this problem we propose to use SpeedIT technology that replaces iterative solvers in OpenFOAM with their GPU-accelerated versions. In following tests we accelerate calculation of pressure equation that usually takes most of the time in simulations of incompressible flows. We compare the performance of OpenFOAM & SpeedIT run on GPU to standard OpenFOAM on CPU using various preconditioners on a typical PC equipped with NVIDIA GPU card that is  CUDA compatible. This report is also used to present a new version of SpeedIT 2.1 that contains a new set of preconditioners.

2. Methodology

SpeedIT is a library which implements set of accelerated solvers with various preconditioners. Thanks to CUSP library in SpeedIT 2.1 we were able to utilize algebraic multigrid preconditioner with smoothed aggregation (AMG) . This preconditioner significally reduces number of iterations during the pressure calculation which imply shorter time of calculation. SpeedIT Plugin to OpenFOAM® was used to substitute OpenFOAM’s iterative solvers with the one provided by SpeedIT. Tests were performed on following machines:

A) CPU: Intel Core 2 Duo E8400, 3GHz, 8GB RAM @ 800MHz
GPU: Nvidia GTX 460, VRAM 1GB
Software: Ubuntu 11.04 x64, OpenFOAM 2.0.1, CUDA Toolkit 4.1
B) CPU: Intel Q8400, 2,66GHz, 8GB RAM @ 800MHz
GPU: Nvidia Tesla C2070, VRAM: 6GB.
Software: Ubuntu 11.04 x64, OpenFOAM 2.0.1, CUDA Toolkit 4.1

To solve pressure equation with OpenFOAM on CPU either GAMG solver or CG with DIC preconditioner was used for different number of cores. On GPU SpeedIT was run together with AMG preconditioner. We have tested following test cases for fixed number of time steps.

  1. Cavity 3D 512K cells, icoFoam, on 1 and 2 Cores with PCG solver and DIC preconditioner, GAMG solver, FDIC preconditioner, Gauss-Seidel smoother, and SpeedIT 2.1 with AMG preconditioners.
Picture 1. Cavity 3D, velocity streamlines
  1. Aorta 200K cells, simpleFoam, on 1 and 2 Cores with PCG solver and DIC preconditioner, GAMG solver, FDIC preconditioner, Gauss-Seidel smoother, and SpeedIT 2.1 with AMG preconditioner.
Picture 2. Aorta, velocity streamlines
  1. Ahmed case with 2.5M cells simulated with original simpleFoam, on 1, 2, 3 and 4 Cores with GAMG solver, Gauss-Seidel smoother and SpeedIT with AMG preconditioner.
    Picture 3. Ahmed 25º, velocity streamlines and pressure field.

Cases 1 and 2 were executed on machine A, and case 3 on machine B.

3. Validation

To validate our solution we have ploted pressure field along x axis for cases 1 and 2. From Figs. 1-3 it is quite clear that solutions are correct for simulations with different preconditioners.

Figure 1. Cavity 3D cross section along x axis. Solution for all preconditioners.
Figure 2. Aorta cross section along x axis. Solution for all preconditioners
Figure 3. Aorta cross section along x axis. Solution for all preconditioners.

4. Results
Cavity 3D

Figure 4. Execution time of Cavity 3D case for different preconditoners and number of cores.
Figure 5. Mean number of iterations for GAMG, AMG and DIC preconditioner during pressure calculations.
Figure 6. Acceleration defined as a ratio SpeedIT vs CPU with different preconditioners

Aorta

Figure 7. Execution time of Aorta case for different preconditoners and number of cores
Figure 8. Mean number of iterations for GAMG, SpeedIT with AMG and DIC preconditioner during pressure calculations
Figure 9. Acceleration defined as a ratio GPU (SpeedIT) vs. CPU with different preconditioners.

Ahmed 25º

Figure 10. Execution time for Ahmed case for GPU with AMG preconditioner and different number of cores with GAMG solver

Figs. 1-3 prove that SpeedIT leads to the same solution as OpenFOAM. SpeedIT new AMG preconditioner can be competitive with OpenFOAM GAMG preconditioner working on 1 or 2 core CPU.  The main advantage of the AMG solver is that significantly reduces number of iterations when solving the pressure equation. Comparing to widely used DIC preconditioner SpeedIT 2.1 gives about 10 time less iterations (Fig. 5, and Fig 8 ) which in effect gives a speedup up to 3.5x. What was interesting we found that GAMG is failing when calculations are performed in single precision while AMG is still functioning. Fig. 11 presents the mean number of iterations for the Cavity3D case in single precision. GAMG solver gives as much as 1000 of iterations during pressure field calculations.

Figure 11. Mean number of iterations for Cavity 3D case in single precision.

5. Acknowledgments

We would like to thank NVIDIA for hardware support and 4-ID network for providing the Ahmed test case. Ahmed test case was based on Motorbike tutorial from OpenFOAM 2.0. We also acknowledge Dominik Szczerba from IT’IS Foundation for providing the geometry of the human aorta.