Where CFD meets GPU

Acceleration of OpenFOAM with SpeedIT 2.1

marta — Tue, 24 Apr 2012 09:18:55 +0000

Acceleration of OpenFOAM with SpeedIT 2.1
Comparison to GAMG and DIC preconditioners

Vratis Ltd., Wroclaw, Poland
April 5, 2012

1. Objective

OpenFOAM® simulations take a significant amount of time leading to higher costs of simulations. GPGPU technology has a potential to overcome this problem. As a solution of this problem we propose to use SpeedIT technology that replaces iterative solvers in OpenFOAM with their GPU-accelerated versions. In following tests we accelerate calculation of pressure equation that usually takes most of the time in simulations of incompressible flows. We compare the performance of OpenFOAM & SpeedIT run on GPU to standard OpenFOAM on CPU using various preconditioners on a typical PC equipped with NVIDIA GPU card that is CUDA compatible. This report is also used to present a new version of SpeedIT 2.1 that contains a new set of preconditioners.

2. Methodology

SpeedIT is a library which implements set of accelerated solvers with various preconditioners. Thanks to CUSP library in SpeedIT 2.1 we were able to utilize algebraic multigrid preconditioner with smoothed aggregation (AMG) . This preconditioner significally reduces number of iterations during the pressure calculation which imply shorter time of calculation. SpeedIT Plugin to OpenFOAM® was used to substitute OpenFOAM’s iterative solvers with the one provided by SpeedIT. Tests were performed on following machines:

A) CPU: Intel Core 2 Duo E8400, 3GHz, 8GB RAM @ 800MHz
GPU: Nvidia GTX 460, VRAM 1GB
Software: Ubuntu 11.04 x64, OpenFOAM 2.0.1, CUDA Toolkit 4.1
B) CPU: Intel Q8400, 2,66GHz, 8GB RAM @ 800MHz
GPU: Nvidia Tesla C2070, VRAM: 6GB.
Software: Ubuntu 11.04 x64, OpenFOAM 2.0.1, CUDA Toolkit 4.1

To solve pressure equation with OpenFOAM on CPU either GAMG solver or CG with DIC preconditioner was used for different number of cores. On GPU SpeedIT was run together with AMG preconditioner. We have tested following test cases for fixed number of time steps.

Cavity 3D 512K cells, icoFoam, on 1 and 2 Cores with PCG solver and DIC preconditioner, GAMG solver, FDIC preconditioner, Gauss-Seidel smoother, and SpeedIT 2.1 with AMG preconditioners.

Picture 1. Cavity 3D, velocity streamlines

Aorta 200K cells, simpleFoam, on 1 and 2 Cores with PCG solver and DIC preconditioner, GAMG solver, FDIC preconditioner, Gauss-Seidel smoother, and SpeedIT 2.1 with AMG preconditioner.

: Picture 2. Aorta, velocity streamlines

Ahmed case with 2.5M cells simulated with original simpleFoam, on 1, 2, 3 and 4 Cores with GAMG solver, Gauss-Seidel smoother and SpeedIT with AMG preconditioner.

Picture 3. Ahmed 25º, velocity streamlines and pressure field.

Cases 1 and 2 were executed on machine A, and case 3 on machine B.

3. Validation

To validate our solution we have ploted pressure field along x axis for cases 1 and 2. From Figs. 1-3 it is quite clear that solutions are correct for simulations with different preconditioners.

: Figure 1. Cavity 3D cross section along x axis. Solution for all preconditioners.

: Figure 2. Aorta cross section along x axis. Solution for all preconditioners

Figure 3. Aorta cross section along x axis. Solution for all preconditioners.

4. Results
Cavity 3D

Figure 4. Execution time of Cavity 3D case for different preconditoners and number of cores.

: Figure 5. Mean number of iterations for GAMG, AMG and DIC preconditioner during pressure calculations.

Figure 6. Acceleration defined as a ratio SpeedIT vs CPU with different preconditioners

Aorta

Figure 7. Execution time of Aorta case for different preconditoners and number of cores

Figure 8. Mean number of iterations for GAMG, SpeedIT with AMG and DIC preconditioner during pressure calculations

Figure 9. Acceleration defined as a ratio GPU (SpeedIT) vs. CPU with different preconditioners.

Ahmed 25º

Figure 10. Execution time for Ahmed case for GPU with AMG preconditioner and different number of cores with GAMG solver

Figs. 1-3 prove that SpeedIT leads to the same solution as OpenFOAM. SpeedIT new AMG preconditioner can be competitive with OpenFOAM GAMG preconditioner working on 1 or 2 core CPU. The main advantage of the AMG solver is that significantly reduces number of iterations when solving the pressure equation. Comparing to widely used DIC preconditioner SpeedIT 2.1 gives about 10 time less iterations (Fig. 5, and Fig 8 ) which in effect gives a speedup up to 3.5x. What was interesting we found that GAMG is failing when calculations are performed in single precision while AMG is still functioning. Fig. 11 presents the mean number of iterations for the Cavity3D case in single precision. GAMG solver gives as much as 1000 of iterations during pressure field calculations.

Figure 11. Mean number of iterations for Cavity 3D case in single precision.

5. Acknowledgments

We would like to thank NVIDIA for hardware support and 4-ID network for providing the Ahmed test case. Ahmed test case was based on Motorbike tutorial from OpenFOAM 2.0. We also acknowledge Dominik Szczerba from IT’IS Foundation for providing the geometry of the human aorta.

Multi-GPU simulation of the motorbike in OpenFOAM and SpeedIT technology

marta — Tue, 17 Apr 2012 08:09:03 +0000

Multi-GPU simulations of the motorbike in OpenFOAM with SpeedIT technology

Vratis Ltd., Wroclaw, Poland

March 28, 2012

1. Objective

OpenFOAM® simulations take a significant amount of time leading to higher costs of simulations. GPGPU technology has a potential to overcome this problem. However, due to a limited memory of a single GPU card, realistic simulations may be not possible. As a solution to this problem we propose to use a SpeedIT Multi-GPU technology where we accelerate calculation of pressure equation, which usually takes most of the time in simulations of incompressible flows. We compare the performance of SpeedIT Multi-GPU to standard OpenFOAM runs on CPU in various test scenarios for up to 32 millions cells on clusters with up to 16 GPU cards.

2. Methodology

SpeedIT is a library that implements iterative solvers on GPU using MPI to exchange data between domains. SpeedIT Plugin to OpenFOAM® was used to call GPU-accelerated iterative solvers in OpenFOAM which was responsible for decomposition of the case. Preliminary tests (see the report) for cavity3D performed at PLGRID cluster with varying number of cells showed (see Fig.1-2) that technology has a potential in reducing the simulation time. The tests performed at CINECA cluster aimed at solving larger simulations, with geometries up to 80M cells as well as testing a more efficient preconditioners, such as AMG. CINECA PLX cluster was equipped with 548 Intel Xeon E5645 and 548 Tesla M2070 cards with 6GB memory and 448 CUDA cores. Following test was performed in both multi-CPU and multi-GPU environment for a fixed number of time steps:

80M case, a ramp, simpleFoam.
32M cells motorBike test, simpleFoam.

: Figure 1: Acceleration defined as a ratio nGPU vs. nCPU for different cavity3D runs with icoFoam and diagonal preconditioner.

Figure 2: Acceleration defined as a ratio nGPU vs. nCPU for AhmedBody and Cabin runs with simpleFoam.

Results

Tests performed on CINECA cluster were inspired by industry. First one was delivered by one of government agencies. It was a ramp and had 80M cells. We used PLX cluster to decompose the case and mesh it. Unfortunately, due to technical issues we were not able to run the simulations yet.

Second test was a standard OpenFOAM test, called motorBike modified by SGI so that it had 32 million cells. Test were performed in multi-GPU and multi-CPU environment. For the time being SpeedIT library can offer CG solver with diagonal preconditioner for the solution of the pressure equation on GPU. For the computations on CPU we also used the CG solver with diagonal preconditioner. We also used GAMG solver since it is mostly used in real life simulations. Results are presented in Fig. 3. As one can see computation on GPUs can be up to 8 times faster comparing to computations on the same number of processor cores. When compared against GAMG solver SpeedIT multi-GPU can also provide acceleration of factor x1.1-x1.4 (without the first time step, the acceleration was x1.5).

Figure 3: Acceleration defined as a ratio nGPU vs. nCPU for motorBike against diagonal preconditioner and CG or GAMG solvers used to solve the pressure equation.

Acknowledgments

We kindly acknowledge NVIDIA and CINECA for the support in performing the simulations as well as SGI for providing the test case.

Disclaimer

This offering is not approved or endorsed by OpenCFD Limited, the producer of the OpenFOAM software and owner of the OPENFOAM® and OpenCFD® trade marks (see the Disclaimer).
The views and statements expressed in this blog are of Vratis Ltd. and are not necessarily the views of or endorsement by 3rd parties named in this activity.
OPENFOAM® is a registered trade mark of OpenCFD Limited, the producer of the OpenFOAM software.

Performance of SpMV in CUSPARSE, CUSP and SpeedIT

lmiroslaw — Sat, 12 Nov 2011 20:30:51 +0000

Introduction

Sparse Matrix-Vector multiplication (SpMV) is one of BLAS operations that are often used in scientific calculations. In order to show that SpeedIT belongs to the fastest implementations of this routine we have tested SpMV on 23 randomly chosen matrices from University Florida Matrix Collection. Their properties are described in Tab.1. Tab 2 and Tab.3 present time of SpMV in single and double precision while Figs.1-8 present the results in a graphical form. Since the performance is strongly affected by the matrix size we have divided them into two groups: small and large matrices. The tests were performed on a Tesla C2050 GPU card from NVIDIA.

SpeedIT is available in two formats. CSR and a proprietary CMR format, either of which can be easily chosen by the user.

Fig.1. Average time of SPMV in Single Precision for small matrices. Fig.1. Time of SpMV in Single Precision for small matrices. Resulting time is an average from 1000 runs.

Fig.2. Average time of SPMV in Single Precision for large matrices.

Fig.2. Time of SpMV in Single Precision for large matrices. Resulting time is an average from 1000 runs.

Fig.3. Average time of SPMV in Double Precision for small matrices.

Fig.4. Time of SpMV in Double Precision for small matrices. Resulting time is an average from 1000 runs.

Fig.4. Average time of SPMV in Double Precision for large matrices.Fig.4. Time of SpMV in Double Precision for large matrices. Resulting time is an average from 1000 runs.

Fig. 5. Speed-up of SpeedIT CMR vs. CUSPARSE and CUSP in Single Precision.Fig.5. Speed-up of SpeedIT CMR in Single Precision vs. CUSPARSE and CUSP.

Fig. 6. Speed-up of SpeedIT CSR vs. CUSPARSE and CUSP in Single Precision. Fig.6. Speed-up of SpeedIT CSR in Single Precision vs. CUSPARSE and CUSP.

Fig. 7. Speed-up of SpeedIT CMR vs. CUSPARSE and CUSP in Double Precision.Fig.7. Speed-up of SpeedIT CMR in Double Precision vs. CUSPARSE and CUSP.

Fig.8. Speed-up of SpeedIT CSR vs. CUSPARSE and CUSP in Double Precision. Fig.8. Speed-up of SpeedIT CSR in Double Precision vs. CUSPARSE and CUSP.

Conclusions

The highest speed-up of SpMV implemented in SpeedIT CMR vs. CUSPARSE is about 2x while vs. CUSP is more than 4x.
The highest speed-up of SpMV implemented in SpeedIT CSR against and CUSP is about 1.4x.
SpeedIT performs better for large matrices ( > 100 000 NNZ) and CMR format is more efficient.

Appendix

Tab 1. Description of matrix properties used in SpMV tests. NNZ and NZ correspond to the number of non-zero and zero elements. Small matrices are depicted in green. Remaining matrices are termed large in the following tests.