How to run SpeedIT with OpenFOAM?

Introduction

SpeedIT plugin for OpenFOAM is a set of libraries which allows you to accelerate OpenFOAM on GPU. SpeedIT will release the computational power dreaming in NVIDIA Graphics Processing Unit (GPU) that supports CUDA technology. The SpeedIT library provides a set of accelerated solvers and functions for sparse linear systems of equations which are:

  • Preconditioned Conjugate Gradient
  • Preconditioned Stabilized Bi-Conjugate Gradient
  • Accelerated Sparse Matrix-Vector Multiplication
  • Diagonal Preconditioner
  • Algebraic Multigrid (AMG) based on Smoothed Aggregation
  • Approximate Inverse (AINV)

Requirements

Software dependencies

OpenFOAM

OpenFOAM is an environment where SpeedIT plugin operates. OpenFOAM can be downloaded from http://www.openfoam.com/download/. Install OpenFOAM by following the instructions on the OpenFOAM page.

IMPORTANT: Make sure you have done step which sets the OpenFOAM environment variables

CUDA

Download CUDA library from http://developer.nvidia.com/cuda-downloads and install it. Add CUDA include directory to your PATH variable:

PATH=$PATH:/path/to/cuda/include

Depends on your system (32/64-bit) add CUDA lib or lib64 to LD_LIBRARY_PATH i.e:

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/cuda/lib:/path/to/cuda/lib64

SpeedIT

OpenFOAM plugin requires SpeedIT to work. SpeedIT is available commercially and SpeedIT Classic with limited functionality can be download at no cost.  SpeedIT can be downloaded from http://speedit.vratis.com

Cuwrap

Cuwrap is an intermediate library which achieve compatibility between CUDA and OpenFOAM interfaces. It is distributed with the OpenFOAM plugin. You can find it in the folder cuwrap. It is necessary to build this library if you want to use SpeedIT with OpenFoam.

To build this library first in cuwrap folder open Makefile file.

Depends on your configuration set proper paths to CUDA environment.
For 32-bit suystem and default CUDA installation the header of the file should looks following:

CUDA_HOME="/usr/local/cuda" 

CUDA_LIB="$CUDA_HOME/lib" 

NVIDIA_CURRENT="/usr/lib" 

CUDA_INC="$CUDA_HOME/include" 

CUDA_BIN="$CUDA_HOME/bin"

For 64-bit systems:

CUDA_HOME="/usr/local/cuda" 

CUDA_LIB="$CUDA_HOME/lib64" 

NVIDIA_CURRENT="/usr/lib64" 

CUDA_INC="$CUDA_HOME/include" 

CUDA_BIN="$CUDA_HOME/bin"

After setting paths from the cuwrap folder run make command. It shall build the library. If  OpenFOAM is configured properly the library should be created inside $FOAM_USER_LIBBIN folder.

Plugin Installation

  1. Create directory $HOME/OpenFOAM
  2. Create additional directories by typing:
    mkdir $WM_PROJECT_USER_DIR | mkdir $FOAM_RUN
  3. From the plugin directory

     run ./Allwmake COMMERCIAL
  4. If compilation is successfully completed you should have new file libexternalsolv.so and libcuwrap.so in $FOAM_USER_LIBBIN directory.

Plugin use

  1. Copy (or make symbolic links)following libraries to $FOAM_USER_LIBBIN directory:

  • libcublas.so

  • libcudart.so

  • libcuwrap.so

  • libspeedit.so

libcublas.so, libcudart.so, are from NVIDIA CUDA toolkit

libcuwrap.so is distributed with the plugin.

NOTE: Remember to use proper version of libraries depends on your system architecture. 32-bit library in 32-bit operating systems and 64-bit library for 64-bit operating systems.

    1. Go in to the directory with your OpenFOAM case, i.e. $FOAM_RUN/tutorials/incompressible/icoFoam/cavity
    2. Append

      libs
          (
            "libcudart.so"
            "libcublas.so"
            "libcuwrap.so"    
            "libspeedit.so”
            "libexternalsolv.so"
      );

      to the end of your system/controDict file for every FOAM case, for which you want to use external, accelerated solvers.

  • In file system/fvSolution change solver names for solvers, for which you are going to enable acceleration. Remember to use proper names for accelerated solvers. You may replace:

    PBiCG with SI_PBiCG
    PCG   with SI_PCG
  • For accelerated solvers choose an appropriate preconditioner in file system/fvSolution. You may use following preconditioners:

    1. SI_DIAGONAL – Diagonal preconditioner

    2. SI_AMG – Algebraic Multigrid preconditioner

    3. SI_AINV – Approximate Inverse preconditioner

    4. SI_AINV_SC – Approximete Inverse Scaled preconditioner

    5. SI_AINV_NS – Approximate Inverse Non-Symmetric preconditioner

  • When accelerated solvers are used you have to specify additional keyword “matrix” in solver definition. It can take 2 values CMR or CSR which stands for:

    1. CSR – Compressed Sparse Row format.

    2. CMR – Compressed Multi-Row Storage format (see our article for details)

When CSR is used, then all preconditioners mentioned in point 5 are allowed. When CMR matrix is used then only SI_DIAGONAL is working at the moment.

Run icoFOAM from $FOAM_RUN/tutorials/incompressible/icoFoam/cavity.

Accelerated solvers should be available from now.

Example of fvSolution:


/*——————————–*- C++ -*———————————-*\

 

| ========= | |

 

| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |

 

| \\ / O peration | Version: 1.7.1 |

 

| \\ / A nd | Web: www.OpenFOAM.com |

 

| \\/ M anipulation | |

 

\*—————————————————————————*/

 

FoamFile

 

{

 

version 2.0;

 

format ascii;

 

class dictionary;

location “system”;

object fvSolution;

}

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

solvers {

p  {

solver SI_PCG;

preconditioner SI_AMG;

matrix CSR;

tolerance 1e-06;

relTol 0;

}

{

solver SI_PBiCG;

preconditioner SI_DIAGONAL;

matrix CSR;

tolerance 1e-05;

relTol 0;

}

PISO  {

nCorrectors 2;

nNonOrthogonalCorrectors 0;

pRefCell 0;

pRefValue 0;

}

// ************************************************************************* //

Share

Leave a Reply