Setup
Requirements
Originally, GENGA was designed to run on NVIDIA GPUs. The current version supports also AMD GPUs, as well as parallel multicore CPU systems by using OpenMP.
The system requirements are:
Nvidia GPUs:
CUDA toolkit
GPU with compute capability of 3.0 or higher
AMD GPUs:
ROCm and HIP
python3, for translating the CUDA source code to HIP
multicore CPUs:
g++ compiler
OpenMP
python3, for translating the CUDA source code to HIP
Install CUDA
To be able to use the code on NVIDIA GPUs, one has to install the CUDA Toolkit first as described here.
We strongly recommend using a recent CUDA version to get the full performance and correct results.
If an old CUDA version is used (< CUDA 9.0) then the def_OldShuffle parameter in the define.h file must be set to 1.
(See Use old Shuffle)
Linux
nvidia-smi to check CUDA and the available GPUs.GCC version
It can happen that the used CUDA version needs an older GCC version than the current one on the system. In that case, either a newer CUDA version, or an older gcc version should be installed. Use the following compile option to tell CUDA to use an older GCC version (for example 7.0):
-ccbin=g++-7
Install ROCM and HIP for AMD GPUs
When an AMD GPU is used, then ROCM and HIP needs to be installed.
/opt/rocm/bin/hipconfig --full to check the installationrocm-smi to check the GPUexport HIP_PLATFORM=nvidia or export HIP_PLATFORM=amdDetermine the NVIDIA compute capability
GENGA must be compiled for a specific GPU compute capability. The compute capability corresponds to the GPU generation, a list of all NVIDIA GPUS with their compute capabilities can be found here: https://developer.nvidia.com/cuda-gpus .
The compute capability can also be checked with the provided tool CheckGPU:
Step 1: compile the CheckGPU code with:
nvcc -o CheckGPU CheckGPU.cu
Step 2: run:
./CheckGPU
This will list the compute capabilities of all found GPUs.
Compile GENGA
If an old CUDA version is used (< CUDA 9.0) then the def_OldShuffle parameter in the define.h file must be set to 1.
(See Use old Shuffle)
The source code of GENGA and a Makefile is included in the source directory. To compile GENGA, go to the source directory and type:
make SM=xx
into a terminal, where xx corresponds to the compute capability of the GPU (NVIDIA) or the target ID (AMD).
Use e.g. ‘make SM=60’ for compute capability of 6.0, or ‘make SM=65’ for compute capability of 6.5.
For example use:
make SM=35 for Tesla K20
make SM=52 for GeForce GTX 980
make SM=60 for Tesla P100
make SM=61 for GeForce GTX 1080 ti
make SM=75 for GeForce RTX 2080 ti
make SM=86 for GeForce RTX 3090
make SM=gfx906 for AMD Radeon VII
make SM=gfx90a for AMD Instinct MI200
When compiling GENGA with the openGL real time visualization, go to the GengaGL directory. (See GengaGL: Real time visualization with openGL)
When GENGA is compiled for a newer compute capability then the GPU is able to run, then the following error message will appear by running GENGA: FGAlloc error = 13 = invalid device symbol.
Compile GENGA with HIP for AMD GPUs
GENGA provides a tool to translate the source code from CUDA to HIP. The HIP version can run on AMD and on NVIDIA GPUs. The translation tool is located in the HIP directory.
Run:
python3 GengaHIP.py
to translate the code. GengaHIP will copy the translated source code to the HIP directory.
Type:
make
to compile GENGA with HIP.
Compile GENGA with OpenMP for multicore CPUs
GENGA provides a tool to translate the source code from CUDA to an OpenMP CPU version. The translation tool is located in the cpu directory.
Run:
python3 port.py
to translate the code. This will copy the translated source code to the cpu directory.
Type:
make
to compile GENGA with OpenMP.
On systems with hyperthreading enabled, it can be usefull to select the desired CPU cores to run on. This can be done by typing:
export OMP_PLACES="{0,1,2,3, ...}"
to the terminal, before running the code, were the numbers indicate all the core id’s that should be used.
Compile GENGA on Windows
If using Cygwin on Windows, then GENGA can be compiled the same way as in Linux with:
make SM=xx.
If using the Windows Command Prompt, type:
nmake -f Makefile.win SM=xx.
Note, that the Windows C++ compiler cl
must be installed, and the compiler path must be loaded in the command prompt. If
this is not the case, it can be loaded similar to this command:
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\Tools\vsdevcmd.bat"
or:
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat"
, where the exact path and file name must eventually be changed.