May 13, 2022
Daniel Lindblad
Community Blog
No comment

Nektar++ on the JUWELS supercomputer

In this post we will explain how to compile and run Nektar++ on the JUWELS supercomputer. The JUWELS supercomputer consists of two modules, the cluster module and the booster module. The booster module contains Nvidia GPU accelerators, and is not going to be covered in this post. Instead, we will focus on the booster module, which has the following specifications:

  • 2271 Standard compute nodes
    • 2x Intel Xeon Platinum 8168 Skylake CPUs per node (24 cores per CPU)
    • 96GB of DDR4 RAM per node
  • 240 Large memory compute nodes
    • 2x Intel Xeon Platinum 8168 Skylake CPUs per node (24 cores per CPU)
    • 192GB of DDR4 RAM per node

More details about the hardware can be found in the official documentation https://apps.fz-juelich.de/jsc/hps/juwels/index.html

Access

Access to the JUWELS supercomputer is managed through the JuDoor portal: https://judoor.fz-juelich.de/login. Please note that access to the JUWELS supercomputer is very restricted. In particular, you must first be part of a project that has been allocated compute time on JUWLES. After this, you must upload a public ssh key to JuDoor. Finally, your computer must have a static IP address, which you must specify on JuDoor. All connections to JUWELS which are not coming from a known and static IP address will be rejected. Once everything has been set up, you can use ssh to connect to JUWELS 

$$ ssh <username>@juwels-cluster.fz-juelich.de

This will take you to one of the login nodes, which can be used for compiling Nektar++ and to submit jobs. Note, however, that you should never run any parallel or compute intensive jobs on the login nodes.

For more details on how to access JUWELS, please refer to the official documentation https://apps.fz-juelich.de/jsc/hps/juwels/access.html#ssh-login.

File system

When you use ssh to access JUWELS, you will get access to one of the login-nodes. To begin with, you will be located in your home directory. If you need to go back to your home directory at any point, you can type

$$ cd $HOME

Here, we used the environment variable $HOME, which contains the address of your home directory. The home directory should only be used to store small configuration files (such as personal bash settings). All remaining files, including the Nektar++ source files, should be stored under $PROJECT_<project_id>/<username>. Here, <project_id> is the name of the project through which you got access to JUWELS and <username> is the same name that you used to log on to JUWELS. You can find both these names on JuDoor.

Compiling Nektar++

Before compiling Nektar++, we need to load the right modules. JUWELS uses a hierarchical organisation of modules. This means that the list of available modules depends on which modules that were loaded before. At the top of this “module tree” is the Stages/20** module. When compiling Nektar++, we will use the Stages/2020 module since it contains versions of SCOTCH and HDF5 that work with Nektar++. If you want to try more up-to-date compilers and libraries, feel free to load a later Stages/20** module, and the corresponding modules available under this module, but be aware that Nektar++ might not work with the most recent libraries available on JUWELS. Also note that a very limited number of versions of different libraries are available under each Stages module. Hence, if you need an older version of some library, you might not find the corresponding module once you loaded one of the latest Stages/20** modules.

When selecting modules, you start with a Stages/20** module, followed by a compiler module, followed by a MPI module, followed by a BLAS/LAPACK module, and then finally the remaining modules that you need. After loading each module, you can check which modules that are available to you by typing

$$ module avail

This will give you a list of available modules. For more details on a specific module, including its dependencies, you can type

$$ module spider <module_name>

Note that you can also use module spider <module_name> to search for modules, but be aware that this will give you a list of all modules that matches the search string, including those that are not compatible with the modules that you currently have loaded. For more details on how to select modules on JUWELS, please see the official documentation https://apps.fz-juelich.de/jsc/hps/juwels/software-modules.html

To load the modules that we need for compiling Nektar+, we will use a shell-script. This way, we can easily run the shell script each time we are going to compile Nektar++, without having to worry about forgetting some specific module. In this tutorial, we will put this file in the project directory, inside the folder where we will also install Nektar++ (at this point we assume that you have created a folder inside the $PROJECT_<project_id> directory whose name matches your username)

$$ cd $PROJECT_<project_id>/<username>

$$ mkdir Programs

$$ cd Programs

By using your favourite editor, create a file called personal_modules_gcc.sh, add add the following

# Load modules
module load Stages/2020
module load GCC/10.3.0
module load ParaStationMPI/5.4.10-1
module load imkl/2021.2.0
module load SCOTCH/6.1.0
module load HDF5/1.10.6
module load CMake/3.18.0

# Set environment variables
export CMAKE_LIBRARY_PATH=$LIBRARY_PATH
export CC=mpicc
export CXX=mpicxx

Once the file containing all the modules and environment variables have been defined, we can source it as follows

$$ source $PROJECT_<project_id>/<username>/Programs/personal_modules_gcc.sh

This will load the modules and set the environment variables defined in personal_modules_gcc.sh. It is important to note that the environment variables defined in this shell script are used by CMake to find the right compilers and libraries. If these environment variables are not specified, the code may still compile, but CMake might have selected the default C++ compiler on the system. Also note that the environment variables must be defined using the export command. Simply assigning the names as “CC=mpicc” won’t work, as these so-called shell variables will not be passed to CMake later on when we configure the build (unless they are specified on the same line as the ccmake command).

The next step is to clone the GitLab repository. To do this, we start by moving to the right directory

$$ cd $PROJECT_<project_id>/<username>/Programs

Next, Nektar++ may be downloaded by cloning the GitLab repository using https

$$ git clone https://gitlab.nektar.info/nektar/nektar.git nektar++

Please note that ssh-based authentication is not possible on JUWELS. Next, we create a build directory where the code will be installed

$$ cd nektar++

$$ mkdir build

$$ cd build

Inside the build directory, start the CMake configuration as follows

$$ ccmake -DNEKTAR_USE_MKL=ON -DNEKTAR_USE_SYSTEM_BLAS_LAPACK=OFF -DNEKTAR_USE_MPI=ON -DNEKTAR_USE_HDF5=ON -DCMAKE_CXX_FLAGS:STRING="-std=c++11 -O3 -march=native" -DCMAKE_C_FLAGS:STRING="-O3 -march=native" -DNEKTAR_ENABLE_SIMD_AVX2=ON -DNEKTAR_ENABLE_SIMD_SSE2=ON -DNEKTAR_ENABLE_SIMD_VMATH=ON -DTHIRDPARTY_BUILD_BOOST=ON -DTHIRDPARTY_BUILD_GSMPI=ON ../

After this, press c to configure the build. If you want to add/remove some features from the build, you can do this too once the initial configuration is finished. Once you are happy with the configuration, press g to generate the build. Once the build configuration and generation is complete, you can compile the code using

$$ make -j8 install

A couple of things should be noted with regards to the CMake settings that we pass as command-line-arguments above:

  • -DNEKTAR_USE_SYSTEM_BLAS_LAPACK=OFF prevents the use of the default BLAS/LAPACK implementation on the system.
  • -DNEKTAR_USE_MKL=ON ensures that we use the Intel Math Kernel Library (MKL) BLAS/LAPACK implementation instead. This implementation is generally much faster than the system BLAS/LAPACK implementation, which we have disabled
  • -DCMAKE_CXX_FLAGS:STRING="-std=c++11 -03 -march=native" sets the compiler flags for the C++ compiler. In particular, the "-match=native" flag tells the GCC compiler to generate machine-specific code that is optimised for the current architecture. It also instructs the compiler to enable AVX vectorisation. If this flag (or a similar flag) is not set, then the following CMake options won’t take effect:
    • -DNEKTAR_ENABLE_SIMD_AVX2=ON
    • -DNEKTAR_ENABLE_SIMD_SSE2=ON
    • -DNEKTAR_ENABLE_SIMD_VMATH=ON

Running Nektar++ on JUWELS

The JUWELS supercomputer uses Slurm to manage the jobs. Below follows a simple script for running Nektar++ on 10 nodes

#!/bin/bash
#SBATCH --account=<project_id>
#SBATCH --nodes=10
#SBATCH --ntasks-per-node=48
#SBATCH --output=<name_of_simulation>-%j.out
#SBATCH --error=<name_of_simulation>-%j.err
#SBATCH --time=24:00:00
#SBATCH --partition=batch

# *** start of job script ***
# Note: The current working directory at this point is
# the directory where sbatch was executed.

source $PROJECT_<project_id>/<username>/Programs/personal_modules_gcc.sh

export SOLVER=$PROJECT_<project_id>/<username>/Programs/nektar++/build/<path_to_solver>

srun $SOLVER mesh.xml session.xml --io-format Hdf5 --use-hdf5-node-comm

A couple of things should be noted about this job script

  • <project_id> is the project id that you find on JuDoor
  • The total number of MPI processes will be equal to the number of nodes times the number of tasks per node, in our case this is 480
  • --ntasks-per-node=48 disables multithreading. If you want to, you can also set this variable to 96. In this case, 96 logical processes will be started on each node, which gives a total of 960 MPI processes in our case. This may give a speedup over using 480 MPI processes, and since JUWELS only charges your project based on the number of nodes you are using, the cost is the same as running 480 processes.
  • The longest time that a simulation can be run is 24h
  • By selecting --partition=batch, the standard memory nodes will be used. If you need to use the high-memory nodes, change this variable to mem192
  • srun automatically selects the number of MPI processes based on the flags that you set in your job script. If you want to override these settings, you can pass most flags directly to srun.

To start a job, put the job script above in the same directory as your mesh and session files, then submit the job by typing

$$ sbatch submit_nektar.slurm

Here, submit_nektar.slurm is the name of the job script. Also note that this command must be executed from the same directory as your mesh and session files are located in.

For more details on how to write a job script, please see the official documentation https://apps.fz-juelich.de/jsc/hps/juwels/batchsystem.html.