Nektar++ on ECMWF Cray XC30

The ECMWF HPC facilities are composed by two identical Cray XC30 systems that support both operational forecast requirements of the centre and research.

Compilation instructions

ECMWF Cray XC30, like ARCHER, uses the module system to load various system modules. We choose the GNU compiler suite both for boost compatibility and also to avoid compile issues with the Cray suite. Additionally the CRAYPE_LINK_TYPE is set to dynamic so that shared libraries can be built.

setenv CRAYPE_LINK_TYPE dynamic
module swap PrgEnv-pgi PrgEnv-gnu
module load cmake
module load git fftw/3.3.4.1
module load boost/1.55.0

These options should be put in your ~/.profile file to avoid typing them each session.

Once loaded the above module, we can compile nektar++. Inside the directory containing the software – i.e. nektar++ – we create a builds directory. From within this new directory we then run cmake with a few extra options.

cd nektar++
mkdir builds
cd builds
CC=cc CXX=CC cmake -DNEKTAR_USE_SYSTEM_BLAS_LAPACK=OFF -DNEKTAR_USE_MPI=ON -DNEKTAR_USE_MKL=ON ..

Notes for the above:

  • cc and CC are the C and C++ wrappers for the Cray utilities and determined by the PrgEnv module.
  • SYSTEM_BLAS_LAPACK is disabled since, by default, we can use the libsci package which contains an optimised version of BLAS and LAPACK and not require any additional arguments to cc.

At this point it is possible to run ccmake .. to e.g. disable unnecessary solvers. Now we can run make as usual to compile the code

make -j4 install

Do not try to run regression tests – the binaries at this point are cross-compiled for the compute nodes and will not execute properly on the login nodes.

Once the compilation has successfully finished, you need to provide the path containing the boost libraries. You can do so by setting the following two environment variables:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/apps/boost/1.55.0/GNU/48/lib