Nektar++ on Copper (Cray XE6)

Copper is a Cray XE6 system and supports research programs of the United the States Department of Defense (DoD). The system is provided for computationally based science and engineering and is intended for high performance computing, large-scale storage, and utility computing.

Compilation instructions

Copper, like ARCHER, uses the module system to load various system modules. We choose the GNU compiler suite both for boost compatibility and also to avoid compile issues with the Cray suite. Additionally the CRAYPE_LINK_TYPE is set to dynamic so that shared libraries can be built.

setenv CRAYPE_LINK_TYPE dynamic
module swap PrgEnv-pgi PrgEnv-gnu
module load cmake
module load git fftw/3.3.4.1
module load costinit

These options should be put in your ~/.profile file to avoid typing them each session.

Once loaded the above module, we can compile nektar++. Inside the directory containing the software – i.e. nektar++ – we create a builds directory. From within this new directory we then run cmake with a few extra options. Note the use of CC and CXX to select the special Copper-specific compilers.

cd nektar++
mkdir builds
cd builds
CC=cc CXX=CC cmake -DNEKTAR_USE_SYSTEM_BLAS_LAPACK=OFF -DNEKTAR_USE_MPI=ON ..

Notes for the above:

  • cc and CC are the C and C++ wrappers for the Cray utilities and determined by the PrgEnv module.
  • SYSTEM_BLAS_LAPACK is disabled since, by default, we can use the libsci package which contains an optimised version of BLAS and LAPACK and not require any additional arguments to cc.

At this point you can run ccmake .. to e.g. disable unnecessary solvers. Now run make as usual to compile the code

make -j4 install

Do not try to run regression tests – the binaries at this point are cross-compiled for the compute nodes and will not execute properly on the login nodes.

Once the compilation has successfully finished, you need to provide the path containing the boost libraries. You can do so by setting the following two environment variables:

setenv ${NEKTAR_HOME} path_to_nektar++
setenv LD_LIBRARY_PATH $LD_LIBRARY_PATH\:${NEKTAR_HOME}/builds/dist/nektar++-4.4.0

Running your job

Jobs are submitted to the queue using PBS files. Below a simple example of script running the incompressible Navier-Stokes solver. Make sure to substitute the name of project.xml with your own .xml file.

#!/bin/ksh
#PBS -l select=1:ncpus=32:mpiprocs=8
#PBS -l walltime=00:01:00
#PBS -N nektar++
#PBS -q standard
#PBS -A Add_your_project_ID_here
#PBS -j oe

# One node, 32 mpi processes per node,
# for 8 total mpi processes

cd $PBS_O_WORKDIR

aprun -n 8 /lustre/usr/local/u/vassilis/sw/nektar++/build/dist/bin/IncNavierStokesSolver project.xml | tee project_out.txt