Building and running Nektar++ in Parallel on Windows 10 with MPI

Nektar++ has been primarily developed on Unix-like operating systems and, consequently, building and running the software on these platforms is generally more straightforward and offers the widest set of features. Windows support has been included in the software for many years but, as highlighted in the Nektar++ user guide section on building the software on Windows, the build process can be convoluted and only serial execution has previously been supported.

In Windows 10, the Windows Subsystem for Linux provides the option to run a Linux environment within Windows and this provides one approach to building parallel Nektar++ binaries on Windows. However, for users who are not familiar with Linux, or for users wanting to work within a Visual Studio environment or run across multiple nodes in a Windows cluster, this is not a practical option.

This tutorial will take you through the process of completing a parallel build of Nektar++ on Windows using a Microsoft toolset based on Visual Studio 2019 and Microsoft MPI.

NOTE: Building and running Nektar++ in parallel under Windows is not yet a supported configuration. If you encounter issues with the instructions here, you are welcome to post a message to the nektar-users mailing list and we will do our best to assist. (You can find details of how to join the nekar-users mailing list here)

Prerequisites

This tutorial assumes that you are using a 64-bit version of Windows 10.

Before tackling the parallel Nektar++ build, ensure the you have the following prerequisites installed and set up:

Visual Studio 2019: If you don’t already have an installation of Visual Studio 2019 on your system, you can install the free Visual Studio Community version.

Microsoft MPI: Install both Microsoft MPI (msmpisetup.exe) and the SDK (msmpisdk.msi) via the download page.

Git: If you don’t already have git available on your system, install Git for Windows from https://gitforwindows.org/. When you install Git, ensure that you select either the option to “Add CMake to the system PATH for all users” or to “Add CMake to the system PATH for the current user“. This requires changing the default option.

CMake: Install a Windows win64-x64 version of CMake 3.16 or greater from the CMake download page.

Boost: Install the 64-bit Boost 1.72 binaries compiled for Visual Studio 2019, using the fileboost_1_72_0-msvc-14.2-64.exe, from Boost 1.72’s SourceForge download page.

Adding and updating system environment variables

You will need to add new and modify existing system environment variables as part of this tutorial. If you are not familiar with the process of doing this in Windows 10, you can access the “Environment Variables” by clicking on the Start button and typing “env“. You should see an “Edit the system environment variables” option appear. Click this and then click the “Environment Variables” button to open the dialog for editing the environment variables. Alternatively you can reach the same dialog by selecting “Settings” under the Start menu and navigating to System > About > System info (under Related Settings on the right hand panel), selecting “Advanced System Settings” and in the “Advanced” tab click the “Environment Variables” button.

When adding environment variables, you will see that there are two lists of environment variables: “User variables for <username>” and “System variables“. If you add new environment variables to the “User variables for <username>” list, they will only be available in Command Prompts that you open when logged in as the current user. If you add them to the “System variables” list, they will be made available to all users of the system.

NOTE: If you edit or add environment variables, you will need to open a new Command Prompt for these changes to take effect.

Configuring your system environment

NOTE: You will need to use a Visual Studio command prompt for running all of the terminal commands shown in this tutorial. The command prompt link can be found in your Start menu under the Visual Studio 2019 group. You should use the “Developer Command Prompt for VS2019” or “x64 Native Tools Command Prompt for VS2019“.

CMake and Git should have been added your system path as part of their respective installation processes. The Bin directory of Microsoft MPI should also have been automatically added to your system path and the mpiexec command should be available on the command line.

You now need to manually add one environment variable to your configuration:

Add a BOOST_HOME environment variable pointing to the directory of your boost install, e.g. C:\local\boost_1_72_0

Configuring and building Nektar++

Create a directory on your system into which you can clone the Nektar++ Git repository and undertake your build. The directory used in this tutorial is C:\work.

Change into this directory and clone the Nektar++ source code from GitLab:

C:
cd \work 
git clone https://gitlab.nektar.info/nektar/nektar.git
cd nektar

The directory in which the Nektar++ source code is stored, in this caseC:\work\nektar will be referred to from now on as %NEKTAR_HOME%.

NOTE: Updates to support an MPI build of Nektar++ on Windows 10 have recently been merged into the master branch of the Nektar++ repository so we will build directly from within the master branch.

You can now create a build directory, change into it and configure the build for MPI using cmake:

md build
cd build
cmake -G "Visual Studio 16 2019" -DNEKTAR_USE_MPI=ON ..

cmake will produce a fair amount of output telling you what libraries it’s been looking for and what it has found.

You should see a message in the output telling you that MPI was found along with a path to the msmpi.lib file from Microsoft MPI, e.g.

-- Found MPI: C:/Program Files (x86)/Microsoft SDKs/MPI/Lib/x64/msmpi.lib

You should also see a number of lines beginning:

-- -- Found Boost <library_name> library: ...

These lines should include a path to a library file that should be within the directory where you installed your Boost binaries.

Assuming that the required libraries have been found and the configure step has completed successfully with a message Build files have been written to:..., you can now proceed with the build and installation:

msbuild INSTALL.vcxproj /p:Configuration=Release

To undertake the build in parallel using multiple cores, add the /m:<num procs> switch where <num procs> is the number of processes you want to run for the build.

Post install configuration and testing

Once the build and install process has completed, you should find executables for the solvers and tools in %NEKTAR_HOME%\build\dist\bin.

Before the test suite and the executables can be run, it is necessary to add some directories to your system path. In the system “Environment Variables” settings, add the full path to the following directories (replacing %NEKTAR_HOME% with the actual directory path):

%NEKTAR_HOME%\ThirdParty
%NEKTAR_HOME%\build\ThirdParty\dist\bin
%NEKTAR_HOME%\build\dist\bin
C:\local\boost_1_72_0\lib64-msvc-14.2 (adjust this path as necessary if you’ve installed boost to another location)

You will need to open a new terminal window after adding these environment variables in order for them to take effect.

You should now be able to run the test suite successfully. Change to%NEKTAR_HOME%\build and run:

ctest -C Release

The tests will take a little while to run but you should see them passing as the output from each test starts to appear.

NOTE: 527 tests should be run and all tests should pass. In the event that one or more tests fail, you can run an individual test and show failure information by running: ctest -C Release -I <test_num>,<test_num> --output-on-failure

Undertaking a parallel solve

Now that you have a working MPI parallel Windows build of Nektar++, this final section of the tutorial will show you how to undertake a parallel test run using an example 2D mesh.

Create a directory in which to undertake your test run, for example C:\work\nektar-test, and change into this directory.

Into this directory, download the example 2D Cyl.xml mesh that will allow us to simulate flow of an incompressible fluid around a cylindrical obstruction.

By default the Cyl.xml file is configured to undertake a long, 300,000 timestep simulation. It is recommended that you undertake a much shorter test run to verify that parallel execution is working correctly. To do this, edit Cyl.xml, modifying the FinalTime parameter on line 3271 of the file, within the <PARAMETERS> block in the <CONDITIONS> section. ReduceFinalTime from 300 to 5 to undertake a 5000 timestep run.

You can now use Nektar++’s IncompressibleNavierStokesSolver to run the simulation in parallel. Use the -np flag to tell mpiexec how many processes to run. For example, within the C:\work\nektar-test directory, to use 4 processes, run:

mpiexec -np 4 IncNavierStokesSolver C:\work\nektar-test\Cyl.xml

In the information that is output at the start of the run, you should see Num. Processes: 4 (or a different number if you passed a different number of processes to the -np switch for mpiexec.

Conclusion

You now have a working parallel build of Nektar++ on your Windows 10 system.

If you’re new to Nektar++, take a look at the tutorials on the website.

If you’re already an experienced Nektar++ user, we hope you find the ability to run Nektar++ in parallel on Windows beneficial to your work using the software.