%
\section{SIMDLib}

This directory contains the tinysimd library, a header only library.
The library is a light, zero-overhead wrapper based on template meta-programming that automatically selects a SIMD intrinsics from the most specialized x86-64 instruction set extension available among SSE2 (limited support), AVX2 and AVX512 (not tested) or SVE for the ARM AArch64 architecture.
The library is designed to be easily extended to other architectures.

To use the library one needs to import the tinysimd.hpp header.
The type traits routines, needed for templated programming are available in the traits.hpp header.
It is highly discouraged to perform IO with vector types. If IO is needed for debugging, one needs to import the io.hpp header.

To enable the vector types in {\nek} you need to set to \verb+ON+ the desired extension (for instance \verb+NEKTAR_ENABLE_SIMD_AVX2+).
This will automatically set the appropriate compiler flags. However notice that currently these are set correctly only for gcc and that you might need to delete the cached variable \verb+CMAKE_CXX_FLAGS+ before configuring cmake.
You can check that the desired vector extension was compiled properly by running the \verb+VecDataUnitTests+ which prints out the extension in use.

SVE (for instance \verb+NEKTAR_ENABLE_SIMD_SVE+) is a vector length agnostic ISA extension.
However, in order to wrap the SVE intrinsic types with c++ classes, we fix the size at compile time.
Therefore you need to set appropriate vector size (\verb+NEKTAR_SVE_BITS+) according to your target machine.

Note, the extensions are advanced options and only the options relevant to the compiling machine architecture are made available (see NektarSIMD.cmake for more details).

Vector types are largely used with the same semantic as built-in c++ types.

\paragraph{A simple example: } if avx2 is available, then this scalar code
\begin{lstlisting}[language=C++]
#include <array>
std::array<double,4> a = {-1.0, -1.0, -1.0, -1.0};
std::array<double,4> b;
for (int i = 0; i < 4; ++i){
    b[i] = abs(a[i]);
}
\end{lstlisting}

is equivalent to this vector computation

\begin{lstlisting}[language=C++]
#include <LibUtilities/SimdLib/tinysimd.hpp>
using vec_t = tinysimd::simd<double>;
vec_t a = -1.0;
vec_t b = abs(a);
\end{lstlisting}

which the compiler translates to the corresponding intrisics

\begin{lstlisting}[language=C++]
#include <immintrin.h>
__m256d a = -1.0;
__m256d sign_mask = _mm256_set1_pd(1<<63);
__m256d b = _mm256_andnot_pd(sign_mask, a);
\end{lstlisting}

\paragraph{A realistic example: } an example of a more realistic usage can be found in the SIMD version of the Vmath routines

\lstinputlisting[language=C++, firstline=47, lastline=88]{src/library/LibUtilities/BasicUtils/VmathSIMD.hpp}

Note that there are 2 loops, a vectorized loop and a spillover loop (which is used when the input array size is not a multiple of the vector width).
For more complex methods the core of the loop is replaced by a call to a kernel that can accept both a vector type or a scalar type.
In general the loops are characterized 3 sections: a load to local variables from the input arrays, a call to one or more kernels, a store from the local variables to the output arrays.
The load and store operations need to specify the flag \verb+is_not_aligned+ if the referenced memory is not guaranteed to be aligned to the vector width boundaries.
Otherwise a segmentation fault is just waiting to happen!

As an example of a method with a complex body with calls to multiple kernels refer to RoeSolverSIMD.cpp.

\paragraph{Usage with matrix free operators: }
the usage of the tineysimd library in the matrix free operators differs from the above due to the interleaving of $n$ elements degree of freedoms (where $n$ is the vector width) in a contiguous chunk of memory.
You can refer to \cite{moxey2020efficient} for more details.

\paragraph{General optimization guidelines: }
a key factor to improve performance on modern architectures is to limit as much as possible data transfer from DRAM to cache
\begin{itemize}
\item use local temporary variables to store intermediate values
\item do not call \verb+Vmath+ functions more than once, make a loop over the points instead
\item if you do call a \verb+Vmath+ function, call the \verb+VmathArray+ version (it might be optimized via \verb+VmathSIMD+ call)

\end{itemize}
