\chapter{Optimisation}

One of the most frequently asked questions when performing almost any scientific
computation is: how do I make my simulation faster? Or, equivalently, why is my
simulation running so slowly?

The spectral element method is no exception to this rule. The purpose of this
section is to highlight some of the easiest parameters that can be tuned to
attain optimum performance for a given simulation.

Details are kept as untechnical as possible, but some background information on
the underlying numerical methods is necessary in order to understand the various
options available and the implications that they can have on your simulation.

In the current version of the library we now attempt to turn on some
of these optimisations automatically and so you will likely observe a
\inltt{session.opt} file appear in your directory which can be viewed to see
what settings are being selected.

\section{Collections and MatrixFree operations}
The Collections and associated MatrixFree libraries adds optimisations
to perform certain elemental operations collectively by applying an
operator using either matrix-matrix or unrolled matrix free operations,
rather than a sequence of matrix-vector multiplications. Certain
operators benefit more than other from this treatment, so the
following implementations are available:
\begin{itemize}
    \item StdMat: Perform operations using collated matrix-matrix type
      elemental operation.
    \item SumFac: Perform operation using collated matrix-matrix type sum
        factorisation (i.e. direction by direction) /operations.
    \item IterPerExp: Loop through elements, performing matrix-vector
      operation utilising StdRegions building blocks.
     \item MatrixFree: call matrix free implementations that can
       utilise vectorisation by performing SIMD (single instruction
       multiple data) operations over multiple elements concurrently.
    \item NoCollections: Use the original LocalRegions implementation
      to perform the operation which involves looping over the
      elements which may subsequently call the StdRegions
      implementations.
\end{itemize}
All configuration relating to Collections is given in the \inltt{COLLECTIONS}
Xml element within the \inltt{NEKTAR} XML element.

\subsection{Automatic tuning and the --writeoptfile command line option}
By default we now try to select the optimal choice of implementation
when you first run a solver. If you run the solver in verbose mode you
will observe an output of the form:

\begin{lstlisting}[style=XmlStyle]
Collection Implementation for Tetrahedron ( 4 4 4 ) for ngeoms = 428
	    Op.    :	opt. Impl.	 (IterLocExp,  IterStdExp,  StdMat,     SumFac,      MatrixFree)
	 BwdTrans: 	MatFree   	 (0.000344303, 0.000336822, 0.000340444, 0.000185503, 6.80494e-05)
	 Helmholtz: 	MatFree   	 (0.00227906, 0.00481378,      --    ,      --    , 0.000374155)
	 IPWrtBase: 	MatFree   	 (0.000364424, 0.000318054, 0.000291705, 0.000138584, 8.37257e-05)
	 IPWrtDBase: 	MatFree   	 (0.00378674, 0.00308545, 0.00100464, 0.000653242, 0.000283372)
	 PhysDeriv : 	MatFree   	 (0.000881537, 0.000774604, 0.00407994, 0.000540257, 0.000185529)
Collection Implemenation for Prism ( 4 4 4 ) for ngeoms = 136
	    Op.    :	opt. Impl.	 (IterLocExp,  IterStdExp,  StdMat,     SumFac,      MatrixFree)
	 BwdTrans: 	MatFree   	 (0.000131559, 0.000130099, 0.000237854, 8.40501e-05, 2.78436e-05)
	 Helmholtz: 	MatFree   	 (0.000988519, 0.00133484,      --    ,      --    , 0.000166906)
	 IPWrtBase: 	MatFree   	 (0.000113946, 0.000105544, 0.00022007, 5.74802e-05, 3.18842e-05)
	 IPWrtDBase: 	MatFree   	 (0.00148209, 0.000717362, 0.000885148, 0.000257414, 0.00011241)
	 PhysDeriv : 	MatFree   	 (0.000295485, 0.000247841, 0.00186362, 0.000219107, 7.38712e-05)
\end{lstlisting}


This shows the selected collection operation, in this case
\inltt{MatrixFree}, for the different operators implmentations and the
various approaches. Note that IterLocExp is equivalent to NoCollection
and IterStdExp is directly related to the IterPerExp option.

This choice of optimisation is then written into a file called
\inltt{Session.opt} where \inltt{Session} is name of the user defined
xml file. We note that the optimal choice is currently based on the
volumetric elements of the mesh (i.e. Tris and Quads in 2D and Tets,
Pyramids, Prisms and Hexs in 3D) and not on the boundary
conditions. In the case of a parallel run the root process will write
the file based on the optimisation on this processor. In the case one
type of element is not on the root processor the output form the
highest rank process with this element shape will be outputted.  Once
this file is present it will be read directly rather than re-running
the auto-tuning. 


\subsection{Manually selecting the COLLECTIONS section} 

The \inltt{COLLECTIONS} section can be set manually within the
\inltt{COLLECTIONS} tag as shown in the following example. Note this
section can be added in either the input \inltt{Session.xml} file or
the \inltt{Session.opt} file that is auto-generated.

Different implementations may be chosen for different element shapes and
expansion orders.  Specifying \inltt{*} for \inltt{ORDER} sets the
default implementation for any expansion orders not explicitly defined.

\begin{lstlisting}[style=XmlStyle]
<COLLECTIONS>
    <OPERATOR TYPE="BwdTrans">
        <ELEMENT TYPE="T" ORDER="*"   IMPTYPE="IterPerExp" />
        <ELEMENT TYPE="T" ORDER="1-5" IMPTYPE="StdMat" />
    </OPERATOR>
    <OPERATOR TYPE="IProductWRTBase">
        <ELEMENT TYPE="Q" ORDER="*"   IMPTYPE="SumFac" />
    </OPERATOR>
</COLLECTIONS>
\end{lstlisting}

\subsubsection{Default implementation}
The default implementation for all operators may be chosen through
setting the \inltt{DEFAULT} attribute of the \inltt{COLLECTIONS} XML
element to one of \inltt{StdMat}, \inltt{SumFac}, \inltt{IterPerExp},
\inltt{NoCollection} or \inltt{Matrixfree}. The \inltt{StdMat} sets up
a standard matrix for the element in the collection as the underlying
operator. The following uses the collated matrix-matrix type elemental
operation for all operators and expansion orders:

\begin{lstlisting}[style=XmlStyle]
<COLLECTIONS DEFAULT="StdMat" />
\end{lstlisting}

The \inltt{NoCollection} option iterates over each expansion
in the local region calling the local operator which is implemented in
a sum factorization method within the element. The \inltt{IterPerExp}
holds a standard expansion and then also holds an expanded copy of the
geometric factors within the collection operator. \inltt{SumFac} is a
sum factorization implementation which undertakes each direction of
the method over multiple elements in the collection. Finally
\inltt{MatrixFree} implements a vectorisation suitable version of the
sum factorisation which has minimal memory movement but requires some
initial data re-orientation when vectorising over multiple elements. 

\subsubsection{Auto-tuning}
The choice of implementation for each operator, for the given mesh and
expansion orders, can be selected selected automatically through an
attribute in the \inltt{COLLECITON} section. To enable this, add the
following to the \nekpp session file:

\begin{lstlisting}[style=XmlStyle]
<COLLECTIONS DEFAULT="auto" />
\end{lstlisting}

This will collate elements from the given mesh and given expansion orders,
run and time each implementation strategy in turn, and select the fastest
performing case. Note that the selections will be mesh- and order- specific.
The selections made via auto-tuning are output if the \inltt{--verbose}
command-line switch is given.

\subsection{Collection size}
The maximum number of elements within a single collection can be
enforced using the \inltt{MAXSIZE} attribute.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "../user-guide"
%%% End:
