\chapter{Memory management in NekPy}
\label{sec:nekpy:memory}

A significant amount of effort has been invested into developing efficient
memory management techniques, to enable the sharing of memory between Python and
C++ for the ubiquitous \nek{} \inltt{Array} structure which is used throughout
the code.  The computations carried out in \nek{} are usually highly demanding
on the machines they are run on, therefore any unnecessary data duplication or
memory leaks would be detrimental to software performance.

This section first outlines some precusor information relating to the basics of
memory management in C++ and Python, before explaining the strategy used in
NekPy to effectively use the resources when passing data between the different
layers of the bindings.

\section{Memory management in C++ and Python}

\subsection{C++}

In C++ the memory used by the application is divided into five containers \cite{C++Memory}:
\begin{itemize}
    \item text segment -- containing the set of instructions for the program to execute;
    \item data segment -- containing static and global variables initialised with values, 
    e.g. \texttt{float pi = 3.14;} -- the size of the data segment is pre-determined at 
    the time of the compilation of the program and depends on the size of the variables 
    in the source code;
    \item bss (block started by symbol) segment -- containing static and global variables 
    not explicitly initialised with any value, e.g. \texttt{int n;} -- the size of the 
    data segment is pre-determined at the time of the compilation of the program and 
    depends on the size of the variables in the source code;
    \item stack -- a LIFO (last in, first out) structure containing the function calls 
    and variables intialised in the functions -- the size of the stack is pre-determined 
    at the time of compilation of the program and depends on the operating system and 
    development environment;
    \item heap -- containing all dynamically allocated memory (e.g. using instruction 
    \texttt{new} in C++).
\end{itemize}

The access to data on the heap is maintained by pointers which store the memory address of 
said data. If the pointer ceases to exist (e.g. because the function which contained it 
finished running and hence was taken off the stack) the programmer has no way of referencing 
the data on the heap again. The unused, inaccessible data will stay in the memory, consuming 
valuable resources if not properly deallocated using \texttt{delete} instruction. Issues may 
arise if the same memory address is held by two different pointers -- the deallocation of 
memory can render one of the pointers empty and possibly lead to errors.

In order to facilitate memory management, C++11 standard introduced shared pointers 
(\texttt{shared\_ptr})~\cite{C++SharedPtr}. Shared pointers maintain a reference counter 
which is increased when another shared pointer refers to the same memory address. The memory 
will only be deallocated when all shared pointers referencing the memory are destroyed, as shown 
in Figure~\ref{fig:shared_ptr}.

\begin{figure}[h!]
    \centering
    \includegraphics[width=0.6\textwidth]{img/shared_ptr}
    \caption{Memory management with C++11 \texttt{shared\_ptr}. The two declared shared pointers 
    reference an array of 4 integers. Reference counter shown in green.}
    \label{fig:shared_ptr}
\end{figure}

\subsubsection{Python}

Python manages memory though a private heap containing all Python objects~\cite{PythonMemory}. 
An internal memory manager is used to ensure the memory is properly allocated and deallocated 
while the script runs. In contrast to C++, Python's memory manager tries to optimise the memory 
usage as much as possible. For example, the same object reference is allocated to a new variable 
if the object already exists in memory. Another important feature of Python's memory manager is 
the use of garbage collector based on reference counting. When the object is no longer referenced 
by any variable the reference counter is set to zero, which triggers the garbage collector to free
the memory (possibly at some later time). A disadvantage of this solution is slower execution time,
since the garbage collector routines have to be called periodically. Features of the Python memory
manager are schematically shown in~Figure~\ref{fig:python_gc}

\begin{figure}[h!]
    \centering
    \includegraphics[width=0.6\textwidth]{img/python_gc}
    \caption{Memory management in Python. Memory is optimised as much as possible and garbage 
    collector deallocates the memory if the reference counter (shown in green) reaches 0.}
    \label{fig:python_gc}
\end{figure}

It is unusual for the Python programmer to manually modify the way Python uses
memory resources -- however it is sometimes necessary, as is the case with this
project. In particular, the Python C API exposes several macros to handle
reference counting, and developers can increase or decrease the reference
counter of the object using \texttt{Py\_INCREF} and \texttt{Py\_DECREF}
respectively~\cite{PythonManualMemory}. In \texttt{pybind11}, these calls are
wrapped in the use of either \texttt{incref} and \texttt{decref} functions
inside either a \texttt{py::handle} or \texttt{py::object}. The difference
between these classes is that \texttt{handle} makes no adjustment to the
underlying \texttt{PyObject} reference count, whereas \texttt{py::object}
increments/decrements the counter as the object goes in and out of C++ scope.

\section{Passing C++ memory to Python}

To highlight the technique for passing C++ memory natively into Python's
\texttt{ndarray}, we consider first the case of the native \nek{} matrix
structure. In many situations, matrices created by \nek{} (usually a
\texttt{shared\_ptr} of \texttt{NekMatrix<D, StandardMatrixTag>} type) need to
be passed to Python - for instance, when performing differentiation using
e.g. Gauss quadrature rules a differentiation matrix must be obtained. In order
to keep the program memory efficient, the data should not be copied into a NumPy
array but rather be referenced by the Python interface. This, however,
complicates the issue of memory management.

Consider a situation where C++ program no longer needs to work with the
generated array and the memory dedicated to it is deallocated. If this memory
has already been shares with Python, the Python interface may still require the
data contained within the array. However since the memory has already been
deallocated from the C++ side, this will typically cause an out-of-bounds memory
exception. To prevent such situations a solution employing reference counting
must be used.

\subsubsection{Converter method}

\texttt{pybind11} provides the methods to convert a C++ type element to one
recognised by Python as well as to maintain appropriate reference
counting. Listing \ref{lst:c_to_python} shows an abridged version of the
converter method with comments on individual parameters. The object requiring
conversion is a \texttt{shared\_ptr} of \texttt{NekMatrix<D, StandardMatrixTag>}
type (named \texttt{mat}).

\begin{lstlisting}[caption={Converter method for converting the C++ arrays into Python NumPy arrays.}, label={lst:c_to_python}, language=C++]
namespace pybind11::detail
{

template <typename Type, typename T> struct standard_nekmatrix_caster
{
public:
    PYBIND11_TYPE_CASTER(Type, const_name("NekMatrix<T>"));

    bool load(handle src, bool)
    {
        // Perform Python to C++ conversion: returns true if successful, false
        // otherwise.
    }

    // Conversion part 2 (C++ -> Python)
    static handle cast(
        const Nektar::NekMatrix<T, Nektar::StandardMatrixTag> &src,
        return_value_policy, handle)
    {
        // Perform C++ to Python conversion.         
    }
};

template <typename Type>
struct type_caster<Nektar::NekMatrix<Type, Nektar::StandardMatrixTag>>
    : standard_nekmatrix_caster<
          Nektar::NekMatrix<Type, Nektar::StandardMatrixTag>, Type>
{
};

} // namespace pybind11::detail
\end{lstlisting}

Firstly we give a brief overview of the general process undertaken during type
conversion. \texttt{pybind11} maintains a registry of known C++ to Python
conversion types, which by default allows for fundamental data type conversions
such as \texttt{double} and \texttt{float}. In this manner, many C++ functions
can be automatically converted, for example when they are used in \inltt{.def}
calls when registering a Python class. Clearly the \inltt{convert} function here
contains much of the functionality. In order to perform automatic conversion
between the \texttt{NekMatrix} and a 2D \texttt{ndarray}, we register the
conversion function inside the \texttt{pybind11} namespace via a template
specialisation of \texttt{type\_caster}. We also note that throughout the
conversion code (and elsewhere in NekPy), we make use of the \texttt{pybind11}
NumPy bindings. These are a lightweight wrapper around the NumPy C API, which
simplifies the syntax somewhat and avoids direct use of the API.

The custom caster has two methods, \texttt{load} and \texttt{cast} that perform
conversion from Python to C++, and C++ to Python respectively. We now go through
each listing.

\begin{lstlisting}[caption={Python to C++ converter for \texttt{NekMatrix}}, label={lst:c_to_python_load}, language=C++]
bool load(handle src, bool)
{
    // Perform some checks: the source should be an ndarray.
    if (!array_t<T>::check_(src))
    {
        return false;
    }

    // The array should be a c-style, contiguous array of type T.
    auto buf = array_t<T, array::c_style | array::forcecast>::ensure(src);
    if (!buf)
    {
        return false;
    }

    // There should be two dimensions.
    auto dims = buf.ndim();
    if (dims != 2)
    {
        return false;
    }

    // Copy data across from the Python array to C++.
    std::vector<size_t> shape = {buf.shape()[0], buf.shape()[1]};
    value = Nektar::NekMatrix<T, Nektar::StandardMatrixTag>(
        shape[0], shape[1], buf.data(), Nektar::eFULL, buf.data());

    return true;
}
\end{lstlisting}

For \texttt{NekMatrix} we do not yet have custom C++-side code to handle Python as
a data source. The \texttt{cast} routine therefore simply copies the data from the
Python handle. The thing to note in this instance is that the class member
variable \texttt{value} stores the resulting C++ value as a type \texttt{Type}.

In terms of conversion from Python to C++, we use the following \texttt{cast}
function.

\begin{lstlisting}[caption={C++ to Python converter for \texttt{NekMatrix}}, label={lst:c_to_python_load}, language=C++]
static handle cast(
    const Nektar::NekMatrix<T, Nektar::StandardMatrixTag> &src,
    return_value_policy, handle)
{
    using NMat = Nektar::NekMatrix<T, Nektar::StandardMatrixTag>;

    // Construct a new wrapper matrix to hold onto the data. Assign a
    // destructor so that the wrapper is cleaned up when the Python object
    // is deallocated.
    capsule c(new NMat(src.GetRows(), src.GetColumns(), src.GetPtr(),
                       Nektar::eWrapper),
              [](void *ptr) {
                  NMat *mat = (NMat *)ptr;
                  delete mat;
              });

    // Create the NumPy array, passing the capsule. When we go out of scope,
    // c's reference count will have been reduced by 1, but array increases
    // the reference count when it assigns the base to the array.
    array a({src.GetRows(), src.GetColumns()},
            {sizeof(T), src.GetRows() * sizeof(T)}, src.GetRawPtr(), c);

    // This causes the array to be released without decreasing its reference
    // count, which we do since if we just returned a, then the array would
    // be deallocated immediately when this function returns.
    return a.release();
}
\end{lstlisting}

This conversion can be done more efficiently, since we may construct a
\texttt{NekMatrix} using a reference count of the original matrix, by accessing
its \texttt{Array<OneD, T>} object via the \texttt{GetPtr} method. To ensure
this array lives for the entire time of the NumPy array, we wrap this into a
Python capsule object. The capsule is designed to hold a C pointer and a
callback function that is called when the Python object is deallocated.

A new matrix is therefore constructed and passed to the capsule. This will
prevent it being destroyed if it is in use in Python, even if on the C++ side
the memory is deallocated. The callback function simply deletes the allocated
pointer when it is no longer required, cleaning up the memory
appropriately. This process is shown in Figure~\ref{fig:c_to_python}. It is
worth noting that the steps (c) and (d) can be reversed and the pointer created
by the Python binding can be removed first.  In this case the memory will be
deallocated only when the underlying \texttt{Array<OneD, T>} created by C++ is
deallocated.

\begin{figure}[h!]
    \centering
    \begin{subfigure}{0.6\textwidth}
        \includegraphics[width=0.99\textwidth]{img/c_to_python_a}
        \caption{Nektar++ creates a data array referenced by a \texttt{shared\_ptr}}
        \label{fig:c_to_python_a}
    \end{subfigure}
    \begin{subfigure}{0.6\textwidth}
        \includegraphics[width=0.99\textwidth]{img/c_to_python_b}
        \caption{Array is passed to Python binding which creates a new \texttt{shared\_ptr} 
        to the data}
        \label{fig:c_to_python_b}
    \end{subfigure}
    \begin{subfigure}{0.6\textwidth}
        \includegraphics[width=0.99\textwidth]{img/c_to_python_c}
        \caption{Nektar++ no longer needs the data - its \texttt{shared\_ptr} is removed but 
        the memory is not deallocated}
        \label{fig:c_to_python_c}
    \end{subfigure}
    \begin{subfigure}{0.6\textwidth}
        \includegraphics[width=0.99\textwidth]{img/c_to_python_d}
        \caption{When the data is no longer needed in the Python interface the destructor is 
        called and \texttt{shared\_ptr} is removed.}
        \label{fig:c_to_python_d}
    \end{subfigure}
    \caption{Memory management of data created in C++ using \texttt{shared\_ptr} and passed 
    to Python. Reference counter shown in green.}
    \label{fig:c_to_python}
\end{figure}

Next, the converter method returns a NumPy array using the \texttt{array} object
from \texttt{pybind11}. Note that the \texttt{capsule} object is passed as the
\texttt{ndarray} base argument -- i.e. the object that owns the data. In this
case, when all \texttt{ndarray} objects that refer to the data become
deallocated, NumPy will not directly deallocate the data but instead release its
reference to the capsule. The capsule will then be deallocated, which will
deallocate the \texttt{NekMatrix} and thus decrement the counter in the
\texttt{Array<OneD, >}. We also note that data ordering is important for matrix
storage; \nek{} stores data in column-major order, whereas NumPy arrays are
traditionally row-major.  The stride of the array has to be passed into the
\texttt{np::from\_data} in a form of a tuple \texttt{(a, b)}, where \texttt{a}
is the number of bytes needed to skip to get to the same position in the next
row and \texttt{b} is the number of bytes needed to skip to get to the same
position in the next column.

Finally, in order to stop Python from immediately destroying the resulting
\texttt{ndarray}, we call the \texttt{release} function on the array, which
prevents the reference counter from being decremented when the array is
returned.

\subsubsection{Testing}

The process outlined above requires little manual intervention from the
programmer. There are no explicit calls to Python C API as all operations are
carried out by \texttt{pybind11}. Therefore, the testing focused mostly on
correctness of returned data, in particular the order of the array. To this end,
the \emph{Differentiation} tutorials were used as tests. In order to correctly
run the tutorials the Python wrapper needs to retrieve the differentiation
matrix which, as mentioned before, has to be converted to a datatype Python
recognises. The test runs the differentiation tutorials and compares the final
result to the fixed expected value. The test is automatically run as a part of
\texttt{ctest} command if both the Python wrapper and the tutorials have been
built.

\subsection{Passing Python data to C++} 

Conversely, a similar problem exists when data is created in Python and has to
be passed to the C++ program. In this case, as the data is managed by Python,
the main reference counter should be maintained by the Python object and
incremented or decremented as appropriate using \texttt{incref} and
\texttt{decref} methods respectively. Although we do not support this process
for the \texttt{NekMatrix} as described above, we do use this process for the
\texttt{Array<OneD, T>} structure. When the array is no longer needed by the C++
program the reference counter on the Python side should be decremented in order
for Python garbage collection to work appropriately -- however this should only
happen when the array was created by Python in the first place.

The files implementing the below procedure are:

\begin{itemize}
  \item \texttt{LibUtilities/Python/BasicUtils/SharedArray.cpp} and
  \item \texttt{LibUtilities/BasicUtils/SharedArray.hpp}
\end{itemize}

\subsubsection{Modifications to \texttt{Array<OneD, const DataType>} class template}

In order to perform the operations described above, the C++ array structure
should contain information on whether or not it was created from data managed by
Python. To this end, two new attributes were added to C++ \texttt{Array<OneD,
  const DataType>} class template in the form of a \texttt{struct}:

\begin{lstlisting}[language=C++]
struct PythonInfo {
    void *m_pyObject; // Underlying PyObject pointer
    void (*m_callback)(void *); // Callback
};
\end{lstlisting}

where:

\begin{itemize}
  \item \texttt{m\_pyObject} is a pointer to the \texttt{PyObject} containing
  the data, which should be an \texttt{ndarray};
  \item \texttt{m\_callback} is a function pointer to the callback function
  which will decrement the reference counter of the \texttt{PyObject}.
\end{itemize}

Inside \texttt{Array<OneD, T>}, this struct is held as a double pointer, i.e.:
\begin{lstlisting}[language=C++]
PythonInfo **m_pythonInfo;
\end{lstlisting}
This is done because if the \texttt{ndarray} was created originally from a C++
to Python conversion (as outlined in the previous section), we need to convert
the Array -- and any other shared arrays that hold the same C++ memory -- to
reference the Python array. If we did not do this, then it is possible that the
C++ array could be destroyed whilst it is still used on the Python side, leading
to an out-of-bounds exception. By storing this as a double pointer, in a similar
fashion to the underlying reference counter \texttt{m\_count}, we can ensure
that all C++ arrays are updated when necessary. We can keep track of Python
arrays by checking \texttt{*m\_pythonInfo}; if this is not set to
\texttt{nullptr} then the array has been constructed throught the Python to C++
converter.

Adding new attributes to the arrays might cause a significantly increased memory
usage or additional unforeseen overheads, although this was not seen in
benchmarking. However to avoid all possibility of this, a preprocessor directive
has been added to only include the additional arguments if \texttt{NekPy} has
been built (using the option \texttt{NEKTAR\_BUILD\_PYTHON}).

A new constructor has been added to the class template, as seen in Listing
\ref{lst:new_const}.  \texttt{m\_memory\_pointer} and
\texttt{m\_python\_decrement} have been set to \texttt{nullptr} in the
pre-existing constructors. A similar constructor was added for \texttt{const}
arrays to ensure that these can also be passed between the languages. Note that
no calls to Nektar++ array initialisation policies are made in this constructor,
unlike in the pre-existing ones, as there is no need for the new array to copy
the data.

\begin{lstlisting}[caption={New constructor for initialising arrays created through the Python to C++ converter method.}, label={lst:new_const}, language=C++]
Array(unsigned int dim1Size, DataType* data, void* memory_pointer, void (*python_decrement)(void *)) :
    m_size( dim1Size ),
    m_capacity( dim1Size ),
    m_data( data ),
    m_count( nullptr ),
    m_offset( 0 )                               
{
    m_count = new unsigned int(); 
    *m_count = 1;

    m_pythonInfo = new PythonInfo *();
    *m_pythonInfo = new PythonInfo();
    (*m_pythonInfo)->m_callback = python_decrement;
    (*m_pythonInfo)->m_pyObject = memory_pointer;
}
\end{lstlisting}

Changes have also been made to the destructor, as shown in
Listing~\ref{lst:destructor}, in order to ensure that if the data was initially
created in Python the callback function would decrement the reference counter of
the NekPy array object. The detailed procedure for deleting arrays is described
further in this section.

\begin{lstlisting}[caption={The modified destructor for C++ arrays.}, label={lst:destructor}, language=C++]
~Array()
{
    if( m_count == nullptr )
    {
        return;
    }

    *m_count -= 1;
    if( *m_count == 0 )
    {
        if (*m_pythonInfo == nullptr)
        {
            ArrayDestructionPolicy<DataType>::Destroy( m_data, m_capacity );
            MemoryManager<DataType>::RawDeallocate( m_data, m_capacity );
        }
        else
        {
            (*m_pythonInfo)->m_callback((*m_pythonInfo)->m_pyObject);
            delete *m_pythonInfo;
        }

        delete m_pythonInfo;
        delete m_count; // Clean up the memory used for the reference count.
    }
}
\end{lstlisting}

\subsubsection{Creation of new arrays}

The following algorithm has been proposed to create new arrays in Python and
allow the C++ code to access their contents:

\begin{enumerate}
  \item The NumPy array object to be converted is passed as an argument to a C++
  method available in the Python wrapper.
  \item The converter method is called to convert a Python NumPy array into C++
  \texttt{Array<OneD, const DataType>} object.
  \item If the NumPy array was created through the C++-to-Python process (which
  can be determined by checking the \texttt{base} of the NumPy array), then:
  \begin{itemize}
    \item extract the \nek{} Array from the capsule;
    \item convert the Array (and all of its other references) to a Python array
    so that any C++ arrays that share this memory also know to call the
    appropriate decrement function;
    \item set the NumPy array's \texttt{base} to an empty object to destroy the
    original capsule.
  \end{itemize}
  \item Otherwise, the converter creates a new \texttt{Array<OneD, const DataType>} object
  with the following attribute values:
  \begin{itemize}
    \item \texttt{data} points to the data contained by the NumPy array,
    \item \texttt{memory\_pointer} points to the NumPy array object,
    \item \texttt{python\_decrement} points to the function decrementing the
    reference counter of the \texttt{PyObject},
    \item subsequently, \texttt{m\_count} is equal to 1.
  \end{itemize}
  \item The Python reference counter of the NumPy array object is increased.
  \item If any new references to the array are created in C++ the
  \texttt{m\_count} attribute is increased accordingly. Likewise, if new
  references to NumPy array object are made in Python the reference counter
  increases.
\end{enumerate}

The process is schematically shown in Figure \ref{fig:array_creation_deletion_a} 
and \ref{fig:array_creation_deletion_b}.

\subsubsection{Array deletion}

The array deletion process relies on decrementing two reference counters: one 
on the Python side of the program (Python reference counter) and the other one 
on C++ side of the program. The former registers how many Python references to 
the data there are and if there is a C++ reference to the array. The latter 
(represented by \texttt{m\_count} attribute) counts only the number of references 
on the C++ side and as soon as it reaches zero the callback function is triggered to 
decrement the Python reference counter so that registers that the data is no longer 
referred to in C++. Figure \ref{fig:array_deletion} presents the overview of the 
procedure used to delete the data.

In short, the fact that C++ uses the array is represented to Python as just an 
increment to the object reference counter. Even if the Python object goes out 
of scope or is explicitly deleted, the reference counter will always be non-zero 
until the callback function to decrement it is executed, as shown in Figure 
\ref{fig:array_creation_deletion_c}. Similarly, if the C++ array is deleted first, 
the Python object will still exist as the reference counter will be non-zero 
(see Figure \ref{fig:array_creation_deletion_d}).

\subsubsection{Converter method}

As with conversion from C++ to Python, a converter method was registered to make
Python NumPy arrays available in C++ with \texttt{pybind11}, which can be found in
the \texttt{SharedArray.hpp} bindings file.

\begin{figure}[h!]
    \centering
    \begin{subfigure}{0.9\textwidth}
        \includegraphics[width=0.99\textwidth]{img/array_creation_deletion_a}
        \caption{The NumPy array is created in Python. Note that the NumPy 
        object and the data it contains are represented by two separate memory addresses.}
        \label{fig:array_creation_deletion_a}
    \end{subfigure}
    \begin{subfigure}{0.9\textwidth}
        \includegraphics[width=0.99\textwidth]{img/array_creation_deletion_b}
        \caption{The C++ array is created through the converter method: its attributes 
        point to the appropriate memory addresses and the reference counter of the 
        memory address of NumPy array object is incremented.}
        \label{fig:array_creation_deletion_b}
    \end{subfigure}
    \begin{subfigure}{0.9\textwidth}
        \includegraphics[width=0.99\textwidth]{img/array_creation_deletion_c}
        \caption{If the NumPy object is deleted first, the reference counter is 
        decremented, but the data still exists in memory.}
        \label{fig:array_creation_deletion_c}
    \end{subfigure}
    \begin{subfigure}{0.9\textwidth}
        \includegraphics[width=0.99\textwidth]{img/array_creation_deletion_d}
        \caption{If the C++ array is deleted first, the callback function decrements 
        the reference counter of the NumPy object but the data still exists in memory.}
        \label{fig:array_creation_deletion_d}
    \end{subfigure}
    \caption{Diagram showing the process of creation and deletion of arrays. Reference 
    counters shown in green.}
    \label{fig:array_creation_deletion}
\end{figure}

\clearpage

\begin{figure}[h!]
    \centering
    \includegraphics{img/deletion_flowchart}
    \caption{Flowchart describing the procedure for array deletion from either side of 
    the code.}
    \label{fig:array_deletion}
\end{figure}

\subsubsection{Testing}

As the process of converting arrays from Python to C++ required making direct calls to 
C API and relying on custom-written methods, more detailed testing was deemed necessary. 
In order to thoroughly check if the conversion works as expected, three tests were 
conducted to determine whether the array is:

\begin{enumerate}
\item referenced (not copied) between the C++ program and the Python wrapper,
\item accessible to C++ program after being deleted in Python code,
\item accessible in Python script after being deleted from C++ objects.
\end{enumerate}

Python files containing test scripts are currently located in 
\path{library\Demos\Python\tests}. They should be converted into unit tests 
that should be run when Python components are built.
