Building Nalu-Wind Manually

Although we recommend installing Nalu-Wind with Spack, if you prefer not to build using Spack, below are instructions which describe the process of building Nalu-Wind by hand. These instructions are an approximation, due to the many differences that can exist across machines.

Linux and OSX

The instructions for Linux and OSX are mostly the same, except on each OS you may be able to use a package manager to install some dependencies for you. Using Homebrew on OSX is one option listed below. Compilers and MPI are expected to be already installed. If they are not, please follow the OpenMPI build instructions. Currently we are recommending OpenMPI v1.10.7 or MPICH 3.3 and GCC v7.3.0. Start by creating a ${NALU_ROOT_DIR} to work in.

Homebrew

If using OSX, you can install many dependencies using Homebrew. Install Homebrew on your local machine and reference the list below for some packages Homebrew can install for you. This allows you to skip the steps describing the build process for each of these applications. You will need to find the location of the applications in which Homebrew has installed them, to use when building Trilinos and Nalu-Wind.

brew install openmpi
brew install cmake
brew install libxml2
brew install boost
brew tap homebrew/science
brew install superlu43

CMake v3.12.4

CMake is provided here.

Prepare:

cd ${NALU_ROOT_DIR}/packages
tar xf cmake-3.12.4.tar.gz

Build:

cd ${NALU_ROOT_DIR}/packages/cmake-3.12.4
./configure --prefix=${NALU_ROOT_DIR}/install/cmake
make
make install

SuperLU v4.3

SuperLU is provided here.

Prepare:

cd ${NALU_ROOT_DIR}/packages
tar xf superlu_4.3.tar.gz

Build:

cd ${NALU_ROOT_DIR}/packages/SuperLU_4.3
cp MAKE_INC/make.linux make.inc

To find out what the correct platform extension PLAT is:

uname -m

Edit make.inc as shown below (diffs shown from baseline).

PLAT          = _x86_64
SuperLUroot   = /your_path/install/SuperLU_4.3 i.e., ${NALU_ROOT_DIR}/install/SuperLU_4.3
BLASLIB       = -L/usr/lib64 -lblas
CC            = mpicc
FORTRAN       = mpif77

On some platforms, the ${NALU_ROOT_DIR} may be mangled. In such cases, you may need to use the entire path to install/SuperLU_4.3.

Next, make some new directories:

mkdir ${NALU_ROOT_DIR}/install/SuperLU_4.3
mkdir ${NALU_ROOT_DIR}/install/SuperLU_4.3/lib
mkdir ${NALU_ROOT_DIR}/install/SuperLU_4.3/include
cd ${NALU_ROOT_DIR}/packages/SuperLU_4.3
make
cp SRC/*.h ${NALU_ROOT_DIR}/install/SuperLU_4.3/include

Libxml2 v2.9.8

Libxml2 is found here.

Prepare:

cd ${NALU_ROOT_DIR}/packages
tar -xvf libxml2-2.9.8.tar.gz

Build:

cd ${NALU_ROOT_DIR}/packages/libxml2-2.9.8
CC=mpicc CXX=mpicxx ./configure -without-python --prefix=${NALU_ROOT_DIR}/install/libxml2
make
make install

Boost v1.68.0

Boost is found here.

Prepare:

cd ${NALU_ROOT_DIR}/packages
tar -zxvf boost_1_68_0.tar.gz

Build:

cd ${NALU_ROOT_DIR}/packages/boost_1_68_0
./bootstrap.sh --prefix=${NALU_ROOT_DIR}/install/boost --with-libraries=signals,regex,filesystem,system,mpi,serialization,thread,program_options,exception

Next, edit project-config.jam and add a ‘using mpi’, e.g,

using mpi: /path/to/mpi/openmpi/bin/mpicc

./b2 -j 4 2>&1 | tee boost_build_one
./b2 -j 4 install 2>&1 | tee boost_build_intall

YAML-CPP 0.6.2

YAML is provided here. Versions of Nalu before v1.1.0 used earlier versions of YAML-CPP. For brevity only the latest build instructions are discussed and the history of the Nalu-Wind git repo can be used to find older installation instructions if required.

Prepare:

cd ${NALU_ROOT_DIR}/packages
git clone https://github.com/jbeder/yaml-cpp
cd yaml-cpp && git checkout yaml-cpp-0.6.2

Build:

cd ${NALU_ROOT_DIR}/packages/yaml-cpp
mkdir build
cd build
cmake -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_CXX_FLAGS=-std=c++11 -DCMAKE_CC_COMPILER=mpicc -DCMAKE_INSTALL_PREFIX=${NALU_ROOT_DIR}/install/yaml-cpp ..
make
make install

Zlib v1.2.11

Zlib is provided here.

Prepare:

cd ${NALU_ROOT_DIR}/packages
tar -zxvf zlib-1.2.11.tar.gz

Build:

cd ${NALU_ROOT_DIR}/packages/zlib-1.2.11
CC=gcc CXX=g++ CFLAGS=-O3 CXXFLAGS=-O3 ./configure --prefix=${NALU_ROOT_DIR}/install/zlib
make
make install

HDF5 v1.10.4

HDF5 1.10.4 is provided here.

Prepare:

cd ${NALU_ROOT_DIR}/packages
tar -zxvf hdf5-1.10.4.tar.gz

Build:

cd ${NALU_ROOT_DIR}/packages/hdf5-1.10.4
./configure CC=mpicc FC=mpif90 CXX=mpicxx CXXFLAGS="-fPIC -O3" CFLAGS="-fPIC -O3" FCFLAGS="-fPIC -O3" --enable-parallel --with-zlib=${NALU_ROOT_DIR}/install/zlib --prefix=${NALU_ROOT_DIR}/install/hdf5
make
make install
make check

NetCDF v4.6.1 and Parallel NetCDF v1.8.0

In order to support all aspects of Nalu-Wind’s parallel models, this combination of products is required.

Parallel NetCDF v1.8.0

Parallel NetCDF is provided on the Argon Trac Page.

Prepare:

cd ${NALU_ROOT_DIR}/packages
tar -zxvf parallel-netcdf-1.8.0.tar.gz

Build:

cd parallel-netcdf-1.8.0
./configure --prefix=${NALU_ROOT_DIR}/install/parallel-netcdf CC=mpicc FC=mpif90 CXX=mpicxx CFLAGS=-O3 CXXFLAGS=-O3 --disable-fortran
make
make install

NetCDF v4.6.1

NetCDF is provided here.

Prepare:

cd ${NALU_ROOT_DIR}/packages
tar -zxvf netcdf-4.6.1.tar.gz

Build:

cd netcdf-4.6.1
./configure --prefix=${NALU_ROOT_DIR}/install/netcdf CC=mpicc FC=mpif90 CXX=mpicxx CFLAGS="-I${NALU_ROOT_DIR}/install/parallel-netcdf/include -O3" LDFLAGS=-L${NALU_ROOT_DIR}/install/parallel-netcdf/lib --enable-pnetcdf --enable-parallel-tests --enable-netcdf-4 --disable-shared --disable-fsync --disable-cdmremote --disable-dap --disable-doxygen --disable-v2
make -j 4
make check
make install

Trilinos

Trilinos is managed by the Trilinos project and can be found on Github.

Prepare:

cd ${NALU_ROOT_DIR}/packages
git clone https://github.com/trilinos/Trilinos.git
cd ${NALU_ROOT_DIR}/packages/Trilinos
mkdir build
cd build

Now create a do-config-trilinos script with the following recommended options:

#!/bin/bash

# The base directory where mpi is located.
# From here you should be able to find include/mpi.h bin/mpicxx, bin/mpiexec, etc.
MPI_ROOT_DIR=/PathToMPI
NALU_ROOT_DIR=/PathToNaluProjectDir

# Note: Don't forget to set your LD_LIBRARY_PATH to $mpi_base_dir/lib
#       You may also need to add to LD_LIBRARY_PATH the lib directory for the compiler
#       used to create the mpi executables.

# TPLS needed by trilinos, possibly provided by HomeBrew on a Mac
#BOOST_ROOT_DIR=/usr/local/Cellar/boost/1.56.0/include/boost/
#SUPERLU_ROOT_DIR=/usr/local/Cellar/superlu/4.3
BOOST_ROOT_DIR=${NALU_BUILD_DIR}/install/boost
SUPERLU_ROOT_DIR=${NALU_BUILD_DIR}/install/SuperLU_4.3
NETCDF_ROOT_DIR=${NALU_BUILD_DIR}/install/netcdf
HDF5_ROOT_DIR=${NALU_BUILD_DIR}/install/hdf5
PARALLEL_NETCDF_ROOT_DIR=${NALU_BUILD_DIR}/install/parallel-netcdf
ZLIB_ROOT_DIR=${NALU_BUILD_DIR}/install/zlib
TRILINOS_ROOT_DIR=${NALU_BUILD_DIR}/install/trilinos

EXTRA_ARGS=$@

# Cleanup old cache before we configure
# Note:  This does not remove files produced by make.  Use "make clean" for this.
find . -name "CMakeFiles" -exec rm -rf {} \;
rm -f CMakeCache.txt

cmake \
  -DCMAKE_INSTALL_PREFIX=${TRILINOS_ROOT_DIR} \
  -DCMAKE_BUILD_TYPE:STRING=RELEASE
  -DMPI_USE_COMPILER_WRAPPERS:BOOL=ON
  -DMPI_CXX_COMPILER:FILEPATH=${CXX}
  -DKokkos_ENABLE_DEPRECATED_CODE:BOOL=OFF
  -DTpetra_INST_SERIAL:BOOL=ON
  -DTrilinos_ENABLE_CXX11:BOOL=ON
  -DTrilinos_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON
  -DTpetra_INST_DOUBLE:BOOL=ON
  -DTpetra_INST_COMPLEX_DOUBLE:BOOL=OFF
  -DTrilinos_ENABLE_TESTS:BOOL=OFF
  -DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF
  -DTrilinos_ASSERT_MISSING_PACKAGES:BOOL=OFF
  -DTrilinos_ALLOW_NO_PACKAGES:BOOL=OFF
  -DTrilinos_ENABLE_Epetra:BOOL=OFF
  -DTrilinos_ENABLE_Tpetra:BOOL=ON
  -DTrilinos_ENABLE_KokkosKernels:BOOL=ON
  -DTrilinos_ENABLE_ML:BOOL=OFF
  -DTrilinos_ENABLE_MueLu:BOOL=ON
  -DXpetra_ENABLE_Kokkos_Refactor:BOOL=ON
  -DMueLu_ENABLE_Kokkos_Refactor:BOOL=ON
  -DTrilinos_ENABLE_EpetraExt:BOOL=OFF
  -DTrilinos_ENABLE_AztecOO:BOOL=OFF
  -DTrilinos_ENABLE_Belos:BOOL=ON
  -DTrilinos_ENABLE_Ifpack2:BOOL=ON
  -DTrilinos_ENABLE_Amesos2:BOOL=ON
  -DTrilinos_ENABLE_Zoltan2:BOOL=ON
  -DTrilinos_ENABLE_Ifpack:BOOL=OFF
  -DTrilinos_ENABLE_Amesos:BOOL=OFF
  -DTrilinos_ENABLE_Zoltan:BOOL=ON
  -DTrilinos_ENABLE_STK:BOOL=ON
  -DTrilinos_ENABLE_Gtest:BOOL=ON
  -DTrilinos_ENABLE_SEACASExodus:BOOL=ON
  -DTrilinos_ENABLE_SEACASEpu:BOOL=ON
  -DTrilinos_ENABLE_SEACASExodiff:BOOL=ON
  -DTrilinos_ENABLE_SEACASNemspread:BOOL=ON
  -DTrilinos_ENABLE_SEACASNemslice:BOOL=ON
  -DTrilinos_ENABLE_SEACASIoss:BOOL=ON
  -DTPL_ENABLE_MPI:BOOL=ON
  -DTPL_ENABLE_Boost:BOOL=ON
  -DBoostLib_INCLUDE_DIRS:PATH=${BOOST_ROOT_DIR}/include
  -DBoostLib_LIBRARY_DIRS:PATH=${BOOST_ROOT_DIR}/lib
  -DBoost_INCLUDE_DIRS:PATH=${BOOST_ROOT_DIR}/include
  -DBoost_LIBRARY_DIRS:PATH=${BOOST_ROOT_DIR}/lib
  -DTPL_ENABLE_SuperLU:BOOL=ON
  -DSuperLU_INCLUDE_DIRS:PATH=${SUPERLU_ROOT_DIR}/include
  -DSuperLU_LIBRARY_DIRS:PATH=${SUPERLU_ROOT_DIR}/lib
  -DTPL_ENABLE_Netcdf:BOOL=ON
  -DNetCDF_ROOT:PATH=${NETCDF_ROOT_DIR}
  -DTPL_Netcdf_PARALLEL:BOOL=ON
  -DTPL_ENABLE_Pnetcdf:BOOL=ON
  -DPNetCDF_ROOT:PATH=${PARALLEL_NETCDF_ROOT_DIR}
  -DPnetcdf_INCLUDE_DIRS:PATH=${PARALLEL_NETCDF_ROOT_DIR}/include
  -DPnetcdf_LIBRARY_DIRS:PATH=${PARALLEL_NETCDF_ROOT_DIR}/lib
  -DTPL_ENABLE_HDF5:BOOL=ON
  -DHDF5_ROOT:PATH=${HDF5_ROOT_DIR}
  -DHDF5_NO_SYSTEM_PATHS:BOOL=ON
  -DTPL_ENABLE_Zlib:BOOL=ON
  -DZlib_INCLUDE_DIRS:PATH=${ZLIB_ROOT_DIR}/include
  -DZlib_LIBRARY_DIRS:PATH=${ZLIB_ROOT_DIR}/lib
  -DTPL_ENABLE_BLAS:BOOL=ON
  $EXTRA_ARGS \
  ../

Build

Place into the build directory, the do-config-trilinos script created from the recommended Trilinos configuration listed above.

do-config-trilinos will be used to run cmake to build trilinos correctly for Nalu-Wind.

Make sure all other paths to netcdf, hdf5, etc., are correct.

./do-config-trilinos
make
make install

HYPRE

Nalu-Wind can use HYPRE solvers and preconditioners, especially for Pressure Poisson solves. However, this dependency is optional and is not enabled by default. Users wishing to use HYPRE solver and preconditioner combination must compile HYPRE library and link to it when building Nalu-Wind.

# 1. Clone hypre sources
https://github.com/LLNL/hypre.git
cd hypre/src

# 2. Configure HYPRE package and pass installation directory
./configure --prefix=${NALU_ROOT_DIR}/install/hypre --without-superlu --without-openmp --enable-bigint

# 3. Compile and install
make && make install

Note

Make sure that --enable-bigint option is turned on if you intend to run linear systems with $> 2$ billion rows. Otherwise, nalu executable will throw an error at runtime for large problems.
Users must pass -DENABLE_HYPRE option to CMake during Nalu-Wind configuration phase. Optionally, the variable -DHYPRE_DIR` can be used to pass the path of HYPRE install location to CMake.

ParaView Catalyst

Optionally enable ParaView Catalyst for in-situ visualization with Nalu-Wind. These instructions can be skipped if you do not require in-situ visualization with Nalu-Wind. The first thing you will need to do is build Paraview yourself using their SuperBuild instructions.

Build Nalu-Wind ParaView Catalyst Adapter

Next you will need to build the Catalyst adapter for Trilinos to hook into Paraview. The adapter is located in the Trilinos repo at Trilinos/packages/seacas/libraries/ioss/src/visualization/ParaViewCatalystIossAdapter. To install:

cd Trilinos/packages/seacas/libraries/ioss/src/visualization/ParaViewCatalystIossAdapter
cmake -DParaView_DIR:PATH=/path/to/paraview/lib/cmake/paraview_version -DCMAKE_INSTALL_PREFIX:PATH=${NALU_ROOT_DIR}/install
make
make install

Nalu-Wind

Nalu-Wind is provided here. The master branch of Nalu-Wind typically matches with the master branch or develop branch of Trilinos. If it is necessary to build an older version of Nalu-Wind, refer to the history of the Nalu git repo for instructions on doing so.

Prepare:

git clone https://github.com/Exawind/nalu-wind.git

Build

Create a nalu-wind/build directory and execute something similar to following commands. The general commands for configuring and building Nalu-Wind are listed below. We show a script which uses modules which populate the <PACKAGE>_ROOT_DIR locations for the NREL Eagle machine, but it will need to be modified with the specific TPL locations you have used.

#!/bin/bash -l

# Instructions:
# Make a directory in the Nalu-Wind directory for building,
# Copy this script to that directory and edit the
# options below to your own needs and run it.

CXX_COMPILER=mpicxx
C_COMPILER=mpicc
FLAGS="-O2 -march=native -mtune=native"
OVERSUBSCRIBE_FLAGS="--use-hwthread-cpus --oversubscribe"
  
set -e

TRILINOS_ROOT_DIR=${NALU_ROOT_DIR}/install/trilinos
YAML_CPP_ROOT_DIR=${NALU_ROOT_DIR}/install/yaml-cpp

# Clean before cmake configure
set +e
rm -rf CMakeFiles
rm -f CMakeCache.txt
set -e

# Extra TPLs that can be included in the cmake configure:
#  -DENABLE_PARAVIEW_CATALYST:BOOL=ON \
#  -DPARAVIEW_CATALYST_INSTALL_PATH:PATH=${CATALYST_IOSS_ADAPTER_ROOT_DIR} \
#  -DENABLE_OPENFAST:BOOL=ON \
#  -DOpenFAST_DIR:PATH=${OPENFAST_ROOT_DIR} \
#  -DENABLE_HYPRE:BOOL=ON \
#  -DHYPRE_DIR:PATH=${HYPRE_ROOT_DIR} \
#  -DENABLE_TIOGA:BOOL=ON \
#  -DTIOGA_DIR:PATH=${TIOGA_ROOT_DIR} \

(set -x; cmake \
  -DCMAKE_CXX_COMPILER:STRING=${CXX_COMPILER} \
  -DCMAKE_CXX_FLAGS:STRING="${FLAGS}" \
  -DMPI_CXX_COMPILER:STRING=${CXX_COMPILER} \
  -DMPIEXEC_PREFLAGS:STRING="${OVERSUBSCRIBE_FLAGS}" \
  -DTrilinos_DIR:PATH=${TRILINOS_ROOT_DIR} \
  -DYAML_DIR:PATH=${YAML_CPP_ROOT_DIR} \
  -DCMAKE_BUILD_TYPE:STRING=RELEASE \
  -DENABLE_TESTS:BOOL=ON \
  ..)

(set -x; nice make -j 16)

This process will create naluX within the nalu-wind/build location.