ExaWind-builder – Quick reference

Activating ExaWind environment

# Adjust variables appropriately
export EXWDIR=/projects/exawind
export COMPILER=gcc

# Source environment in an interactive shell or within a job submission script
${EXWDIR}/scripts/exawind-env-${COMPILER}.sh

# Activate additional modules (e.g., netcdf to use ncdump command)
exawind_load_deps netcdf-c
# Check whether ncdump is available
which ncdump

# Purge all exawind variables/functions from environment
exawind_purge_env

Please refer to Running ExaWind executables and Customizing exawind-builder for more details.

Compiling codes

On systems where ExaWind software stack has been pre-installed and maintained by the ExaWind team, you can execute the following commands to quickly build code.

# Adjust variables appropriately
export EXWDIR=/projects/exawind
export CODE=nalu-wind
export COMPILER=gcc

#### One-time setup
mkdir -p ${HOME}/exawind/source
cd ${HOME}/exawind/source
git clone --recurse-submodules https://github.com/exawind/${CODE}.git
cd ${CODE}
mkdir build-${COMPILER}
ln -s ${EXWDIR}/scripts/${CODE}-${COMPILER}.sh

#### Build executables
# Option 1 - Execute both configure and make steps
cd ${HOME}/exawind/source/${CODE}/build-${COMPILER}
./${CODE}-${COMPILER}.sh

# Option 2 - Execute steps sequentially with arguments
cd ${HOME}/exawind/source/${CODE}/build-${COMPILER}
./${CODE}-${COMPILER}.sh cmake -DENABLE_TESTS=OFF
./${CODE}-${COMPILER}.sh make -j 12

# Option 3 - Force full reconfigure
cd ${HOME}/exawind/source/${CODE}/build-${COMPILER}
./${CODE}-${COMPILER} cmake_full -DENABLE_TESTS=ON
./${CODE}-${COMPILER}.sh make

Detailed documentation sections:

• Compiling executables using build scripts

• Specifying tasks with build scripts

• Customizing exawind-builder

Basic usage

This section describes the basic steps to configure exawind-builder and use the scripts provided to build software used within the ExaWind project. We begin with the simplest example, building one of the ExaWind codes on a system with pre-installed dependencies, and then proceed with more complex workflows. These examples assume that the ExaWind simulation environment has been previously installed on the system you are working on. To determine if a previous installation exists or if you need to install one yourself please refer to How to use ExaWind builder?. To install exawind-builder and dependencies please refer to installation guide.

Compiling executables using build scripts

This tutorial describes the steps involved in biulding your own executable of Nalu-Wind using exawind-builder scripts. You can replace nalu-wind with any of the other codes in ExaWind software suite and follow the same steps to build code. Tutorials on complex workflows will refer back to this tutorial. This tutorial will assume that the path to the pre-installed exawind project directory is /projects/exawind/.

One time setup

Assuming you have not setup your own local directory structure before, we will create a working setup that will be used to do development builds.

1. Create exawind directory layout

   # Path to exawind installation
   export EXAWIND_DIR=/projects/exawind
   
   # Choose directory where you want to work on exawind codes
   export MY_EXAWIND_DIR=${HOME}/exawind
   
   # Create directory structure if you haven't done this previously
   mkdir -p ${HOME}/exawind/source
   

   Note

   If you installed your own exawind (see Installing exawind-builder) and are not using a central installation, then ${EXAWIND_DIR} and ${MY_EXAWIND_DIR} would point to the same location and you can skip this step.

2. Clone desired code repository if you do not have a previously checked out version. In this example, we will use nalu-wind. Please replace nalu-wind with your desired code, e.g., amr-wind in the following steps.

   # Switch to source directory
   cd ${MY_EXAWIND_DIR}/source
   
   # Clone the repository
   git clone --recurse-submodules https://github.com/exawind/nalu-wind.git
   
   # Switch to a different branch if necessary
   

3. Create build directory and link build script. In this tutorial, we will use the gcc compiler. Replace gcc with clang or intel to switch compiler.

   # Switch to previously cloned nalu-wind repository
   cd ${MY_EXAWIND_DIR}/source/nalu-wind
   
   # Create a build directory if one doesn't exist
   mkdir build-gcc
   cd build-gcc
   
   # Create a symbolic link to the build script within the build directory
   ln -s /projects/exawind/scripts/nalu-wind-gcc.sh
   

Note

In this tutorial we assume that the path to the exawind project maintained by ExaWind team is /projects/exawind/. Please change this appropriately based on your system.

Configuring and compiling software

After following the one-time setup steps described above, you can modify the your local copy and compile code using the process described below.

To build code, simply execute the following command

# Switch to build directory if necessary (refer to one-time steps for details)
cd ${MY_EXAWIND_DIR}/source/nalu-wind/build-gcc

# Execute script to compile nalu-wind
./nalu-wind-gcc.sh

When invoked without any arguments, the script will first execute cmake with appropriate arguments to configure the project and then call make to compile the project. On successful compilation, you will have executables in the build directory.

Understanding exawind-builder output

When an exawind-builder script is executed, e.g., as described in previous section, it will output lots of informational messages to the screen. Under most situations, the build process would just work and you can ignore the output. However, in some circumstances, you might encounter errors. This section briefly describes the output from build script that can be used to troubleshoot the error, or submit bug reports to relevant ExaWind projects.

When you execute a build script (e.g., ./nalu-wind-gcc.sh) you will see output as shown below. Note that the paths will be different depending on your system and directory layout.

==> Loading options from /projects/exawind/exawind/exawind-config-gcc.sh
==> Using spack configuration: /projects/exawind/exawind/spack
==> spack: locating mpi%gcc
==> mpi = /projects/exawind/exawind/spack/opt/spack/linux-centos7-skylake/gcc-8.4.0/mpich-3.3.1-dn34cqtj7tlnxzwamooud6rxbdbkro42
==> spack: locating cmake%gcc
==> cmake = /usr/local
==> spack: locating netlib-lapack%gcc
==> netlib-lapack = /projects/exawind/exawind/spack/opt/spack/linux-centos7-skylake/gcc-8.4.0/netlib-lapack-3.8.0-bmrqbsbwfqaqkjipdhbbm6t4eewxkvr4
==> No user environment actions defined
==> Loading dependencies for nalu-wind...
==> trilinos = /projects/exawind/exawind/install/gcc/trilinos
==> spack: locating yaml-cpp%gcc
==> yaml-cpp = /projects/exawind/exawind/spack/opt/spack/linux-centos7-skylake/gcc-8.4.0/yaml-cpp-0.6.2-k3me2qqeadfw3jzvgwkiagn3hhw23ekv
==> spack: locating boost%gcc
==> boost = /projects/exawind/exawind/spack/opt/spack/linux-centos7-skylake/gcc-8.4.0/boost-1.68.0-ttkfazswxptatzfrohvpn7pjoz5ggqx6
==> spack: locating hypre%gcc
==> hypre = /projects/exawind/exawind/spack/opt/spack/linux-centos7-skylake/gcc-8.4.0/hypre-develop-crw7bxgflmfwoxkv52qqe5tulzqjvnwx
==> tioga = /projects/exawind/exawind/install/gcc/tioga

The messages with the ==> prefix are output from exawind-builder. These messages show the configuration files that are being loaded followed by the paths to the required dependencies that are loaded to enable configuring and building nalu-wind. The output from CMake configure process is simultaneously shown on screen as well as redirected to cmake_output.log. Similarly, the output from make step is shown on screen as well as saved to file make_output.log. Outputs stored in make_output.log is useful when troubleshooting errors in parallel builds as it captures all messages.

Running ExaWind executables

Exawind-builder provides scripts that can be sourced to create appropriate execution environments for the codes either in an interactive console or within job submission scripts. These scripts recreate the exact environment that was used to build the codes. The environment scripts are stored in within the scripts directory (see Exawind directory structure for more details) and are of the form exawind-env-$COMPILER.sh. In the following examples, replace /projects/exawind with the correct exawind installation path.

# Load the environment corresponding to GCC compiler suite
source /projects/exawind/scripts/exawind-env-gcc.sh

In addition to loading the default modules, sourcing this file will also introduce a bash command exawind_load_deps that can be used to load additional codes within the bash environment.

exawind_load_deps nalu-wind amr-wind trilinos wind-utils
nalu_wind_exe=${NALU_WIND_ROOT_DIR}/bin/naluX
amr_wind_exe=${AMR_WIND_ROOT_DIR}/bin/amr_wind
epu_exe=${TRILINOS_ROOT_DIR}/bin/epu

# Generate an ABL mesh using wind utilities executable
${EXAWIND_INSTALL_DIR}/wind-utils/bin/abl_mesh -i abl_mesh.yaml

For example, to access ncdump available in the netcdf module on any system, the user can execute the following

# Activate exawind environment
source /projects/exawind/scripts/exawind-env-gcc.sh
# load the NetCDF module or spack build
exawind_load_deps netcdf

# Now ncdump should be available in your PATH
ncdump -h <exodus_file>

Within interactive sessions, you can deactivate the exawind environment that was created by sourcing the environment file by executing exawind_purge_env command.

# Deactivate exawind environment
exawind_purge_env

Warning

Note that it is not necessary to source the environment for building software. The build scripts will automatically source the environment. We strongly discourage sourcing exawind environment within .bash_profile and .bashrc scripts. Loading default environment will not allow you to switch compilers or change build options to support different types of builds (e.g., using a different hypre library to link against your nalu-wind build).

To ease the process of quickly activating the user environment, we recommend using functions within your .bashrc scripts. An example is shown below:

# Helper function to load exawind environment in a bash shell
# Execute `load_exawind_env intel` at prompt to load intel environment
function load_exawind_env {
    # Absolute path to exawind project directory
    local exawind_project_dir=/projects/exawind

    # Parse argument to determine compiler type, default is gcc if none provided
    local compiler_type=${1:-gcc}

    source ${exawind_project_dir}/scripts/exawind-env-${compiler_type}.sh
}

Example job submission script

This section shows an example of using the newly built executables within a job script. The example assumes SLURM job manager.

#!/bin/bash

# Example job submission script
#SBATCH --job-name=nalu-wind-exe
#SBATCH --account=exawind
#SBATCH --nodes=30
#SBATCH --time=48:00:00
#SBATCH --output=out.%x_%j

# Path to exawind installation
exawind_dir=/projects/exawind
# Compiler build used
compiler=gcc
# Nalu-Wind exe location
nalu_dir=${HOME}/exawind/source/nalu-wind/build-${compiler}
nalu_exec=${nalu_dir}/naluX

# Input and log files (assume current working directory)
input_file=abl_neutral.yaml
log_file=abl_neutral.log

# Copy the exawind-config if present so that we recreate the exact environment
if [ -f ${nalu_dir}/exawind-config.sh ] ; then
  cp ${nalu_dir}/exawind-config.sh .
fi
# Purge all modules to ensure a clean environment
module purge
# Source exawind environment
source ${HOME}/exawind/scripts/exawind-env-gcc.sh

ranks_per_node=36
mpi_ranks=$(expr $SLURM_JOB_NUM_NODES \* $ranks_per_node)
export OMP_NUM_THREADS=1  # Max hardware threads = 4
export OMP_PLACES=threads
export OMP_PROC_BIND=spread


echo "Job name       = $SLURM_JOB_NAME"
echo "Num. nodes     = $SLURM_JOB_NUM_NODES"
echo "Num. MPI Ranks = $mpi_ranks"
echo "Num. threads   = $OMP_NUM_THREADS"
echo "Working dir    = $PWD"

srun -n ${mpi_ranks} -c 1 --cpu-bind=cores ${nalu_exec} -i ${input_file} -o ${log_file}

Specifying tasks with build scripts

Often during code, commit, build, debug cycle, it is necessary to control the steps executed using the build scripts. For example, after fixing minor typos, it is not necessary to execute the CMake configure step and only make needs to be executed. Similarly, after changes to CMake configuration files, it might be desirable to purge the CMake cache and execute a fresh configure. Finally, the user might want to run the unit/regression tests and or execute the executable in the same environment that was used to build the code. Exawind-builder scripts take additional arguments that can be used to control the tasks executed as show below.

./nalu-wind-gcc.sh [TASK] [ARGUMENTS]

You can use the -h flag to request a brief help message as shown below

Exawind build script

Usage:
    ./nalu-wind-gcc.sh <task> <arguments>

With no tasks provided, the script will configure the project and compile the code

Available tasks:
    cmake       - configure the project
    cmake_full  - configure project after removing CMakeCache
    make        - compile the code
    ctest       - run tests (if available)
    run         - run arbitrary command using the environment used to compile the code

The available tasks are:

• cmake: Configure the project using CMake and generate build files. Exawind-builder can generate both GNU Makefiles as well as Ninja build scripts. This capability is controlled by the variable EXAWIND_MAKE_TYPE.

• cmake_full: Remove CMakeCache.txt and CMakeFiles before executing CMake configuration step.

• make: Build the project libraries and executables. Note that make is used regardless of whether GNU Makefile or Ninja build system is used.

• ctest: Execute CTest for this project, if available.

• run: Run arbitrary shell command within the same environment (modules and dependencies loaded) as when the project was compiled.

User can control the behavior of these tasks by passing extra [ARGUMENTS] that are passed directly to the task invoked. Some examples are shown below

# Change CMake build type to DEBUG and turn on shared library build
./nalu-wind-gcc.sh cmake -DCMAKE_BUILD_TYPE=DEBUG -DBUILD_SHARED_LIBS=ON

# Turn on verbose output with make and only build naluX (and not unittestX)
./nalu-wind-gcc.sh make VERBOSE=1 naluX

# Only execute one regression test and enable output on failure
./nalu-wind-gcc.sh ctest --output-on-failure -R ablNeutralEdge

# Run the unit test executable from within exawind environment
./nalu-wind-gcc.sh run ./unittestX

Warning

Even though Makefiles are present in the build directory and can be invoked through make, we recommend that you always execute the make step through the build script. This will avoid inconsistencies between the build and the configuration environment that could lead to build or runtime errors.

Note

• Replace nalu-wind-gcc.sh with amr-wind-gcc.sh when working on AMR-Wind. Similarly use nalu-wind-intel.sh when building with Intel compiler suite.

• By default, make will execute several jobs in parallel. Users can control the maximum number of parallel jobs by either setting the environment variable EXAWIND_NUM_JOBS, or using ./nalu-wind-gcc.sh make -j 12 to override the defaults.

• cmake_full accepts all valid CMake arguments that cmake command does.

• The cmake_output.log within the build directory contains the output of the last cmake command that was executed. This output is also echoed to the screen.

• The make_output.log contains the output from the last invocation of make. This output is also simultaneously echoed to the screen.

Customizing exawind-builder

The previous section showed how the execution of CMake and Make can be customized to a limited extent by passing command line arguments with specific tasks. However, for more complex customizations it is recommended that the user use the configuration file to control the build process. This approach allows the user to consolidate common build options, e.g., enabling/disabling OpenMP/CUDA, release/debug builds across all projects consistently through the exawind-config.sh and fine tuning options from the config file within the current working directory. This will allow the user to repeat the build process consistently during development and aid in debugging when things don’t work as expected. The various customizations possible are described below. The code examples shown below must be added to exawind-config.sh within the current working directory (either the build directory or the directory from which an HPC job is executed).

Customizing build options

See project-specific documentation in Reference to see what variables can be used to enable/disable various options for different projects.

# Control accelerator device options
ENABLE_OPENMP=OFF
ENABLE_CUDA=ON
ENABLE_HIP=OFF
ENABLE_DPCPP=OFF

# Set debug/release options
BUILD_TYPE=RELEASE

# Disable TIOGA and OpenFAST, but enable HYPRE as TPLs
ENABLE_TIOGA=OFF
ENABLE_OPENFAST=OFF
ENABLE_HYPRE=ON

# Switch build system
EXAWIND_MAKE_TYPE=ninja # Valid options are auto, make, ninja

# Set number of parallel jobs to execute during make step
EXAWIND_NUM_JOBS=18

Customizing installation location

By default, exawind-builder will setup CMAKE_INSTALL_PREFIX such that make install step will install at $EXAWIND_PROJECT_DIR/$EXAWIND_EXEC_TARGET/$EXAWIND_CODE directory. If you have installed exawind-builder yourself and are not using a shared installation you will not need to change this location. However, when using an upstream exawind-builder you might not have write permissions to that installation location. In this situation you can instruct exawind-builder to choose a custom installation location for when executing make install.

In this example we will assume that you are using an upstream exawind-builder installation at /projects/exawind and you are building your development versions in $HOME/exawind. By default, EXAWIND_INSTALL_DIR will point to a directory within EXAWIND_PROJECT_DIR (which in this case is within /projects/exawind/). To override this when building trilinos, for example, you can use the variable TRILINOS_INSTALL_PREFIX to provide the installation location. Next section shows how this installed trilinos libraries can be used to link against codes that depend on trilinos.

# Custom install location for trilinos with date timestamp
export TRILINOS_INSTALL_PREFIX=${HOME}/exawind/install/gcc8-cuda10/trilinos-$(date %Y-%m-%d)

Customizing dependencies

During development, the user might desire to use a different branch of a dependency than what the default system-wide installation provides. For example, the user might want to use a different branch of OpenFAST when developing advanced FSI capability within Nalu-Wind. The user can bypass the module search/load process by defining ROOT_DIR variable for the corresponding dependency. The following example shows how to customize the TPLs used for building nalu-wind

# Override TPLs used to build nalu-wind
export OPENFAST_ROOT_DIR=${EXAWIND_INSTALL_DIR}/openfast-dbg
export HYPRE_ROOT_DIR=${EXAWIND_INSTALL_DIR}/hypre-cuda

# Example using trilinos from nightly-testing build
export TRILINOS_ROOT_DIR=/projects/exawind-nightly-testing/install/trilinos

# Example using trilinos from custom install location (see previous section)
export TRILINOS_ROOT_DIR=${HOME}/exawind/install/gcc8-cuda10/trilinos-2020-10-08

Tip

To specify the installation location use PROJECT_INSTALL_PREFIX and to use an installed version of the code as a depencency in another project use PROJECTNAME_ROOT_DIR variable.

Customizing build environment init process

The builder provides two options that allows the user to further configure the default environment that is enabled for a given system/compiler combination.

1. To load additional modules, the user can use EXAWIND_EXTRA_USER_MODULES variable to provide the list of modules (in module or spack syntax as appropriate) and have them loaded after the base modules have been loaded.

   # Example showing how to always load HDF5 and NetCDF modules
   EXAWIND_EXTRA_USER_MODULES=( hdf5 netcdf-c )
   

2. Fine-grained customization is achieved by defining by overriding the function exawind_env_user_actions() in the exawind-config.sh configuration file.

   # Load additional modules and print out some variables
   exawind_env_user_actions ()
   {
     # You can inject additional module paths
     module use /opt/hpc_system/modules
   
     # load additional modules
     module load paraview
   
     # manipulate environment variables seen by exawind-builder
     echo ${CXX}
     echo ${TRILINOS_ROOT_DIR}
   }
   

Customizing module load

exawind-builder provides a default list of modules on most systems that work for most use cases. However, the user might desire to use different modules for their custom builds. This is achieved by configuring the modules to be loaded in the EXAWIND_MODMAP variable. The following example shows how the user might switch to a different versions of GCC, MPI, and CUDA modules for building code

EXAWIND_MODMAP[gcc]=gcc/8.4.0
EXAWIND_MODMAP[mpi]=mpich/3.3.1
EXAWIND_MODMAP[cuda]=cuda/10.2.89

Similarly, EXAWIND_MODMAP can also be used to select from multiple versions of software installed via spack. For example, if EXAWIND_DEP_LOADER is set to spack then you can provide custom versions to load

EXAWIND_MODMAP[trilinos]=trilinos@2020-12-01 +cuda
EXAWIND_MODMAP[hypre]=hypre@develop+mpi~int64+cuda+curand

Warning

PROJECTNAME_ROOT_DIR variables take precedence over modules/spack packages listed in EXAWIND_MODMAP. You should verify that the right package is loaded. See Understanding exawind-builder output on more details on how to detect the packages that are used during the build process.

Swapping Spack installations

To aid testing/debugging, exawind-builder has the ability to switch to a different spack installation, e.g., nightly-testing spack installation, using the SPACK_ROOT variable.

# Switch spack installation to the one used for nightly-testing
SPACK_ROOT=/projects/exawind/nightly-testing/spack

# (Optional) customize spack packages in case there are multiple versions
EXAWIND_MODMAP[trilinos]=trilinos@develop%gcc@7.4.0 +cuda+cuda_rdc+wrapper
EXAWIND_MODMAP[hypre]=hypre@develop%gcc@7.4.0

Note

When using a different spack installation, you should not attempt to link to codes built against the old spack install that are in EXAWIND_INSTALL_DIR. You should unset all PROJECTNAME_ROOT_DIR and use packages from spack install, or rebuild new versions of the packages.

Customizing CMake configuration

Previously command line arguments were provided to cmake task to customize the CMake configuration step. While that is suitable for one off modifications, a more robust way to customize the configuration step is to provide a custom CMake function from within the exawind-config.sh script. The following example shows a real-world use case to pass jsrun arguments to CTest when executing regression tests on ORNL Summit system.

# Customize CMake behavior by permanently passing extra arguments
function exawind_cmake {
    # Still allow user to pass arguments through command line
    local extra_args="$@"
    # Pass GPU arguments to CUDA-aware MPI
    local jsrun_args=$'--smpiargs=\x22-gpu\x22'

    exawind_cmake_base \
        -DMPIEXEC_EXECUTABLE='"$(which jsrun) ${jsrun_args}"' \
        -DMPIEXEC_NUMPROC_FLAG="-n" \
        -DMPIEXEC_PREFLAGS='"-a 1 -c 1 -g 1"' \
        ${extra_args}
}

Exawind-builder configuration files

During execution, exawind-builder reads configuration from various files that provide fine-grained control of the build process. The default name for the configuration file is exawind-config, but this can be configured by modifying the EXAWIND_CFGFILE variable. exawind-builder will load the following files in the specified order

${EXAWIND_PROJECT_DIR}/exawind-config-${EXAWIND_SYSTEM}      # Common system settings
${EXAWIND_PROJECT_DIR}/exawind-config-${EXAWIND_EXEC_TARGET} # Execution target settings
${EXAWIND_CONFIG}                                            # File pointed to by the variable
$(pwd)/exawind-config.sh                                     # File in the local working directory

The configuration variables in the subsequent files will override the default values as well as configuration variables set in the previous files. Please replace the path appropriately (EXAWIND_PROJECT_DIR), if you used a non-standard location for installation. See also EXAWIND_CONFIG.

On systems with a shared installation of exawind-builder, please look at the exawind-config-*.sh files within EXAWIND_PROJECT_DIR. The EXAWIND_EXEC_TARGET is typically just the compiler name, e.g., gcc. For CUDA builds, it is either cuda or gcc9-cuda11.

Note

1. It is recommended that the user use local configuration files within build directories to set variables instead of modifying the build scripts within the exawind/scripts directory.

2. If you are using a shared instance of exawind-builder (e.g., on NREL), then please use exawind-config.sh within your build directory to override common configuration parameters.

Tutorial: Custom build of Trilinos and Nalu-Wind

This tutorial presents a complete walkthrough of the steps involved in building custom versions of trilinos and nalu-wind to target execution on NVIDIA GPUs. We will assume that the default system build targets host and doesn’t activate CUDA support by default. In this tutorial we will activate CUDA within the exawind-builder environment, compile Trilinos first with CUDA support, and then build nalu-wind to link to this custom trilinos build.

Create ``exawind-config.sh``

1. (Option 1 – the quick way): You can use this method on systems where Exawind team has already created the necessary configuration for CUDA builds.

   # Set the exawind build target type
   EXAWIND_EXEC_TARGET=gcc8-cuda10
   
   # Install path for trilinos
   TRILINOS_INSTALL_PREFIX=${MY_EXAWIND_DIR}/install/gcc8-cuda10
   
   # Trilinos lookup path for building Nalu-Wind
   TRILINOS_ROOT_DIR=${TRILINOS_INSTALL_PREFIX}
   

   See EXAWIND_PROJECT_DIR to see the available configurations.

2. (Option 2 – the hard way): Provides more control

   ENABLE_CUDA=ON
   EXAWIND_MODMAP[gcc]=gcc/8.4.0
   EXAWIND_MODMAP[cuda]=cuda/10.2.89
   
   # Install path for trilinos
   TRILINOS_INSTALL_PREFIX=${MY_EXAWIND_DIR}/install/gcc8-cuda10/trilinos-2020-10-08
   
   # Trilinos lookup path for building Nalu-Wind
   TRILINOS_ROOT_DIR=${TRILINOS_INSTALL_PREFIX}
   

Build Trilinos

1. Build and install trilinos (see Compiling executables using build scripts for more details)

   cd ${MY_EXAWIND_DIR}/source/trilinos/
   mkdir build-gcc8-cuda10
   cd build-gcc8-cuda10
   
   # Copy exawind-config file
   cp ${HOME}/exawind-config.sh .
   
   # link build script
   ln -s ${EXAWIND_PROJECT_DIR}/scripts/trilinos-gcc.sh
   
   # Configure and build
   ./trilinos-gcc.sh
   # Install to destination directory
   ./trilinos-gcc.sh make install
   

If CUDA support was activated, the scripts will print out message similar to what is shown below. You can also look at cmake_output.log to make sure that Trilinos/Kokkos CUDA support was activated.

==> cuda/10.2.89 = /nopt/nrel/ecom/hpacf/compilers/2020-07/spack/opt/spack/linux-centos7-skylake_avx512/gcc-8.4.0/cuda-10.2.89-rmccd4tpc5gxbbrjeeohphuuujb4cz2o
==> Activated Eagle CUDA programming environment

Build Nalu-Wind

1. Build nalu-wind (see Compiling executables using build scripts for more details)

   cd ${MY_EXAWIND_DIR}/source/nalu-wind/
   mkdir build-gcc8-cuda10
   cd build-gcc8-cuda10
   
   # Copy exawind-config file
   cp ${HOME}/exawind-config.sh .
   
   # link build script
   ln -s ${EXAWIND_PROJECT_DIR}/scripts/nalu-wind-gcc.sh
   
   # Configure and build
   ./nalu-wind-gcc.sh
   # Install to destination directory
   ./nalu-wind-gcc.sh make install
   

The script should output the expected path of the tilinos build that is being used

==> trilinos = ${HOME}/install/gcc8-cuda10/trilinos-2020-10-08