Reference

Configuration variables

This section documents all the available options that the user can use to customize the build process. It is divided into common options (most begin with EXAWIND_ prefix) and code-specific parameters under individual projects.

ExaWind Builder configuration

EXAWIND_SYSTEM

The system code used to determine modules to be loaded. Please see available systems for a list of valid systems supported by exawind-builder.

EXAWIND_COMPILER

The compiler to be used for the build. Valid options are gcc, clang, intel, and xl. Not all compiler options are available on all systems. Please consult available compilers on individual systems.

EXAWIND_CODE

The software that is being compiled. This is used to load the project-specific CMake configuration as well as to perform conditional evaluation of code within $EXAWIND_PROJECT_DIR/exawind-config.sh

EXAWIND_SRCDIR

Absolute path to the location of exawind-builder. This is automatically generated by the script and should not be changed.

EXAWIND_PROJECT_DIR

The root directory where all ExaWind code is located. In the Introduction section the examples used $HOME/exawind.

By default, the build script generator will initialize this to be the parent directory of EXAWIND_SRCDIR. If you prefer to keep exawind-builder and your base install directory separate, then export this variable to the appropriate value before invoking new-script.sh command.

EXAWIND_INSTALL_DIR

The default location where custom builds are installed. Default value is $EXAWIND_PROJECT_DIR/install. By default, each project is installed within its own directory ($EXAWIND_INSTALL_DIR/$PROJECT_NAME). User can change this by setting appropriate value for the project install variable PROJECTNAME_INSTALL_PREFIX.

EXAWIND_CONFIG

Absolute path to the ExaWind builder configuration file. Default value is $EXAWIND_PROJECT_DIR/exawind-config.sh.

Note

The variables described above are set when generating build scripts and rarely needs to be changed by the user.

EXAWIND_CFGFILE

The basename of the file where configuration is stored. The default value is exawind-config.

EXAWIND_MODMAP

A dictionary containing the exact resolution of the module that must be loaded. For example, on NREL Peregrine the builder will load trilinos/develop module by default. However, if the user prefers the develop branch with OpenMP enabled, then they can override it by providing the following either in the build script or the exawind-config.sh configuration file.

# Use develop branch of trilinos that has OpenMP enabled
EXAWIND_MODMAP[trilinos]=trilinos/develop-omp

For system configuration using Spack, the compiler flag (e.g., %gcc) is automatically added to the spec.

EXAWIND_MOD_LOADER

This variable determins whether spack or module is used to load dependencies when compiling codes. The default value is set by the system that users are using exawind-builder on and rarely needs to be changed by the user.

EXAWIND_EXTRA_USER_MODULES

A list of extra modules that must be loaded before performing any actions.

EXAWIND_NUM_JOBS

The maximum number of parallel build jobs to execute when make is invoked. Setting this variable within the build script is equivalent to passing -j X at the command line for make.

EXAWIND_CUDA_WRAPPER

Absolute path to the location of nvcc_wrapper script provided by Kokkos. The default path is assumed to be $EXAWIND_PROJECT_DIR/exawind-builder/utils/nvcc_wrapper

EXAWIND_CUDA_SM

Variable used to set the target architecture for CUDA compilations. This variable is a numeric value. For example, when targeting Volta cards the desired nvcc option is -arch=sm_70, then set this variable to 70. Currently, used by non-Trilinos codes like HYPRE, PIFUS, and TIOGA.

EXAWIND_ARCH_FLAGS

Additional architecture specific optimization flags, e.g., to enable SIMD optimizations.

KOKKOS_ARCH

The architectures for which Kokkos builds are optimized. See Kokkos Wiki for further information. Multiple architectures can be separated by commas.

CUDA_LAUNCH_BLOCKING

Variable set to control Kokkos configuration. Defaults to 1.

See Kokkos Wiki for more details.

CUDA_MANAGED_FORCE_DEVICE_ALLOC

Variable necessary when CUDA UVM is enabled (currently required for certain Trilinos packages) that manages device allocation. Default value is 1.

See Kokkos Wiki for more details.

SPACK_ROOT

Absolute path to the spack installation, if using spack to manage dependencies. The default path is $EXAWIND_PROJECT_DIR/spack.

SPACK_COMPILER

Variable controlling the compiler used by spack to install dependencies.

Variables controlling project properties

These variables all start with the project name. The convention is that the project name is converted to all upper case and any dashes are replaced by underscores. For example, parallel-netcdf becomes PARALLEL_NETCDF_ROOT_DIR, SuperLU becomes SUPERLU_ROOT_DIR and so on.

PROJECTNAME_ROOT_DIR

The use can declare a variable (e.g., OPENFAST_ROOT_DIR) to provide a path to a custom installation of a particular dependency and bypass the module search and load process. A typical example is to provide the following line either in the build script or the exawind-config.sh configuration file.

export OPENFAST_ROOT_DIR=${EXAWIND_INSTALL_DIR}/openfast-dev-debug

The primary purpose of this variable is to indicate this as the search path during the build process of other projects.

Currently the following ROOT_DIR variables are used within the scripts:

AMREX_ROOT_DIR
AMR_WIND_ROOT_DIR
BOOST_ROOT_DIR
FFTW_ROOT_DIR
HDF5_ROOT_DIR
HYPRE_ROOT_DIR
MASA_ROOT_DIR
NALU_WIND_ROOT_DIR
NETCDF_ROOT_DIR
OPENFAST_ROOT_DIR
PARALLEL_NETCDF_ROOT_DIR
SUPERLU_ROOT_DIR
TIOGA_ROOT_DIR
TRILINOS_ROOT_DIR
YAML_CPP_ROOT_DIR
ZLIB_ROOT_DIR
PROJECTNAME_INSTALL_PREFIX

The location where make install will install the project. The default value for this variable is ${EXAWIND_INSTALL_DIR}/${PROJECT_NAME}

PROJECTNAME_SOURCE_DIR

This variable is used in situations where the build directory is not a subdirectory located at the root of the project source directory. The default value is just the parent directory from where the script is executed.

Variables controlling build process

This section describes various environment variables that control the build process for individual projects.

Common build variables

BUILD_TYPE

Control the type of build, e.g., Release, Debug, RelWithDebInfo, etc.

BUILD_SHARED_LIBS

Control whether shared libraries or static libraries are built. Valid values: ON or OFF.

BLASLIB

Path to BLAS/LAPACK libraries.

EXAWIND_MKL_LIBNAMES

List of MKL libraries to link to when compiling with Intel MKL. Used as an alterative to BLASLIB, see also EXAWIND_MKL_LIBDIRS.

EXAWIND_MKL_LIBDIRS

Path to Intel MKL libraries, always used in conjunction with EXAWIND_MKL_LIBNAMES.

ENABLE_OPENMP

Boolean flag indicating whether OpenMP is enabled. (default: ON)

ENABLE_CUDA

Boolean flag indicating whether CUDA is enabled. The default value is OFF on most architectures. Exceptions are: ORNL Summit, SNL ascicgpu.

ENABLE_HIP

Boolean flag indicating whether AMD/HIP is enabled. Default is OFF on most architectures.

ENABLE_DPCPP

Boolean flag indicating whether Intel OneAPI/DPC++ support is enabled. Default is OFF on most architectures.

AMR-Wind

ENABLE_HYPRE

Build with HYPRE support

ENABLE_MASA

Build with MASA support

ENABLE_NETCDF

Build with NetCDF support

AMR_WIND_ENABLE_MPI

Boolean flag indicating whether MPI support is enabled when building AMR-Wind.

AMR_WIND_ENABLE_FCOMPARE

Boolean flag indicating whether fcompare is used with tests.

AMR_WIND_TEST_WITH_FCOMPARE

Boolean flag indicating whether regresstion tests check against gold files.

Nalu-Wind

ENABLE_FFTW

Boolean flag indicating whether FFTW library is activated when building Nalu-Wind. (default: ON)

ENABLE_OPENFAST

Boolean flag indicating whether OPENFAST TPL is activated when building Nalu-Wind. (default: ON)

ENABLE_HYPRE

Boolean flag indicating whether HYPRE TPL is activated when building Nalu-Wind. (default: ON)

ENABLE_TIOGA

Boolean flag indicating whether TIOGA TPL is activated when building Nalu-Wind. (default: ON)

ENABLE_TESTS

Boolean flag indicating whether tests are enabled when building Nalu-Wind. (default: ON)

OpenFAST

FAST_CPP_API

Boolean flag indicating whether the C++ API is enabled. (default: ON)

Other variables used: BUILD_SHARED_LIBS, BUILD_TYPE, and BLASLIB.

Trilinos

Trilinos uses ENABLE_OPENMP, ENABLE_CUDA and BLASLIB if configured. OpenMP is enabled by default, and CMake attempts to automatically detect BLAS/LAPACK.

CUDA is enabled by default on summitdev, snl-ascicgpu, and is optionally available on eagle.

EXAWIND_USE_BLASLIB

Use BLASLIB variable declared by Intel MKL for BLAS/LAPACK. The default is ON. The user can set this to OFF to force builds with Spack’s netlib-lapack package.

HYPRE

HYPRE uses ENABLE_OPENMP and ENABLE_CUDA if configured. Both OpenMP and CUDA are disabled by default for HYPRE builds.

ENABLE_BIGINT

Boolean flag indicating whether 64-bit integer support is enabled. (default: ON)

Function reference

User customization functions

exawind_env_user_actions()

A function that is called after the base environment is loaded to allow the user to further customize the environment that will be used to configure and compile software.

Core functions

exawind_purge_env()

Purge an Exawind environment created by sourcing bash files, e.g., exawind-env-gcc.sh. If this function exists in your environment, chances are that the environment was sourced previously. Calling this function will unset all exawind-builder related environment variables and functions, and reset the environment to a clean state. Use this function if you see spurious errors caused by conflicting environment variable settings.

exawind_save_func old new

Create a new function with the implementation of the old one. This is used to override functions within bash but still retain the ability to call the old function within the new one.

# Example to override the cmake_function
exawind_save_func exawind_cmake exawind_cmake_orig

exawind_cmake ()
{
    echo "Executing CMAKE"
    exawind_cmake_orig "$@"
}
exawind_env()

Activates the environment for a particular system and compiler combination. The actual function is implemented in system specific files and are of the form exawind_env_${EXAWIND_COMPILER}.

exawind_cmake [arg1 [arg2 ...]]

Invoke CMake configuration step for a particular project with additional arguments. If the project defines exawind_cmake_${EXAWIND_SYSTEM} then that function is invoked, else it invokes exawind_cmake_base(). All software codes are required to provide the base function.

exawind_cmake_full()

Removes CMakeCache.txt and CMakeFiles directory before invoking exawind_cmake().

exawind_make [args...]

Invokes make to compile the project. With no arguments, it will invoke make -j ${EXAWIND_NUM_PROCS} otherwise it will pass user arguments to make. Note, if passing arguments you must also pass -j <N> for parallel builds, e.g., make VERBOSE=1 -j 12.

exawind_ctest [args...]

Invokes CTest runs if the software supports tests via CTest.

exawind_ctest --output-on-failure -R ablNeutralEdge
exawind_run [args...]

Runs an arbitrary command within the environment used to build the code

exawind_guess_make_type()

Helper function to determine whether to use make or ninja when compiling the code.

System specific functions

exawind_spack_env()

Configure Spack environment and set up module loading

exawind_env_${EXAWIND_COMPILER}

Configuration for the ${EXAWIND_COMPILER} if supported on this particular system.

exawind_load_modules [dep ...]

Uses module load command to load modules. This is a specialization of exawind_load_deps() on systems that have all dependencies available via modules. Examples are: NREL Eagle, Peregrine, and Rhodes.

exawind_load_spack [dep ...]

Uses spack load command to load dependencies. This is a specialization of exawind_load_deps() on most systems which uses spack to manage all dependencies.

exawind_load_deps dep [dep ...]

Loads the required dependencies either via spack or module load. Users should use this command elsewhere.

exawind_default_install_dir dep

Check if the default installation location for a project (${EXAWIND_INSTALL_DIR}/${PROJECT_NAME}) exists and if so set ${PROJECTNAME_ROOT_DIR}.

Project specific functions

exawind_cmake_base [args...]

Base implementation of CMake configure for the project.

exawind_project_env()

Additional project configuration. Usually this just is a simple call to exawind_load_deps() with the list of required dependencies.

exawind_cmake_${EXAWIND_SYSTEM}

Optional system-specific configuration. For example, on Mac OSX nalu-wind declares the following function to enable running CTest on more than four MPI ranks with OpenMPI v3.0.0 or greater.

exawind_cmake_osx ()
{
    local extra_args="$@"
    exawind_cmake_base \
        -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=ON \
        -DMPIEXEC_PREFLAGS:STRING='"--use-hwthread-cpus --oversubscribe"' \
        ${extra_args}
}