Reference¶

Configuration variables¶

This section documents all the available options that the user can use to customize the build process. It is divided into common options (most begin with EXAWIND_ prefix) and code-specific parameters under individual projects.

ExaWind Builder configuration¶

EXAWIND_SYSTEM¶: The system code used to determine modules to be loaded. Please see available systems for a list of valid systems supported by exawind-builder.

EXAWIND_COMPILER¶: The compiler to be used for the build. Valid options are gcc, clang, intel, and xl. Not all compiler options are available on all systems. Please consult available compilers on individual systems.

EXAWIND_CODE¶: The software that is being compiled. This is used to load the project-specific CMake configuration as well as to perform conditional evaluation of code within $EXAWIND_PROJECT_DIR/exawind-config.sh

EXAWIND_SRCDIR¶: Absolute path to the location of exawind-builder. This is automatically generated by the script and should not be changed.

EXAWIND_PROJECT_DIR¶

The root directory where all ExaWind code is located. In the Introduction section the examples used $HOME/exawind.

By default, the build script generator will initialize this to be the parent directory of EXAWIND_SRCDIR. If you prefer to keep exawind-builder and your base install directory separate, then export this variable to the appropriate value before invoking new-script.sh command.

EXAWIND_INSTALL_DIR¶: The default location where custom builds are installed. Default value is $EXAWIND_PROJECT_DIR/install. By default, each project is installed within its own directory ($EXAWIND_INSTALL_DIR/$PROJECT_NAME). User can change this by setting appropriate value for the project install variable PROJECTNAME_INSTALL_PREFIX.

EXAWIND_CONFIG¶: Absolute path to the ExaWind builder configuration file. Default value is $EXAWIND_PROJECT_DIR/exawind-config.sh.

Note

The variables described above are set when generating build scripts and rarely needs to be changed by the user.

EXAWIND_CFGFILE¶: The basename of the file where configuration is stored. The default value is exawind-config.

EXAWIND_MODMAP¶

A dictionary containing the exact resolution of the module that must be loaded. For example, on NREL Peregrine the builder will load trilinos/develop module by default. However, if the user prefers the develop branch with OpenMP enabled, then they can override it by providing the following either in the build script or the exawind-config.sh configuration file.

# Use develop branch of trilinos that has OpenMP enabled
EXAWIND_MODMAP[trilinos]=trilinos/develop-omp

For system configuration using Spack, the compiler flag (e.g., %gcc) is automatically added to the spec.

EXAWIND_MOD_LOADER¶: This variable determins whether spack or module is used to load dependencies when compiling codes. The default value is set by the system that users are using exawind-builder on and rarely needs to be changed by the user.

EXAWIND_EXTRA_USER_MODULES¶: A list of extra modules that must be loaded before performing any actions.

EXAWIND_NUM_JOBS¶: The maximum number of parallel build jobs to execute when make is invoked. Setting this variable within the build script is equivalent to passing -j X at the command line for make.

EXAWIND_CUDA_WRAPPER¶: Absolute path to the location of nvcc_wrapper script provided by Kokkos. The default path is assumed to be $EXAWIND_PROJECT_DIR/exawind-builder/utils/nvcc_wrapper

EXAWIND_CUDA_SM¶: Variable used to set the target architecture for CUDA compilations. This variable is a numeric value. For example, when targeting Volta cards the desired nvcc option is -arch=sm_70, then set this variable to 70. Currently, used by non-Trilinos codes like HYPRE, PIFUS, and TIOGA.

EXAWIND_ARCH_FLAGS¶: Additional architecture specific optimization flags, e.g., to enable SIMD optimizations.

KOKKOS_ARCH¶: The architectures for which Kokkos builds are optimized. See Kokkos Wiki for further information. Multiple architectures can be separated by commas.

CUDA_LAUNCH_BLOCKING¶

Variable set to control Kokkos configuration. Defaults to 1.

See Kokkos Wiki for more details.

CUDA_MANAGED_FORCE_DEVICE_ALLOC¶

Variable necessary when CUDA UVM is enabled (currently required for certain Trilinos packages) that manages device allocation. Default value is 1.

See Kokkos Wiki for more details.

SPACK_ROOT¶: Absolute path to the spack installation, if using spack to manage dependencies. The default path is $EXAWIND_PROJECT_DIR/spack.

SPACK_COMPILER¶: Variable controlling the compiler used by spack to install dependencies.

Variables controlling project properties¶

These variables all start with the project name. The convention is that the project name is converted to all upper case and any dashes are replaced by underscores. For example, parallel-netcdf becomes PARALLEL_NETCDF_ROOT_DIR, SuperLU becomes SUPERLU_ROOT_DIR and so on.

PROJECTNAME_ROOT_DIR¶

The use can declare a variable (e.g., OPENFAST_ROOT_DIR) to provide a path to a custom installation of a particular dependency and bypass the module search and load process. A typical example is to provide the following line either in the build script or the exawind-config.sh configuration file.

export OPENFAST_ROOT_DIR=${EXAWIND_INSTALL_DIR}/openfast-dev-debug

The primary purpose of this variable is to indicate this as the search path during the build process of other projects.

Currently the following ROOT_DIR variables are used within the scripts:

AMREX_ROOT_DIR
AMR_WIND_ROOT_DIR
BOOST_ROOT_DIR
FFTW_ROOT_DIR
HDF5_ROOT_DIR
HYPRE_ROOT_DIR
MASA_ROOT_DIR
NALU_WIND_ROOT_DIR
NETCDF_ROOT_DIR
OPENFAST_ROOT_DIR
PARALLEL_NETCDF_ROOT_DIR
SUPERLU_ROOT_DIR
TIOGA_ROOT_DIR
TRILINOS_ROOT_DIR
YAML_CPP_ROOT_DIR
ZLIB_ROOT_DIR

PROJECTNAME_INSTALL_PREFIX¶: The location where make install will install the project. The default value for this variable is ${EXAWIND_INSTALL_DIR}/${PROJECT_NAME}

PROJECTNAME_SOURCE_DIR¶: This variable is used in situations where the build directory is not a subdirectory located at the root of the project source directory. The default value is just the parent directory from where the script is executed.

Variables controlling build process¶

This section describes various environment variables that control the build process for individual projects.

Common build variables¶

BUILD_TYPE¶: Control the type of build, e.g., Release, Debug, RelWithDebInfo, etc.

BUILD_SHARED_LIBS¶: Control whether shared libraries or static libraries are built. Valid values: ON or OFF.

BLASLIB¶: Path to BLAS/LAPACK libraries.

EXAWIND_MKL_LIBNAMES¶: List of MKL libraries to link to when compiling with Intel MKL. Used as an alterative to BLASLIB, see also EXAWIND_MKL_LIBDIRS.

EXAWIND_MKL_LIBDIRS¶: Path to Intel MKL libraries, always used in conjunction with EXAWIND_MKL_LIBNAMES.

ENABLE_OPENMP¶: Boolean flag indicating whether OpenMP is enabled. (default: ON)

ENABLE_CUDA¶: Boolean flag indicating whether CUDA is enabled. The default value is OFF on most architectures. Exceptions are: ORNL Summit, SNL ascicgpu.

ENABLE_HIP¶: Boolean flag indicating whether AMD/HIP is enabled. Default is OFF on most architectures.

ENABLE_DPCPP¶: Boolean flag indicating whether Intel OneAPI/DPC++ support is enabled. Default is OFF on most architectures.

AMR-Wind¶

ENABLE_HYPRE¶: Build with HYPRE support

ENABLE_MASA¶: Build with MASA support

ENABLE_NETCDF¶: Build with NetCDF support

AMR_WIND_ENABLE_MPI¶: Boolean flag indicating whether MPI support is enabled when building AMR-Wind.

AMR_WIND_ENABLE_FCOMPARE¶: Boolean flag indicating whether fcompare is used with tests.

AMR_WIND_TEST_WITH_FCOMPARE¶: Boolean flag indicating whether regresstion tests check against gold files.

Nalu-Wind¶

ENABLE_FFTW¶: Boolean flag indicating whether FFTW library is activated when building Nalu-Wind. (default: ON)

ENABLE_OPENFAST¶: Boolean flag indicating whether OPENFAST TPL is activated when building Nalu-Wind. (default: ON)

ENABLE_HYPRE¶: Boolean flag indicating whether HYPRE TPL is activated when building Nalu-Wind. (default: ON)

ENABLE_TIOGA¶: Boolean flag indicating whether TIOGA TPL is activated when building Nalu-Wind. (default: ON)

ENABLE_TESTS¶: Boolean flag indicating whether tests are enabled when building Nalu-Wind. (default: ON)

OpenFAST¶

FAST_CPP_API¶: Boolean flag indicating whether the C++ API is enabled. (default: ON)

Other variables used: BUILD_SHARED_LIBS, BUILD_TYPE, and BLASLIB.

Trilinos¶

Trilinos uses ENABLE_OPENMP, ENABLE_CUDA and BLASLIB if configured. OpenMP is enabled by default, and CMake attempts to automatically detect BLAS/LAPACK.

CUDA is enabled by default on summitdev, snl-ascicgpu, and is optionally available on eagle.

EXAWIND_USE_BLASLIB¶: Use BLASLIB variable declared by Intel MKL for BLAS/LAPACK. The default is ON. The user can set this to OFF to force builds with Spack’s netlib-lapack package.

HYPRE¶

HYPRE uses ENABLE_OPENMP and ENABLE_CUDA if configured. Both OpenMP and CUDA are disabled by default for HYPRE builds.

ENABLE_BIGINT¶: Boolean flag indicating whether 64-bit integer support is enabled. (default: ON)

Function reference¶

User customization functions¶

exawind_env_user_actions()¶: A function that is called after the base environment is loaded to allow the user to further customize the environment that will be used to configure and compile software.

Core functions¶

exawind_purge_env()¶: Purge an Exawind environment created by sourcing bash files, e.g., exawind-env-gcc.sh. If this function exists in your environment, chances are that the environment was sourced previously. Calling this function will unset all exawind-builder related environment variables and functions, and reset the environment to a clean state. Use this function if you see spurious errors caused by conflicting environment variable settings.

exawind_save_func old new

Create a new function with the implementation of the old one. This is used to override functions within bash but still retain the ability to call the old function within the new one.

# Example to override the cmake_function
exawind_save_func exawind_cmake exawind_cmake_orig

exawind_cmake ()
{
    echo "Executing CMAKE"
    exawind_cmake_orig "$@"
}

exawind_env()¶: Activates the environment for a particular system and compiler combination. The actual function is implemented in system specific files and are of the form exawind_env_${EXAWIND_COMPILER}.

exawind_cmake [arg1 [arg2 ...]]: Invoke CMake configuration step for a particular project with additional arguments. If the project defines exawind_cmake_${EXAWIND_SYSTEM} then that function is invoked, else it invokes exawind_cmake_base(). All software codes are required to provide the base function.

exawind_cmake_full()¶: Removes CMakeCache.txt and CMakeFiles directory before invoking exawind_cmake().

exawind_make [args...]: Invokes make to compile the project. With no arguments, it will invoke make -j ${EXAWIND_NUM_PROCS} otherwise it will pass user arguments to make. Note, if passing arguments you must also pass -j <N> for parallel builds, e.g., make VERBOSE=1 -j 12.

exawind_ctest [args...]

Invokes CTest runs if the software supports tests via CTest.

exawind_ctest --output-on-failure -R ablNeutralEdge

exawind_run [args...]: Runs an arbitrary command within the environment used to build the code

exawind_guess_make_type()¶: Helper function to determine whether to use make or ninja when compiling the code.

System specific functions¶

exawind_spack_env()¶: Configure Spack environment and set up module loading

exawind_env_${EXAWIND_COMPILER}: Configuration for the ${EXAWIND_COMPILER} if supported on this particular system.

exawind_load_modules [dep ...]: Uses module load command to load modules. This is a specialization of exawind_load_deps() on systems that have all dependencies available via modules. Examples are: NREL Eagle, Peregrine, and Rhodes.

exawind_load_spack [dep ...]: Uses spack load command to load dependencies. This is a specialization of exawind_load_deps() on most systems which uses spack to manage all dependencies.

exawind_load_deps dep [dep ...]: Loads the required dependencies either via spack or module load. Users should use this command elsewhere.

exawind_default_install_dir dep: Check if the default installation location for a project (${EXAWIND_INSTALL_DIR}/${PROJECT_NAME}) exists and if so set ${PROJECTNAME_ROOT_DIR}.

Project specific functions¶

exawind_cmake_base [args...]: Base implementation of CMake configure for the project.

exawind_project_env()¶: Additional project configuration. Usually this just is a simple call to exawind_load_deps() with the list of required dependencies.

exawind_cmake_${EXAWIND_SYSTEM}

Optional system-specific configuration. For example, on Mac OSX nalu-wind declares the following function to enable running CTest on more than four MPI ranks with OpenMPI v3.0.0 or greater.

exawind_cmake_osx ()
{
    local extra_args="$@"
    exawind_cmake_base \
        -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=ON \
        -DMPIEXEC_PREFLAGS:STRING='"--use-hwthread-cpus --oversubscribe"' \
        ${extra_args}
}