.. _reference: Reference ########## Configuration variables ======================= This section documents all the available options that the user can use to customize the build process. It is divided into common options (most begin with ``EXAWIND_`` prefix) and code-specific parameters under individual projects. ExaWind Builder configuration ----------------------------- .. envvar:: EXAWIND_SYSTEM The system code used to determine modules to be loaded. Please see :ref:`available systems ` for a list of valid systems supported by exawind-builder. .. envvar:: EXAWIND_COMPILER The compiler to be used for the build. Valid options are `gcc`, `clang`, `intel`, and `xl`. Not all compiler options are available on all systems. Please consult :ref:`available compilers ` on individual systems. .. envvar:: EXAWIND_CODE The software that is being compiled. This is used to load the project-specific CMake configuration as well as to perform conditional evaluation of code within :file:`${EXAWIND_PROJECT_DIR}/exawind-config.sh` .. envvar:: EXAWIND_SRCDIR Absolute path to the location of ``exawind-builder``. This is automatically generated by the script and should not be changed. .. envvar:: EXAWIND_PROJECT_DIR The root directory where all ExaWind code is located. In the :ref:`introduction` section the examples used :file:`${HOME}/exawind`. By default, the :ref:`build script generator ` will initialize this to be the parent directory of :envvar:`EXAWIND_SRCDIR`. If you prefer to keep ``exawind-builder`` and your base install directory separate, then ``export`` this variable to the appropriate value before invoking ``new-script.sh`` command. .. envvar:: EXAWIND_INSTALL_DIR The default location where custom builds are installed. Default value is :file:`${EXAWIND_PROJECT_DIR}/install`. By default, each project is installed within its own directory (:file:`${EXAWIND_INSTALL_DIR}/${PROJECT_NAME}`). User can change this by setting appropriate value for the :envvar:`project install variable PROJECTNAME_INSTALL_PREFIX`. .. envvar:: EXAWIND_CONFIG Absolute path to the ExaWind builder configuration file. Default value is :file:`${EXAWIND_PROJECT_DIR}/exawind-config.sh`. .. note:: The variables described above are set when :ref:`generating build scripts ` and rarely needs to be changed by the user. .. envvar:: EXAWIND_CFGFILE The basename of the file where configuration is stored. The default value is ``exawind-config``. .. envvar:: EXAWIND_MODMAP A dictionary containing the exact resolution of the module that must be loaded. For example, on NREL Peregrine the builder will load ``trilinos/develop`` module by default. However, if the user prefers the ``develop`` branch with OpenMP enabled, then they can override it by providing the following either in the build script or the :file:`exawind-config.sh` configuration file. .. code-block:: bash # Use develop branch of trilinos that has OpenMP enabled EXAWIND_MODMAP[trilinos]=trilinos/develop-omp For system configuration using Spack, the compiler flag (e.g., ``%gcc``) is automatically added to the spec. .. envvar:: EXAWIND_MOD_LOADER This variable determins whether :program:`spack` or :program:`module` is used to load dependencies when compiling codes. The default value is set by the system that users are using ``exawind-builder`` on and rarely needs to be changed by the user. .. envvar:: EXAWIND_EXTRA_USER_MODULES A list of extra modules that must be loaded before performing any actions. .. envvar:: EXAWIND_NUM_JOBS The maximum number of parallel build jobs to execute when ``make`` is invoked. Setting this variable within the build script is equivalent to passing ``-j X`` at the command line for ``make``. .. envvar:: EXAWIND_CUDA_WRAPPER Absolute path to the location of :file:`nvcc_wrapper` script provided by Kokkos. The default path is assumed to be :file:`${EXAWIND_PROJECT_DIR}/exawind-builder/utils/nvcc_wrapper` .. envvar:: EXAWIND_CUDA_SM Variable used to set the target architecture for CUDA compilations. This variable is a numeric value. For example, when targeting Volta cards the desired ``nvcc`` option is ``-arch=sm_70``, then set this variable to ``70``. Currently, used by non-Trilinos codes like HYPRE, PIFUS, and TIOGA. .. envvar:: EXAWIND_ARCH_FLAGS Additional architecture specific optimization flags, e.g., to enable SIMD optimizations. .. envvar:: KOKKOS_ARCH The architectures for which Kokkos builds are optimized. See `Kokkos Wiki `__ for further information. Multiple architectures can be separated by commas. .. envvar:: CUDA_LAUNCH_BLOCKING Variable set to control Kokkos configuration. Defaults to 1. See `Kokkos Wiki `__ for more details. .. envvar:: CUDA_MANAGED_FORCE_DEVICE_ALLOC Variable necessary when CUDA UVM is enabled (currently required for certain Trilinos packages) that manages device allocation. Default value is 1. See `Kokkos Wiki `__ for more details. .. envvar:: SPACK_ROOT Absolute path to the spack installation, if using spack to manage dependencies. The default path is :file:`${EXAWIND_PROJECT_DIR}/spack`. .. envvar:: SPACK_COMPILER Variable controlling the compiler used by spack to install dependencies. .. _ref-project-vars: Variables controlling project properties ---------------------------------------- These variables all start with the project name. The convention is that the project name is converted to all upper case and any dashes are replaced by underscores. For example, ``parallel-netcdf`` becomes ``PARALLEL_NETCDF_ROOT_DIR``, SuperLU becomes ``SUPERLU_ROOT_DIR`` and so on. .. envvar:: PROJECTNAME_ROOT_DIR The use can declare a variable (e.g., ``OPENFAST_ROOT_DIR``) to provide a path to a custom installation of a particular dependency and bypass the module search and load process. A typical example is to provide the following line either in the build script or the :file:`exawind-config.sh` configuration file. .. code-block:: bash export OPENFAST_ROOT_DIR=${EXAWIND_INSTALL_DIR}/openfast-dev-debug The primary purpose of this variable is to indicate this as the search path during the build process of other projects. Currently the following ``ROOT_DIR`` variables are used within the scripts:: AMREX_ROOT_DIR AMR_WIND_ROOT_DIR BOOST_ROOT_DIR FFTW_ROOT_DIR HDF5_ROOT_DIR HYPRE_ROOT_DIR MASA_ROOT_DIR NALU_WIND_ROOT_DIR NETCDF_ROOT_DIR OPENFAST_ROOT_DIR PARALLEL_NETCDF_ROOT_DIR SUPERLU_ROOT_DIR TIOGA_ROOT_DIR TRILINOS_ROOT_DIR YAML_CPP_ROOT_DIR ZLIB_ROOT_DIR .. envvar:: PROJECTNAME_INSTALL_PREFIX The location where ``make install`` will install the project. The default value for this variable is ``${EXAWIND_INSTALL_DIR}/${PROJECT_NAME}`` .. envvar:: PROJECTNAME_SOURCE_DIR This variable is used in situations where the ``build`` directory is not a subdirectory located at the root of the project source directory. The default value is just the parent directory from where the script is executed. Variables controlling build process ----------------------------------- This section describes various environment variables that control the build process for individual projects. Common build variables ~~~~~~~~~~~~~~~~~~~~~~ .. envvar:: BUILD_TYPE Control the type of build, e.g., Release, Debug, RelWithDebInfo, etc. .. envvar:: BUILD_SHARED_LIBS Control whether shared libraries or static libraries are built. Valid values: ``ON`` or ``OFF``. .. envvar:: BLASLIB Path to BLAS/LAPACK libraries. .. envvar:: EXAWIND_MKL_LIBNAMES List of MKL libraries to link to when compiling with Intel MKL. Used as an alterative to :envvar:`BLASLIB`, see also :envvar:`EXAWIND_MKL_LIBDIRS`. .. envvar:: EXAWIND_MKL_LIBDIRS Path to Intel MKL libraries, always used in conjunction with :envvar:`EXAWIND_MKL_LIBNAMES`. .. envvar:: ENABLE_OPENMP Boolean flag indicating whether OpenMP is enabled. (default: ON) .. envvar:: ENABLE_CUDA Boolean flag indicating whether CUDA is enabled. The default value is OFF on most architectures. Exceptions are: ORNL Summit, SNL ascicgpu. .. envvar:: ENABLE_HIP Boolean flag indicating whether AMD/HIP is enabled. Default is OFF on most architectures. .. envvar:: ENABLE_DPCPP Boolean flag indicating whether Intel OneAPI/DPC++ support is enabled. Default is OFF on most architectures. AMR-Wind ~~~~~~~~~ .. envvar:: ENABLE_HYPRE Build with HYPRE support .. envvar:: ENABLE_MASA Build with MASA support .. envvar:: ENABLE_NETCDF Build with NetCDF support .. envvar:: AMR_WIND_ENABLE_MPI Boolean flag indicating whether MPI support is enabled when building AMR-Wind. .. envvar:: AMR_WIND_ENABLE_FCOMPARE Boolean flag indicating whether ``fcompare`` is used with tests. .. envvar:: AMR_WIND_TEST_WITH_FCOMPARE Boolean flag indicating whether regresstion tests check against gold files. Nalu-Wind ~~~~~~~~~ .. envvar:: ENABLE_FFTW Boolean flag indicating whether FFTW library is activated when building Nalu-Wind. (default: ON) .. envvar:: ENABLE_OPENFAST Boolean flag indicating whether OPENFAST TPL is activated when building Nalu-Wind. (default: ON) .. envvar:: ENABLE_HYPRE Boolean flag indicating whether HYPRE TPL is activated when building Nalu-Wind. (default: ON) .. envvar:: ENABLE_TIOGA Boolean flag indicating whether TIOGA TPL is activated when building Nalu-Wind. (default: ON) .. envvar:: ENABLE_TESTS Boolean flag indicating whether tests are enabled when building Nalu-Wind. (default: ON) OpenFAST ~~~~~~~~ .. envvar:: FAST_CPP_API Boolean flag indicating whether the C++ API is enabled. (default: ON) Other variables used: :envvar:`BUILD_SHARED_LIBS`, :envvar:`BUILD_TYPE`, and :envvar:`BLASLIB`. Trilinos ~~~~~~~~ Trilinos uses :envvar:`ENABLE_OPENMP`, :envvar:`ENABLE_CUDA` and :envvar:`BLASLIB` if configured. OpenMP is enabled by default, and CMake attempts to automatically detect BLAS/LAPACK. CUDA is enabled by default on ``summitdev``, ``snl-ascicgpu``, and is optionally available on ``eagle``. .. envvar:: EXAWIND_USE_BLASLIB Use :envvar:`BLASLIB` variable declared by Intel MKL for BLAS/LAPACK. The default is ``ON``. The user can set this to ``OFF`` to force builds with Spack's netlib-lapack package. HYPRE ~~~~~ HYPRE uses :envvar:`ENABLE_OPENMP` and :envvar:`ENABLE_CUDA` if configured. Both OpenMP and CUDA are disabled by default for HYPRE builds. .. envvar:: ENABLE_BIGINT Boolean flag indicating whether 64-bit integer support is enabled. (default: ON) Function reference ================== User customization functions ---------------------------- .. function:: exawind_env_user_actions A function that is called after the base environment is loaded to allow the user to further customize the environment that will be used to configure and compile software. Core functions -------------- .. function:: exawind_purge_env Purge an Exawind environment created by sourcing bash files, e.g., ``exawind-env-gcc.sh``. If this function exists in your environment, chances are that the environment was sourced previously. Calling this function will unset all exawind-builder related environment variables and functions, and reset the environment to a clean state. Use this function if you see spurious errors caused by conflicting environment variable settings. .. function:: exawind_save_func old new Create a new function with the implementation of the old one. This is used to override functions within bash but still retain the ability to call the old function within the new one. .. code-block:: bash # Example to override the cmake_function exawind_save_func exawind_cmake exawind_cmake_orig exawind_cmake () { echo "Executing CMAKE" exawind_cmake_orig "$@" } .. function:: exawind_env () Activates the environment for a particular system and compiler combination. The actual function is implemented in system specific files and are of the form ``exawind_env_${EXAWIND_COMPILER}``. .. function:: exawind_cmake [arg1 [arg2 ...]] Invoke CMake configuration step for a particular project with additional arguments. If the project defines ``exawind_cmake_${EXAWIND_SYSTEM}`` then that function is invoked, else it invokes :func:`exawind_cmake_base`. All software codes are required to provide the base function. .. function:: exawind_cmake_full Removes :file:`CMakeCache.txt` and :file:`CMakeFiles` directory before invoking :func:`exawind_cmake`. .. function:: exawind_make [args...] Invokes ``make`` to compile the project. With no arguments, it will invoke ``make -j ${EXAWIND_NUM_PROCS}`` otherwise it will pass user arguments to ``make``. Note, if passing arguments you must also pass ``-j `` for parallel builds, e.g., ``make VERBOSE=1 -j 12``. .. function:: exawind_ctest [args...] Invokes CTest runs if the software supports tests via CTest. .. code-block:: bash exawind_ctest --output-on-failure -R ablNeutralEdge .. function:: exawind_run [args...] Runs an arbitrary command within the environment used to build the code .. function:: exawind_guess_make_type Helper function to determine whether to use :program:`make` or :program:`ninja` when compiling the code. System specific functions ------------------------- .. function:: exawind_spack_env Configure Spack environment and set up module loading .. function:: exawind_env_${EXAWIND_COMPILER} Configuration for the :envvar:`${EXAWIND_COMPILER}` if supported on this particular system. .. function:: exawind_load_modules [dep ...] Uses :program:`module load` command to load modules. This is a specialization of :func:`exawind_load_deps` on systems that have all dependencies available via modules. Examples are: NREL Eagle, Peregrine, and Rhodes. .. function:: exawind_load_spack [dep ...] Uses :program:`spack load` command to load dependencies. This is a specialization of :func:`exawind_load_deps` on most systems which uses spack to manage all dependencies. .. function:: exawind_load_deps dep [dep ...] Loads the required dependencies either via spack or module load. Users should use this command elsewhere. .. function:: exawind_default_install_dir dep Check if the default installation location for a project (``${EXAWIND_INSTALL_DIR}/${PROJECT_NAME}``) exists and if so set :envvar:`${PROJECTNAME_ROOT_DIR}`. Project specific functions -------------------------- .. function:: exawind_cmake_base [args...] Base implementation of CMake configure for the project. .. function:: exawind_project_env Additional project configuration. Usually this just is a simple call to :func:`exawind_load_deps` with the list of required dependencies. .. function:: exawind_cmake_${EXAWIND_SYSTEM} Optional system-specific configuration. For example, on Mac OSX ``nalu-wind`` declares the following function to enable running CTest on more than four MPI ranks with OpenMPI v3.0.0 or greater. .. code-block:: bash exawind_cmake_osx () { local extra_args="$@" exawind_cmake_base \ -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=ON \ -DMPIEXEC_PREFLAGS:STRING='"--use-hwthread-cpus --oversubscribe"' \ ${extra_args} }