Adding new system configuration

This section documents the process of adding a new system configuration to exawind-builder. Currently, exawind-builder has two major modes of operation: the bootstrap mode, and the software configuration and build mode. The bootstrap mode sets up the basic :ref:<exawind_dir_layout>, configures spack (if necessary), and builds all the dependencies required to compile Trilinos. In the software build mode the it allows users to configure (using CMake) and build Trilinos and Nalu-Wind. The basic steps can be summarized as follows

Preparation

  • Determine a unique name for the system. The recommended naming system is org-system. For example, to create a configuration for ORNL’s Summit system, we will use ornl-summit as the system name in exawind-builder. The convention within exawind-builder is to use this system name consistently to name things: directories containing system-specific configuration, filenames for system specific environment functions, etc. This will be described in detail in the later sections of this documentation. The system specific name will be assigned to EXAWIND_SYSTEM and will be referred to as ${EXAWIND_SYSTEM} in the following sections.

  • Collect necessary data to create a system configuration

Configuration for bootstrap mode

  • Create the minimal build environment necessary for running bootstrap mode, i.e., building dependencies with Spack.

  • Create a Spack configuration allowing use of as many of the available system modules but building the rest within Spack.

Configuration for build mode

  • Create necessary system environment functions to allow users to build Nalu-Wind using different compiler configurations and, optionally, with GPU support.

Note

  • Important: It must be noted that the configuration steps for bootstrap mode are optional. Users can use -s spack for system and have Spack build the entire dependency stack on a new system. The disadvantage of this approach is the long build time for dependencies (particularly MPI) and not being able to use the libraries that have been optimized for the target system (again MPI that might be build with Infiniband, SLURM support, etc.)

  • exawind-builder currently doesn’t follow the recommended system naming convention for several legacy systems (NREL Peregrine and Eagle, NERSC Cori, etc.). The builder evolved from several one-off build scripts and the old names have been retained to preserve backwards compatibility.

  • Nalu-Wind tracks the develop branch of Trilinos for its latest version. This is necessary because ExaWind project has performance and scalability as its primary objectives, and this often requires the latest improvements to Trilinos solvers and Sierra Toolkit (STK) packages.

  • The exawind-builder documentation often only mentions Trilinos as a prerequisite for building Nalu-Wind. However, the process described for building Trilinos and Nalu-Wind should be used to build other prerequisites such as HYPRE, OpenFAST, and TIOGA.

Determine system configuration

  1. Determine what compiler suites you want to support/use on the system, e.g., GCC, Intel, LLVM/Clang, IBM XL, etc.

  2. Determine what software is already available on the system that can be used and what we will need to build ourselves. It is strongly recommended that the user build HDF5, NetCDF, and parallel NetCDF (pNetCDF) through Spack always regardless of whether these modules are available on the system. The Exawind team has experienced a lot of issues with these libraries that lead to runtime errors when loading Exodus files in parallel.

  3. Determine whether you want to build shared or static libraries. Ensure that the libraries available on the login or compute nodes used to build the codes are also available on the nodes where the runs will be performed. When in doubt opt for static library builds, this will increase the size of the executable but is the most robust for the end user.

  4. Determine whether you will be able to download packages (during bootstrap phase) through curl or wget, or if you will have issues with SSL certificates or need proxy servers.

  5. Determine how many parallel builds you are allowed to execute on your system. We will use this to limit the launch of parallel jobs by spack and exawind-builder. When in doubt, 4-8 parallel jobs is a safe number.

Create skeleton directory structure

We will create a minimal exawind structure to clone exawind-builder and add the necessary system files.

# Create top-level exawind directory structure
mkdir -p ${HOME}/exawind/{scripts,install,source}
cd ${HOME}/exawind
git clone git@github.com:exawind/exawind-builder.git

Change the protocol from git to https://` if you have issues cloning using git transport over SSH. For the rest of this documentation, exawind-builder will refer to the path $HOME/exawind/exawind-builder, please adjust appropriately if you are using a non-standard installation location for exawind.

Create minimal bootstrap environment

This step involves loading the necessary compiler, MPI, and CMake modules for use with Spack when running the bootstrap script. This step is optional and is only necessary if the login environment on a system does not correspond to what the user intends to use to build the software. If a specific environment must be setup before running bootstrap, then we will create a system specific file $EXAWIND_SYSTEM.bash in exawind-builder/etc/boostrap directory. The following example shows the contents of nrel-eagle.bash that loads modules necessary to execute bootstrap command on NREL’s Eagle cluster.

#!/bin/bash

# Remove any user modules that might conflict
module purge

# Default build is using GCC compilers
module load gcc/7.3.0
# Load the latest OpenMPI version (build with CUDA support)
module load openmpi/3.1.3

Tip

  • To avoid strange linking errors during the build mode, it is recommended that the bootsrap environment match the final environment you will use in the system environment specification.

  • If your system is behind a firewall, it might be necessary to configure appropriate proxies for HTTP and HTTPS (e.g., SNL systems), look at etc/bootstrap/snl-ghost.bash for examples.

  • If you experience spurious build errors, you might need to configure the temporary directory used by the build systems by configuring the TMPDIR variable to point to a scratch directory.

Create Spack configuration

In this step we will create exact specifications for the compilers spack will use, pin the package versions for all the dependencies, instruct spack which pre-installed dependencies on the system we will use, and (optionally) tell spack about insecure SSL transport requirements and/or limits on the parallel jobs. A system-specific spack configuration is generated by creating a subdirectory exawind-builder/etc/spack/$EXAWIND_SYSTEM/. We will always create two files compilers.yaml and packages.yaml and an optional config.yaml within this directory based on specific requirements for the system.

Spack compiler configuration

The easiest way to determine the compiler configurations available is to load the necessary modules on your system and run spack’s compiler detection command as shown below:

# Load all necessary modules
# Clone a throwaway spack repo if necessary
cd ${HOME}/tmp
git clone https://github.com/spack/spack.git
# Activate the spack environment (assuming bash shell)
source spack/share/spack/setup-env.sh

# Let spack detect compilers
spack compiler find

The above step creates a file $HOME/.spack/$(spack arch -p)/compilers.yaml that can be used as the basis for creating your compiler configuration. This YAML file contains a list of compilers that was detected by spack. Please edit this file and keep only the compilers you want to add to exawind-builder. We recommend removing older versions of GCC etc. that you don’t plan to use. If your desired compiler is not found/detected, you will need to add entries manually. In this case, you should note and reuse the variables operating_system and target from the spack output. Copy the completed file over to exawind-builder/etc/spack/$EXAWIND_SYSTEM/compilers.yaml

See Spack compilers configuration docs for more details.

Note

Make sure you backup and remove the $HOME/.spack/$(spack arch -p) directory as the settings lurking here will take precendence over the ones we will set up using exawind-builder.

Spack package configuration

In this step, we will inform spack the modules/paths of pre-built system libraries we will want to use and the compilers we want spack to be aware of when building packages. Start with exawind-builder/etc/spack/spack/packages.yaml as the basis for building your packages.yaml file. Take a look at other packages.yaml examples in the exawind-builder/etc/spack/ sub-directories to see examples of using system libraries. The general steps involve updating the version, setting buildable: false and providing the list of modules or paths where the library is located. The steps are:

  • Set the order and precendence of compilers

  • Set default package providers for mpi (OpenMPI, MPICH, Intel-MPI, etc.), blas, lapack

  • Set default variants, use ~shared here to enforce static libraries for all packages spack builds. A good default value is +mpi build_type=Release.

Also see Spack build customization for more information.

Spack config.yaml

This file is optional and is necessary when you want to change some of the default behaviors of spack. The variables that often require changing are:

  • build_jobs – Set this to the number of maximum parallel build jobs you are allowed to run on the system.

  • verify_ssl – On some systems, you might have to set this to false to be able to download packages.

Please see Spack docs for other variables that can be configured for your system.

Create system environment configuration

In this step we will create the files necessary to recreate the build environment when building the software. The system-specific configuration is implemented as bash functions stored in the file exawind-builder/envs/$EXAWIND_SYSTEM.bash. This file must implement at least one function exawind_env_${EXAWIND_COMPILER} where EXAWIND_COMPILER is the default compiler option supported for this system. A barebones environment file for a system with only GCC compiler support is shown here:

#!/bin/bash

# Source the default spack functionality
source ${__EXAWIND_CORE_DIR}/envs/spack.bash

# Set the maximum parallel build jobs we can execute
export EXAWIND_NUM_JOBS_DEFAULT=8
# Set the default compiler to GCC
export EXAWIND_COMPILER_DEFAULT=gcc

exawind_env_gcc ()
{
    module purge
    module load gcc/7.3.0
    module load openmpi/3.1.3

    # Load other dependencies
    exawind_load_deps cmake netlib-lapack
}

exawind_env_clang ()
{
    echo "ERROR: No CLANG environment set up for ${EXAWIND_SYSTEM}"
    exit 1
}

exawind_env_intel ()
{
    echo "ERROR: No Intel environment set up for ${EXAWIND_SYSTEM}"
}

Note

  • Please consult the variable reference to see other variables that can be configured for a system. Do not set the following variables within a system environment file: EXAWIND_SYSTEM, EXAWIND_COMPILER, EXAWIND_CODE, EXAWIND_SRCDIR, EXAWIND_PROJECT_DIR, EXAWIND_INSTALL_DIR, EXAWIND_CONFIG, EXAWIND_CFGFILE, SPACK_ROOT.

  • For more complicated build environment support, take a look at the NREL Eagle environment file.

Run bootstrap

At this point, exawind-builder has all the information necessary for your system. Run bootstrap to tell exawind-builder to fetch spack and install all the dependencies.

# Run bootstrap
cd ${HOME}
# Run bootstrap from your local exawind-builder
exawind/exawind-builder/bootstrap.sh -c gcc -s ${EXAWIND_SYSTEM}

In case you run into errors and want to tweak the configuration, please delete the spack directory $HOME/exawind/spack and start a fresh build to ensure that the final configuration in exawind-builder for your system will execute without any errors for other users.

If bootstrap succeeds, you should have build scripts in $HOME/exawind/scripts for the compiler of your choice. Proceed to Compiling executables using build scripts to build Trilinos and Nalu-Wind.

Once you have successfully built Nalu-Wind and executed regression tests on the new system, please consider submitting a pull request to allow other users to benefit from this configuration when using exawind-builder.