Introduction

Exawind-builder is a set of bash functions that can be compiled to generate build scripts for the software used in ExaWind project on the different systems of interest. It separates machine-specific configuration from the software-specific configuration (tracking library dependencies and CMake configuration) so that they can be modularized and combined for different systems and compilers.

Pre-configured systems

Pre-built configurations exist for the following systems. Use the system name shown on the following table when generating scripts targeting that particular system. Within the build scripts the system is accessed using the environment variable EXAWIND_SYSTEM.

System Name

Description

spack

Spack (system agnostic)

anl-jlse-skylake

ANL JLSE Skylake

anl-jlse-gpu_v100_smx2

ANL JLSE V100 nodes

ornl-summit

OLCF Summit

eagle

NREL Eagle

cori

NERSC Cori

snl-waterman.

Sandia waterman cluster (also snl-waterman-atdm)

snl-ghost

Sandia Ghost cluster

snl-skybridge

Sandia Skybridge cluster

snl-ascicgpu

Sandia ASC GPU development machines

snl-ceerws

Sandia blade workstations

snl-ews

Sandia engineering workstations

pnnl-constance

PNNL Constance system

rhodes

NREL nightly build and test system

Supported compilers for pre-configured systems

The following compilers are configured for each machine. In situations where multiple compilers are present, we recommend that the users use the first one. The latter ones have not received enough testing and might have issues. Within the build scripts and elsewhere in the user manual the compiler suite is referred using the environment variable EXAWIND_COMPILER.

Environment

Compilers

anl-jlse-skylake

gcc

anl-jlse-gpu_v100_smx2

gcc, cuda

ornl-summit

gcc, cuda

eagle

gcc

cori

intel

snl-waterman

gcc, cuda

snl-ghost

intel

snl-skybridge

intel

snl-ascicgpu

gcc, cuda

snl-ceerws

gcc

snl-ews

gcc

pnnl-constance

gcc

rhodes

gcc, intel

Mac OSX

clang, gcc

ExaWind software suite

Exawind-builder provides CMake configurations for the following codes used within the ExaWind project. Please consult Reference section for configuration variables availble to customize configuration of each project. Within the build scripts, the code that is being configured/compiled is referred using the environment variable EXAWIND_CODE.

Exawind directory structure

Exawind-builder recommends the organizing code under a standard directory structure for ExaWind project. While it is not necessary to follow this directory structure, and the user is free to call the build scripts from any location, the standard directory structure will be used in the rest of the manual. A brief description of the standard layout is presented here.

All source code, build directories, installation directories, and the exawind-builder package itself is assumed to be located within exawind base directory. Within this directory the main subdirectories are shown below:

exawind/
├── exawind-builder
├── exawind-config-ornl-summit.sh
├── exawind-config-gcc7-cuda10.sh
├── exawind-config.sh
├── install
│   └── gcc8
│       ├── amr-wind
│       ├── amrex
│       ├── hypre
│       ├── nalu-wind
│       ├── openfast
│       ├── tioga
│       └── trilinos
│   └── gcc8-cuda10
│       ├── amr-wind
│       ├── amrex
│       ├── hypre
│       ├── nalu-wind
│       ├── openfast
│       ├── tioga
│       └── trilinos
│   └── intel18
│       ├── amr-wind
│       ├── amrex
│       ├── nalu-wind
│       ├── openfast
│       ├── tioga
│       └── trilinos
├── scripts
│   ├── amr-wind-gcc.sh
│   ├── amrex-gcc.sh
│   ├── arborx-gcc.sh
│   ├── exawind-env-gcc.sh
│   ├── exawind-sim-gcc.sh
│   ├── hypre-gcc.sh
│   ├── hypre-mini-app-gcc.sh
│   ├── umpire-gcc.sh
│   ├── nalu-wind-gcc.sh
│   ├── openfast-gcc.sh
│   ├── pifus-gcc.sh
│   ├── pyamrex-gcc.sh
│   ├── pystk-gcc.sh
│   ├── tioga-gcc.sh
│   ├── tioga-utils-gcc.sh
│   ├── trilinos-gcc.sh
│   └── wind-utils-gcc.sh
├── spack
└── source
    ├── amr-wind
    ├── hypre
    ├── nalu-wind
    ├── openfast
    ├── tioga
    ├── trilinos
    └── wind-utils

The sub-directories are:

  • exawind-builder: The build script package cloned from the git repository that contains scripts to configure and build codes on different systems. This directory must be considered read-only unless you are adding features to exawind-builder. This directory is not necessary if you are using one of the central installations of ExaWind that are maintained by the ExaWind team. To determine whether you need this directory please refer How to use ExaWind builder? section.

  • spack: Optional location for Spack if using Spack to manage dependencies. Like exawind-builder, this directory is not necessary if you are on a system where ExaWind suite is maintained by the ExaWind team.

  • source: Local git repository checkouts of the ExaWind codes of interest to the user. This is the recommended location for all the development versions of the various codes (e.g., nalu-wind, openfast, etc.).

  • scripts: On an existing exawind-builder installation, this directory contains the driver scripts that user will use on a daily basis. This directory contains two types of scripts:

    • Build scripts that are used to configure and compile local git checkouts of ExaWind codes that are checked out in the source directory. These scripts have a naming convention project-compiler.sh (in scripts ${EXAWIND_CODE}-${EXAWIND_COMPILER}.sh). For example, the build script that is used to compile hypre using GCC compiler suite is named hypre-gcc.sh. Similarly the script that is used to compile nalu-wind` using LLVM/Clang suite is named nalu-wind-clang.sh.

    • Environment scripts that can be sourced to load all the necessary modules and update user paths to run the codes. These scripts are useful during interactive sessions or to source within job submission scripts. These scripts are named exawind-env-${EXAWIND_COMPILER}.sh. For example, to source the environment that was used to build the code using GCC compiler suite, the user would add

      source ${EXAWIND_POJECT_DIR}/scripts/exawind-env-gcc.sh

  • install: The default install location where make install will install the headers, libraries, and executables.

The ExaWind project directory also contains several configuration files that are used to customize the behavior on different systems and different execution types, e.g., compiling and executing codes on host using Intel compiler vs. compiling and executing codes on GPUs using GCC as host compiler and NVIDIA CUDA to compile device code.

How to use ExaWind builder?

Depending on the system on which you intend to use ExaWind software, there are two options for using exawind-builder.

  1. If the desired system is one that is currently actively used by the ExaWind software team, then exawind-builder, as well as all necessary third-party libraries (TPLs), are already installe and configured on the system. Therefore, installation step is not required. Users can proceed directly to cloning and building the desired codes.

    Currently, ExaWind simulation environment, along with exawind-builder, is pre-installed and available on the following systems:

    • OLCF Summit

    • NREL Eagle

    • NERSC Cori

    If you are on of the systems listed above, please proceed to Basic usage section to learn how to use pre-built codes, or to build your own versions of the ExaWind codes using the TPLs pre-built by the ExaWind team.

  2. If your system is not listed above, then you should first follow installation instructions and on successfull installation proceed to Basic usage.