Building Nalu-Wind Semi-Automatically Using Spack

Mac OS X or Linux

The following describes how to build Nalu-Wind and its dependencies mostly automatically on your Mac using Spack. This can also be used as a template to build Nalu-Wind on any Linux system with Spack.

Step 1

This assumes you have a (Homebrew) installation of GCC installed already (we are using GCC 7.3.0). These instructions have been tested on OSX 10.11, MacOS 10.12, and MacOS 10.13. MacOS 10.12/10.13 will not build CMake or Pkg-Config with GCC because they will pick up system header files that have objective C code in them. Therefore we build Nalu-Wind using Spack on MacOS Sierra by using Homebrew to install cmake and pkg-config and defining these as external packages in Spack (see packages.yaml).

Step 2

Checkout the official Spack repo from github (we will checkout into ${HOME}):

cd ${HOME} && git clone https://github.com/spack/spack.git

Step 3

Add Spack shell support to your .profile or .bashrc etc, by adding the lines:

export SPACK_ROOT=${HOME}/spack
source ${SPACK_ROOT}/share/spack/setup-env.sh

Step 4

Run the setup-spack.sh script from the repo which tries to find out what machine you are on and then copies the corresponding *.yaml configuration files to your Spack installation:

cd ${HOME} && git clone https://github.com/exawind/build-test.git
cd ${HOME}/build-test/configs && ./setup-spack.sh

Step 5

Try spack info nalu-wind to see if Spack works. If it does, check the compilers you have available by:

machine:~ user$ spack compilers
==> Available compilers
-- clang sierra-x86_64 ------------------------------------------
clang@9.0.0-apple

-- gcc sierra-x86_64 --------------------------------------------
gcc@7.3.0  gcc@6.4.0  gcc@5.5.0

Step 6

Install Nalu-Wind with whatever compiler you prefer (it will default to Apple Clang) merely by running spack install nalu-wind or by editing and running the install_nalu_gcc_mac.sh script from the build-test repo:

cd ${HOME}/build-test/install_scripts && ./install_nalu_gcc_mac.sh

That should be it! When using the install script you will see that Spack will install using the constraints we’ve specified in shared_constraints.sh which specifies a much more specific set of Trilinos options for Nalu-Wind that can shorten the build time.

NREL’s Eagle Machine

The following describes how to build Nalu-Wind and its dependencies mostly automatically on NREL’s Eagle machine using Spack. This can also be used as a template to help build Nalu-Wind on any Linux system with Spack.

Step 1

Login to Eagle, and checkout the https://github.com/exawind/build-test.git repo (we will be cloning into the ${HOME} directory):

cd ${HOME} && git clone https://github.com/exawind/build-test.git

Step 2

Checkout the official Spack repo from github:

cd ${HOME} && git clone https://github.com/spack/spack.git

Step 3

Configure your environment in the recommended way. You should purge all modules and load GCC 7.3.0 in your login script. In the example .bashrc in the repo we also load Python. If you have problems building with Spack on Eagle, it is most likely your environment has deviated from this recommended one. Even when building with the Intel compiler in Spack, this is the recommended environment at login.

module purge
module load gcc/7.3.0

Also add Spack shell support to your .bashrc as shown in the example .bashrc in the repo or the following lines:

export SPACK_ROOT=${HOME}/spack
source ${SPACK_ROOT}/share/spack/setup-env.sh

Log out and log back in or source your .bashrc to get the Spack shell support loaded. Try spack info nalu-wind to see if Spack works.

Step 4

Configure Spack for Eagle. This is done by running the setup-spack.sh script provided which tries finding what machine you’re on and copying the corresponding *.yaml file to your Spack directory:

cd ${HOME}/build-test/configs && ./setup-spack.sh

Step 5

Try spack info nalu-wind to see if Spack works.

Step 6

Note the build scripts and packages.yaml configuration files provided here adhere to the official versions of the third party libraries we test with, and that you may want to adhere to using them as well. Also note that when you checkout the latest Spack, it also means you will be using the latest packages available if you do not set constraints at install time and the newest packages may not have been tested to build correctly on NREL machines yet. So specifying versions of the TPL dependencies in your packages.yaml file for Spack is recommended.

Install Nalu-Wind using a non-GPU login node with the example script install_nalu_eagle.sh or edit the script to use the correct allocation and nice ./install_nalu_eagle.sh.

That’s it! Hopefully the install_nalu_eagle.sh script installs the entire set of dependencies and you get a working build of Nalu-Wind on Eagle…after several hours of waiting for it to build.

To build with the Intel compiler, note the necessary change listed in the install_nalu_eagle.sh batch script.

To load Nalu-Wind dependencies (you will need Spack’s OpenMPI for Nalu-Wind now) into your path you will need to spack load openmpi %compiler and spack load nalu-wind %compiler, using %gcc or %intel to specify which to load.

Development Build of Nalu-Wind

When building Nalu-Wind with Spack, Spack will cache downloaded archive files such as *.tar.gz files. However, by default Spack will also erase extracted or checked out (‘staged’) source files after it has built a package successfully. Therefore if your build succeeds, Spack will have erased the Nalu-Wind source code it checked out from Github.

The recommended way to get a version of Nalu-Wind you can develop in is to checkout Nalu-Wind yourself outside of Spack and build this version using the dependencies Spack has built for you. To do so, checkout Nalu-Wind:

git clone https://github.com/exawind/nalu-wind.git

Next, create your own directory to build in, or use the existing build directory in Nalu-Wind to run the CMake configuration. When running the CMake configuration, point Nalu-Wind to the dependencies by using spack location -i <package>. For example in the build directory run:

cmake -DTrilinos_DIR:PATH=$(spack location -i trilinos) \
      -DYAML_DIR:PATH=$(spack location -i yaml-cpp) \
      -DCMAKE_BUILD_TYPE=RELEASE \
      ..
make

There are also do-config scripts available for this according to machine under the configs directory here. These scripts may also provide the capability to access and use pre-built dependencies from a set of modules if they are available on the machine. This should allow you to have a build of Nalu-Wind in which you are able to continuosly modify the source code and rebuild.

Building inside Docker Container

It is also possible to build (and run) Nalu-Wind inside a docker container with a prepared environment. This has the benefit of requiring less setup and usually being faster and can be useful in some situations (e.g. quickly testing something).

For this, use the container ecpe4s/exawind-snapshot:

docker run -it ecpe4s/exawind-snapshot bash

Inside the container you can directly load the pre-installed version of Nalu-Wind with spack:

spack load nalu-wind

ExaWind-Manager is used generate the container and is pre-installed in the container. Additional development of the code can be done through the Exawind-Manager’s developer tools.

If you are uncomfortable using spack for development then you can also set up your own build workflow in more of a “roll-your-own” development environment using CMake and just utilize the pre-installed TPL’s that are available in the container through spack. This really comes down to your development preferences and your comfort level with CMake.