Building Nalu-Wind Semi-Automatically Using Spack
Mac OS X or Linux
The following describes how to build Nalu-Wind and its dependencies mostly automatically on your Mac using Spack. This can also be used as a template to build Nalu-Wind on any Linux system with Spack.
Step 1
This assumes you have a (Homebrew) installation of GCC installed already
(we are using GCC 7.3.0). These instructions have been tested on OSX 10.11, MacOS 10.12, and MacOS 10.13.
MacOS 10.12/10.13 will not build CMake or Pkg-Config with GCC because they will pick up
system header files that have objective C code in them. Therefore we build Nalu-Wind using Spack on MacOS Sierra by
using Homebrew to install cmake and pkg-config and defining these
as external packages in Spack (see
packages.yaml).
Step 2
Checkout the official Spack repo from github (we will checkout into ${HOME}):
cd ${HOME} && git clone https://github.com/spack/spack.git
Step 3
Add Spack shell support to your .profile or .bashrc etc, by adding the lines:
export SPACK_ROOT=${HOME}/spack
source ${SPACK_ROOT}/share/spack/setup-env.sh
Step 4
Run the setup-spack.sh
script from the repo which tries to find out what machine you are on and then copies the corresponding *.yaml
configuration files to your Spack installation:
cd ${HOME} && git clone https://github.com/exawind/build-test.git
cd ${HOME}/build-test/configs && ./setup-spack.sh
Step 5
Try spack info nalu-wind to see if Spack works. If it does, check the
compilers you have available by:
machine:~ user$ spack compilers
==> Available compilers
-- clang sierra-x86_64 ------------------------------------------
clang@9.0.0-apple
-- gcc sierra-x86_64 --------------------------------------------
gcc@7.3.0 gcc@6.4.0 gcc@5.5.0
Step 6
Install Nalu-Wind with whatever compiler you prefer (it will default to Apple Clang) merely by
running spack install nalu-wind or by editing and running the
install_nalu_gcc_mac.sh script from the build-test repo:
cd ${HOME}/build-test/install_scripts && ./install_nalu_gcc_mac.sh
That should be it! When using the install script you will see that Spack will install
using the constraints we’ve specified in shared_constraints.sh which specifies a much more specific
set of Trilinos options for Nalu-Wind that can shorten the build time.
NREL’s Eagle Machine
The following describes how to build Nalu-Wind and its dependencies mostly automatically on NREL’s Eagle machine using Spack. This can also be used as a template to help build Nalu-Wind on any Linux system with Spack.
Step 1
Login to Eagle, and checkout the https://github.com/exawind/build-test.git
repo (we will be cloning into the ${HOME} directory):
cd ${HOME} && git clone https://github.com/exawind/build-test.git
Step 2
Checkout the official Spack repo from github:
cd ${HOME} && git clone https://github.com/spack/spack.git
Step 3
Configure your environment in the recommended way. You should purge all modules and load GCC 7.3.0 in your login script. In the example .bashrc in the repo we also load Python. If you have problems building with Spack on Eagle, it is most likely your environment has deviated from this recommended one. Even when building with the Intel compiler in Spack, this is the recommended environment at login.
module purge
module load gcc/7.3.0
Also add Spack shell support to your .bashrc as shown in the example
.bashrc
in the repo or the following lines:
export SPACK_ROOT=${HOME}/spack
source ${SPACK_ROOT}/share/spack/setup-env.sh
Log out and log back in or source your .bashrc to get the Spack
shell support loaded. Try spack info nalu-wind to see if Spack works.
Step 4
Configure Spack for Eagle. This is done by running the
setup-spack.sh
script provided which tries finding what machine you’re on and copying the corresponding *.yaml
file to your Spack directory:
cd ${HOME}/build-test/configs && ./setup-spack.sh
Step 5
Try spack info nalu-wind to see if Spack works.
Step 6
Note the build scripts and packages.yaml configuration files provided here adhere to the official versions of the third party libraries we test with, and that you may want to adhere to using them as well. Also note that when you checkout the latest Spack, it also means you will be using the latest packages available if you do not set constraints at install time and the newest packages may not have been tested to build correctly on NREL machines yet. So specifying versions of the TPL dependencies in your packages.yaml file for Spack is recommended.
Install Nalu-Wind using a non-GPU login node with the example script
install_nalu_eagle.sh
or edit the script to use the correct allocation and nice ./install_nalu_eagle.sh.
That’s it! Hopefully the install_nalu_eagle.sh
script installs the entire set of dependencies and you get a working build
of Nalu-Wind on Eagle…after several hours of waiting for it to build.
To build with the Intel compiler, note the necessary change listed in the install_nalu_eagle.sh batch script.
To load Nalu-Wind dependencies (you will need Spack’s OpenMPI for Nalu-Wind now) into your path you
will need to spack load openmpi %compiler and spack load nalu-wind %compiler, using
%gcc or %intel to specify which to load.
Development Build of Nalu-Wind
When building Nalu-Wind with Spack, Spack will cache downloaded archive files such as
*.tar.gz files. However, by default Spack will also erase extracted or
checked out (‘staged’) source files after it has built a package successfully.
Therefore if your build succeeds, Spack will have erased the Nalu-Wind source code
it checked out from Github.
The recommended way to get a version of Nalu-Wind you can develop in is to checkout Nalu-Wind yourself outside of Spack and build this version using the dependencies Spack has built for you. To do so, checkout Nalu-Wind:
git clone https://github.com/exawind/nalu-wind.git
Next, create your own directory to build in, or use the existing build directory in Nalu-Wind to
run the CMake configuration. When running the CMake configuration, point Nalu-Wind to
the dependencies by using spack location -i <package>. For example in the
build directory run:
cmake -DTrilinos_DIR:PATH=$(spack location -i trilinos) \
-DYAML_DIR:PATH=$(spack location -i yaml-cpp) \
-DCMAKE_BUILD_TYPE=RELEASE \
..
make
There are also do-config scripts available for this according to machine under the configs directory here. These scripts may also provide the capability to access and use pre-built dependencies from a set of modules if they are available on the machine. This should allow you to have a build of Nalu-Wind in which you are able to continuosly modify the source code and rebuild.
Building inside Docker Container
It is also possible to build (and run) Nalu-Wind inside a docker container with a prepared environment. This has the benefit of requiring less setup and usually being faster and can be useful in some situations (e.g. quickly testing something).
For this, use the container ecpe4s/exawind-snapshot:
docker run -it ecpe4s/exawind-snapshot bash
Inside the container you can directly load the pre-installed version of Nalu-Wind with spack:
spack load nalu-wind
ExaWind-Manager is used generate the container and is pre-installed in the container. Additional development of the code can be done through the Exawind-Manager’s developer tools.
If you are uncomfortable using spack for development then you can also set up your own build workflow in more of a “roll-your-own” development environment using CMake and just utilize the pre-installed TPL’s that are available in the container through spack. This really comes down to your development preferences and your comfort level with CMake.