Regression testing
AMR-Wind comes with a suite of regression tests to ensure that the relevant parts of the code are working correctly, and any code modifications do not introduce errors. These tests are run nightly using an automated system and the results are published on a public dashboard. We ensure that the testing covers multiple operating systems (currently Linux and MacOS), different compilers (GCC, Intel, LLVM Clang, and NVIDIA CUDA), as well as two different versions of AMReX library – a stable monthly snapshot, and the version-of-the-day (VOTD) development branch. In addition to the nightly testing, continuous integration (CI) capability is used to test pull-requests to ensure that they pass all tests before they are merged to the mainline codebase.
Enabling and running regression tests
AMR-Wind uses CTest
and its integration with CMake for the nightly
regression test suite. Developers can enable and use the tests locally on their
machine to assist development of new features and ensuring that the
modifications do not break current capabilities. Regression testing is not
enabled by default in a CMake build and must be enabled explicitly by the user.
To enable regression testing, the user must enable additional flags during the
CMake configure phase: AMR_WIND_ENABLE_TESTS
and
AMR_WIND_TEST_WITH_FCOMPARE
as shown below:
# Switch to AMR-wind source code
cd ${HOME}/exawind/source/amr-wind
# Create a build directory
mkdir build-test
# Run configure with testing flags
cmake -DAMR_WIND_ENABLE_TESTS=ON -DAMR_WIND_TEST_WITH_FCOMPARE=ON ../
# Build code
make -j 4
Tip
The flag AMR_WIND_ENABLE_TESTS
activates CTest facility and enables the
use of CTest to run the regression test facility. However, this in itself
does not check whether the tests are producing the correct output. The second
flag AMR_WIND_TEST_WITH_FCOMPARE
uses AMReX fcompare utility
to test the plotfile outputs against gold files to ensure that the results
haven’t changed.
Upon successful build, you will notice that a new executable fcompare is built, and CTest has been enabled.
# List available tests
ctest -N
# Run available tests (using 8 parallel processes)
# show output if a test fails otherwise suppress output
ctest -j 8 --output-on-failure
# Run tests with that match a particular regex
ctest -R abl_godunov --output-on-failure -j 8
# Rerun only tests that failed previously
ctest --rerun-failed --output-on-failure -j 8
Warning
Regression testing capability is only available using the CMake build process. The legacy GNUMakefile process does not support the regression testing capability via CTest. The user must manually run the cases with amr_wind and then run fcompare to determine issues.
Testing against gold files
To use fcompare to determine if the code has been built properly and generates the correct results, we must provide gold solutions for the tests. Currently, the gold files are not released with the code, so a local copy must be generated by the user. The recommended process is to download the appropriate version of AMR-Wind, build and run the executable once to generate the gold files and copy it into the appropriate directory and use them during the development process. This is not an ideal setup and will be rectified in future. We recommend running the unit tests to ensure that the build process worked correctly to generate the correct executable. The gold files directory is printed out during the configure phase, as shown below:
-- AMR-Wind Information:
-- CMAKE_SYSTEM_NAME = Darwin
-- CMAKE_CXX_COMPILER_ID = AppleClang
-- CMAKE_CXX_COMPILER_VERSION = 11.0.0.11000033
-- CMAKE_BUILD_TYPE = RelWithDebInfo
-- Test golds directory for fcompare: $HOME/exawind/source/amr-wind/test/AMR-WindGoldFiles/Darwin/AppleClang/11.0.0.11000033
-- Configuring done
-- Generating done
-- Build files have been written to: $HOME/exawind/source/amr-wind/build-test
The default gold files directory is
test/AMR-WindGoldFiles/${OS}/${COMPILER}/${COMPILER_VERSION}
.
# Ensure that you are in the build directory
# Run CTest first time (all tests will fail as there are no golds to compare with)
ctest -j 8
# Create initial version of Golds
cp -R test/test_files/* <absolute_path_to_test_golds>
# Rerun CTest again and all tests should pass
ctest -j 8
Example output for a failed test
The following shows an example of a failed test and the typical output generated
by fcompare
that can be used for diagnostics.
❯ ctest -R abl_godunov$ --output-on-failure
Test project ~/exawind/source/amr-wind/build-test
Start 7: abl_godunov
1/1 Test #7: abl_godunov ......................***Failed 9.73 sec
variable name absolute error relative error
(||A - B||) (||A - B||/||A||)
----------------------------------------------------------------------------
level = 0
velx 0.0009695495942 0.0001370997978
vely 0.0009397088188 0.0001544075933
velz 0.0001684407299 0.00408613285
gpx 5.837947396e-05 0.003916799182
gpy 5.947263951e-05 0.003794860517
gpz 5.148686593e-05 0.0001801671463
density 0 0
tracer0 1.591615728e-12 5.155552515e-15
vort 0.0001047506059 0.002168282324
0% tests passed, 1 tests failed out of 1
Label Time Summary:
regression = 38.90 sec*proc (1 test)
Total Test time (real) = 9.76 sec
The following tests FAILED:
7 - abl_godunov (Failed)
Errors while running CTest
During testing, fcompare will calculate the differences for each field in the plot file against gold files. Currently any difference is flagged as an error and causes the test to fail as seen from the above example. The test can also fail if the grids don’t match (e.g., due to different regrid based on refinement criteria) or if certain fields are missing in the plot file.
Test file organization
The regression tests are organized in the directory
amr-wind/tests/test_files
and are arranged in directories corresponding
to the name of the test. Each directory contains the input file
<test-name.i>
and any other files necessary to run the test case. The
test definitions are added to amr-wind/tests/CTestList.cmake
.
Creating new regression tests
New tests can be added using the following steps:
Create a new directory with the desired test name within the tests directory.
Add appropriate input files necessary to run the tests
Add a new entry into
CTestList.cmake
Rerun CMake configure to allow CMake to detect the new tests
Build, test, refine the feature
Commit the new test directory along with relevant source code updates to Git
For example, to create a new test called abl_godnov
(ABL simulation using
Godunov numerical scheme). The entry in the test file is shown below
add_test_re(abl_godunov 4)
The second argument to add_test_re
indicates the number of parallel
processes used to run the test. Currently it is recommended that the tests be
run using 4 MPI ranks.
Test outputs and troubleshooting
During development, it is likely that some tests fail and it is necessary to
examine the outputs and plot files for troubleshooting. CTest stores all outputs
in the same directory structure as the test file but within the build directory.
For example, if the build directory is build-test
then the outputs for
the test abl_godunov will be stored in the directory
build-test/test/test_files/abl_godunov/
. At least two outputs are always
generated: a log file (e.g., abl_godunov.log
) that contains the output
usually printed to the console during amr_wind execution, and plot
file output.