Running on HPC clusters

Instructions for running on various HPC clusters are given below.

Gadi (NCI Australia)

Use the openmpi/4.1.4 module (or newer), and build with gcc/system or gcc/11.1.0, and use cuda/11.7.0 (or newer).

Using VisIt

You can use VisIt in client/server mode with the following server-side patch for the launcher script.

A host file is provided here. You must change the username, project code, and server-side VisIt path.

Setonix (Pawsey)

The recommended build procedure on Setonix is:

source scripts/hpc_profiles/setonix-gpu.profile
cmake -B build -S . -DCMAKE_BUILD_TYPE=Release -DAMReX_GPU_BACKEND=HIP -DHDF5_ROOT="$PAWSEY_HDF5_HOME"
cmake --build build -j

Then a single-node test job can be run with:

sbatch scripts/slurm/setonix-1node.submit

Workaround for interconnect issues

If interconnect issues are observed, it is recommended to add the line :

export FI_CXI_RX_MATCH_MODE=software

to your job scripts.