Merge pull request #580 from oahull0112/openmpi

Update performance recommendation section in running jobs
NREL · Jan 29, 2024 · 81301e7 · 81301e7
2 parents d280984 + f40ef0a
commit 81301e7
Show file tree

Hide file tree

Showing 2 changed files with 15 additions and 4 deletions.
diff --git a/docs/Documentation/Systems/Kestrel/Environments/index.md b/docs/Documentation/Systems/Kestrel/Environments/index.md
@@ -20,6 +20,11 @@ The NREL-built environments function similarly to those on Eagle, and it is up t
 
 NREL-built environments can make use of Cray MPICH via the `cray-mpich-abi`. As long as program is compiled with an MPICH-based MPI (e.g., Intel MPI but *not* Open MPI), the `cray-mpich-abi` can be loaded at runtime, which causes the program to use Cray MPICH for dynamically built binaries.
 
+
+## A note on OpenMPI
+
+Currently, OpenMPI does not run performantly or stably on Kestrel. You should do your best to avoid using OpenMPI. Please reach out to hpc-help@nrel.gov if you need help working around OpenMPI.
+
 ## Summary of available compiler environments
 
 Note: to access compilers not included in the default Cray modules (i.e., compilers within the NREL-built environment), you must run the command `source /nopt/nrel/apps/env.sh`.

diff --git a/docs/Documentation/Systems/Kestrel/running.md b/docs/Documentation/Systems/Kestrel/running.md
@@ -97,11 +97,17 @@ You may need to export these variables even if you are not running your job with
 
 #### Scaling
 
-Currently, some applications on Kestrel are not scaling with the expected performance. For these applications, we recommend:
+Currently, some applications on Kestrel are not scaling with the expected performance. We are actively working with the vendor's engineers to resolve these issues. For now, for these applications, we recommend:
 
-1. Submitting jobs with the fewest number of nodes possible.
+1. Setting the following environment variables:
+```
+export MPICH_SHARED_MEM_COLL_OPT=mpi_bcast,mpi_barrier 
+export MPICH_COLL_OPT_OFF=mpi_allreduce 
+```
+
+2. Submitting jobs with the fewest number of nodes possible.
 
-1. For hybrid MPI/OpenMP codes, requesting more threads per task than you tend to request on Eagle. This may yield performance improvements.
-1. Building and running with Intel MPI or Cray MPICH, rather than OpenMPI.
+3. For hybrid MPI/OpenMP codes, requesting more threads per task than you tend to request on Eagle. This may yield performance improvements.
 
+4. Building and running with Cray MPICH (or Intel MPI/cray-mpich-abi), rather than OpenMPI.