MADNESS performance tuning

Introduction

Performance of MADNESS is affected by many parameters. Some parameters are obvious, like the compiler optimization flags, and some are more obscure, like the MPI library parameters. This section aggregates the know-how for optimizing performance of MADNESS on the most important platforms.

MPI library

MVAPICH

Many aspects of MVAPICH behavior can be controlled by the environment variables, whose complete listing can be found in the Full User Guide (see here ). The following environment variables should be used:

• MV2_ENABLE_AFFINITY : this variable should be set to 0 to support MPI_THREAD_MULTIPLE .
• MV2_USE_LAZY_MEM_UNREGISTER : this variable should be set to 0 to turn off "lazy" cache deregistration. In MVAPICH version 2.0 we found that the cache registration/deregistration code often breaks in typical applications due to strong bursts of messages generated by a typical MADNESS application (internally we found some bugs due to unchecked table overruns).
• MV2_IBA_EAGER_THRESHOLD : this variable controls the maximum size of messages that can be sent using eager protocol. For an application that sends many small messages, such as MADNESS MRA applications like MolDFT, this variable should be set to the value greater than the typical message size. Specifically, for MolDFT and other applications working with 3-d MRA functions this should be set to 1.1 times sizeof(Real) * (2 * k)^3, where k is the multiwavelet order (the factor of 1.1 takes into account the metadata sent with each data block). E.g. for k=10 MV2_IBA_EAGER_THRESHOLD=70000 is sufficient.
• MV2_VBUF_TOTAL_SIZE : MVAPICH User Guide recommends to set this variable to the same value as MV2_IBA_EAGER_THRESHOLD (see above). Other MV2_VBUF_* variables may also need to be tuned for optimal performance.