unit testing standards

[jared321]

Klaus has pointed me toward the unit test architecture section of the User Manual, which is very useful. However, it does not specify how unit tests are supposed to communicate the SUCCESS or FAILURE status of the test. This is necessary so that test developers can do their work with the assurance that their test will communicate properly with Flash Test and therefore with CI servers.

AFAIK, this communication is via a file named unitTest_XXXX. Based on looking at a few Driver_evolveAll files in the unitTest folder, I assume that success is communicated by writing "all results conformed with expected values." If this line is not found in the file, then I suppose that implies failure. Is the value of XXXX important? Does that line have to be the final line in the file?

If this information exists elsewhere, perhaps the User Manual should point to it. If not, then perhaps someone who knows the fine details of this requirement could add it to the existing unit test section of the User Manual so that Flash-X novices are not expected to reverse engineer requirements by looking at existing (and possibly broken) tests.

Assigned to Anshu to determine if this should be done and if so to whom this Issue should be assigned.

@srichers added as he might have some input on this request.

[kweide]

The XXXX represents the MPI rank of the process writing the file.
I assume this is designed this way with tests in mind where several ranks each compute an error value, presumably from sone sort of distributed solution data, and the test as a whole passes IFF the error value of each rank is 0 (or below a threshold).