grid_stat output sometimes gets delayed?

[YaliMao-NOAA]

In a sequential job which ran grid_stat in a loop of forecast hours, I found sometimes/occasionally grid_stat output got delayed.

Attached is a sample log file.

On L4563, 'ls' command lists all stat files which are generated by grid_stat. On this line it shows only 9 stat files are present/listed.
But from L1-L4563, searching keywords 'Output file', you can see 24 stat files supposed to be generated by grid_stat.

That means though stat files are submitted to grid_stat, they are possibly delayed. Then it causes consequential errors.

To confirm my guess, I reran the case and everything was all right.

Thank you for your help!

20210404.txt

[JohnHalleyGotway]

@YaliMao-NOAA I see you have a question about the output being generated by the script you're running to call Grid-Stat. Thanks for sending the log output.

Yes, I do see the behavior you describe... the 'ls' command on line 4563 only shows the existence of 9 grid_stat .stat output files. But searching lines 1 - 4563, there are supposed to be 24 of them.

I did find something a little concerning in this log file that might be related. I only see one instance of a 'cd' (change directory) command in this log file:

0 + cd /scratch2/NCEPDEV/stmp3/Yali.Mao/met_working/wafs_verf_g2g.105368

So I ASSUME that all commands are being run from this directory, but that may or may not be true. When I grep this log file for "Output file", I see that grid_stat is generating 288 output files. However, only 72 of those are unique file names!

grep "Output file" 20210404.txt | wc -l
     288
grep "Output file" 20210404.txt | sort -u | wc -l
      72

If you configure/run Grid-Stat to write to the same output file name, the contents of the new file will clobber the old file. I do see that you're already using the "output_prefix" option to customize the output file names to avoid them being clobbered, but perhaps your output_prefix setting is not sufficient to do so? Now I haven't traced through the log message to back out exactly what you script is doing, but I'd recommend that you take a close look at the uniqueness of the output file names before debugging any further.

Perhaps you are changing directories to some other location, but your script is not reporting that? Or perhaps grid_stat really is clobbering it's previously generated output files and that messing up your 'ls' counting logic?

To make the behavior more clear, I'd recommend setting "-out_dir" to an absolute path instead of a relative one, like the current directory (".").

As we talked about in the METplus NOAA telecon this morning, another option is setting this up as a METplus use case.

Please let me know what you find.

[YaliMao-NOAA]

Thank you for looking into more details.

Each of the same stat file is run 4 times based on different config files, that is the reason there are only 72 unique files among 288 stat output files.

Yes, all commands are run from the same directory. However before a new set of output files with the same names to be generated, the old ones are moved to another location. Please see L4593.

I am going to debug more with more specific file names. But I am still confused why the first 'ls' fails to list all stat output files which should have been generated earlier as shown in the log file. And simply rerunning the same case doesn't have the issue anymore.

[JohnHalleyGotway]

Yali, yes, I don't have an explanation for why output files appear to be missing from your 'ls' command. I don't see anything in the log output you sent that would indicate a problem in the way the grid_stat tool is running. So at this point, that behavior remains unexplained.

Do you have any interest in trying to replicate this logic in a METplus use case? We do have some NOAA support funding that could be used to assist you in the transition. If so, just let me know and I can try to get the ball rolling on this end.

[YaliMao-NOAA]

John,
I forgot to mention that if the line of 'sleep 30' was not included before 'ls', the issue occurred more often, about 4-5 days out of a month. With 'sleep 30', only 1 day had the issue.

I came upon a possible reason. Do you think it could be out of disk space? The job working folder is stmp3 on Hera. Once for a while, we got notifications that some stmp folders were full.

[JohnHalleyGotway]

@YaliMao-NOAA I'm really not sure. I suppose it's possible, but I don't even know what would happen to Grid-Stat if we ran out of disk space while trying to write an output file. Ideally it'd error out, but I'm really not sure.

Sorry I can't be of more help.