When/how often is the metplus_final.conf invoked?

[mpm-meto]

I have set off multiple simultaneous instances of MODE, breaking my time series of forecasts up into chunks that will fit into the wallclock limits of 6h of our scheduler (~2 weeks at a time). I am running through forecasts from three different experiments. So I will have anything between 4 and 6 instances of METplus running at the same time (as many as I dare submit at once!).

I made the decision to have a generic MET config file as it is the same for all (except if I specified the unique expid in each, but I didn't). Instead, I have made the generation of the METplus wrapper so that I can create unique versions with the correct date/time/expid/directories for each before submission using the cat > ... << EOF construct in my scripts. Yet, all the different unique config files get written to the same metplus_final.conf filename, so that my efforts to create unique METplus wrappers may in fact be in vain.

The questions are then this:
(1) when is the metplus_final.conf file read in? Only at the beginning? i.e. once? Clearly the MET config file is read in each time MODE is run.
(2) What is the danger of getting the config files between the simultaneous instances all messed up given that all of them end up being written to the same file name i.e. the file gets overwritten every time I run run_metplus.py (noting that they are not submitted all at once, but staggered by a minute or so).

Having inspected my output, I am getting output in all the output directories and all appears to be well but I am a little more concerned having inspected some of the output more closely. My biggest issue (I found) is that the output is not distinguishable in the sense that because the MET config file is the same for all, the model ID is not unique. This definitely suggests I should make the MET config file unique so that the model name is written to the output file.

I think it would be safer to have a unique metplus_final.conf but I am unsure where I should put the path? Would it be ok to put it in the MODE METplus wrapper so that whatever default is set is overwritten? Any thoughts or experiences of others on the above would be very welcome.

[j-opatz]

Hello,

And thank you for your question. After speaking with some of the METplus engineers, I can provide some of the following guidance.

For your first question, the metplus_final.conf file is not actually read in by METplus (unlike the MET configuration files, as you noted); it's written twice, once at the inital runtime of the program and once at its conclusion, to include all of the default values that were set. Each run will overwrite the file, but shouldn't cause any issues with your instances.

It was suggested that if you'd like to see separate final configuration files for each run that you have, you could accomplish that by changing the METPLUS_CONF variable to something like

METPLUS_CONF = {OUTPUT_BASE}/metplus_final_a.conf

Where "a" is a designated value that helps you understand which metplus_final file run you're looking at. This value would need to be manually changed each time you run your configuration, though. You could also create a variable within the configuration file such as

METPLUS_CONF = {OUTPUT_BASE}/metplus_final_{RUN_ID}.conf
RUN_ID = a
RUN_ID = b

which will allow each run of the configuration file to produce a separate, more easily identifiable final configuration file you can review. It was also suggested that you could change the designated {OUTPUT_BASE} for each run, but that might not be ideal if you're trying to keep all of your final configuration files in the same area.

And since METplus doesn't actually read in the metplus_final.conf and overwrites the file itself each time it's run by default, the answer to your second question is it's OK that you're doing the same in your current usage. You shouldn't experience any issues.

But if you do see any odd behavior, or have any other questions, please feel free to post it to the Discussions and we'll take a look at it. I hope this helps answer your questions!

[j-opatz]

Hi there,

Just wanted to reach out and see if the guidance I provided helped you with your problem. We try to keep discussions active and if they are resolved, we lock the discussion so we can move our attention onto other discussion topics.

If the above guidance did solve the issue, please mark that reply as the answer and we'll go ahead and lock this discussion. If there are any remaining concerns on this topic, please let us know.