IMI preview failing

[mattloman]

Name and institution

Name: Matt Loman
Institution: University of Rochester

Description of your issue or question

IMI version 2.0.0 running on local cluster
config.yml.txt
bluehive.gcclassic.intel.env.txt
imi_output.log.txt

I've run into a few minor issues running IMI on the local cluster at UR that I've been able to resolve but that seemed to be unexpected behavior, and one that I haven't been able to resolve.

Issues I've resolved:

• When I run from the command line, the script reaches === RUNNING SETUP === immediately before I get the following error: ERROR: GCClassic already exists but is not version 14.4.1 even though GCClassic was downloaded automatically using the setup script. I was able to fix this by running git checkout 14.4.1 in the GCClassic directory, but it seems like the setup script is supposed to do that on its own.
• Git has to be added to the environment file in order to run the model, since line 140 of run_imi.sh uses git describe --tags to check the IMI version, but the environment files provided do not include it in the list of necessary software. Similarly with AWS-CLI if any data needs to be downloaded.
• The GEOS-Chem input data directory is assumed to contain specific folders and files when creating the state vector, which causes IMI to fail if they're not present. At least one of these is HEMCO/CH4/v2024-07 which contains HEMCO standalone diagnostic output, and it's not possible to point to a different folder to use in its place. Easily resolved by creating or downloading the folder containing the data that IMI is looking for, but I was surprised that the dry run options didn't cover the creation of the state vector.

Another question: do you have a list of files that need to be edited for proper function on a local cluster (things like ch4_run.template) or would you be able to include such a list in the documentation? It'd be nice if the slurm options set in config.yml could be applied to all of the slurm scripts that IMI submits, but as far as I can tell this is not currently the case.

Unresolved issue:

I've run into the following error while running the preview:

Creating preview plots and statistics... Submitted batch job 26972939
must supply at least one object to concatenate

The error comes from line 50 of preview.sh: cat imi_output.tmp >>${InversionPath}/imi_output.log. It seems as though preview.sh is continuing through the wait command (in the line above) as soon as the slurm script to run imi_preview.py has been successfully submitted without waiting for its completion, so imi_output.tmp doesn't exist to concatenate. Is there a way to edit preview.sh so that this doesn't happen?
Thanks!

[msulprizio]

Thanks for writing @mattloman.

When I run from the command line, the script reaches === RUNNING SETUP === immediately before I get the following error: ERROR: GCClassic already exists but is not version 14.4.1 even though GCClassic was downloaded automatically using the setup script. I was able to fix this by running git checkout 14.4.1 in the GCClassic directory, but it seems like the setup script is supposed to do that on its own.

This is the correct workaround for now. If GCClassic already exists in the IMI directory, we expect users to manually update the version after receiving the error message. The reason we don't do it automatically is to avoid issues with any updates that users have made to their local copy of GCClassic and/or GEOS-Chem. We can consider doing this automatically though if it would be useful to most applications.

Git has to be added to the environment file in order to run the model, since line 140 of run_imi.sh uses git describe --tags to check the IMI version, but the environment files provided do not include it in the list of necessary software. Similarly with AWS-CLI if any data needs to be downloaded.

Thanks for pointing this out. We rely on users to create their own environment files when running the IMI locally and provide the Harvard one as an example. (Side note: feel free to make a PR with the Rochester environment if you think it would be helpful to others). However, we should make an list in the documentation on required software. I'll add that to the to-do list!

The GEOS-Chem input data directory is assumed to contain specific folders and files when creating the state vector, which causes IMI to fail if they're not present. At least one of these is HEMCO/CH4/v2024-07 which contains HEMCO standalone diagnostic output, and it's not possible to point to a different folder to use in its place. Easily resolved by creating or downloading the folder containing the data that IMI is looking for, but I was surprised that the dry run options didn't cover the creation of the state vector.

I'm surprised the dry didn't catch that missing file as well, but perhaps it's run at a time before the HEMCO configuration file is properly updated to use those files. I'll run some tests to see if we can fix that.

It seems as though preview.sh is continuing through the wait command (in the line above) as soon as the slurm script to run imi_preview.py has been successfully submitted without waiting for its completion, so imi_output.tmp doesn't exist to concatenate. Is there a way to edit preview.sh so that this doesn't happen?

We can try to add a fix for this. I'll work on reproducing it on my end and will keep this issue open until we have a resolution.