Sensitivity tests

[elenagjyli]

Name: Elena Gjyli
Institution: University of Aarhus
Hello everyone :),
I used IMI version 2.0 to run an inversion over a northern part of Europe (lat:50->58, log:2->15) for 2 months (March and April of 2019).
After the inversion I run 3 sensitivity tests changing the prior emissions and gamma.
When the sensitivity tests were done I ploted all my results and the values were the same in all 3 cases and the same with the main inversion.
I changed the 'RunName' in the config file.
My path was correctly pointing to the directory where "integrated_inversion" was.
I also changed the names of ''gridded_posterior.nc" and "inversion_result.nc" files to ''gridded_posterior_prior0.5.nc" and "inversion_result_prior0.5.nc" ect.
In the IMI Quick guide (https://imi.readthedocs.io/en/latest/other/common-configurations.html) it says this:
<<Note that the final results of the original inversion (inversion_result.nc and gridded_posterior.nc) will be overwritten if not archived before running the sensitivity inversion>>.
So they should be overwriten with the new values (if the files still had the same name) or created (in the case I changed the name in each sensitivity test).
I tried another aproach in which I deleted the "imi_output.log" of the inversion in each sensitivity test.
Although the sensitivity tests were completed in 2-3 minutes, these comments appear in the imi_output.log' file:
"unable to find run_inversion.sh" and "home/ubuntu/imi_output.dir/prior0.5/inversion: no such file".
What is the exact process during sensitivity tests?
In which directories and which files are used and how?
Thank you in advance for your time :)

[msulprizio]

You may be interested in this pull request (still in review): #297. That pull request also updates the current documentation at https://imi.readthedocs.io/en/latest/advanced/inversion-ensemble.html. It also simplifies the case of changing gamma but still requires manually swapping out emissions if that's desired.

It's difficult to diagnose exactly what is going on without seeing your configuration and log files. If you would like us to have a closer look, please upload those here (you may need to change the file extension to .txt).

[elenagjyli]

I have already followed the instructions mentioned at this link (https://imi.readthedocs.io/en/latest/advanced/inversion-ensemble.html). I believe that the reason that I got the same values from all my 3 sensitivity tests (that were the same with the values from inversion) is because when I changed the 'Gamma' or 'PriorError' then the DOFs value changed and was very low.
So as a procedure it runs, but it gives the same results that inversion gives.
Can this be possible?
Is there a way to see the new number of DOFs everytime I change 'Gamma' or 'PriorError'?

[elenagjyli]

inversion_config.txt
inversion_imi_output.log.txt
preview_imi_output.txt
sensitivity_config.txt
sensitivity_imi_output.txt

I have attached the configuration files from the inversion and the sensitivity test as well as the imi_outpout.log from the preview, the inversion and sensitivity test.

The problem with the sensitivity is that it seems like it runs but it gives these commends:
=== RUNNING INVERSION ===
src/components/inversion_component/inversion.sh: line 80: cd: /home/ubuntu/imi_output_dir/gamma0.5/inversion: No such file or directory
sbatch: error: Unable to open file run_inversion.sh

So the result is I get the same values as those I already had inside the "gridded_posterior.nc" and "inversion_result.nc" files. Which means that they were never overwritten.

What I can't understand is why it asks for the file "...imi_output_dir/gamma0.5/inversion" to be there when the file "...imi_output_dir/gamma0.5" was created by the process automatically?
Shouldn't the sub-file "inversion" be created also automatically inside "gamma0.5"?

Maybe is because it cannot open the file "run_inversion.sh"?
Is my "ReferenceRunDir" wrong?
I am pointing at "home/ubuntu/integrated_inversion".
I also tried to point at "home/ubuntu/imi_output.dir/try1.0/inersion" where the results of the inversion are, but I got the same error.

[keneuoe]

Name: Keneuoe Maliehe
Institution: University of Nottingham

Dear IMI team,

I am posting my issue here because I believe it is related, if not similar. And thanks, @elenagjyli, for uploading your files; your output logs look a lot like mine.

I will provide a bit of context about mine and what I have tried so far, highlighting some of the differences. I also used the IMI 2.0 to complete an initial inversion and have been trying to run sensitivity/ensemble inversions without success.

1. In my sensitivity sensitivity_config.txt, I have SafeMode set to false and my ReferenceRunDir points to the IMI output directory of my initial inversion: "the {OutputPath}/{RunName} from the config.yml file used to generate the jacobian you will be using"- as pointed out my @msulprizio in an email.
2. I am also confused about whether to have a new run directory for each sensitivity inversion or not. So I have tried both approaches, but neither have worked. May you kindly advise what the correct approach is?
3. When I changed the run name, the sensitivity_output_log.txt indicated that there was no inversion directory for the IMI to be able to complete run_inversion.sh. So eventually I changed RunSetup and SetupInversion to true. Of course this doesn't work either, because this way run_inversion.sh expects to find the Jacobians in the JacobianRunDir which does not exist.
4. When I did not create a new run directory and attempted to run the sensitivity in the same directory as the initial inversion, the run_inversion_137.txt indicated that './data_converted_reference/ was missing. I have now created it using it ln -s <ref-run-dir>/inversion/data_converted <new-run-dir>/inversion/data_converted_reference, but with both the <ref-run-dir> and <new-run-dir> being a single directory. This time it exited due to not finding './jacobian_scale_factors.npy, as per my latest run_inversion_153.txt.

When you have time, I will be grateful for some guidance on carrying out these sensitivity inversions.

Thank you in advance for your time.