How can I run model-vs-model diagnostics?

[araman2]

Discussed in #293

^{Originally posted by forsyth2 August 9, 2022}
How can I run model-vs-model diagnostics?

Hi,

I have a question about the model-vs-model comparisons. Can the zppy code directly create climatological outputs for both model simulations, or is it a two-step process ? 1. Create climatological outputs separately for each simulation, then, run zppy the reference_data_path in 'atm_monthly_180x360_aave_mvm' pointing to the climatological outputs from the other simulation. . if you have a reference code for model-vs-model comparison, that would be very helpful.

[forsyth2]

Converted issue #418 into this Q&A discussion.

#420 addresses the question of running two cases simultaneously. At the moment, I personally think it's best to do as a two-step process. Copied from my response in that discussion:

The best solution is to run a zppy cfg for one simulation and then use those paths as reference for the following cfg. Note that the integration test (the [[ atm_monthly_180x360_aave_mvm ]] section of https://github.com/E3SM-Project/zppy/blob/main/tests/integration/generated/test_complete_run_chrysalis.cfg) manages to not need a previous run by simply running the test and ref on different years (1852-1853 and 1850-1851 respectively) of the same simulation.

[araman2]

Thank you. I followed the example in https://github.com/E3SM-Project/zppy/blob/main/tests/integration/generated/test_complete_run_chrysalis.cfg

I'm running model_vs_model comparison for a single year. I'm attaching a snippet of my code for the e3sm_diags section.
Error

e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_2010-2010
...skipping because of dependency status file missing
/compyfs/rama225/F2010.nudge.PI.10xINP_v3atm.compy/zppy_outputs_nudge_F2010_PI_10xINP/post/scripts/climo_atm_monthly_180x360_aave_mvm_2010-2010.status
e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_2011-2011
...skipping because of dependency status file missing
/compyfs/rama225/F2010.nudge.PI.10xINP_v3atm.compy/zppy_outputs_nudge_F2010_PI_10xINP/post/scripts/climo_atm_monthly_180x360_aave_mvm_2011-2011.status
Error interpreting years
Traceback (most recent call last):
File "/share/apps/E3SM/conda_envs/base/envs/e3sm_unified_1.5.1_nompi/bin/zppy", line 10, in
sys.exit(main())
File "/share/apps/E3SM/conda_envs/base/envs/e3sm_unified_1.5.1_nompi/lib/python3.8/site-packages/zppy/main.py", line 103, in main
e3sm_diags_vs_model(config, scriptDir)
File "/share/apps/E3SM/conda_envs/base/envs/e3sm_unified_1.5.1_nompi/lib/python3.8/site-packages/zppy/e3sm_diags_vs_model.py", line 28, in e3sm_diags_vs_model
year_sets = getYears(c["years"])
File "/share/apps/E3SM/conda_envs/base/envs/e3sm_unified_1.5.1_nompi/lib/python3.8/site-packages/zppy/utils.py", line 98, in getYears
raise Exception

Code snippet:

[e3sm_diags]
active = True
environment_commands = "source /share/apps/E3SM/conda_envs/load_e3sm_unified_1.5.1_compy.sh"
#partition = debug
years = "2010:2011:1",
ts_num_years = 1

[[ atm_monthly_180x360_aave ]]
short_name = 'v3atm_PD10xINP_F2010'
grid = '180x360_aave'
sets="lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram","meridional_mean_2d","enso_diags","qbo","area_mean_time_series",
ref_start_yr = "1850"
ref_end_yr = "2013"
ref_final_yr = "2021"
reference_data_path = '/compyfs/diagnostics/observations/Atm/climatology'
obs_ts = '/compyfs/diagnostics/observations/Atm/time-series'
diags_obs_climo = '/compyfs/diagnostics/observations/Atm/climatology/'
climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
output_format_subplot = "pdf",

[[ atm_monthly_180x360_aave_mvm ]]

climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
climo_subsection = "atm_monthly_180x360_aave"
diff_title = "Difference"
partition = "compute"
qos = "regular"
ref_final_yr = 2010
ref_name = "v3atm_10xinp_mvm"
ref_start_yr = 2010
ref_years = "2010-2011"
reference_data_path = "/compyfs/mahf708/E3SMv3_dev/F2010.nPD.NGD_v3atm.0096484.compy/post/atm/180x360_aave/clim_diurnal_8xdaily/1yr"
run_type = "model_vs_model"
short_ref_name = "v3atm_10xINP"
swap_test_ref = False
tag = "model_vs_model"
ts_num_years_ref = 1
ts_subsection = "atm_monthly_180x360_aave"
walltime = "2:00:00"
years = "2010-2011",

[forsyth2]

Does adding a comma to ref_years fix it? ref_years = "2010-2011" -> ref_years = "2010-2011",

Which version of zppy are you using? The above will be an issue if you're using a version that predates #338.

[araman2]

I think the error is also present in e3sm_diags model_vs_obs. Do you see red flags here in the code below? I tried with and without ref start year.

active = True
environment_commands = "source /share/apps/E3SM/conda_envs/load_e3sm_unified_1.5.1_compy.sh"
#partition = debug
years = "2010:2011:1",
ts_num_years = 1

[[ atm_monthly_180x360_aave ]]
short_name = 'F2010.nPI.0.1xINP_v3atm'
grid = '180x360_aave'
sets="lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram","meridional_mean_2d","enso_diags","qbo","area_mean_time_series",
#ref_start_yr = 1850
#ref_final_yr = 2021
reference_data_path = '/compyfs/diagnostics/observations/Atm/climatology'
obs_ts = '/compyfs/diagnostics/observations/Atm/time-series'
diags_obs_climo = '/compyfs/diagnostics/observations/Atm/climatology/'
climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
output_format_subplot = "pdf",

[forsyth2]

Looking at your /compyfs/rama225/F2010.nudge.PI.10xINP_v3atm.compy/zppy_outputs_nudge_F2010_PI_10xINP/post/scripts directory, I see that the ts_land_monthly tasks failed with ncclimo: ERROR Failed to split. cmd_sbs[0] failed..

I'm not sure what's causing that, but I do see you're on a very old version of E3SM Unified (1.5 as opposed to the current 1.8). It's possible some things aren't working nicely because of the version.

[araman2]

I started using 1.5 because we had some issues with negative TREFHT values in all versions after 1.5.1

[forsyth2]

Wasn't that issue solved in the latest release (E3SM Unified 1.8.1)? (Discussed in #348).

[araman2]

I tested the code for a single year using 1.8.1. I get a similar error in e3sm_diags model_vs_obs. It works when I input multiple years. I added dependency files in climatology and time series.
/compyfs/rama225/F2010.PI.NGD_v3atm.10xINP/zppy_scripts/zppy_F2010PI_10xINP_test_mvm.cfg
Are you able to access this code?

[whannah1]

Using a single year is generally problematic unless you make sure to use the "seasonally discontinuous december" (sdd) option for ncclimo, but it's better to just use multiple years.

[forsyth2]

• /compyfs/rama225/F2010.PI.NGD_v3atm.10xINP/zppy_output_10xINP_mvmdemo_1.8.1/post/scripts/climo_atm_monthly_diurnal_8xdaily_180x360_aave_0001-0001.o403612 > Frequency monthly_diurnal_8xdaily is not a valid option > Use frequency = "diurnal_8xdaily" in /compyfs/rama225/F2010.PI.NGD_v3atm.10xINP/zppy_scripts/zppy_F2010PI_10xINP_test_mvm.cfg [same for the other 4 errors of this type]
• For the 2 ts_land_monthly errors, I'm seeing a lot of ncrcat: ERROR nco_xtr_mk() reports user-supplied variable name or regular expression ' <variable> ' is not in and/or does not match contents of input file. Are you generating the necessary data for these variables?

[mahf708]

@forsyth2 and all, sorry this took a long time, but here's an example cfg that works for one-year runs:

# [default]
# input = /global/cfs/cdirs/e3sm/mahf708/E3SMv3_dev/78a5ea1.nudged.mixPI.2DMS.10Nc.pm-cpu
# output = /global/cfs/cdirs/e3sm/mahf708/E3SMv3_dev/78a5ea1.nudged.mixPI.2DMS.10Nc.pm-cpu
# case = 78a5ea1.nudged.mixPI.2DMS.10Nc.pm-cpu
# www = /global/cfs/cdirs/e3sm/www/mahf708/E3SMv3_dev
# qos = preempt
# machine = pm-cpu
# environment_commands = "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.9.0rc12_pm-cpu.sh"

# [climo]
# active = True
# nodes = 2
# walltime = "00:30:00"
# years = "2010:2010:1",

#   [[ atm_monthly_180x360_aave ]]
#   input_subdir = "run"
#   input_files = "eam.h0"
#   mapping_file = /global/cfs/cdirs/e3sm/zender/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
#   frequency = "monthly"

[default]
input = /global/cfs/cdirs/e3sm/mahf708/E3SMv3_dev/78a5ea1.nudged.clmPD.2DMS.10Nc.pm-cpu
output = /global/cfs/cdirs/e3sm/mahf708/E3SMv3_dev/78a5ea1.nudged.clmPD.2DMS.10Nc.pm-cpu
case = 78a5ea1.nudged.clmPD.2DMS.10Nc.pm-cpu
www = /global/cfs/cdirs/e3sm/www/mahf708/E3SMv3_dev
qos = preempt
machine = pm-cpu
environment_commands = "source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.9.0rc12_pm-cpu.sh"

[climo]
active = True
nodes = 2
walltime = "00:30:00"
years = "2010:2010:1",

  [[ atm_monthly_180x360_aave ]]
  input_subdir = "run"
  input_files = "eam.h0"
  mapping_file = /global/cfs/cdirs/e3sm/zender/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  frequency = "monthly"

[e3sm_diags]
active = True
machine = pm-cpu
years = "2010:2010:1",
ts_num_years = 1
ref_start_yr = 1979
ref_final_yr = 2016
multiprocessing = True
num_workers = 128

  [[ atm_monthly_180x360_aave_mvm ]]
  climo_subsection = "atm_monthly_180x360_aave"
  grid = '180x360_aave'
  diff_title = "Difference"
  ref_final_yr = 2010
  ref_start_yr = 2010
  reference_data_path = '/global/cfs/cdirs/e3sm/mahf708/E3SMv3_dev/78a5ea1.nudged.mixPI.2DMS.10Nc.pm-cpu/post/atm/180x360_aave/clim/'
  ref_name = "78a5ea1.nudged.mixPI.2DMS.10Nc.pm-cpu"
  ref_years = "2010-2010",
  run_type = "model_vs_model"
  years = "2010-2010", 
  short_ref_name = '78a5ea1.nudged.mixPI.2DMS.10Nc.pm-cpu'
  short_name = '78a5ea1.nudged.clmPD.2DMS.10Nc.pm-cpu'
  swap_test_ref = False
  tag = "model_vs_model"
  ts_num_years_ref = 1
  ts_subsection = "atm_monthly_180x360_aave"

Idea is to run the commented out stuff in isolation first to get those climos, then to run the currently uncommented stuff second to get the climos and run the mvm e3sm_diags.

[chengzhuzhang]

Thank you for sharing your config here @mahf708. @forsyth2 it might be useful to consider to add a place for example config files for typical use cases that can be refered to in the future. Those complete testing scripts are also useful, but they may contain too much tasks that users need to comment out.

[forsyth2]

I created #491 for this.