Map file name, sub session name, and dependencies

[chengzhuzhang]

Question criteria

• I added a descriptive title here.
• I searched the zppy GitHub Discussions to find a similar question and didn't find it.
• I searched the zppy documentation.

What is the deadline?

NA

Describe your question

The folder names of directories that hold derived files seems to be dynamically set based on the mapping files. It seems like zppy has been having files from dependencies not found issues when using mapping files created following a new convention. For instance derived files are saved in folders with name 180x360_ncoaave instead of 180x360_aave, while tasks from zppy are expecting an old name, i.g. 180x360_aave. Not sure how best to address this.

I also hope to understand better the relationship between the sub session name, and how to set dependencies in configuration files. Should all dependencies listed in the cfg files explicitly?

Are there are any possible answers you came across?

No response

What machine were you running on?

not machine specific

Environment

latest version

What command did you run?

zppy -c configuration_file.cfg

Copy your cfg file

[default]
input = /pscratch/sd/z/zelinka1/E3SMv2/v2.LR.amip_0101_NOAA-OISST-v2.1
output = /pscratch/sd/z/zelinka1/E3SMv2/v2.LR.amip_0101_NOAA-OISST-v2.1_test
case = v2.LR.amip_0101_NOAA-OISST-v2.1
input_subdir = run
www = /global/cfs/cdirs/e3sm/www/zelinka1/E3SMv2/v2.LR.amip_0101_NOAA-OISST-v2.1
mapping_file = /global/cfs/cdirs/e3sm/mapping/maps/map_ne30pg2_to_cmip6_180x360_nco.20200901.nc
partition = pm-cpu
environment_commands = "source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh"


[climo]
active = True
walltime = "00:30:00"
years = "1979:2014:36", 

  [[ atm_monthly_180x360_aave ]]
  frequency = "monthly"

  # [[ atm_monthly_diurnal_8xdaily_180x360_aave ]]
  # frequency = "diurnal_8xdaily"
  # input_files = "eam.h4"
  # vars = "PRECT"


# Regridded atmosphere monthly time series files in N-year chunks
[ts]
active = True
walltime = "0:30:00"
years = "1979:2014:36", 

  [[ atm_monthly_180x360_aave ]]
  input_subdir = "run"
  input_files = "eam.h0"
  frequency = "monthly"
  vars = "CLDHGH,CLDMED,CLDLOW,CLDTOT,CLDTOT_ISCCP,FISCCP1_COSP,CLDTOT_CAL,CLD_CAL,CLD_MISR,CLMODIS,CLOUD,PS,Q,TREFHT,TROP_P,TS,T,LWCF,FLDS,FLNSC,FLNS,FLNTC,FLNT,FLUTC,FLUT,FSDSC,FSDS,FSNSC,FSNS,FSNTOAC,FSNTOA,FSUTOAC,FSUTOA,SOLIN,SWCF,OMEGA500,U,U10,V,ICEFRAC,LANDFRAC,OCNFRAC"

  # Monthly global time series
  [[ atm_monthly_glb ]]
  input_subdir = "run"
  input_files = "eam.h0"
  frequency = "monthly"
  mapping_file = "glb

[e3sm_diags]
active = True
grid = '180x360_aave'
obs_ts = "/global/cfs/cdirs/e3sm/diagnostics/observations/Atm/time-series/"
ref_final_yr = 2014
ref_start_yr = 1979 
reference_data_path = "/global/cfs/cdirs/e3sm/diagnostics/observations/Atm/climatology/"
sets = "lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram","meridional_mean_2d","enso_diags","qbo","annual_cycle_zonal_mean","zonal_mean_2d_stratosphere","aerosol_aeronet", 
short_name = 'v2.LR.amip_0101_NOAA-OISST-v2.1'
output_format = "png",
ts_num_years = 36
walltime = "00:30:00"
years = "1979:2014:36",

  [[ atm_monthly_180x360_aave ]]
  # climo_diurnal_frequency = "diurnal_8xdaily"
  # climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
  qos = "regular"
  walltime = "3:00:00"

[global_time_series]
active = True
atmosphere_only = True # Available in E3SM Unified 1.5.1
years = "1979-2014",
ts_num_years = 36
figstr= "v2.LR.amip_0101_NOAA-OISST-v2.1"
# moc_file needed for ocean component
moc_file = "mocTimeSeries_1870-2014.nc"
experiment_name = "v2.LR.amip_0101_NOAA-OISST-v2.1"
ts_years = "1979-2014",
climo_years = "1979-2014",

What jobs are failing?

Note that this user is using a map file: map_ne30pg2_to_cmip6_180x360_nco.20200901.nc. And a folder with name 180x360_nco was created. The error shows:
cp: cannot stat '/pscratch/sd/z/zelinka1/E3SMv2/v2.LR.amip_0101_NOAA-OISST-v2.1_test/post/atm/180x360_aave/clim/36yr/v2.LR.amip_0101_NOAA-OISST-v2.1_*_1979??_2014??_climo.nc': No such file or directory

What stack trace are you encountering?

No response

[forsyth2]

I also hope to understand better the relationship between the sub session name, and how to set dependencies in configuration files. Should all dependencies listed in the cfg files explicitly?

How are dependencies set?

Terminology:

• What you're calling "sections": Single brackets in the cfg define tasks. E.g., [climo]
• What you're calling "subsections": Double brackets in the cfg define subtasks. E.g., [[ atm_monthly_180x360_aave ]]
• Job: what actually runs on the machines. Note that a task with no subtasks can still launch multiple jobs (e.g., with different year sets). Each subtask can also launch multiple jobs.
• Dependency (job): The job will not begin until this other job runs.
• Dependency (task/subtask): Any of the task/subtask's jobs will depend on (i.e., will wait to start because of) this other task/subtask's jobs (running the same years). Dependencies are added in a task's .py file.

Example: The E3SM Diags task

zppy/zppy/e3sm_diags.py

Lines 150 to 175 in c3f463d

	depend_on_climo = set(
	[
	"lat_lon",
	"zonal_mean_xy",
	"zonal_mean_2d",
	"polar",
	"cosp_histogram",
	"meridional_mean_2d",
	"annual_cycle_zonal_mean",
	"zonal_mean_2d_stratosphere",
	]
	)
	in_sets = set(c["sets"])
	# Check if any requested sets depend on climo:
	if depend_on_climo & in_sets:
	if "climo_subsection" in c.keys() and c["climo_subsection"] != "":
	climo_sub = c["climo_subsection"]
	else:
	climo_sub = c["sub"]
	dependencies.append(
	os.path.join(
	scriptDir,
	"climo_%s_%04d-%04d.status"
	% (climo_sub, c["year1"], c["year2"]),
	),
	)

If sets includes any of ["lat_lon", "zonal_mean_xy", "zonal_mean_2d", "polar", "cosp_histogram", "meridional_mean_2d", "annual_cycle_zonal_mean", "zonal_mean_2d_stratosphere"], then e3sm_diags will depend on climo. Specifically, the climo subsection with the matching name.

For instance, in https://github.com/E3SM-Project/zppy/blob/c3f463da95011a6beeab0c6d9fc96a47355d3ab7/tests/integration/generated/test_complete_run_chrysalis.cfg, the atm_monthly_180x360_aave e3sm_diags subtask will depend on the atm_monthly_180x360_aave climo subtask.

If that's not what you want, you can set the climo_subsection parameter. We can see this in that cfg with the atm_monthly_180x360_aave_mvm e3sm_diags subtask using climo_subsection = "atm_monthly_180x360_aave" to refer to a climo subtask with a different name.

zppy/zppy/e3sm_diags.py

Lines 176 to 219 in c3f463d

	if "diurnal_cycle" in c["sets"]:
	dependencies.append(
	os.path.join(
	scriptDir,
	"climo_%s_%04d-%04d.status"
	% (c["climo_diurnal_subsection"], c["year1"], c["year2"]),
	)
	)
	if "tc_analysis" in c["sets"]:
	dependencies.append(
	os.path.join(
	scriptDir,
	"tc_analysis_%04d-%04d.status" % (c["year1"], c["year2"]),
	)
	)
	# Iterate from year1 to year2 incrementing by the number of years per time series file.
	if "ts_num_years" in c.keys():
	for yr in range(c["year1"], c["year2"], c["ts_num_years"]):
	start_yr = yr
	end_yr = yr + c["ts_num_years"] - 1
	if "ts_subsection" in c.keys() and c["ts_subsection"] != "":
	ts_sub = c["ts_subsection"]
	else:
	ts_sub = c["sub"]
	if (
	("enso_diags" in c["sets"])
	or ("qbo" in c["sets"])
	or ("area_mean_time_series" in c["sets"])
	):
	dependencies.append(
	os.path.join(
	scriptDir,
	"ts_%s_%04d-%04d-%04d.status"
	% (ts_sub, start_yr, end_yr, c["ts_num_years"]),
	)
	)
	if "streamflow" in c["sets"]:
	dependencies.append(
	os.path.join(
	scriptDir,
	"ts_rof_monthly_%04d-%04d-%04d.status"
	% (start_yr, end_yr, c["ts_num_years"]),
	)
	)

The diurnal_cycle set triggers a dependency on a diurnal climo subtask. We use the climo_diurnal_subsection parameter to tell us which subtask that is.

Thetc_analysis set triggers a dependency on the tc_analysis task. That task can't have any subtasks for us to worry about. We know that because there's no [[__many__]] section after:

zppy/zppy/templates/default.ini

Lines 116 to 120 in c3f463d

	[tc_analysis]
	# NOTE: always overrides value in [default]
	input_files = string(default="eam.h2")
	# The scratch directory
	scratch = string(default="")

The enso_diags, qbo, and area_mean_time_series sets trigger a dependency on a ts subtask. Determining which one is the same process as with climo, using the ts_subsection parameter.

The streamflow set triggers a dependency on the rof_monthly ts subtask specifically. It must be named this.

Other tasks

climo.py ,ts.py: no dependencies

tc_analysis.py, mpas_analysis.py: Later runs actually depend on earlier runs:

zppy/zppy/tc_analysis.py

Lines 88 to 91 in c3f463d

	# Note that this line should still be executed even if jobid == -1
	# The later tc_analysis tasks still depend on this task (and thus will also fail).
	# Add to the dependency list
	dependencies.append(statusFile)

https://github.com/E3SM-Project/zppy/blob/c3f463da95011a6beeab0c6d9fc96a47355d3ab7/zppy/mpas_analysis.py#L133-136
mpas_analysis.py also has a check to skip dependency requirements if an earlier run completed successfully previously:

zppy/zppy/mpas_analysis.py

Lines 96 to 101 in c3f463d

	# Check if we can skip because it completed successfully before
	skip = checkStatus(statusFile)
	if skip:
	# Add to the dependency list
	dependencies.append(statusFile)
	continue

global_time_series.py:
Depends on the ts_atm_monthly_glb ts subtask specifically. It must be named this.
https://github.com/E3SM-Project/zppy/blob/c3f463da95011a6beeab0c6d9fc96a47355d3ab7/zppy/global_time_series.py#L76-87
Depends on the mpas_analysis task (only if we're not doing atmosphere-only!)

zppy/zppy/global_time_series.py

Lines 88 to 108 in c3f463d

	if not c["atmosphere_only"]:
	# Add MPAS Analysis dependencies
	ts_year_sets = getYears(c["ts_years"])
	climo_year_sets = getYears(c["climo_years"])
	for ts_year_set, climo_year_set in zip(ts_year_sets, climo_year_sets):
	c["ts_year1"] = ts_year_set[0]
	c["ts_year2"] = ts_year_set[1]
	c["climo_year1"] = climo_year_set[0]
	c["climo_year2"] = climo_year_set[1]
	dependencies.append(
	os.path.join(
	scriptDir,
	"mpas_analysis_ts_%04d-%04d_climo_%04d-%04d.status"
	% (
	c["ts_year1"],
	c["ts_year2"],
	c["climo_year1"],
	c["climo_year2"],
	),
	)
	)

That task can't have any subtasks for us to worry about. Again, because there is no corresponding [[__many__]] section in default.ini.

ilamb.py:
Depends on a ts subtask. Determining which one is done is the same process as with e3sm_diags above, except we're looking for ts_land_subsection rather than ts_subsection.

zppy/zppy/ilamb.py

Lines 57 to 68 in c3f463d

	dependencies.append(
	os.path.join(
	scriptDir,
	"ts_%s_%04d-%04d-%04d.status"
	% (
	c["ts_land_subsection"],
	c["year1"],
	c["year2"],
	c["ts_num_years"],
	),
	),
	)

Depends on a atm subtask (only if we're not doing land-only!). Determining which one is done is the same as above, but using ts_atm_subsection.

zppy/zppy/ilamb.py

Lines 69 to 81 in c3f463d

	if not c["land_only"]:
	dependencies.append(
	os.path.join(
	scriptDir,
	"ts_%s_%04d-%04d-%04d.status"
	% (
	c["ts_atm_subsection"],
	c["year1"],
	c["year2"],
	c["ts_num_years"],
	),
	),
	)

Subsection parameters

Current only available in e3sm_diags:

zppy/zppy/templates/default.ini

Line 130 in c3f463d

climo_diurnal_subsection = string(default="")

zppy/zppy/templates/default.ini

Line 132 in c3f463d

climo_subsection = string(default="")

zppy/zppy/templates/default.ini

Line 228 in c3f463d

ts_subsection = string(default="")

Currently only available in ilamb:

zppy/zppy/templates/default.ini

Line 312 in c3f463d

ts_atm_subsection = string(default="atm_monthly_180x360_aave")

zppy/zppy/templates/default.ini

Line 313 in c3f463d

ts_land_subsection = string(default="land_monthly")

[forsyth2]

For instance derived files are saved in folders with name 180x360_ncoaave instead of 180x360_aave, while tasks from zppy are expecting an old name, i.g. 180x360_aave. Not sure how best to address this.

@chengzhuzhang Which task is causing the cp: cannot stat '/pscratch/sd/z/zelinka1/E3SMv2/v2.LR.amip_0101_NOAA-OISST-v2.1_test/post/atm/180x360_aave/clim/36yr/v2.LR.amip_0101_NOAA-OISST-v2.1_*_1979??_2014??_climo.nc': No such file or directory error? E3SM Diags and ILAMB have parameters for resolving this (see above, final section: "Subsection parameters") -- assuming I'm understanding correctly that the file/directory error is coming directly from the task/subtask names in the cfg.

[chengzhuzhang]

Hi Ryan, thank you for the clarification! I think I learned later with this example is that the grids parameter can be used to customize the search path for intermediate post-processed files?

With the section parameters, it sounds like subsection names can be given freely by the users.

[forsyth2]

Yes, the grid parameter is also useful for properly setting paths in climo, e3sm_diags, and ts.

$ git grep -n "/{{ grid }}/"
templates/climo.bash:151:dest={{ output }}/post/{{ component }}/{{ grid }}/${subdir}/{{ '%dyr' % (yr_end-yr_start+1) }}
templates/e3sm_diags.bash:148:climo_dir_source={{ output }}/post/lnd/{{ grid }}/clim/{{ '%dyr' % (year2-year1+1) }}
templates/e3sm_diags.bash:165:climo_dir_source={{ output }}/post/atm/{{ grid }}/clim/{{ '%dyr' % (year2-year1+1) }}
templates/e3sm_diags.bash:183:climo_diurnal_dir_source={{ output }}/post/atm/{{ grid }}/clim_{{ climo_diurnal_frequency }}/{{ '%dyr' % (year2-year1+1) }}
templates/e3sm_diags.bash:200:ts_dir_source={{ output }}/post/atm/{{ grid }}/ts/monthly/{{ '%dyr' % (ts_num_years) }}
templates/ts.bash:125:  dest={{ output }}/post/{{ component }}/{{ grid }}/ts/{{ frequency }}/{{ '%dyr' % (ypf) }}
templates/ts.bash:144:  #input_dir={{ output }}/post/{{ component }}/{{ grid }}/ts/{{ frequency }}/{{ '%dyr' % (ypf) }}
templates/ts.bash:149:  dest_cmip={{ output }}/post/{{ component }}/{{ grid }}/cmip_ts/{{ frequency }}