Status file containing only "WAITING"

[forsyth2]

@BunnyVon described the following scenario to me: after running /global/cfs/cdirs/e3sm/feng809/e3sm_test/zppy/post.mysimulation.cfg (on Cori), the status files only say WAITING <job #>, even though the jobs showed as running in the queue. Also interestingly, despite setting walltime to 00:30:00, the walltime remained 2:00:00 in the queue.

I would have expected the walltime issue for MPAS-Analysis due to the way defaults are handled (e.g., https://github.com/E3SM-Project/zppy/blob/main/tests/integration/test_defaults.cfg#L17). However, this cfg only runs ts tasks, so I'm not sure what's happening. @golaz -- any ideas on what might be happening?

I created my own version of the cfg (below) and ran on Cori. I do not run into the WAITING issue, however I do get ncclimo: ERROR Failed to split.. @chengzhuzhang Is this related to E3SM-Project/e3sm_diags#593?

# Copied from /global/cfs/cdirs/e3sm/feng809/e3sm_test/zppy/post.mysimulation.cfg                                                         

[default]
input = /global/cscratch1/sd/feng809/E3SM_simulations/20220512.v2.LR.BGC-LNDATM.SPINUP.ne30pg2_r05_EC30to60E2r2.cori-knl
input_subdir = run
output = /global/cfs/cdirs/e3sm/forsyth/zppy_support
case = v2.LR.BGC-LNDATM.SPINUP
www =/global/cfs/cdirs/e3sm/www/forsyth/zppy_support/v2.LR.BGC-LNDATM.SPINUP
partition = knl
years = "1:80:10"
walltime = "00:30:00" # NOTE: Walltime appears to stay 02:00:00                                                                           

# Regridded atmosphere monthly time series files in 10 year chunks                                                                        
[ts]
active = True
years = "1:80:10",

  # Monthly global time series                                                                                                            
  [[ atm_monthly_glb ]]
  input_subdir = "run"
  input_files = "eam.h0"
  frequency = "monthly"
  mapping_file = "glb"
  vars = "TS,FSNT,FLNT,RESTOM,CO2,CO2_FFF,CO2_OCN,CO2_LND"
  # Use default vars                                                                                                                      

  # Monthly land time series                                                                                                              
  [[ land_monthly ]]
  input_subdir = "run"
  input_files = "elm.h0"
  frequency = "monthly"
  vars = "NBP,PCO2,GPP,NPP,NEE,TOTECOSYSC,TOTVEGC,TOTSOMC,TOTLITC"

[chengzhuzhang]

Hey @forsyth2 could you post the ncclimo line and the error in more detail? Thanks.

[chengzhuzhang]

mmm, i might be confused... is ts_atm_monthly_glb_0001-0010-0010.bash for global_time_series task?

[forsyth2]

That's a ts task. The global_time_series task name starts with global_time_series rather than ts. (The global_time_series task requires this global ts task as a dependency however.)

[chengzhuzhang]

Okay, thanks. I'm learning...Correct me if i'm wrong: If @BunnyVon remove RESTOM in this ts task, and zppy should be able to derive RESTOM global mean time-series if to add a follow-up global_time_series task, right? I think that is what Sha is looking for..

[BunnyVon]

I am also learning, too. What are the differences between the ts task and global_time_series task?

[golaz]

The ts task will produce time series netCDF files. The global_time_series task will generate some quick overview time series plots. I realize the naming convention is a little confusing.

[golaz]

@forsyth2 : sounds like you were not able to reproduce the error regarding WAITING <job #> and WALLTIME?

@BunnyVon : does that error occur systematically?

[forsyth2]

1. I see "04:00:00" for TIME when I run squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p' -u forsyth, so I'm not sure what's happening there. I do recall seeing an incorrect TIME when debugging this issue earlier though. Maybe I should make a non-reproducible bug issue for it.
2. Debugging steps:

I ran the below variant of your cfg.

[default]
input = /global/cscratch1/sd/feng809/E3SM_simulations/20220512.v2.LR.BGC-LNDATM.SPINUP.ne30pg2_r05_EC30to60E2r2.cori-knl
input_subdir = run
#output = /global/cfs/cdirs/e3sm/feng809/e3sm_test/zppy/data
output = /global/cfs/cdirs/e3sm/forsyth/zppy_support/debug_20220624.20220512.v2.LR.BGC-LNDATM.SPINUP.ne30pg2_r05_EC30to60E2r2.cori-knl
case = v2.LR.BGC-LNDATM.SPINUP
#www = /global/cfs/cdirs/e3sm/www/sfeng/v2.LR.BGC-LNDATM.SPINUP
www = /global/cfs/cdirs/e3sm/www/forsyth/zppy_support/debug_20220624.20220512.v2.LR.BGC-LNDATM.SPINUP.ne30pg2_r05_EC30to60E2r2.cori-knl
partition = knl
qos = debug
years = "1:20:10",
walltime = "00:30:00"

[ts]
active = True
years = "1:20:10",

  # Monthly global time series
  [[ atm_monthly_glb ]]
  input_subdir = "run"
  input_files = "eam.h0"
  frequency = "monthly"
  mapping_file = "glb"
  #vars = "TS,FSNT,FLNT,RESTOM,CO2,CO2_FFF,CO2_OCN,CO2_LND"
  vars = "TS,FSNT,FLNT,CO2,CO2_FFF,CO2_OCN,CO2_LND"
  # Use default vars

  # Monthly land time series
  [[ land_monthly ]]
  input_subdir = "run"
  input_files = "elm.h0"
  frequency = "monthly"
#  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
  vars = "NBP,PCO2,GPP,NPP,NEE,TOTECOSYSC,TOTVEGC,TOTSOMC,TOTLITC"

# Global time series plots
[global_time_series]
active = True
atmosphere_only = True
input_subdir = "run"
years = "1-20",
ts_num_years = 10
figstr=land-atm spin-up
moc_file=mocTimeSeries_0001-0020.nc
experiment_name=20220512.v2.LR.BGC-LNDATM.SPINUP.ne30pg2_r05_EC30to60E2r2.cori-knl
ts_years = "1-20",
climo_years = "1-20",

I ran the following:

$ /global/cfs/cdirs/e3sm/forsyth/zppy_support/debug_20220624.20220512.v2.LR.BGC-LNDATM.SPINUP.ne30pg2_r05_EC30to60E2r2.cori-knl # cd to output directory from cfg
$ grep -v "OK" *status # Check which jobs failed, see that `global_time_series` failed.
$ emacs global_time_series_0001-0020.o<Job ID> # Look at error

Error 1:

    if sys.argv[8].lower() == "false":
IndexError: list index out of range

I edited global_time_series_0001-0020.bash to set debug=True and I edited global_time_series_0001-0020_dir/coupled_global.py to add print(f"sys.argv={sys.argv}"). I then ran sbatch global_time_series_0001-0020.bash.
Error 2:

 FileNotFoundError: [Errno 2] No such file or directory: '/global/cfs/cdirs/e3sm/forsyth/zppy_support/debug_20220624.20220512.v2.LR.BGC-LNDATM.SPINUP.ne30pg2_r05_EC30to60\
E2r2.cori-knl/post/atm/glb/ts/monthly/Blueyr/glb.xml'

I noticed sys.argv=['coupled_global.py', '/global/cfs/cdirs/e3sm/forsyth/zppy_support/debug_20220624.20220512.v2.LR.BGC-LNDATM.SPINUP.ne30pg2_r05_EC30to60E2r2.cori-knl', '20220512.v2.LR.BGC-LNDATM.SPINUP.ne30pg2_r05_EC30to60E2r2.cori-knl', 'land-atm', 'spin-up', '1', '20', 'Blue', '10', 'true'] in the .o file. In global_time_series_0001-0020.bash I changed figstr=land-atm spin-up to figstr="land-atm-spin-up". I then ran sbatch global_time_series_0001-0020.bash.
Error 3, which is what I got above:

Traceback (most recent call last):
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.6.0_nompi/lib/python3.9/site-packages/cdms2/cudsinterface.py", line 26, in __call__
    if(self.is_gridspec_grid_file() and
AttributeError: 'Dataset' object has no attribute 'is_gridspec_grid_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/global/cfs/cdirs/e3sm/forsyth/zppy_support/debug_20220624.20220512.v2.LR.BGC-LNDATM.SPINUP.ne30pg2_r05_EC30to60E2r2.cori-knl/post/scripts/global_time_series_000\
1-0020_dir/coupled_global.py", line 193, in <module>
    v = ts.globalAnnual(var)
  File "/global/cfs/cdirs/e3sm/forsyth/zppy_support/debug_20220624.20220512.v2.LR.BGC-LNDATM.SPINUP.ne30pg2_r05_EC30to60E2r2.cori-knl/post/scripts/global_time_series_000\
1-0020_dir/readTS.py", line 47, in globalAnnual
    FSNS = self.globalAnnual("FSNS")
  File "/global/cfs/cdirs/e3sm/forsyth/zppy_support/debug_20220624.20220512.v2.LR.BGC-LNDATM.SPINUP.ne30pg2_r05_EC30to60E2r2.cori-knl/post/scripts/global_time_series_000\
1-0020_dir/readTS.py", line 62, in globalAnnual
    v = self.f(var)
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.6.0_nompi/lib/python3.9/site-packages/cdms2/cudsinterface.py", line 33, in __call__
    raise CDMSError("No such variable, " + id)
cdms2.error.CDMSError: No such variable, FSNS

I'm not quite sure what's happening here. @chengzhuzhang @golaz Do you know what could be causing the 'Dataset' object has no attribute 'is_gridspec_grid_file' or cdms2.error.CDMSError: No such variable, FSNS errors? FSNS isn't even listed in the cfg.

[forsyth2]

Note I was testing main (as of the merging of #268) on Cori.

[forsyth2]

I changed the walltime to 4 hours, but the jobs still showed 2 hours.

I created #273

Debugging steps:

I created #274

changed figstr=land-atm spin-up to figstr="land-atm-spin-up"

I created #275.

[chengzhuzhang]

@forsyth2 based on a search of zppy code base. I found that FSNS is needed as a dependency of https://github.com/E3SM-Project/zppy/blob/e1c1b05e0dfda7da38afa6ffe8413066ae9291a8/zppy/templates/readTS.py
My guess is that all the variables needed for this script is supposed to be present in the atm time series var list. Otherwise, the global time series job would fail.

[forsyth2]

@chengzhuzhang Thanks!

@BunnyVon I'm having a hard time determining exactly which variables you need. I suspect this issue is due to a problem in the cfg, not an internal zppy problem.

I tried partially implementing #277 to determine the CDMS errors. That told me to add RESSURF,TREFHT,FSNTOA,FLUT,PRECC,PRECL,QFLX to vars in [[ atm_monthly_glb ]]. That expanded to FSNS,FLNS,SHFLX,QFLX,PRECC,PRECL,PRECSC,PRECSL,TREFHT,FSNTOA,FLUT. However, PRECC,PRECSC,FSNTOA couldn't seem to be found.