ilamb dependency status file missing

[BunnyVon]

Hi,

I wonder if anyone can help me with [ilamb] settings in zppy. Thanks!
I have ilamb set

[ilamb]
active = True
nodes = 8
walltime = "20:00:00"
partition = compute
grid = '180x360_aave'
short_name = 'v2.LR.CBGC-LNDATM.CONTRL'
ts_num_years = 5
years = "1870-2014",

but it has complaints when I run zppy
ilamb_1870-2014 ...skipping because of dependency status file missing /lcrc/group/e3sm/ac.sfeng1/analysis/20221127.v2.LR.BGC-LNDATM.CONTRL.ne30pg2_r05_EC30to60E2r2.chrysalis/zppy/post/scripts/ts_land_monthly_1870-2014-0005.status URL: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.sfeng1/20221127.v2.LR.BGC-LNDATM.CONTRL.ne30pg2_r05_EC30to60E2r2.chrysalis/ilamb

I wonder what "dependency status file missing" is referring to. I also have

[ts]
active = True
years = "1870:2014:5",

[forsyth2]

Try changing years = "1870-2014", in your ilamb section to years = "1870:2014:5".

I'm also seeing the following ncclimo error in /lcrc/group/e3sm/ac.sfeng1/analysis/20221127.v2.LR.BGC-LNDATM.CONTRL.ne30pg2_r05_EC30to60E2r2.chrysalis/zppy/post/scripts/ts_land_monthly_2010-2014-0005.o25661:

ncclimo: ERROR 59 files of clm_md=mth input contain non-integral number of years, 11 files leftover in final year
ncclimo: HINT Provide input filenames in multiples of 12

My guess is that's related to #125 -- there may not be sufficient input data.

[BunnyVon]

Thanks! Yes, you were right about ncclimo error. I only have 11 months for 2014. Any recommendations for the solution without running more simulations? I suspect my boundary condition or input file is only up to 2014-12, E3SM interpolates between months, so it only generated output up to 2014-11.

[forsyth2]

I'm not sure. #314 suggests a potential future improvement in zppy, but it hasn't been implemented yet. @czender or @chengzhuzhang might have more suggestions.

[BunnyVon]

I see. Thank you. Cycle back to the ilamb errors. [ilamb] runs now but crashes with the error below

/var/spool/slurmd/job257960/slurm_script: line 16: /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.8.0rc6_chrysalis.sh: No such file or directory
cp: cannot stat '/lcrc/group/e3sm/ac.sfeng1/analysis/20221127.v2.LR.BGC-LNDATM.CONTRL.ne30pg2_r05_EC30to60E2r2.chrysalis/zppy/post/lnd/180x360_aave/cmip_ts/monthly//*_*_*_*_*_*_1870??-1874??.nc': No such file or directory
cp: cannot stat '/lcrc/group/e3sm/ac.sfeng1/analysis/20221127.v2.LR.BGC-LNDATM.CONTRL.ne30pg2_r05_EC30to60E2r2.chrysalis/zppy/post/atm/180x360_aave/cmip_ts/monthly//*_*_*_*_*_*_1870??-1874??.nc': No such file or directory

which module generate the cmip_ts files? I only have the directory clim/ , clim_diurnal_8xdaily/, and ts/ in atm/180x360_aave, and lnd/native/ts/monthly/5yr/ (elm output is at .5deg)

[BunnyVon]

I remapped the land output as @chengzhuzhang suggested. ilamb could get started. However, I received another error below related to ilamb.

/var/spool/slurmd/job262726/slurm_script: line 16: /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.8.0rc6_chrysalis.sh: No such file or directory

===== RUN ILAMB =====

/lcrc/group/e3sm/ac.sfeng1/analysis/20221127.v2.LR.BGC-LNDATM.CONTRL.ne30pg2_r05_EC30to60E2r2.chrysalis/zppy/post/scripts/tmp.262726.vOOU
/lcrc/group/e3sm/ac.sfeng1/analysis/20221127.v2.LR.BGC-LNDATM.CONTRL.ne30pg2_r05_EC30to60E2r2.chrysalis/zppy/post/scripts
slurmstepd: error: execve(): ilamb-run: No such file or directory
srun: error: chr-0248: task 0: Exited with exit code 2

[xylar]

@BunnyVon, The following path was to a test environment that was available only for a short time.

/lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.8.0rc6_chrysalis.sh

That path should br replaced with:

/lcrc/soft/climate/e3sm-unified/load_e3sm_unified_1.8.0_chrysalis.sh

in your script(s) to use the released version.

[BunnyVon]

ah, my bad. I created a script to run a small job but forgot to change the environment in the original script. Thank you for pointing it out @xylar

[BunnyVon]

@forsyth2 , could you see how I can resolve this issue?
numpy.ma.core.MaskError: Mask and data not compatible: data size is 3648, mask size is 20520.

I turned on "debug" mode. more info can be seen in /lcrc/group/e3sm/ac.sfeng1/analysis/20221127.v2.LR.BGC-LNDATM.CONTRL.ne30pg2_r05_EC30to60E2r2.chrysalis/zppy/post/scripts/tmp.263709.lLaU/_build/ILAMB01.log

[forsyth2]

I get Permission denied when trying to enter /lcrc/group/e3sm/ac.sfeng1/analysis/20221127.v2.LR.BGC-LNDATM.CONTRL.ne30pg2_r05_EC30to60E2r2.chrysalis/zppy/post/scripts/tmp.263709.lLaU.

However, it looks ILAMB finished successfully:

$ ls | grep 263709
ilamb_1870-1917.o263709
tmp.263709.lLaU
$ cat ilamb_1870-1917.status
OK

Indeed, it looks the ILAMB page was generated: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.sfeng1/20221127.v2.LR.BGC-LNDATM.CONTRL.ne30pg2_r05_EC30to60E2r2.chrysalis/v2.LR.BGC-LNDATM.CONTRL/ilamb/180x360_aave_1870-1917/. That said, it looks like it's missing some plots, but I'm not sure which ones you were trying to generate.

I only see one reference to MaskError:

$ grep -n MaskError ilamb_1870-1917.o263709 
129:                      CarbonDioxide/NOAA.Emulated v2.LR.BGC-LNDATM.CONTRL MaskError

In any case, I don't have much experience in dealing with that type of error.