Question about gen-ens-prod ensemble file list

[michelleharrold]

Hello!

I am (unsuccessfully) running gen-ens-prod. I am pretty sure it is user error, so any pro tips are greatly appreciated.

When I am running gen-ens-prod, it is always using my first forecast hour (f001) when grabbing the files to use, even if it is not on f001.

It looks like it is constructing the correct file list, e.g., for f003:

04/05 13:55:11.184 metplus (command_builder.py:658) DEBUG: Looking for FCST_INPUT file /gpfs/fs1/p/ral/jntp/SRW/harrold/expt_dirs/test_gep/2019061500/mem*/postprd/rrfs.t00z.prslevf003.tm00.grib2
04/05 13:55:11.186 metplus (command_builder.py:664) DEBUG: Wildcard file pattern: /gpfs/fs1/p/ral/jntp/SRW/harrold/expt_dirs/test_gep/2019061500/mem*/postprd/rrfs.t00z.prslevf003.tm00.grib2
04/05 13:55:11.186 metplus (command_builder.py:665) DEBUG: 3 files match pattern
04/05 13:55:11.187 metplus (command_builder.py:730) DEBUG: Found file: /gpfs/fs1/p/ral/jntp/SRW/harrold/expt_dirs/test_gep/2019061500/mem1/postprd/rrfs.t00z.prslevf003.tm00.grib2
04/05 13:55:11.187 metplus (command_builder.py:730) DEBUG: Found file: /gpfs/fs1/p/ral/jntp/SRW/harrold/expt_dirs/test_gep/2019061500/mem2/postprd/rrfs.t00z.prslevf003.tm00.grib2
04/05 13:55:11.187 metplus (command_builder.py:730) DEBUG: Found file: /gpfs/fs1/p/ral/jntp/SRW/harrold/expt_dirs/test_gep/2019061500/mem3/postprd/rrfs.t00z.prslevf003.tm00.grib2
04/05 13:55:11.187 metplus (command_builder.py:961) DEBUG: Writing list of filenames...
04/05 13:55:11.189 metplus (command_builder.py:965) DEBUG: Adding file to list: /gpfs/fs1/p/ral/jntp/SRW/harrold/expt_dirs/test_gep/2019061500/mem1/postprd/rrfs.t00z.prslevf003.tm00.grib2
04/05 13:55:11.189 metplus (command_builder.py:965) DEBUG: Adding file to list: /gpfs/fs1/p/ral/jntp/SRW/harrold/expt_dirs/test_gep/2019061500/mem2/postprd/rrfs.t00z.prslevf003.tm00.grib2
04/05 13:55:11.189 metplus (command_builder.py:965) DEBUG: Adding file to list: /gpfs/fs1/p/ral/jntp/SRW/harrold/expt_dirs/test_gep/2019061500/mem3/postprd/rrfs.t00z.prslevf003.tm00.grib2
04/05 13:55:11.189 metplus (command_builder.py:968) DEBUG: Wrote list of filenames to /gpfs/fs1/p/ral/jntp/SRW/harrold/expt_dirs/test_gep/2019061500/stage/APCP_01/file_lists/20190615000000_3_gen_ens_prod.txt

But when it comes to actually using files:

OUTPUT:
DEBUG 2: OpenMP running on 1 thread(s).
DEBUG 1: Default Config File: /glade/p/ral/jntp/MET/MET_releases/10.1.0/share/met/config/GenEnsProdConfig_default
DEBUG 1: User Config File: /gpfs/fs1/p/ral/jntp/MET/METplus/METplus-4.1.0/parm/met_config/GenEnsProdConfig_wrapped
DEBUG 1: Ensemble Files[3]:
DEBUG 1:    /gpfs/fs1/p/ral/jntp/SRW/harrold/expt_dirs/test_gep/2019061500/mem1/postprd/rrfs.t00z.prslevf001.tm00.grib2
DEBUG 1:    /gpfs/fs1/p/ral/jntp/SRW/harrold/expt_dirs/test_gep/2019061500/mem2/postprd/rrfs.t00z.prslevf001.tm00.grib2
DEBUG 1:    /gpfs/fs1/p/ral/jntp/SRW/harrold/expt_dirs/test_gep/2019061500/mem3/postprd/rrfs.t00z.prslevf001.tm00.grib2
DEBUG 1: Ensemble Control: None
DEBUG 2:
DEBUG 2: --------------------------------------------------------------------------------
DEBUG 2:
DEBUG 2: Processing ensemble field: APCP/A01

Therefore, all of my output has the same valid time, which is obviously problematic. Off the top of your head, am I missing a setting somewhere in the conf file?

For reference I am running METv10.1.0 and METplusv4.1.0 on Cheyenne.

Thank you!
Michelle

[michelleharrold]

Update: When I look at the command it is running for f003:

04/05 13:55:11.192 metplus INFO: COMMAND: /gpfs/fs1/p/ral/jntp/MET/MET_releases/10.1.0/bin/gen_ens_prod -v 2 -ens /gpfs/fs1/p/ral/jntp/SRW/harrold/expt_dirs/test_gep/2019061500/stage/APCP_01/file_lists/20190615000000_1_gen_ens_prod.txt -out /gpfs/fs1/p/ral/jntp/SRW/harrold/expt_dirs/test_gep/2019061500/metprd/gen_ens_prod/gen_ens_prod_FV3_GFS_v15p2_CONUS_25km_APCP_01h_20190615_030000V.nc -config /gpfs/fs1/p/ral/jntp/MET/METplus/METplus-4.1.0/parm/met_config/GenEnsProdConfig_wrapped -config /gpfs/fs1/p/ral/jntp/MET/METplus/METplus-4.1.0/parm/met_config/GenEnsProdConfig_wrapped -config /gpfs/fs1/p/ral/jntp/MET/METplus/METplus-4.1.0/parm/met_config/GenEnsProdConfig_wrapped

It looks like it is using the wrong file list, and it has -config three times (same number as the forecast hour -- this is consistent behavior for all forecast hours).

[JohnHalleyGotway]

@michelleharrold from the behavior you describe it sounds like there's a potential issue in the Gen-Ens-Prod wrapper for METplus rather than the Gen-Ens-Prod tool itself. The file names listed in the file list for each forecast hour don't match what you'd expect. Please correct me if that's not accurate.

Can you please point us to your METplus config file and recent log files on Cheyenne for easier debugging?

[michelleharrold]

@JohnHalleyGotway - I would concur it is likely the Gen-Ens-Prod wrapper (I have run Gen-Ens-Prod on the command line, and have gotten correct results).

Here is the log file: /gpfs/fs1/p/ral/jntp/SRW/harrold/expt_dirs/test_gep/log/metplus.log.gep_2019061500_APCP_01h
Here is the config file: /gpfs/fs1/p/ral/jntp/SRW/harrold/ufs-srweather-app/regional_workflow/ush/templates/parm/metplus/GenEnsProd_APCP01h.conf (this is part of the UFS SRW App regional workflow, so if you have any questions on envars, etc., let me know).

Thank you so much,
Michelle

[georgemccabe]

This is a bug in the wrapper. It is not resetting the arguments after each run. I created #1555 to address it.

[georgemccabe]

I put a patched version of the wrappers on Cheyenne for testing: /glade/u/home/mccabe/METplus.bugfix_1555
This fix will be incorporated in the METplus 4.1.1 bugfix release.

[michelleharrold]

Thanks, @georgemccabe!