Configuring SeriesAnalysis for data stored in different directories

[mrinalbiswas]

I am trying to run SeriesAnalysis where the data are stored in different directories based on storm_id. e.g. 01l.2024, 02l.2024 etc. Under each directory, the file names are like: 05l.2024081600.hfsb.parent.atm.f024.grb2. I configured the conf file like:

SERIES_ANALYSIS_RUNTIME_FREQ = RUN_ONCE

LOOP_BY = INIT
INIT_TIME_FMT = %Y%m%d%H
INIT_BEG=2024061718 --> Beginning of 01l (first storm)
INIT_END=2024110718 --> End of the last storm
INIT_INCREMENT = 6H

SERIES_ANALYSIS_CUSTOM_LOOP_LIST = 01,02,03,04,05,06,07,08,09,10,11,12,13,14,15,16

FCST_SERIES_ANALYSIS_INPUT_DIR = /path_to_data/{custom?fmt=%S}l.2024
FCST_SERIES_ANALYSIS_INPUT_TEMPLATE = {custom?fmt=%S}l.{init?fmt=%Y%m%d%2H}.hfsb.parent.atm.f024.grb2
OBS_SERIES_ANALYSIS_INPUT_DIR = /path_to_analysis/GFS
OBS_SERIES_ANALYSIS_INPUT_TEMPLATE = gfs.0p25.{init?fmt=%Y%m%d%H?shift=86400}.f000.grib2

I am getting SeriesAnalysis output per storm_id. Is there any way I can get the SeriesAnalysis computed for all the storm_id's mentioned in the SERIES_ANALYSIS_CUSTOM_LOOP_LIST? My goal is to get a single NETCDF file for all storms combined.

[CPKalb]

Hi Biswas:

Apologies for the slow response. Myself and George are doing some testing on this and we will get back to you.

Tina

[CPKalb]

Hi Biswas:

The way to get Series-Analysis output for all storm id's combined is to use a list inside FCST_SERIES_ANALYSIS_INPUT_TEMPLATE, rather than just a single value. So, what I mean by this is the following:

1. omit the CUSTOM_LOOP_LIST
2. Remove the custom directory from FCST_SERIES_ANALYSIS)INPUT_DIR:

FCST_SERIES_ANALYSIS_INPUT_DIR = /path_to_data

3. Change FCST_SERIES_ANALYSIS_INPUT_TEMPLATE so that it looks like this:

FCST_SERIES_ANALYSIS_INPUT_TEMPLATE = 01l.2024/01l.{init?fmt=%Y%m%d%2H}.hfsb.parent.atm.f024.grb2, 02l.2024/02l.{init?fmt=%Y%m%d%2H}.hfsb.parent.atm.f024.grb2, 03l.2024/03l.{init?fmt=%Y%m%d%2H}.hfsb.parent.atm.f024.grb2, 04l.2024/04l.{init?fmt=%Y%m%d%2H}.hfsb.parent.atm.f024.grb2, 05l.2024/05l.{init?fmt=%Y%m%d%2H}.hfsb.parent.atm.f024.grb2, 06l.2024/06l.{init?fmt=%Y%m%d%2H}.hfsb.parent.atm.f024.grb2, 07l.2024/07l.{init?fmt=%Y%m%d%2H}.hfsb.parent.atm.f024.grb2, 08l.2024/08l.{init?fmt=%Y%m%d%2H}.hfsb.parent.atm.f024.grb2, 09l.2024/09l.{init?fmt=%Y%m%d%2H}.hfsb.parent.atm.f024.grb2, 10l.2024/10l.{init?fmt=%Y%m%d%2H}.hfsb.parent.atm.f024.grb2, 11l.2024/11l.{init?fmt=%Y%m%d%2H}.hfsb.parent.atm.f024.grb2, 12l.2024/12l.{init?fmt=%Y%m%d%2H}.hfsb.parent.atm.f024.grb2, 13l.2024/13l.{init?fmt=%Y%m%d%2H}.hfsb.parent.atm.f024.grb2, 14l.2024/14l.{init?fmt=%Y%m%d%2H}.hfsb.parent.atm.f024.grb2, 15l.2024/15l.{init?fmt=%Y%m%d%2H}.hfsb.parent.atm.f024.grb2, 16l.2024/16l.{init?fmt=%Y%m%d%2H}.hfsb.parent.atm.f024.grb2

4. Put the same number of files in the observation list

OBS_SERIES_ANALYSIS_INPUT_TEMPLATE = gfs.0p25.{init?fmt=%Y%m%d%H?shift=86400}.f000.grib2, gfs.0p25.{init?fmt=%Y%m%d%H?shift=86400}.f000.grib2, gfs.0p25.{init?fmt=%Y%m%d%H?shift=86400}.f000.grib2, gfs.0p25.{init?fmt=%Y%m%d%H?shift=86400}.f000.grib2, gfs.0p25.{init?fmt=%Y%m%d%H?shift=86400}.f000.grib2, gfs.0p25.{init?fmt=%Y%m%d%H?shift=86400}.f000.grib2, gfs.0p25.{init?fmt=%Y%m%d%H?shift=86400}.f000.grib2, gfs.0p25.{init?fmt=%Y%m%d%H?shift=86400}.f000.grib2, gfs.0p25.{init?fmt=%Y%m%d%H?shift=86400}.f000.grib2, gfs.0p25.{init?fmt=%Y%m%d%H?shift=86400}.f000.grib2, gfs.0p25.{init?fmt=%Y%m%d%H?shift=86400}.f000.grib2, gfs.0p25.{init?fmt=%Y%m%d%H?shift=86400}.f000.grib2, gfs.0p25.{init?fmt=%Y%m%d%H?shift=86400}.f000.grib2, gfs.0p25.{init?fmt=%Y%m%d%H?shift=86400}.f000.grib2, gfs.0p25.{init?fmt=%Y%m%d%H?shift=86400}.f000.grib2, gfs.0p25.{init?fmt=%Y%m%d%H?shift=86400}.f000.grib2

Depending on some of the other settings you have in your configuration file, you may need METplus version 6 for this to work.

I tested with a smaller list of files and that was working for me. Does this work for you and give you the desired output?

Tina

[mrinalbiswas]

Hi Tina,
Many thanks for looking into the issue. When I am running with one storm e.g. 01l, it runs fine but when I am adding more storms it fails with:
11/25 01:48:30.959Z metplus.ec12828a (command_builder.py:264) ERROR: (runtime_freq_wrapper.py:478) A problem occurred trying to obtain input files
11/25 01:48:30.959Z metplus.ec12828a INFO: No commands were run. Skip writing all_commands file
11/25 01:48:30.959Z metplus.ec12828a (run_util.py:298) ERROR: SeriesAnalysis had 1 error.

The final conf file is on casper at /glade/derecho/scratch/biswas/HAFS_EVAL/HAFS/metplus_final.conf.20241125014830.

I had these like you suggested:
FCST_SERIES_ANALYSIS_INPUT_TEMPLATE = 01l.2024/01l.{init?fmt=%Y%m%d%2H}.hfsb.parent.atm.f024.grb2, 02l.2024/02l.{init?fmt=%Y%m%d%2H}.hfsb.parent.atm.f024.grb2
OBS_SERIES_ANALYSIS_INPUT_TEMPLATE = gfs.0p25.{init?fmt=%Y%m%d%H?shift=86400}.f000.grib2, gfs.0p25.{init?fmt=%Y%m%d%H?shift=86400}.f000.grib2

Many thanks for looking into the issue. I am using version 6 as suggested.
Biswas

[CPKalb]

Hi Biswas:

Could you point me to your METplus configuration and data? It would be helpful for me to be able to do some testing with your exact configuration to determine what my be going on.

Thanks,
Tina

[mrinalbiswas]

Hi Tina,
Sure. My conf file: /glade/derecho/scratch/biswas/HAFS_EVAL/HAFS/series_analysis/serana_tina.conf. The data directory is mentioned in the conf file.
Thanks for your help
Biswas

[CPKalb]

Hi Biswas:

The problem here is that your different storms are over different time frames. In your configuration file, INIT_BEG = 2024061718. However, the only storm with data for that time is number 1. All other storms start later. So, METplus is searching for data for storm number 2, 3, etc at 2024061718, but it doesn't find that data (since it doesn't exist). This is what is causing the error and preventing Series-Analysis from running.

There is, unfortunately, no easy way to get this data into Series-Analysis since it is over differing time frames. The only way would be to create a listing of the forecast to a text file, and another one that contains the observation file names matched to the forecast data. These text files can be input to Series-Analysis using FCST_SERIES_ANALYSIS_INPUT_FILE_LIST and OBS_SERIES_ANALYSIS_INPUT_FILE_LIST (rather than the input directories and templates).

METplus does have the UserScript option which can build listings of files. I tried setting this up, but it doesn't work smoothly because some storms do overlap in time, creating file lists that are unequal in length (longer for the forecast and shorter for the observations). I've attached that configuration below, as well as the output. You can see that the observation file listing is shorter than the forecast file listing, which won't work for Series-Analysis. Having "missing" in the forecast file list also doesn't work.

I can really only think of 2 possible ways to get this data to run through Series-Analysis:

1. Creating a listing of the needed files and save those to a text file outside of METplus (via bash, python, etc)

or

2. Create a file listing for each storm (using a custom_loop_list) with METplus UserScripts. Once this is done you would have to concatenate the files together and remove lines that say missing from the forecast file and the corresponding lines in the observation files. Removing missing lines could be done with sed commands that don't require many lines of code. I have a script that does something like this, but I haven't tried to add it to the METplus workflow yet.

In both of these cases above, the resulting text file could be used to input the correct data to Series-Analysis. However, both of these will take some work beyond just specifying configurations. Are you familiar with UserScript's filename template options?

Tina

UserScript File list creation:
userscript_test.conf.txt

Resulting Forecast file listing:
user_script_files_FCST_INPUT_init_ALL_valid_ALL_lead_ALL.txt

Resulting Obs file listing:
user_script_files_OBS_INPUT_init_ALL_valid_ALL_lead_ALL.txt