MET V11.1.0 Point-Stat reading wrong GRIB record

[jwraby]

There are 2 records in the WRF output post-processed using the UPP which contain 2-D grid values of the variable SHTFL (sensible heat flux at surface). The record selected by Point-Stat contains only missing data. In the run log "Data plane information" the GRIB record 4826 has max = -999 and min = -999, but GRIB record 4832 has max = 6.29 and min = -1.02. It appears that Point-Stat chose record 4826 as evidenced by the FBAR value for SHTFL in the Point-Stat CNT output which is -999. The same problem is happening with the variable GFLUX (ground heat flux) which also has two records, one of which contains missing data. Both of these variables are surface variables (Z0). Is there a way to specify the GRIB record used by Point-Stat for computing the error statistics? If needed, I can provide the Point-Stat log file, the post-processed WRF output GRIB file, the NetCDF observations file, and any other files to Julie Prestopnik or another person by email using the Army file exchange system. Let me know what additional files you need and to whom I should send the data. Thanks.

[georgemccabe]

Hi @jwraby,
You can specify the grib record by using using the RNNN as the level value where NNN is the record number.

The “level” entry specifies a level type and value:
...
RNNN for a specific GRIB record number

This information can be found in the MET User's Guide in the fcst section (search for File-format specific settings for the “field” entry). In this section, there are many other configuration settings that can be used to filter GRIB data. One of these settings may be a better approach to obtain the desired record, as it is possible that the record number may change. If you would like help determining which option to use, you could provide the wgrib2 output that corresponds to these 2 records to help us identify discernible information.

[JohnHalleyGotway]

Thanks @georgemccabe for pointing @jwraby to the GRIB2 filtering options. I agree that specifying the exact record number (e.g. name="SHTFL"; level="R4832";) would do the trick in this specific case but is not a great longterm solution since the number and order of GRIB2 records are not always consistent from file to file. I'll just add a couple more details.

Note also that running plot_data_plane at verbosity level 3 or 4 prints a log message telling you exactly what GRIB2 record number is being read and the range of values. For example, using a sample file:

plot_data_plane nam_2012040900_F012.grib2 plot.ps 'name="TMP"; level="Z2";' -v 4
...
DEBUG 3: MetGrib2DataFile::data_plane() -> Found exact match for 'TMP/Z2' in GRIB2 record 6 field 1 of GRIB2 file '/d1/projects/MET/MET_test_data/unit_test/model_data/grib2/nam/nam_2012040900_F012.grib2'
DEBUG 4: 
DEBUG 4: Data plane information:
DEBUG 4:       plane min: 244.341
DEBUG 4:       plane max: 300.091
DEBUG 4:       scan mode: 64
DEBUG 4:      valid time: 20120409_120000
DEBUG 4:       lead time: 120000
DEBUG 4:       init time: 20120409_000000
DEBUG 4:     bitmap flag: 255

So looking at the log messages will clarify what record is being read.

If records 4826 and 4832 really are "duplicates", meaning that there is NO difference in their metadata, then you're kind of stuck. You could either "fix" that GRIB2 file by writing a script to strip out any fields containing only missing data (for example) or you could stick with specifying the explicit record number.

But hopefully there really is some difference in the Product Description Section we can use to tell MET which record to read. And that's "Section 4" in the GRIB2 record. I usually use the wgrib2 utility to inspect these things. Here's some example commands:

1. Run it with no args to see the VERY LONG usage statement:

wgrib2

2. Use -d 6 to only print info for record number 6 and -V for verbose output:

> wgrib2 nam_2012040900_F012.grib2 -d 6 -V
6:606179:vt=2012040912:2 m above ground:12 hour fcst:TMP Temperature [K]:
    ndata=262792:undef=0:mean=281.474:min=244.341:max=300.091
    grid_template=30:winds(grid):
	Lambert Conformal: (614 x 428) input WE:SN output WE:SN res 8
	Lat1 12.190000 Lon1 226.541000 LoV 265.000000
	LatD 25.000000 Latin1 25.000000 Latin2 25.000000
	LatSP 0.000000 LonSP 0.000000
	North Pole (614 x 428) Dx 12191.000000 m Dy 12191.000000 m 

3. Use -Sec4 to print the length of the Product Description Section:

> wgrib2 $MET_TEST_INPUT/model_data/grib2/nam/nam_2012040900_F012.grib2 -d 6 -Sec4
6:606179:Sec4 len=34 #vert coordinate=0 Product Defn Template=4.0 size=34 expected size=34

4. Use -get_byte to print the 34 entries of the PDS for this record (i.e. read bytes from section 4, starting at byte number 1, and read 34 of them):

> wgrib2 nam_2012040900_F012.grib2 -d 6 -get_byte 4 1 34
6:606179:4-1=0,0,0,34,4,0,0,0,0,0,0,2,0,84,0,0,0,1,0,0,0,12,103,0,0,0,0,2,255,0,0,0,0,0

You could do these same steps for records 4826 and 4832 to look for differences in the PDS section of those records. If there are differences, MET could differentiate between them. If not, it probably can't.

[jwraby]

Thank you @georgemccabe and @JohnHalleyGotway for your quick responses to my question. The information you've both provided is helpful to understand more about how to get MET to read a specific record and the potential complexities involved in doing so. I noticed the same Data plane info in the Point-Stat log file for SHTFL (record 4825. I mistakenly mentioned record 4826 above):
DEBUG 4: Data plane information:
DEBUG 4: plane min: -999
DEBUG 4: plane max: -999
DEBUG 4: valid time: 20220705_121500
DEBUG 4: lead time: 001500
DEBUG 4: init time: 20220705_120000
DEBUG 4: accum time: 000000

For record 4832 the following is listed:
DEBUG 4: Data plane information:
DEBUG 4: plane min: -1.02919
DEBUG 4: plane max: 6.29081
DEBUG 4: valid time: 20220705_121500
DEBUG 4: lead time: 001500
DEBUG 4: init time: 20220705_120000
DEBUG 4: accum time: 000000

My post-processed WRF output is in GRIB1 format vice GRIB2. I use the wgrib code to explore the contents of the file. Would it help you to have the file to look at? I would need your email address to transfer the file(s) to you.

[jwraby]

Updated info after executing the commands suggested by @JohnHalleyGotway to explore the contents of the post-processed forecast file. The Step 2 command (adapted for wgrib) worked to display the following output for both records:
For record 4832 (SHTFL):
/p/home/jraby/WGRIB/wgrib -d 4832 -V WRFPRS_d05.00_15
rec 4832:140988466:date 2022070512 SHTFL kpds5=122 kpds6=1 kpds7=0 levels=(0,0) grid=255 sfc 15min fcst:
SHTFL=Sensible heat flux [W/m^2]
timerange 10 P1 0 P2 1 TimeU 13 nx 168 ny 168 GDS grid 3 num_in_ave 0 missing 0
center 7 subcenter 0 process 125 Table 2 scan: WE:SN winds(grid)
Lambert Conf: Lat1 32.512000 Lon1 253.099000 Lov 253.233000
Latin1 32.625000 Latin2 32.625000 LatSP 0.000000 LonSP 0.000000
North Pole (168 x 168) Dx 0.150000 Dy 0.150000 scan 64 mode 136
min/max data -1.02919 6.29081 num bits 10 BDS_Ref -102.919 DecScale 2 BinScale 0

For record 4825 (SHTFL):
/p/home/jraby/WGRIB/wgrib -d 4825 -V WRFPRS_d05.00_15
rec 4825:140963112:date 2022070512 SHTFL kpds5=122 kpds6=1 kpds7=0 levels=(0,0) grid=255 sfc 15min fcst:
SHTFL=Sensible heat flux [W/m^2]
timerange 10 P1 0 P2 1 TimeU 13 nx 168 ny 168 GDS grid 3 num_in_ave 0 missing 0
center 7 subcenter 0 process 125 Table 2 scan: WE:SN winds(grid)
Lambert Conf: Lat1 32.512000 Lon1 253.099000 Lov 253.233000
Latin1 32.625000 Latin2 32.625000 LatSP 0.000000 LonSP 0.000000
North Pole (168 x 168) Dx 0.150000 Dy 0.150000 scan 64 mode 136
min/max data -999 -999 num bits 0 BDS_Ref -999 DecScale 0 BinScale 0

The commands in Steps 3 and 4 would not work using wgrib.

Unless I am missing something, I don't see any differences between the 2 records except that the last line of output which for 4825 shows the max and min values are -999 an the number of bits is 0, BDF_Ref is -999 and DecScale is 0.

Would MET be able to differentiate between the two records given this information without having to set the level to the specific record?

I re-ran Point-Stat after editing the config file "level" for the fcst field to R4832 as suggested by @georgemccabe and it successfully produced the expected output for SHTFL. Same for the GFLUX variable after editing the config file to specify the record number in the level entry.

[jwraby]

I conducted a survey of several different post-processed WRF forecast files for different days and times looking for instances where the record numbers for SHTFL and GFLUX changed, but I found no instances. It appears that I can rely on being able to specify the specific record numbers for both variables to ensure that the data is valid instead of missing (-999s).