pb2nc quality markers

[willmayfield]

Hi all,

We've been investigating the difference in using PB2NC_QUALITY_MARK_THRESH with values "9" and "3" with NAM prepbufr observation data for a 48 hour period. The tests on Cheyenne are here ('metprd_old' dir): /glade/work/briannen/qc_sensitivity_tests/expt_dirs/CONUS_25km_RRFSv1alpha_qc_flag_*

I wrote a script to count the frequency of the values in 'obs_qty' for the resulting .nc files, and got these counts:

With PB2NC_QUALITY_MARK_THRESH=9:

0	1	2	3	4	5	6	7	8	9
2516684	605807	599399	113598	4517	22	0	0	0	0

With PB2NC_QUALITY_MARK_THRESH=3:

0	1	2	3	4	5	6	7	8	9
2516684	1146738	161473	8	0	0	0	0	0	0

We found this to be confusing as to why the counts for obs_qty change for marks 1, 2, and 3, but maybe we're not understanding something about how the observations are being handled. Maybe it has to do with the event_stack_flag? If anyone has any insights about how this works, we would be grateful!

Thanks!

Will

@michelleharrold @jwolff-ncar @briannen

[hsoh-u]

The value at the obs_qty variable is not the quality mark values if the "obs_qty_table" variable exists. It the offset of the "obs_qty_table".

    int obs_qty(nobs) ;
        obs_qty:long_name = "index of quality flag" ;

The "obs_qty_table" is not the same for the PB2NC output because the first QC value from the input PREPBUFR becomes obs_qty_table[0].

For the pb2nc output, /glade/work/briannen/qc_sensitivity_tests/expt_dirs/CONUS_25km_RRFSv1alpha_qc_flag_3/metprd_old/pb2nc/prepbufr.ndas.2019061500.nc:

 obs_qty_table =
  "2",
  "NA",
  "3" ;

The quality mark for the value 0 at obs_qty is "2" which is obs_qty_table[0]
The quality mark for the value 1 at obs_qty is "NA" which is obs_qty_table[1]
The quality mark for the value 2 at obs_qty is "3" which is obs_qty_table[2]

For the pb2nc output, /glade/work/briannen/qc_sensitivity_tests/expt_dirs/CONUS_25km_RRFSv1alpha_qc_flag_9/metprd_old/pb2nc/prepbufr.ndas.2019061500.nc:

 obs_qty_table =
  "2",
  "9",
  "NA",
  "3",
  "8" ;

The quality mark for the value 0 at obs_qty is "2" which is obs_qty_table[0]
The quality mark for the value 1 at obs_qty is "9" which is obs_qty_table[1]
The quality mark for the value 2 at obs_qty is "NA" which is obs_qty_table[2]
The quality mark for the value 4 at obs_qty is "3" which is obs_qty_table[3]
The quality mark for the value 5 at obs_qty is "8" which is obs_qty_table[4]

[hsoh-u]

Here are counts per QC for above two files (prepbufr.ndas.2019061500.nc):

QC value   QC_9     QC_3
   NA:   35,719   35,719
    2:   87,219   87,219
    3:    1,115    1,115
    8:       22        -
    9:    2,548        -

[willmayfield]

Ah-ha, thank you Howard! This helps a lot...I will see if this clarifies the differences in our test.

[JohnHalleyGotway]

Indeed, thanks very much for clarifying @hsoh-u! Will had asked me about this last week, and I forgot that detail. I'm relieved that we don't have an insidious bug to chase down!

[JohnHalleyGotway]

@willmayfield if/when you confirm that Howard's explanation resolves these issues, please select his response in this discussion as being a good answer. Look for the "Mark as answer" button. Once we see that you've done that, we'll "Lock conversation" on this discussion to prevent future posts.

That's basically how we "resolve" open discussions.