Confidence intervals for scores from aggregate_stat job for GridStat output.

[AnastasiaBundel]

Hello!
Is there a way to calculate confidence intervals (CIs) for the scores, which don't have parametric CIs (RMSE, e.g.), when aggregating the scores from GridStat output?
Because, as far as I understand, GridStat output doesn't include MPR tables ( #991 ), and so, bootstrap CIs cannot be calculated.

Below are the jobs I use:
jobs = [

"-job aggregate_stat -line_type FHO -out_line_type CTS -fcst_thresh >=0.1 -obs_thresh >=0.1 -dump_row $output/$exp/stat_analysis/job_aggregate_stat_CTS.stat",

"-job aggregate_stat -line_type SL1L2 -out_line_type CNT -dump_row $output/$exp/stat_analysis/job_aggregate_stat_CNT.stat"

];

Thanks,
Anastasia.

[JohnHalleyGotway]

Hello Anastasia,

I see you have a question about whether or not the Stat-Analysis tool can compute confidence intervals. While it does compute parametric CI's and populate the *_NCL and *_NCU output columns, it does NOT, in general, compute bootstrap confidence intervals during the aggregate_stat job type. So the *_BCL and *_BCU output columns generally contain NA.

The only exception to that is when aggregating MPR lines from the Point-Stat output file. Stat-Analysis calls the same library code that Point-Stat does to bootstrap those MPR lines. So you can get bootstrap CI's for a job like:

 -job aggregate_stat -line_type MPR -out_line_type CNT -n_boot_rep 1000

But when aggregating FHO and SL1L2 lines to compute CTS and CNT output, respectively, as in your examples, no bootstrapping logic is implemented. This isn't the first time this request has arisen. Here's an existing GitHub issue for this feature request:
dtcenter/MET#59

And that issue has existing for a whopping 10 years! Our development team has never done this task because we use METviewer instead. The METviewer database and display system DOES have the ability to bootstrap this type of aggregation and compute CI's. The DTCenter relies heavily on METviewer for this type of analysis, and that feature request has never risen high enough in priority.

Hope that helps clarify.

[AnastasiaBundel]

Hello John,

Many thanks for a detailed answer!
Sorry, I didn't find the issue, I searched by keywords like bootstrap through discussions and didn't find it.
Yes, I understood that BCL-BCU are not calculated because the matched pairs from GridStat are not output in ascii tables (while they can be saved in a netcdf file, so I didn't understand why this option cannot be added easily, same as in PointStat).
But it is good news that the bootstrap CIs can be added in METviewer.
We began from using MET, but we will get at METviewer as well.
Thanks again,
Anastasia.