variables related to TC products of NHC v.s. METplus

[YanJin-NOAA]

Hi,
I am recently working on the comparison between the TC products of NHC and METplus. Here is the list of the output TC-related variables I made: https://docs.google.com/spreadsheets/d/1th3Ix71dUhXHWc8jvxMIkF6hAQSK44xAhnNYNgVuXDM/edit#gid=0

The yellow columns are NHC's TC variables, and the green columns are METplus's TC variables. Compared to NHC, in METplus some variables are the same or can be easily calculated. However, some variables are non-existent in METplus. For example, bias errors (kt), minimum center pressure bias (hPa), intensity relative skill (%) relative to model 1/2/3/4/5, track forecast skill (%), track standard deviation (nm), intensity FSP errors (%), etc.

I wonder if we can plot the above variables using METplus? My understanding is that, in METplus, as long as the variables exist in tc_pairs output file, we can use those variables and plot the figures. Otherwise there is no way to plot the variables. Is my understanding correct? Or if there are other ways to plot those variables (generated by NHC but not currently exist in METplus)?

Appreciate your time and help.

[JohnHalleyGotway]

Hi Yan,

I see you have questions about comparing MET-TC to the NHC verification system. You see output from NHC vx and are wondering how it maps to functionality in MET. Let me start by saying that MET-TC is wholly based on the NHC vx system. The initial version of MET-TC was added in version 4.1 of MET, back in 2013. They were designed to replicate and extend the functionality of the NHC vx system. And @KathrynNewman is the scientist who advised a lot on these enhancements.

Next, let me briefly describe the functionality of the MET-TC tools, as of version 10.0.0:

• tc_dland pre-computes the distance to the nearest land point. Users generally don't need to rerun this.
• tc_pairs matches up forecast tracks and probabilities to analysis tracks (i.e. typically BEST tracks) and writes matched pairs info.
• tc_stat reads the matched pairs output from tc_pairs and computes additional summary statistics (including FSP and the other bias statistics you listed in your question).
• tc_gen applies a categorical approach to compare forecast/observed Tropical Cyclone Genesis events.
• tc_rmw and rmw_analysis tools do a coordinate transform to extract data relative to the center of the storm.

So the short answer to your question is that the tc_stat tool is the missing piece. That's the one that computes the additional statistics/errors you were asking about. Please take a look at the Documentation and Online Tutorial for the tc_stat tool.

Please also read about the plot_tcmpr.R script. Its an R-script we've been running for many years to (1) call the tc_stat tool and (2) plot the result. And those plot types do include FSP.

So these tools/scripts have been available for a long time. But there have also been some recent developments...

• METdatadb has been enhanced to load the output from TC-Pairs into a database.
• METexpress has recently been enhanced to support a new TC app for displaying track error information. @mollybsmith-noaa can answer questions about that.

We do have additional plans to enhance METviewer to also plot TC outputs. But the tasks for that work haven't been clearly defined yet.

Hope that helps clarify!

[YanJin-NOAA]

Hi John,

Thanks very much for your explanation. I understand that tc_stat tools reads in TCMPR output and do further filtering and stats. I checked the TCMPR output variables: https://met.readthedocs.io/en/main_v9.1/Users_Guide/tc-pairs.html#id3, and wonder in NHC there are forecast skills, e.g.: intensity relative skill, track forecast skill, how can I calculate those skills based on the TCMPR output? Also, in NHC there is a variable "minimum center pressure bias (hPa)", however, I didn't find it in the TCMPR list. Same thing for "intensity FSP". Did I miss something here?

Thanks very much.
Best,
Yan

[JohnHalleyGotway]

Yan,

First, let me point you to the documentation of the TC-Stat summary job type.

By way of example, here's a sample output file from TC-Pairs containing the TCMPR line type: alal2010.txt

In order for GitHub to let me attach it, I renamed the .tcst suffix to be .txt instead.

Next, try running the following TC-Stat job on this data:

tc_stat -lookin alal2010.txt -job summary -column TI -amodel HWFI,GFSI,GHMI,SPC3 -event_equal true -match_points true -by AMODEL -out job_summary_ti.txt

And here's the output file: job_summary_ti.txt

First, "TI" is a shortcut for the absolute value of maximum wind differences and track error. The documentation describes other shortcuts for TRACK, WIND, AC, and XY. Or you can explicitly list them yourself. For example this would give you the absolute value of AMSLP - BMSLP:

-column 'ABS(AMSLP-BMSLP)'

However, in my sample data AMSLP (column 31) is set to bad data (NA) on every line. So I can't actually compute that difference.

Next, note that event equalization is turned on for my job (-event_equal true). This is required to compute the frequency of superior performance. In job_summary_ti.txt we see that for the wind speed errors, i.e. 'ABS(AMAX_WIND-BMAX_WIND)', there were 8 cases (FSP_TOTAL). 4 of those 8 were "ties", meaning that the diffs were too small to pick a winner. The GFSI model was best 3 (FSP_BEST) out of the 4 times, and SPC3 was best 1 of the 4 times. The job summary output is described in this table.

The trouble is that I don't know exactly how to map this output into the skill measures you listed:
intensity relative skill, track forecast skill, minimum central pressure bias, intensity FSP

But understanding the TC-Stat summary output for each of the TCMPR columns, differences of columns, or absolute value of the differences of columns, is a good first step.

Next, you could run the plot_tcmpr.R script using data for only those 4 models and create many plot types:

export MET_INSTALL_DIR=MET-main_v10.0/met
Rscript MET-main_v10.0/met/scripts/Rscripts/plot_tcmpr.R -lookin alal2010.txt -filter '-amodel HWFI,GFSI,GHMI,SPC3' -plot BOXPLOT,POINT,MEAN,MEDIAN,RELPERF,RANK,SKILL_MN,SKILL_MD

You get several png output files: TK_ERR_plots.tar.gz

Since I have so little sample data, these plots aren't very interesting. But the more data you pass to it (using the -lookin option), the better your plots. By default, it plots the track error column (TK_ERR), but you can change that or plot more columns/diffs/abs value of diffs using the "-dep" command line option. Just run the plot_tcmpr.R script with no arguments to see the full usage statement.

Rscript MET-main_v10.0/met/scripts/Rscripts/plot_tcmpr.R

And I'll also list it below for completeness:

Usage: plot_tcmpr.R
        -lookin tcst_file_list
        [-config path]
        [-outdir path]
        [-prefix string]
        [-title string]
        [-subtitle string]
        [-xlab string]
        [-ylab string]
        [-xlim min,max]
        [-ylim min,max]
        [-filter options]
        [-tcst path]
        [-dep list]
        [-scatter_x list]
        [-scatter_y list]
        [-skill_ref string]
        [-series string [list]]
        [-series_ci list]
        [-legend list]
        [-lead list]
        [-plot list]
        [-rp_diff string]
        [-demo_yr year]
        [-hfip_bsln string]
        [-footnote_flag]
        [-no_ee]
        [-no_log]
        [-plot_config path]
        [-save_data path]
        [-save]
        where "-lookin"        is a list of files with TCMPR lines to be used.
              "-config"        is a plotting configuration file.
              "-outdir"        is the output directory.
              "-prefix"        is the output file name prefix.
              "-title"         overrides the default plot title.
              "-subtitle"      overrides the default plot subtitle.
              "-xlab"          overrides the default plot x-axis label.
              "-ylab"          overrides the default plot y-axis label.
              "-xlim"          is the min,max bounds for plotting the X-axis.
              "-ylim"          is the min,max bounds for plotting the Y-axis.
              "-filter"        is a list of filtering options for the tc_stat tool.
              "-tcst"          is a tcst data file to be used instead of running the tc_stat tool.
              "-dep"           is a comma-separated list of dependent variable columns to plot.
              "-scatter_x"     is a comma-separated list of x-axis variable columns to plot.
              "-scatter_y"     is a comma-separated list of y-axis variable columns to plot.
              "-skill_ref"     is the identifier for the skill score reference.
              "-series"        is the column whose unique values define the series on the plot,
                                 optionally followed by a comma-separated list of values, including:
                                 ALL, OTHER, and colon-separated groups.
              "-series_ci"     is a list of true/false for confidence intervals.
                                 optionally followed by a comma-separated list of values, including:
                                 ALL, OTHER, and colon-separated groups.
              "-legend"        is a comma-separted list of strings to be used in the legend.
              "-lead"          is a comma-separted list of lead times (h) to be plotted.
              "-plot"          is a comma-separated list of plot types to create:
                                 BOXPLOT, POINT, MEAN, MEDIAN, RELPERF, RANK, SKILL_MN, SKILL_MD
              "-rp_diff"       is a comma-separated list of thresholds to specify
                                 meaningful differences for the relative performance plot.
              "-demo_yr"       is the demo year
              "-hfip_bsln"     is a string indicating whether to add the HFIP baseline
                                 and which version (no, 0, 5, 10 year goal)
              "-footnote_flag" to disable footnote (date).
              "-no_ee"         to disable event equalization.
              "-no_log"        to disable log file generation.
              "-plot_config"   to read model-specific plotting options from a configuration file.
              "-save_data"     to save the filtered track data to a file instead of deleting it.
              "-save"          to call save.image().

[JohnHalleyGotway]

@YanJin-NOAA I'd like to mark this discussion as being answered and "lock it" to prevent future posts. That's how we encourage users to create new discussions for new questions. But I wanted to give you an opportunity to comment on it before doing so. Please feel free to select one of the responses as being the best answer to your original question. That'll help future users with similar questions find answers.