Level inconsistency when processing chemistry fields from inline data (RRFS-CMAQ) #1053
Replies: 9 comments 9 replies
-
|
@edwardstrobach, OK, I've logged onto WCOSS to take a look at this issue. As I understand it, your question is really this... How should I set up the MET config file to read the first record from the following grib2 file? /gpfs/dell2/ptmp/Edward.Strobach/metplus_aqmax/archive/r111_v1/20190815/out_ozmax1.grb2 Unfortunately, I can't find that file. In fact, the /gpfs/dell2/ptmp/Edward.Strobach/metplus_aqmax/archive directory is empty. Please tell me where I can find that file to debug more... or just attach it in a zip file to this discussion. Looking in the wgrib2 output you sent: I am most concerned about the forecast hour values listed for the first record: -2147483641--2147483618 hour ave fcst To me that indicates a poorly formatted GRIB2 header. |
Beta Was this translation helpful? Give feedback.
-
|
Hi John,
That is correct. The impression I get from this exercise is that the issue
seems related to how the level is being specified. You might find
something different, however. Yes, those files are stored in ptmp which
gets wiped frequently. I reran the script and the ptmp directory has been
repopulated, so you should be able to view the contents within that
directory now.
|
Beta Was this translation helpful? Give feedback.
-
|
Hello, did you happen to get a chance at looking at the files? If not, then I'll likely have to rerun again. Thanks |
Beta Was this translation helpful? Give feedback.
-
|
My apologies Ed, I did not. I was out of the office last week on vacation and dropped the ball. But I'm available to debug today and the remainder of this week. |
Beta Was this translation helpful? Give feedback.
-
|
No problem John. Everyone needs a vacation. Let me reproduce the issue and get back to you. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Hi John, I reproduced the issue (I hope). The output from the "/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210819170724" has the same error as the original file. As originally done, I separate the grib file and create grib output for each field. You can look at referenced files here: I've been running a test script from this location: /gpfs/dell2/emc/modeling/save/Edward.Strobach/MET/METplus/scripts/run_metplus_aqmax_test.sh You can see that the wgrib file output from the forecast file that is used is this: You can also see that the error references that forecast file. Notice the similarities in name. This has led me to believe that this could be a "level" issue. In the original script the level was sigma. This is hybrid. Don't know what the specifier should be as a result: WARNING: Note also that this occurs during the pointstat step. |
Beta Was this translation helpful? Give feedback.
-
|
Ed, thanks for posting the data. I pulled down the sample file and was able to plot it using plot_data_plane right away:
Here's the log file to which you pointed me: Scrolling up from the warning/error messages, I see METplus reporting this configuration setting: But that valid time you're requesting (20190816_04) does NOT match the valid time reported by wgrib2 (20190817_03): So that's why MET does not find any data... because you're telling it to read data for a timestamp that is not present in the input file. Hope that helps clarify. |
Beta Was this translation helpful? Give feedback.
-
|
Okay. I see. The inline data does indeed have a shift by one hour
relative to my old set-up. I did account for those changes elsewhere when
creating the separate grib files. I guess I can adjust that using
FCST_VAR1_OPTIONS
…On Thu, Aug 19, 2021 at 2:21 PM johnhg ***@***.***> wrote:
Ed, thanks for posting the data. I pulled down the same file and was able
to plot it using plot_data_plane right away:
MET-main_v10.0/met/bin/plot_data_plane 20190815/out_ozmax1.grb2 plot.ps 'name="OZMAX1"; level="L1";' -v 4
[image: Screen Shot 2021-08-19 at 12 12 34 PM]
<https://user-images.githubusercontent.com/21087144/130122178-1233f51d-609a-463c-8ae2-d56d9a7f70d6.png>
So I'm confident that MET is able to read data from these files. The most
likely problem is some issue in your configuration.
Here's the log file to which you pointed me:
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210819170724
Scrolling up from the warning/error messages, I see METplus reporting this
configuration setting:
export FCST_FIELD="{ name=\"OZMAX1\"; level=\"L1\"; valid_time= \"20190816_04\"; }";
But that valid time you're requesting (20190816_04) does NOT match the
valid time reported by wgrib2 (20190817_03):
wgrib2 -V 20190815/out_ozmax1.grb2
1:0:vt=2019081703:1 hybrid level:16-39 hour ave fcst:OZMAX1 Ozone Daily Max from 1-hour Average [ppbV]:
...
So that's why MET does not find any data... because you're telling it to
read data for a timestamp that is not present in the input file.
Hope that helps clarify.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1053 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ALO6JVDLDEHVOJF6GRHYMJLT5VDRPANCNFSM5BNBBEKA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>
.
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks. I'm able to create point stat files and stat files, but they are
not being populated. There's probably some mismatch still. I'm surprised
that I had to do a 23 hour shift from the original scripts I used. The
files not being populated seems related to continued mismatch. I will
focus on one variable (OZMAX1) and try to resolve that issue. The steps
indicate success, but there has to be something in the file that points to
no matched pairs and the reason for that. I do see now that my FCST_FIELD
is correctly specified at least:
08/19 19:32:44.514 metplus.PointStat (command_builder.py:290) DEBUG:
FCST_FIELD={ name="OZMAX1"; level="L1"; valid_time= "20190817_03"; }
On Thu, Aug 19, 2021 at 2:42 PM Edward Strobach - NOAA Affiliate <
***@***.***> wrote:
… Okay. I see. The inline data does indeed have a shift by one hour
relative to my old set-up. I did account for those changes elsewhere when
creating the separate grib files. I guess I can adjust that using
FCST_VAR1_OPTIONS
On Thu, Aug 19, 2021 at 2:21 PM johnhg ***@***.***> wrote:
> Ed, thanks for posting the data. I pulled down the same file and was able
> to plot it using plot_data_plane right away:
>
> MET-main_v10.0/met/bin/plot_data_plane 20190815/out_ozmax1.grb2 plot.ps 'name="OZMAX1"; level="L1";' -v 4
>
> [image: Screen Shot 2021-08-19 at 12 12 34 PM]
> <https://user-images.githubusercontent.com/21087144/130122178-1233f51d-609a-463c-8ae2-d56d9a7f70d6.png>
> So I'm confident that MET is able to read data from these files. The most
> likely problem is some issue in your configuration.
>
> Here's the log file to which you pointed me:
>
> /gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210819170724
>
> Scrolling up from the warning/error messages, I see METplus reporting
> this configuration setting:
>
> export FCST_FIELD="{ name=\"OZMAX1\"; level=\"L1\"; valid_time= \"20190816_04\"; }";
>
> But that valid time you're requesting (20190816_04) does NOT match the
> valid time reported by wgrib2 (20190817_03):
>
> wgrib2 -V 20190815/out_ozmax1.grb2
> 1:0:vt=2019081703:1 hybrid level:16-39 hour ave fcst:OZMAX1 Ozone Daily Max from 1-hour Average [ppbV]:
> ...
>
> So that's why MET does not find any data... because you're telling it to
> read data for a timestamp that is not present in the input file.
>
> Hope that helps clarify.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#1053 (reply in thread)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ALO6JVDLDEHVOJF6GRHYMJLT5VDRPANCNFSM5BNBBEKA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>
> .
>
|
Beta Was this translation helpful? Give feedback.
-
|
FYI,
I believe this is happening because the obs. are coming from 08/15/2019
(i.e.,
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqmmax/aqmax1/r111_v1/
prepbufr.aqm.2019081500.nc). The output from the run is from 08/15/2019.
The wgrib2 command indicates that:
1:0:d=2019081512:OZMAX1:1 hybrid level:16-39 hour ave fcst:
I guess the 08/17/2019 03 is represented by the addition of the two numbers
in blue. That would result in 20190817_03. Would you recommend that I
change my valid time to match that for the pb2nc step? Maybe I'm wrong,
but since it is a 16-39 hour average, it would seem that I would need data
from both 08/16/2019 and 08/17/2019 to successfully get all the matched
pairs I need? Sorry for all the emails, just trying to understand better.
|
Beta Was this translation helpful? Give feedback.
-
|
Ed, just to clarify, running wgrib2 with/without the -V option is useful: Here, "d=2019081512" indicate the model initialization time. And here, "vt=2019081703" indicates the valid time. Note the "-vt" option also prints the valid time. As for how to compare the model output to the point observations, its really up to NOAA/EMC folks to decide. I'd recommend that you discuss these details with @PerryShafran-NOAA. Do you want to verify against the raw point observation values? Perhaps compute matched pairs for all observations within the time window? Perhaps only use observation values close to the end of the time window? Or perhaps you want to compute the average obs value at each station over a 24-hour time window that matches the 16-39 hour forecast time window? I believe that all should be possible, but I'd recommend giving it some thought and discussing internally. |
Beta Was this translation helpful? Give feedback.
-
|
I would not think that we would want to change what we have been doing in terms of this type of verification. I'm trying to follow what has changed since y'all helped set up this type of verification for me previously. |
Beta Was this translation helpful? Give feedback.
-
|
I'm keeping my offline scripts (similar to yours) separate from the inline
scripts. I need to test inline verification as well. John mentioned the
different things I could do. I think computing the average obs value at
each station over a 24-hour time window that matches the 16-39 hour
forecast time window is the way to go. I'll work on that tomorrow and see
if anything comes from that. Thanks again John for your help.
|
Beta Was this translation helpful? Give feedback.
-
|
Hello, I'm following up with this discussion with some new findings. PointStat does run to completion successfully when making the suggested change but there are no matched pairs and thus nothing is being stored in the point stat files. The log file (/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210914144647) shows that the recommended change passed through: 09/14 14:46:50.688 metplus.PointStat (command_builder.py:290) DEBUG: FCST_FIELD={ name="OZMAX1"; level="L1"; valid_time= "20190817_03"; } Originally the valid_time was 20190816_04. You are correct about the level not being the issue. The message below also shows that for the whole region there are zero matched pairs: The point stat file is created as mentioned earlier, but it is empty: [Edward.Strobach@m71a3 scripts]$ ls -ltr /gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqmmax/stat/aqmax1/r111_v1//point_stat_R111_V1_AIRNOW_390000L_20190817_030000V.stat It probably has something to do with how the data is written. The script that I'm adjusting tailored for the offline file structure. Here is a bit of a comparison showing these differences: Offline (the one that works) maximum ozone from 1-hour average [Edward.Strobach@m71a1 aqm.20210910]$ /gpfs/dell1/nco/ops/nwprod/grib_util.v1.1.1/exec/wgrib2 -v aqm.t12z.max_1hr_o3.148.grib2 1:0:vt=2021091105:1 sigma level:-2147483641--2147483618 hour ave fcst:OZMAX1 Ozone Daily Max from 1-hour Average [ppbV]: 2:141165:vt=2021091205:1 sigma level:17-40 hour ave fcst:OZMAX1 Ozone Daily Max from 1-hour Average [ppbV]: 3:278591:vt=2021091305:1 sigma level:41-64 hour ave fcst:OZMAX1 Ozone Daily Max from 1-hour Average [ppbV]: The first record has a valid time of 2021091105, which marks the ending hour of the -7-16 averaging interval. The second record has a valid time of 2021091205, which marks the ending hour of the 17-40 averaging interval. And finally, the third record has a valid time of 2021091305, which marks the ending hour of the 41-64 averaging interval. These correspond to days 1, 2, and 3. Running pointstat generates three of these files corresponding to each day, all of which are populated: -rw-r--r-- 1 Edward.Strobach emcmodel 108426 Sep 14 07:23 point_stat_PROD_AIRNOW_160000L_20210912_040000V.stat Inline System (the one that doesn't work) When I run wgrib2 on the subsetted file for OZMAX1 I see the following: Right now there is only one record, not 3. However, there is a very important and subtle difference between how the offline is being processed v. how inline is being processed. Offline The corresponding vt record shows the following in the 1-hour max ozone file: Notice it is off by 1 hour. Inline 09/14 14:46:50.688 metplus.PointStat (command_builder.py:290) DEBUG: FCST_FIELD={ name="OZMAX1"; level="L1"; valid_time= "20190817_03"; } while the corresponding vt is While the inline system have the same definition, the offline system is off by 1 hour. I tried using valid_time = 20190817_02 based on what I'm seeing for the offline system as a way to test for consistency. That did not work, however. I'm unsure how to proceed. |
Beta Was this translation helpful? Give feedback.
-
|
Oh, and I do want to clarify...I don't believe this is an issue with metplus; rather, it is an issue with how the records are being written. I think there needs to be a 1 hour shift in the record in order for metplus to detect matched pairs. Do you agree? |
Beta Was this translation helpful? Give feedback.
-
|
Hi Ed, yes, I definitely agree that the issue is in how that averaging period is defined... the 17-40 averaging window certainly differs from 16-39. And if you're explicitly specifying the "valid_time" to be evaluated then data for that time would definitely need to exist to make it all work. However, I really can't tell you what is correct here. Should it be 17-40 or should it be 16-39? I have no idea. I'd say that's up to you and the data producers to decide. You have some options for how to handle this via METplus. Obviously, if you can determine what is correct and make the metadata correct, that'd be the best choice. But if you can't make the metadata correct, you could account for the discrepancy in the METplus setup. As you already know, specifying the valid_time in the configuration string tells MET to only use data for that valid time. And that's a useful way of selecting a record from a GRIB file that includes multiple OZMAX1 records: By default, Point-Stat defines the matching time window relative to the valid time of the forecast. For example, a matching time window 1 hour +/- the forecast valid time is given by: However, the Point-Stat "-obs_valid_beg" and "-obs_valid_end" command line options can be used to override that default: If you need to account for a 1-hour offset between the forecast valid time and the verifying observations, you could do so either using the "obs_window" config option or explicitly set the "-obs_valid_beg/end" command line options. Either would work. For example, if you want a +/- 1 hour window but for 1 hour earlier, you could just subtract 1 hour (3600 seconds) from each entry: So again, if you can fix the data to make it more correct, do so. If not, you should be able to account for the discrepancy in your configuration files. I wish I could tell you what the right answer is, but I can't. I imagine that @PerryShafran-NOAA struggled through the same details when he originally set up this verification workflow. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks John for explanations and the discussion of the obs_window feature. Just to clarify, I can keep the OBS_WINDOW specification for the pb2nc step the same but should specify -7200 for obs_window_beg and 0 for obs_window_end for point stat only to account for the hour shift. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, you certainly can keep using the "obs_window" config option. I was just explaining how it works and how you can set it to account for an offset in time between the forecast and observation data. Exactly how you set it is up to you. I don't know if you shift the values forward one hour or backward one hour, but with some trial and error, it should be pretty obvious. |
Beta Was this translation helpful? Give feedback.
-
@edwardstrobach, please note that I've edited this Discussion by migrating the content over from #1070 here. Is this what you intended? Thanks @JohnHalleyGotway
I'm processing grib output from the RRFS-CMAQ system with new max/ave.
output from post upp. I've made all the adjustments but am facing an issue
that appears linked to how the levels are being specified. The original
message is attached below. In the past I set the forecast level to "L1".
I don't think it works for this case. Notice that the original output was
on sigma levels while this output is on hybrid levels.
Hi Lin,
I made some structural changes in the code to accommodate the differences
in file structure. In the past we separated the fields out and grouped
days 1, 2, and 3 for each file. For instance, for OZMAX8, there will be
three entries in a grib file corresponding to day 1, 2, and 3. In your
example you lump all fields into one file. In the future we may want to
adapt the offline approach to separating data so that we don't have to
change the code.
Unfortunately, the error still occurs. You can see this in my log file (
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210722174824).
I only applied this to OZMAX1, so please look at that field, not the
others. The references in the log suggest that everything is pointed to
the right locations. For instance, I parsed out the OZMAX1 field from
your post output and put it a ptmp directory that can be found here (
/gpfs/dell2/ptmp/Edward.Strobach/metplus_aqmax/archive/r111_v1/20190815/out_ozmax1.grb2
)
This is what the header info reads for that file; notice the OZMAX1 field
is there:
[Edward.Strobach@v71a3 scripts]$
/gpfs/dell1/nco/ops/nwprod/grib_util.v1.1.1/exec/wgrib2
/gpfs/dell2/ptmp/Edward.Strobach/metplus_aqmax/archive/r111_v1/20190815/out_ozmax1.grb2
1:0:d=2019081512:OZMAX1:1 hybrid level:16-39 hour ave fcst:
The step to generate the observations for verification, using the prepbufr
data and converting that to netcdf file format (that is stored here:
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqmmax/aqmax1/r111_v1/prepbufr.aqm.2019081500.nc
http://prepbufr.aqm.2019081500.nc), worked and is supposed to be used
and combined with the out_ozmax1.grb2 file to generate point stat files.
Unfortunately, that doesn't happen because of this error:
WARNING:WARNING: process_fcst_climo_files() -> no fields matching
OZMAX1/L1 found in file:
/gpfs/dell2/ptmp/Edward.Strobach/metplus_aqmax/archive/r111_v1/20190815/out_ozmax1.grb2WARNING:ERROR
:ERROR : process_fcst_climo_files() -> no requested forecast data found!
Exiting...ERROR :07/22 17:48:32.013 metplus.PointStat (command_runner.py:
Note also that I never changed the FCST name or level information. I
shouldn't have to since they are supposedly consistent. Note, however,
that when wgrib2'ing the file that we have used for the offline system,
that I get this:
[Edward.Strobach@v71a1 20190807]$
/gpfs/dell1/nco/ops/nwprod/grib_util.v1.1.1/exec/wgrib2
aqm.t12z.max_1hr_o3.148.grib2 1:0:d=2019080712:OZMAX1:1 sigma
level:-2147483641--2147483618 hour ave fcst:2:144034:d=2019080712:OZMAX1:1
sigma level:17-40 hour ave fcst:
I highlighted in blue the difference in the level information. The
vertical coordinate from the inline system is a hybrid sigma-pressure
coordinate system, I believe, while the offline system is pure sigma. I'm
not sure what the level distinction should be. Could it be that the "L1"
specification is resulting in this error? What then would be the
appropriate specification. Will need help to figure this out I think.
Once this is resolved, then I think it should work and we can then generate
the stats needed. Thanks for working and coordinating on this issue. Let
me know what needs to be done or if more information is needed.
Beta Was this translation helpful? Give feedback.
All reactions