re-grid the ERA5 data #1334
-
Hi, I'm now using ECMWF ERA5 Reanalysis data ds633.0 as obs to verify my forecast fields u, v, t, qv and windspeed. These ERA5 data has 37 pressure levels, with two data format of .grb and .nc. The different field (u,v,t,q ..) is in a different file name, but put one day data in one file. The first step I would like to do is re-grid the ERA5 data to my WRF domain, but failed. Please help me on this, see the log file and config file as below. log file: /glade/scratch/zhuming/metplus4/logs/metplus.log.20211222102748 My other question: Thanks! cheyenne5:/glade/collections/rda/data/ds633.0/e5.oper.an.pl/201507>l *2015072900_2015072923.nc |
Beta Was this translation helpful? Give feedback.
Replies: 9 comments 3 replies
-
Hi @zhumingying. I took at look on Cheyenne as this file:
I have a couple of observations about the NetCDF header of this file:
The original GRIB1 data is also in that directory, and you could use that instead.
So the MET tools should be able to read these NetCDF files. When getting started with new gridded datasets in MET, its always a good idea to start by running the plot_data_plane tool to plot some data from the file:
Note that it's probably even easier to use the GRIB data since you can explicitly request the pressure level you'd like:
Note that each GRIB file contains 24 hours of data. I used the "valid_time" config option to select a specific hour. So MET can read this data directly from the input NetCDF and GRIB files. Your next question is how to regrid to your WRF domain. And I'll assume that you're trying to run grid_stat to compare your WRF output to this ERA reanalysis.
Of these, I'd recommend starting with (2) since it's easier. Please see an example of this in the METplus User's Guide: Specifically, you'd set "GRID_STAT_REGRID_TO_GRID = FCST" to tell grid_stat to regrid the obs data to the forecast domain. For temperature data, you'd probably want to use bilinear interpolation:
The regrid_data_plane tool is not designed to regrid an entire input file. Instead, it only regrids individual 2D fields, listed one by one. So I think you'll find that running first would make the logic more messy. In general, I'd recommend:
|
Beta Was this translation helpful? Give feedback.
-
Hi Zhuming: I took a look at your log file. One problem is that you are processing 10 levels, but only gave 1 output name. METplus needs a unique output name for each combination of variables and levels. So, you would need somethin like: I started my testing by commenting out the output field name. However, I then got an error that the script could not find the observation files. It looks like you are missing the second date when specifying your observation files. Also, since each file is always labeled starting at the 00 hour and ending at 23 (and the timesteps are inside the file), you probably need to change your filename template to look like this: Then, add in the valid_time mentioned above to select specific time steps inside each file. If you are using the netCDF files, you would specify this by: If using grib files, then you would specify as: You can do multiple fields (VAR1, VAR2, etc) as long as they are in the same input file. Christina |
Beta Was this translation helpful? Give feedback.
-
Hi John,
I tried to revise my config file and run METPlus with your suggestions, but
still a fatal error occurred. I can not find what the error is?
Could you help me with this?
Thanks lot!
Zhuming
log file: /glade/scratch/zhuming/metplus4/logs/metplus.log.20220103085357
[metplus.log.20220103085357.txt](https://github.com/dtcenter/METplus/files/7803159/metplus.log.20220103085357.txt)
config file: /glade/work/zhuming/METplus4/GridStat_wrf_ERA5_GRIB.conf
[GridStat_wrf_ERA5_GRIB.conf.txt](https://github.com/dtcenter/METplus/files/7803163/GridStat_wrf_ERA5_GRIB.conf.txt)
|
Beta Was this translation helpful? Give feedback.
-
Hi Zhuming: Try changing these lines: Also, since you are using the grib files, you will need to change your level specification: and I'm not sure exactly which pressure levels you want, but for grib files, you need to specify them with the letter P and then the level. Also, you will need to add this lines: Thanks, |
Beta Was this translation helpful? Give feedback.
-
Hi @zhumingying, I found a filename template in your config file that was missing some curly braces, so it wasn't being interpreted properly and causing the fatal error. There is an open METplus GitHub issue (#608) to improve the error checking on the code that processes the filename templates. This would help alert you to which template has the issue instead of crashing. To answer your original question, no, you will not be able to process all of the fields in a single config file using VAR<n> syntax. Each field of the observation data is in a different file, which will require a separate call to the grid_stat tool to process each. However, there are a lot of common configuration variables used for processing each field that can stored in a single config file. The different settings for each field can be stored in their own config file that can be passed in after the "base" config file to process each field. I have set up config files to demonstrate this on cheyenne. I first started by creating a config file to process the U field. Your configuration processes many init times, forecast leads, and pressure levels, so I started by trying to process a single run time (init/lead) and single pressure level. I ensured that everything ran properly before adding more. Note that I excluded many of the config variables from the example below. Also note that excluding the INIT_END time (in METplus v4.0.0+) will only run for the INIT_BEG time.
Once I confirmed that I was able to get this to produce output, I added in the other pressure levels and forecast lead times (one at a time) to ensure no WARNING message came from grid_stat).
I then copied the config file for processing U to modify for processing another field (V). I reduced the times/levels again to ensure everything worked first, searching the output log file for WARNING or ERROR messages that came from grid_stat. Once I had a working config file for V, I ran diff on the 2 config files to see what had changed between them:
Note that I configured these cases to write to a different directory for each field to avoid overwriting the output files and referenced the OBS_VAR1_NAME in the output template:
If you prefer each file to be written to the same directory, you could exclude the var name from the output template and instead set it in the GRID_STAT_OUTPUT_PREFIX to change the output filenames instead:
Using this information, I created a conf file called GridStat_wrf_ERA5_GRIB_base.conf that has the variables that differ between the 2 fields commented out. Then I created files for each field that only contain the variables that differ. Here is an example of GridStat_wrf_ERA5_GRIB_U_only.conf:
To run for U, I call run_metplus.py passing in first the base config file, then the U config file, then finally by user config file:
I ran this for U, V, T, and Q for init 20150715 0Z for leads 0-48 x 3. The output is in /glade/u/home/mccabe/data/out-zhuming. Let me know if you have any questions about any of this or run into any other issues. |
Beta Was this translation helpful? Give feedback.
-
Hi @zhumingying, have you had a chance to review the config files and output on Cheyenne? Do you have any other questions or issues regarding this subject? |
Beta Was this translation helpful? Give feedback.
-
Hi George,
Yes, It works well for U,V, T, and Q. Thanks a lot.
Now I would like to verify wind_speed. I first created an ERA5 global wind
speed NetCDF file using NCL, and then did GridStat verification the same as
U,V .., but it failed. Could you have a look?
Thanks!
Zhuming
log file: /glade/scratch/zhuming/metplus4/logs/metplus.log.20220110224208
config file: /glade/work/zhuming/METplus4/GridStat_mpas_ERA5_GRIB_ws.conf
…On Mon, Jan 10, 2022 at 5:37 PM George McCabe ***@***.***> wrote:
Hi @zhumingying <https://github.com/zhumingying>, have you had a chance
to review the config files and output on Cheyenne? Do you have any other
questions or issues regarding this subject?
—
Reply to this email directly, view it on GitHub
<#1334 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFASISMLTWNTAGVJVIF6LKTUVN3S7ANCNFSM5KTEGJVA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
Hi George,
It works well.
Thanks!
Zhuming
…On Tue, Jan 11, 2022 at 5:32 PM George McCabe ***@***.***> wrote:
Here is the error message from your log file:
ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &) const
-> needed 4 arguments for variable WS, got 2
It looks like the values you have set for OBS_VAR1_LEVELS are using the
GRIB format, while the data you are reading is NetCDF.
Here is the header from the WS field in your NetCDF file:
float WS(time, level, latitude, longitude) ;
The WS field is a 4-D variable, so you will need to format the levels to
extract the appropriate data from the variable. Since the first dimension
is time and the last 2 are lat/lon, you should be able to use a similar
format as the forecast data to define the NetCDF dimensions to read. It
looks like there are 37 levels in the file, so you will need to match up
the values to what is used in the forecast file to ensure that the same
level is used for the verification.
FCST:
pressure = 925, 850, 700, 600, 500, 400, 300, 250, 200, 100 ;
OBS:
level = 1, 2, 3, 5, 7, 10, 20, 30, 50, 70, 100, 125, 150, 175, 200, 225,
250, 300, 350, 400, 450, 500, 550, 600, 650, 700, 750, 775, 800, 825,
850, 875, 900, 925, 950, 975, 1000 ;
The first level in the FCST is 925, so you will need to find the index for
925 in the OBS, which is 33. The next level is 850, so you will need to
find the index for 850 in the OBS, which is 30.
OBS_VAR1_LEVELS =
({valid?fmt=%Y%m%d_%H%M%S},33,*,*),
({valid?fmt=%Y%m%d_%H%M%S},30,*,*),
({valid?fmt=%Y%m%d_%H%M%S},25,*,*),
({valid?fmt=%Y%m%d_%H%M%S},23,*,*),
({valid?fmt=%Y%m%d_%H%M%S},21,*,*),
({valid?fmt=%Y%m%d_%H%M%S},19,*,*),
({valid?fmt=%Y%m%d_%H%M%S},17,*,*),
({valid?fmt=%Y%m%d_%H%M%S},16,*,*),
({valid?fmt=%Y%m%d_%H%M%S},14,*,*),
({valid?fmt=%Y%m%d_%H%M%S},10,*,*)
—
Reply to this email directly, view it on GitHub
<#1334 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFASISNFNCOCB6FY466RQATUVTD3HANCNFSM5KTEGJVA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
Here is the error message from your log file:
It looks like the values you have set for OBS_VAR1_LEVELS are using the GRIB format, while the data you are reading is NetCDF. Here is the header from the WS field in your NetCDF file:
The WS field is a 4-D variable, so you will need to format the levels to extract the appropriate data from the variable. Since the first dimension is time and the last 2 are lat/lon, you should be able to use a similar format as the forecast data to define the NetCDF dimensions to read. It looks like there are 37 levels in the file, so you will need to match up the values to what is used in the forecast file to ensure that the same level is used for the verification. FCST:
OBS:
The first level in the FCST is 925, so you will need to find the index for 925 in the OBS, which is 33. The next level is 850, so you will need to find the index for 850 in the OBS, which is 30.
|
Beta Was this translation helpful? Give feedback.
Here is the error message from your log file:
It looks like the values you have set for OBS_VAR1_LEVELS are using the GRIB format, while the data you are reading is NetCDF.
Here is the header from the WS field in your NetCDF file:
The WS field is a 4-D variable, so you will need to format the levels to extract the appropriate data from the variable. Since the first dimension is time and the last 2 are lat/lon, you should be able to use a similar format as the forecast data to define the NetCDF dimensions to read. It loo…