-
Notifications
You must be signed in to change notification settings - Fork 253
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Inconsistent land-sea mask among sfc*, oro* and static information files for control_p8 RT #1423
Comments
It looks like the CCPP/Noah MP DDT trace was not posted. Can you please include that in the issue so we can get some help debugging? |
@rsdunlapiv Here is the DDT trace. Also note that I changed its extension from html to txt since GitHub does not support it when attaching the file to the issue. Anyone who would like to check the trace, please change its extension to html. |
So the failures are:
This is failing at this line.
This is failing at this line. @junwang-noaa do you have anyone on your side who can look into this issue? Or do we need to try to debug this as part of the external land model project? |
@rsdunlapiv BTW, this is just for restart run. The initial run has no issue with those files (copied over coupled simulation input directory). Maybe something going on wrong with the restart files and mismatched in the land-sea mask. As I know from @junwang-noaa, there is a code in FV3 side that sets the land sea mask. https://github.com/NOAA-EMC/fv3atm/blob/develop/atmos_model.F90#L3156 |
@junwang-noaa @barlage JFYI, I did some testing in new external NoahMP model and I tried to define the mask like following (
Then, model seems working without modification that I done previously in BTW, i also checked the mask calculation in following line as an alternative method to use exactly same mask with the FV3, but could not reproduce same mask in the external land component side since it requires additional variables like Anyway, I still need to test solution that defines mask in NoahMP side for the restart case ( |
@uturuncoglu I think it is good to define the noahmp mask using noahmp%domain%frac and vegtype. Please confirm with @barlage. |
@junwang-noaa @barlage @jkbk2004 I think the most less complicated short term solution will be option 3. It is your decision. |
@uturuncoglu Did you ever try the control_p8 test case with |
@DeniseWorthen no I did not |
I switch to easier solution by modifying land-sea mask code in NoahMP NUOPC cap and removing modifications that uses some files from coupled RT for control under UFS (commit: a8b669d). Note that this is just a temporary solution and those input files needs to be fixed but this fix will allow us to work on this issue and address in near future with separate PR. |
@junwang-noaa @barlage I back to this issue again when I try to implement the fully coupled configuration (fv3+noahmp). At this point, I am both checking land fraction as well as the data in vegetation_greenness file and if land fraction is greater than 0 and vegetation_greenness has data over that point (not -999) then I am defining that point as land. But now issue with this logic. For example, in tile 3 the point (27,27) and (42,1) is defined as land if you look at oro file (the value is greater than 0) but vegetation_greenness has -999 for them. So, the external land model assign those point are water but in FV3 output (sfc* files) the land variable (land sea mask) show them as land. So, when I send this data to CCPP/physics these two points have unrealistic data since external land component does not calculate fluxes for those. I think we need to take action for land sea mask inconsistency across the input files otherwise it would be hard for me to make progress in fully coupled -land configuration. If you need I could provide more information. Is there any plan to make input files consistent interns of land sea mask? What do you suggest to proceed with this issue? |
@junwang-noaa @barlage I think the control_p8 is not reading those static information from static files but from sfc_data.tile* files. So, maybe the ccpp version noahmp is fine since all the information is coming from sfc_data.tile* files. Am I right? I'll try to setup configurations that uses active atmosphere to read static information from the sfc_data.tile* to see what happens. |
@junwang-noaa @barlage I tested the idea and get the static information from sfc_data.tile* files. It seems it fixed the consistency issue and I have no any missing points since all the information uses consistent land sea mask but I have a question. The information in sfc_data.tile* files are masked based on slmsk but there are some grid points set as water but have land fraction > 0 (if you look at oro* files you will see that some points in the coastline has no zero land fraction bu it set as water). How those points handled in the compositing step in ccpp/physics. Maybe ccpp/physics just get data from land when slmsk = 1 and other grid points from atm, ocn or ice. If you don't mind can you clarify this. I don't want to do anything wrong when composting fluxes coming from different components. |
Maybe I am wrong. It seems that control_p8 copies static information from @[INPUTDATA_ROOT]/FV3_fix_tiled/ (tests/fv3_conf/control_run.IN) and we need to get the static information from those files. If so, we still need to fix the land sea mask consistency issue in the input files and also recreate sfc_data* and gfs_data* files that are consistent with the new land sea mask information. Anyway, let me know what you think. |
@uturuncoglu my understanding is that the slmsk is a composite variable with value 0 or 1 for ocean and land. For fraction grid runs, it might be good to use land_fraction >0 (means grid box has land, in oro, Sfcprop(nb)%landfrac ) to check land point and the corresponding land features such as veg greenness. I think the land fields in sfc file and fix files are created from oro data, so they should be consistent. |
@uturuncoglu @junwang-noaa The sfc_data tile files should (need to) be fully consistent with the oro files. I have some scripts here (hera:/home/Michael.Barlage/data/check/consistency) that I use to confirm this since inconsistencies happen if one doesn't use the same fix files through the process. We want to run the land model on any point that has landfrac > 0 (in the fractional grid case) so the sfc_data files need veg/soil/etc on each of those points even though the lsmsk may not be land. If the same oro files are used throughout the process, there shouldn't be a problem. Note that this check script will report at the end if there are any defined land points that don't have soil/veg information. All that said, now I'm confused at what is happening in control_p8. The result of the C96 using the fix, oro and sfc files that I had from a control_p8 are shown below. What I see is that the oro slmsk and sfc veg/soil are consistent so these files will work in a non-fractional case. The fix files have 18322 valid veg types which is consistent with the most recent C96 fractional grid, but there are definitely some mismatches for the fractional grid using these files.
|
@junwang-noaa The sfc and oro files are consistent in terms of masking. I setup land cap to create its mask base on the mask information that is found in oro file (I did not use land fraction for this case) but i think I have still some issue. I could not set mask in land component based on land fraction because if I do so, I would end up having some points that land_fraction >0 but it is marked as water (in land se mask) and for those points I could not provide static information such as veg greenness (I am assuming to get it from sfc* file too) and land model would not work. Also, I could not get veg greenness from the static files (C96.vegetation_greenness.) since the land sea mask is not consistent with oro and sfc* files. This was my previous approach and I end up to have missing values for point (27,27) and (42,1) in tile 3. Anyway, I enabled debug output in CCPP side (aux2d) and look at fields that I am sending to CCPP from external land component. It seems that I have some unrealistic large values over coastlines. I think this is expected since I setup mask in external land component just by taking account the slmsk. I am trying to figure out the issue at this point but I think it is an issue related with the inconsistent land-sea mask across the input files. |
@barlage Thanks for the extra information. Yes, there are some mismatch issue in fractional grid case which prevents me to run the external land component all the points that has land fraction > 0. It is strange that CCPP/physics has no problem with it even if there is a mismatch for fractional grid case. Maybe there is an issue but nobody realized it until this point. Not sure. |
@uturuncoglu Based on the numbers I'm seeing in my output, it might be that the oro/fix/sfc files have not used the same processing. My understanding is that the goal is that the fix/sfc files should have the same veg/soil and that these should have the same count as land_frac>0 in the oro files. That way the sfc files can be used in both the fractional and non-fractional cases. I'm tagging @HelinWei-NOAA since he helped to create these fix files, I believe. |
@uturuncoglu @junwang-noaa @barlage I found an inconsistent issue. control_p8 still used the old oro_data in /scratch1/NCEPDEV/nems/emc.nemspara/RT/NEMSfv3gfs/input-data-20221101/FV3_input_data/INPUT_L127. We have couple of updates on the oro data, but the data over that directory never get updated. However the land-sea mask or other fixed fields like veg types used by control_p8 were created based on the updated oro data. |
@HelinWei-NOAA Do you think that updating the oro_data* files will fix the problem? In the external land component I am creating mask by using following code,
this ensures that the grid point will have both land fraction > 0 and also static data such as veg greenness. Since, land fraction is coming from oro_data file and vegtype from static file, inconsistencies in those files creates issue for me and lead to less land cells then used in CCPP/Physics. BTW, do we also need to create sfc_data* file from scratch. |
@HelinWei-NOAA Do you have the new oro data? I can make a new data input directory with the updated data (oro, sfc, veg etc) for testing before we update it in ufs-weather model. |
@junwang-noaa You can copy the latest oro data from /scratch1/NCEPDEV/nems/emc.nemspara/RT/NEMSfv3gfs/input-data-20221101/FV3_fix_tiled/C96/ |
@junwang-noaa Is it possible test in my case? I am working on Cheyenne and not have access to Hera. So, If you could copy the fixed files to Cheyenne and let me know what needs to be changed, then I could also test in my end. |
@barlage @HelinWei-NOAA I am not sure but gfs_data.* need to be consistent too. |
@barlage I think the ICs also need to be recreated with the right oro data |
@HelinWei-NOAA @barlage @yangfanglin May I ask who can help create the new sfc_data from the new oro data? The current sfc_data is under: /scratch1/NCEPDEV/nems/emc.nemspara/RT/NEMSfv3gfs/input-data-20221101/FV3_input_data/INPUT_L127. Thanks! |
@junwang-noaa Bing Fu created all ICs including sfc_data for prototype tests. Who created the current sfc_data file? George? |
Here is the procedure how to create sfc_data created by Fanglin |
I thought all the "fix" files saved under, for instance on Cactus at /lfs/h2/emc/global/noscrub/emc.global/FIX/fix/orog/20220805, have followed the procedure and are all consistent. The UFS_UTIL/CHGRES has been pointing to the latest fix files as well. |
@yangfanglin The input data for RT control_p8 test is still using the old oro/sfc files. I am going to follow your document to create new sfc_data from the updated oro data. By the way, where are the oro_data_ls.tileX.nc and oro_data_ss.tileX.nc fix files? I do not find them under /lfs/h2/emc/global/noscrub/emc.global/FIX/fix/orog/20220805/C96/. Thanks |
@junwang-noaa Here the "X" represents 1,2 ...6 tiles. You can find them under /lfs/h2/emc/global/noscrub/emc.global/FIX/fix/ugwd/20220805 |
@junwang-noaa for these two oro data used for the gravity wave drag scheme, I don't think they have any update corresponding to the oro_data update. |
I had once talked with Mike Toy about recreating the fix files for uGWD.v2. But Mike said there is no need to do this since the GWD code was written to following the "model" grids/masks. It should not cause any inconsistency. |
@junwang-noaa You only need to do the step C. Also I don't know for your case if you have to run that script twice (once with ceil oro data, once with ceil oro data)(step C1). You can ask Bing if you have any questions.
|
@junwang-noaa Also I remember the sfc_data for the coupled RTs was created by Bing. Maybe it is better to let him create for the uncoupled RTs as well. |
sfc_data for coupled and uncoupled is different? or just different initial time? |
@HelinWei-NOAA That is a good question. I do see under /lfs/h2/emc/global/noscrub/emc.global/FIX/fix/orog/20220805 there are three different directories for, for instance C768, C768 Can someone check if the files under the three directories are different ? I think for unification and consistency we should be using the C768.mx025_frac for both coupled or uncoupled, and for both fractional and non-fractional grid options. |
@junwang-noaa @HelinWei-NOAA @barlage I just wonder if we have any progress in this issue. Are we plaining to update input files for standalone cases soon? |
Just a update on this issue. Mike is leading project to update the script to create new ICs, once the script is ready, we can create consistent fix files and ICs for the test case. |
@junwang-noaa Is this related to George's current PR #2005 ? |
@junwang-noaa @DeniseWorthen @barlage Just to update about this old issue. As I mentioned before, I have still issue with the land component even if I use V2 sfc/oro files. I first write a very basic NCL script to check number of land and ocean points for all the input files of
This basically prints out
It looks like number of land and ocean points are same. But, I think this is not correct way to check the land and sea points. Then, I used simple script that you can find in the description of this issue and try to look at mask difference with it. For oro file (as an example tile 1), it looks like the oro file mask is not matching with fixed files around coastline (based on slmsk variable), Then, I did same with but in this case with surface file (again using same variable slmsk). It strangely match with the fixed files and I am not seeing any issue with the mask. I double check that in my run directory is V2 but maybe it is not consistent with oro file. To be sure, I plot the difference of slmsk variable between sfc and oro files and it seems I am correct and they are not matching. You could see it from following plot, I think maybe I need to switch back to use original files and not to use V2 file for initial conditions for land coupled configurations. At this point, the active atmosphere coupled RTs seems to use V2 but not actually since @DeniseWorthen found a issue - #2267 and the configurations are running since they are still using original files due to a bug in
control_run.IN -
Anyway, if there is a oro file which is consistent with V2 surface file, I could also try with it. Let me know what you think? Maybe fv3 is updating the mask internally and writes to sfc file and then there is no way to have land-sea mask consistent with the oro file but now sure. |
@junwang-noaa @DeniseWorthen @barlage JFYI, The component land model calculates its mask using land fraction coming from orography file and vegetation type fining from fixed file (https://github.com/NOAA-EMC/noahmp/blob/ec38ea3d902644cd4519d5fe060316859ccdc108/drivers/nuopc/lnd_comp_domain.F90#L245). So, those two needs to be consistent to have correct masking that matches with sfc file since fully coupled configurations are using sfc file for initial conditions. So, land component model needs to have consistent view of land sea mask for those files (oro, fixed files and sfc file) to function properly. |
Description
The land sea mask information used by
control_p8 ( and its derivatives such as
control_restart_p8) is not consistent between the oro* files, sfc* files and the files that provides static information to the land model (i.e.
C96.snowfree_albedo.tile*.nc`). This issue is appeared during the implementation of side-by-side RT for external land component under UFS model.To Reproduce:
This is tested on NCAR's Cheyenne machine using both Intel and Gnu compilers.
Steps to reproduce the behavior.
git clone --recursive https://github.com/ufs-community/ufs-weather-model.git
control_p8
RT - ```cd ufs-weather-model/tests; ./rt.sh -k -n control_p8`slmsk
) in the orography and variable in any file that has static information to see the difference4.Additional context
fv3_conf/control_run.IN
script to overwrite default files from coupled control as a workaround and it seems external land component RT based oncontrol_p8
is working without any issue with this fix.But this change breaks the
control_restart_p8
and causes error in CCPP/noahmp (see attached trace collected by DDT). The error does not appear if I use the original file (not from coupled control). I am not sure why CCPP/noahmp does not have issue with inconsistent land sea mask and missing static information.I think that the control_p8 should use the same atmosphere ICs as the
cpld_control_p8
, this includes the gfs_data*, oro_data* and sfc_data*. They are both using same static files such asC96.slope_type.tile*
.Output
Screenshots
Plots for mask from static file, orography and the difference,
The text was updated successfully, but these errors were encountered: