Ninth reconciliation PR from production/RRFS.v1 + Add CCPP host variable to track initialization of GFS_phys_time_vary (#2544) + develop of WW3 up to 2024-01-11 to dev/ufs-weather-model #1342 (#2549)

[grantfirl]

Commit Queue Requirements:

• Fill out all sections of this template.
• All sub component pull requests have been reviewed by their code managers.
• Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
• Commit 'test_changes.list' from previous step

---

Description:

This PR combines 3 into 1.

The first is identical to #2405 and #2428 and is the work of @JiliDong-NOAA

The second is #2544: The set of PRs listed belows adds support for multiple physics instances running the GFS_phys_time_vary scheme. No changes in the ufs-weather-model repository itself. See associated PRs below for more information.

The third is #2549: Merge develop of WW3 up to 2024-01-11 to dev/ufs-weather-model

Commit Message:

* UFSWM - fix REFS ensemble member and add GF convection cold start parameter; Add CCPP host variable to track initialization of GFS_phys_time_vary; Update WW3 with develop up to 2024-01-11
  * FV3 - fix REFS ensemble member and add GF convection cold start parameter; Add CCPP host variable to track initialization of GFS_phys_time_vary, fix bug for MP Thompson init flag
    * ccpp-physics - fix REFS ensemble member and add GF convection cold start parameter; In GFS_phys_time_vary.fv3: make is_initialized a host model variable to support multiple instances of CCPP physics in a model run
  * WW3 - Update one commit hash

Priority:

• High: Needed for SRW App

Git Tracking

UFSWM:

Closes (replaces) NCAR/ccpp-physics#1000

Sub component Pull Requests:

• FV3: Ninth reconciliation PR from production/RRFS.v1 NOAA-EMC/fv3atm#899
  • ccpp-physics: Ninth reconciliation PR from production/RRFS.v1 ccpp-physics#238
• WW3: Merge develop up to 2024-01-11 to dev/ufs-weather-model  NOAA-EMC/WW3#1342

UFSWM Blocking Dependencies:

• None

---

Changes

Regression Test Changes (Please commit test_changes.list):

• No Baseline Changes.

Input data Changes:

New input data: New WW3 Input data currently located on hera at: /scratch1/NCEPDEV/climate/Jessica.Meixner/PR_WW3/UFSPRww3Jan11/WW3_input_data_20240111

Library Changes/Upgrades:

• No Updates

---

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • Jet
  • Gaea C5
  • Gaea C6
  • Derecho
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
• opnReqTest (complete task if unnecessary)

[grantfirl]

@jkbk2004 I combined #2544 into this one because neither changes baselines and I thought it would lighten the merge queue load a bit. Please let me know if you'd like this reversed. Also, I noticed on your updated merge queue that you'd like to combine #2549 as well. I'm happy to do this. Please let me know if you'd like me to do that.

[FernandoAndrade-NOAA]

@jkbk2004 I combined #2544 into this one because neither changes baselines and I thought it would lighten the merge queue load a bit. Please let me know if you'd like this reversed. Also, I noticed on your updated merge queue that you'd like to combine #2549 as well. I'm happy to do this. Please let me know if you'd like me to do that.

Hi @grantfirl, yes we'd like to bring #2549 into this PR as well , thank you!

[grantfirl]

@jkbk2004 @FernandoAndrade-NOAA OK, the PRs have been combined and should be ready to test. They have each been tested independently and one can still look at the RT logs from the original PRs if necessary. The RT log left in PR branch right now is only valid for the pre-test that I did for the original, un-combined PR, although it will get re-written with the final tests before merge.

[BrianCurtis-NOAA]

@jkbk2004 @FernandoAndrade-NOAA Are the WW3 inputs synced to Hera?

[FernandoAndrade-NOAA]

@jkbk2004 @FernandoAndrade-NOAA Are the WW3 inputs synced to Hera?

They should be available on Hera at
/scratch2/NAGAPE/epic/UFS-WM_RT/NEMSfv3gfs/input-data-20240501/WW3_input_data_20240111

[BrianCurtis-NOAA]

@FernandoAndrade-NOAA can you make the changes mentioned in NOAA-EMC/WW3#1342 (comment) to this PR?

[BrianCurtis-NOAA]

@jkbk2004 and @FernandoAndrade-NOAA you'll need to change the date on the hera input-data-20240501 directory for the WW3_input_data as well. This is still showing as 20240111

[jkbk2004]

@BrianCurtis-NOAA do you mean to update the date of /scratch2/NAGAPE/epic/UFS-WM_RT/NEMSfv3gfs/input-data-20240501 ? In that case, we need to keep input-data-20240501 and create new input directory to add WW3_input_data_20240111. There are a few WW3_input_data_* already in input-data-20240501: we are using WW3_input_data_20240214. Adding WW3_input_data_20240111 under input-data-20240501 looks ok. @JessicaMeixner-NOAA any comment?

[JessicaMeixner-NOAA]

/scratch2/NAGAPE/epic/UFS-WM_RT/NEMSfv3gfs/input-data-20240501

@BrianCurtis-NOAA - good catch about the extra files. However, the createmoddefs directory should be kept, the workdir should be removed.

[BrianCurtis-NOAA]

/scratch2/NAGAPE/epic/UFS-WM_RT/NEMSfv3gfs/input-data-20240501

@BrianCurtis-NOAA - good catch about the extra files. However, the createmoddefs directory should be kept, the workdir should be removed.

Thanks for the clarification!

[BrianCurtis-NOAA]

@JessicaMeixner-NOAA in createmoddefs, do we need the work or worktemp directories?

[JessicaMeixner-NOAA]

@JessicaMeixner-NOAA in createmoddefs, do we need the work or worktemp directories?

@BrianCurtis-NOAA - thanks again for finding this, no we do not need either of those directories.

[FernandoAndrade-NOAA]

Testing is complete, I'll leave a note in subcomponents to continue with merging.

[DeniseWorthen]

If the permissions are not fixed for the new WW3 input data on Hercules (#2560), as soon as the first person tries to run against the new UWM hash, their RT will fail.