Skip to content

Commit

Permalink
Merge pull request #175 from gold2718/UpdateNoresmBranch
Browse files Browse the repository at this point in the history 
Summary: Update CAM with ESCOMP changes between cam6_3_158 and cam6_4_041
Contributors: gold2718
Reviewers: DirkOlivie, mvertens, oyvindseland
Purpose of changes: Update CAM with new NCAR configurations (#173)
Github PR URL: #175
Changes made to build system: Replace manage_externals with git-fleximod
Changes made to the namelist:
- CAM3 entries removed
- New entries for gravity wave moving mountain parameterization
- GEOSchem updates
- RRTMGP added as a radiation scheme
- New SCAM variables
- Wetdep variables made into independent namelist
- New (temporary?) dust emission variables to coordinate with CTSM

Changes to the defaults for the boundary datasets: None (for LT)
Substantial timing or memory changes: None
Change highlights:
- Adjustments to chemistry compset names
- Bring in re-organized atmospheric_physics repo
- Update to GEOS-Chem 14.4.3 and HEMCO 3.9.0, and add Cloud-J 7.7.3 and HETP 1.0
- Generalize aerosol wet removal
- Add missing total energy in physics state from dycore in snapshots and cleanup total water
- Use same cloud water for radiation and COSP.
- fix psl values sent to coupler in cam7
- Updates to age of air diagnostic tracers
- Add capability to use Leung dust emission scheme
- Repartition dust deposition fluxes passed to surface models
- Deposition fixes for aquaplanet and simple model configurations.
- SCAM-SE feature addition plus bugfixes and some refactoring
- Remove 0.5*timestep from call to ZM
- Correction to aerosol convective removal
- New CLUBB external
- Add climate-chemistry compset
- Initial Gravity Wave moving mountain
- Replaced outdated log-gamma function with intrinsic
- Fix clubb interface bug (dry/moist mixing ratio conversion)
- Limit vertical domain used by COSP.
- Activate additional clubb diffusion in cam6
- Change name of physics package 'cam_dev' to 'cam7'
- Replace manage_externals with git-fleximod
- Update subcomponents where ESCOMP component is used directly in CAM
- Update top level externals to match NorESMhub/NorESM/noresm_develop
- Diagnostic rainbows
- New PUMAS external with fixed GPU directives
  • Loading branch information
@gold2718
gold2718 authored Nov 12, 2024
2 parents 9baa370 + 9bc40e1 commit a2c1885
Show file tree
Hide file tree
Showing 534 changed files with 45,359 additions and 38,116 deletions.
254 changes: 31 additions & 223 deletions .github/scripts/branch_pr_issue_closer.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,6 @@

import re
import sys
import subprocess
import shlex
import argparse

from github import Github
Expand All @@ -31,42 +29,6 @@
#HELPER FUNCTIONS
#################

#+++++++++++++++++++++++++++++++++++++++++
#Curl command needed to move project cards
#+++++++++++++++++++++++++++++++++++++++++

def project_card_move(oa_token, column_id, card_id):

"""
Currently pyGithub doesn't contain the methods required
to move project cards from one column to another, so
the unix curl command must be called directly, which is
what this function does.
The specific command-line call made is:
curl -H "Authorization: token OA_token" -H \
"Accept: application/vnd.github.inertia-preview+json" \
-X POST -d '{"position":"top", "column_id":<column_id>}' \
https://api.github.com/projects/columns/cards/<card_id>/moves
"""

#create required argument strings from inputs:
github_oa_header = ''' "Authorization: token {0}" '''.format(oa_token)
github_url_str = '''https://api.github.com/projects/columns/cards/{0}/moves'''.format(card_id)
json_post_inputs = ''' '{{"position":"top", "column_id":{}}}' '''.format(column_id)

#Create curl command line string:
curl_cmdline = '''curl -H '''+github_oa_header+''' -H "Accept: application/vnd.github.inertia-preview+json" -X POST -d '''+\
json_post_inputs+''' '''+github_url_str

#Split command line string into argument list:
curl_arg_list = shlex.split(curl_cmdline)

#Run command using subprocess:
subprocess.run(curl_arg_list, check=True)

#++++++++++++++++++++++++++++++
#Input Argument parser function
#++++++++++++++++++++++++++++++
Expand Down Expand Up @@ -101,7 +63,7 @@ def end_script(msg):
"""
Prints message to screen, and then exits script.
"""
print("\n{}\n".format(msg))
print(f"\n{msg}\n")
print("Issue closing check has completed successfully.")
sys.exit(0)

Expand Down Expand Up @@ -137,11 +99,10 @@ def _main_prog():

ghub = Github(token)

#++++++++++++++++++++
#+++++++++++++++++++++
#Open ESCOMP/CAM repo
#++++++++++++++++++++
#+++++++++++++++++++++

#Official CAM repo:
cam_repo = ghub.get_repo("ESCOMP/CAM")

#+++++++++++++++++++++++++++++
Expand All @@ -162,6 +123,9 @@ def _main_prog():
#Search for merge text, starting at beginning of message:
commit_msg_match = pr_merge_pattern.match(commit_message)

#Initialize variables:
pr_num = 0

#Check if match exists:
if commit_msg_match is not None:
#If it does then pull out text immediately after message:
Expand All @@ -174,7 +138,7 @@ def _main_prog():
first_word = post_msg_word_list[0]

#Print merged pr number to screen:
print("Merged PR: {}".format(first_word))
print(f"Merged PR: {first_word}")

try:
#Try assuming the word is just a number:
Expand Down Expand Up @@ -251,27 +215,28 @@ def _main_prog():
pr_msg_lower = merged_pull.body.lower()

#search for at least one keyword:
word_matches = []
if keyword_pattern.search(pr_msg_lower) is not None:
#If at least one keyword is found, then determine location of every keyword instance:
word_matches = keyword_pattern.finditer(pr_msg_lower)
else:
endmsg = "Pull request was merged without using any of the keywords. Thus there are no issues to close."
end_script(endmsg)

#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#Extract issue and PR numbers associated with found keywords in merged PR message
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#Extract issue and PR numbers associated with found keywords in merged PR message
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

#create issue pattern ("the number symbol {#} + a number"),
#which ends with either a space, a comma, a period, or
#the end of the string itself:
issue_pattern = re.compile(r'#[0-9]+(\s|,|$)|.')

#Create new "close" issues list:
close_issues = list()
close_issues = []

#Create new "closed" PR list:
close_pulls = list()
close_pulls = []

#Search text right after keywords for possible issue numbers:
for match in word_matches:
Expand Down Expand Up @@ -299,13 +264,13 @@ def _main_prog():
#so set the issue number to one that will never be found:
issue_num = -9999

#Check that number is actually for an issue (as opposed to a PR):
if issue_num in open_issues:
#Add issue number to "close issues" list:
close_issues.append(issue_num)
elif issue_num in open_pulls:
#If in fact a PR, then add to PR list:
#Check if number is actually for a PR (as opposed to an issue):
if issue_num in open_pulls:
#Add PR number to "close pulls" list:
close_pulls.append(issue_num)
elif issue_num in open_issues:
#If in fact an issue, then add to "close issues" list:
close_issues.append(issue_num)

#If no issue numbers are present after any of the keywords, then exit script:
if not close_issues and not close_pulls:
Expand All @@ -322,183 +287,26 @@ def _main_prog():
print("PRs referenced by the merged PR: "+", ".join(\
str(pull) for pull in close_pulls))

#+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#Determine name of project associated with merged Pull Request
#+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

#Pull-out all projects from repo:
projects = cam_repo.get_projects()

#Initalize modified project name:
proj_mod_name = None

#Loop over all repo projects:
for project in projects:
#Pull-out columns from each project:
proj_columns = project.get_columns()

#Loop over columns:
for column in proj_columns:

#check if column name is "Completed Tags"
if column.name == "Completed tags":
#If so, then extract cards:
cards = column.get_cards()

#Loop over cards:
for card in cards:
#Extract card content:
card_content = card.get_content()

#Next, check if card number exists and matches merged PR number:
if card_content is not None and card_content.number == pr_num:
#If so, and if Project name is None, then set string:
if proj_mod_name is None:
proj_mod_name = project.name
#Break out of card loop:
break

#If already set, then somehow merged PR is in two different projects,
#which is not what this script is expecting, so just exit:
endmsg = "Merged Pull Request found in two different projects, so script will do nothing."
end_script(endmsg)

#Print project name associated with merged PR:
print("merged PR project name: {}".format(proj_mod_name))

#++++++++++++++++++++++++++++++++++++++++
#Extract repo project "To do" card issues
#++++++++++++++++++++++++++++++++++++++++

#Initalize issue counting dictionary:
proj_issues_count = dict()

#Initalize issue id to project card id dictionary:
proj_issue_card_ids = dict()

#Initialize list for issues that have already been closed:
already_closed_issues = list()

#Loop over all repo projects:
for project in projects:

#Next, pull-out columns from each project:
proj_columns = project.get_columns()

#Loop over columns:
for column in proj_columns:
#Check if column name is "To do"
if column.name == "To do":
#If so, then extract cards:
cards = column.get_cards()

#Loop over cards:
for card in cards:
#Extract card content:
card_content = card.get_content()

#Next, check if card issue number matches any of the "close" issue numbers from the PR:
if card_content is not None and card_content.number in close_issues:

#If so, then check if issue number is already in proj_issues_count:
if card_content.number in proj_issues_count:
#Add one to project issue counter:
proj_issues_count[card_content.number] += 1

#Also add issue id and card id to id dictionary used for card move, if in relevant project:
if project.name == proj_mod_name:
proj_issue_card_ids[card_content.number] = card.id

else:
#If not, then append to project issues count dictionary:
proj_issues_count[card_content.number] = 1

#Also add issue id and card id to id dictionary used for card move, if in relevant project:
if project.name == proj_mod_name:
proj_issue_card_ids[card_content.number] = card.id

#Otherwise, check if column name matches "closed issues" column:
elif column.name == "closed issues" and project.name == proj_mod_name:
#Save column id:
column_target_id = column.id

#Extract cards:
closed_cards = column.get_cards()

#Loop over cards:
for closed_card in closed_cards:
#Extract card content:
closed_card_content = closed_card.get_content()

#Check if card issue number matches any of the "close" issue numbers from the PR:
if closed_card_content is not None and closed_card_content.number in close_issues:
#If issue number matches, then it likely means the same
#commit message or issue number reference was used in multiple
#pushes to the same repo (e.g., for a PR and then a tag). Thus
#the issue should be marked as "already closed":
already_closed_issues.append(closed_card_content.number)

#Remove all issues from issue dictionary that are "already closed":
for already_closed_issue_num in already_closed_issues:
if already_closed_issue_num in proj_issues_count:
proj_issues_count.pop(already_closed_issue_num)

#If no project cards are found that match the issue, then exit script:
if not proj_issues_count:
endmsg = "No project cards match the issue being closed, so the script will do nothing."
end_script(endmsg)
#++++++++++++++++++++++++++++++++++++++++++++++
#Attempt to close all referenced issues and PRs
#++++++++++++++++++++++++++++++++++++++++++++++

#+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#Check if the number of "To-do" project cards matches the total number
#of merged PRs for each 'close' issue.
#
#Then, close all issues for which project cards equals merged PRs
#
#If not, then simply move the project card to the relevant project's
#"closed issues" column.
#+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

#Loop over project issues and counts that have been "closed" by merged PR:
for issue_num, issue_count in proj_issues_count.items():

#If issue count is just one, then close issue:
if issue_count == 1:
#Extract github issue object:
cam_issue = cam_repo.get_issue(number=issue_num)
#Close issue:
cam_issue.edit(state='closed')
print("Issue #{} has been closed.".format(issue_num))
else:
#Extract card id from id dictionary:
if issue_num in proj_issue_card_ids:
card_id = proj_issue_card_ids[issue_num]
else:
#If issue isn't in dictionary, then it means the issue
#number was never found in the "To do" column, which
#likely means the user either referenced the wrong
#issue number, or the issue was never assigned to the
#project. Warn user and then exit with a non-zero
#error so that the Action fails:
endmsg = 'Issue #{} was not found in the "To Do" Column of the "{}" project.\n' \
'Either the wrong issue number was referenced, or the issue was never ' \
'attached to the project.'.format(issue_num, proj_mod_name)
print(endmsg)
sys.exit(1)

#Then move the card on the relevant project page to the "closed issues" column:
project_card_move(token.strip(), column_target_id, card_id)

#++++++++++++++++++++++++++++++++++++++++++++++++++++++
#Finally, close all Pull Requests in "close_pulls" list:
#++++++++++++++++++++++++++++++++++++++++++++++++++++++
#Loop over referenced issues:
for issue_num in close_issues:
#Extract github issue object:
cam_issue = cam_repo.get_issue(number=issue_num)
#Close issue:
cam_issue.edit(state='closed')
print(f"Issue #{issue_num} has been closed.")

#Loop over referenced PRs:
for pull_num in close_pulls:
#Extract Pull request object:
cam_pull = cam_repo.get_pull(number=pull_num)

#Close Pull Request:
cam_pull.edit(state='closed')
print("Pull Request #{} has been closed.".format(pull_num))
print(f"Pull Request #{pull_num} has been closed.")

#++++++++++
#End script
Expand Down
21 changes: 21 additions & 0 deletions .github/workflows/fleximod_test.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
on: pull_request

jobs:
fleximod-test:
runs-on: ubuntu-latest
strategy:
matrix:
# oldest supported and latest supported
python-version: ["3.7", "3.x"]
steps:
- id: checkout-CESM
uses: actions/checkout@v4
- id: run-fleximod
run: |
$GITHUB_WORKSPACE/bin/git-fleximod update
$GITHUB_WORKSPACE/bin/git-fleximod test
# - name: Setup tmate session
# if: ${{ failure() }}
# uses: mxschmitt/action-tmate@v3


27 changes: 0 additions & 27 deletions .gitignore
View file
Original file line number Diff line number Diff line change
@@ -1,30 +1,3 @@
# Ignore externals
ccs_config
chem_proc
cime
components
manage_externals.log
src/chemistry/oslo_aero
src/physics/ali_arms/
src/physics/camnor_phys
src/physics/carma/base
src/physics/clubb
src/physics/cosp2/src
src/physics/silhs
src/chemistry/geoschem/geoschem_src
src/physics/pumas
src/physics/pumas-frozen
src/physics/rrtmgp/data/
src/physics/rrtmgp/ext/
src/dynamics/fv3/atmos_cubed_sphere
src/hemco
libraries/FMS
libraries/mct
libraries/parallelio
src/atmos_phys
src/dynamics/mpas/dycore
share

# Ignore compiled python
buildnmlc
buildcppc
Expand Down
Loading

0 comments on commit a2c1885

Please sign in to comment.