Add v1 data #46
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -16,6 +16,7 @@ simulations. | |
| :caption: Contents: | ||
|
|
||
| v2/index | ||
| v1/index | ||
|
|
||
|
|
||
| .. Indices and tables | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| E3SMv1 (Water Cycle) | ||
| ==================== | ||
|
|
||
| The `E3SM version 1 water cycle simulation campaign <https://e3sm.org/research/water-cycle/v1-water-cycle/>`_ includes standard set of | ||
| Coupled Model Intercomparison Project Phase 6 (`CMIP6 <https://www.geosci-model-dev.net/9/1937/2016/>`_) | ||
| Diagnosis, Evaluation, and Characterization of Klima (DECK) simulations. | ||
| These E3SM simulations at standard resolution have been completed and the data is publicly available. | ||
|
|
||
| Data Resolution: | ||
|
|
||
| The standard resolution configuration of E3SM v1.0 water cycle configuration has approximate horizontal resolution of | ||
| 1 deg latitude by 1 deg longitude in atmosphere (110 km grid spacing), | ||
| with ocean and sea ice grid of 60 km in the mid-latitudes and 30 km at the equator and poles, | ||
| and river transport at 55 km horizontal resolution. | ||
| This model configuration is described in | ||
| `“v1 1 deg CMIP” <https://e3sm.org/model/scientifically-validated-configurations/v1-configurations/v1-1-deg-cmip6/?preview=true>`_ page | ||
| in `Scientifically Validated Configurations <https://e3sm.org/model/scientifically-validated-configurations/>`_. | ||
|
|
||
| Reference Paper: | ||
|
|
||
| For more details, | ||
| refer to `Coupled E3SM v1 Model Overview <https://e3sm.org/?p=5470>`_ or | ||
| directly to the scientific paper (`doi:10.1029/2018MS001603 <https://doi.org/10.1029/2018MS001603>`_), | ||
| which documents the E3SM model version 1, | ||
| its almost 3000 years of DECK simulations and discusses the model’s performance. | ||
|
|
||
| Experiments: | ||
|
|
||
| The datasets include the following experiments: | ||
|
|
||
| * piControl – Pre-industrial control (piControl) simulation (500 years) | ||
| * historical – Historical simulations 1850-2014 (165 years) 5 ensembles | ||
| * 1pctCO2 – Prescribed 1% / year CO2 increase (1pctCO2) simulation (150 years) | ||
| * abrupt-4xCO2 – Abrupt CO2 quadrupling (abrupt-4xCO2) simulation (150 years) | ||
| * abrupt-4xCO2-ext300yrs – Abrupt CO2 quadrupling (abrupt-4xCO2) simulation (140-300 years) | ||
| * amip – atmosphere only AMIP simulation 1870-2014 (145 years) 3 ensembles | ||
| * amip_1850_allF – atmosphere only AMIP with all forcings held at 1850 values, 1870-2014 (145 years) 3 ensembles | ||
| * amip_1850_aeroF – atmosphere only AMIP with all aerosol forcings held at 1850 values, 1870-2014 (145 years) 3 ensembles | ||
|
|
||
| .. toctree:: | ||
| :maxdepth: 2 | ||
| :caption: Contents: | ||
|
|
||
| simulation_data/index | ||
| reproducing_simulations/index |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| *********************** | ||
| Reproducing Simulations | ||
| *********************** | ||
|
|
||
| .. toctree:: | ||
| :maxdepth: 2 | ||
| :caption: Contents: | ||
|
|
||
| reproduction_table |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| *************** | ||
| Simulation Data | ||
| *************** | ||
|
|
||
| .. toctree:: | ||
| :maxdepth: 2 | ||
| :caption: Contents: | ||
|
|
||
| simulation_table |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| v1 | ||
| ========================================== | ||
|
|
||
| .. toctree:: | ||
| :maxdepth: 2 | ||
| :caption: Contents: | ||
|
|
||
| WaterCycle/index |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -8,10 +8,6 @@ $ cd utils | |
| $ ./generate_html.bash | ||
| ``` | ||
|
|
||
| Creating reproduction scripts | ||
| ============================= | ||
|
|
||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,10 @@ | ||
| #python generate_tables.py | ||
| cd ../docs/ && make html | ||
| rm -rf /global/cfs/cdirs/e3sm/www/$USER/data_docs | ||
| mv _build /global/cfs/cdirs/e3sm/www/$USER/data_docs | ||
| chmod -R o+rx /global/cfs/cdirs/e3sm/www/$USER/data_docs | ||
|
|
||
| echo "https://portal.nersc.gov/project/e3sm/$USER/data_docs/html/v2/WaterCycle/simulation_data/simulation_table.html" | ||
| echo "https://portal.nersc.gov/project/e3sm/$USER/data_docs/html/v2/WaterCycle/reproducing_simulations/reproduction_table.html" | ||
| echo "https://portal.nersc.gov/project/e3sm/$USER/data_docs/html/v1/WaterCycle/simulation_data/simulation_table.html" | ||
| echo "https://portal.nersc.gov/project/e3sm/$USER/data_docs/html/v1/WaterCycle/reproducing_simulations/reproduction_table.html" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -33,20 +33,43 @@ def get_data_size_and_hpss(hpss_path: str) -> Tuple[str, str]: | |
| hpss = "" | ||
| return (data_size, hpss) | ||
|
|
||
| def get_esgf(source_id: str, model_version: str, experiment: str, ensemble_num: str, link_type: str) -> str: | ||
| if experiment and ensemble_num: | ||
| # See https://github.com/E3SM-Project/CMIP6-Metadata/pull/9#issuecomment-1246086256 for the table of ensemble numbers | ||
| # Note that `[1:]`` removes `v` from `model_version` | ||
| # | ||
| # Query parameters: | ||
| # v2: | ||
| # Native (model_version, experiment, ensemble_member) | ||
| # CMIP6 (source_id, experiment_id, variant_label) | ||
| # v1 on https://e3sm.org/data/get-e3sm-data/released-e3sm-data/v1-1-deg-data-cmip6/: | ||
| # Native (experiment, and optionally model_version) | ||
| # CMIP6 (activity_id, source_id, experiment_id) | ||
|
|
||
| # TODO: figure out if we're returning the correct things | ||
|
There was a problem hiding this comment. I've listed the numbers of items returned in the comments below, for e3sm.org vs the current implementation in this PR. As the table below reveals, the number of returned items don't always match up as expected. |
||
| # Comparison to links on https://portal.nersc.gov/project/e3sm/forsyth/data_docs/html/v1/WaterCycle/simulation_data/simulation_table.html | ||
| # simulation | e3sm.org CMIP6 | e3sm.org Native | data_docs CMIP6 | data_docs Native | | ||
| # piControl | 346 | 67 | 346 | 15 | | ||
| # historical | 3414 | 190 | 1420 | 70 | 5 rows on data_docs | ||
| # 1pctCO2 | 339 | 38 | 339 | 14 | | ||
| # abrupt-4xCO2 | 293 | 50 | 0 | 14 | 2 rows on data_docs | ||
| # abrupt-4xCO2-ext300yrs | x | 14 | | ||
| # amip | 527 | 30 | 527 | 30 | 3 rows on data_docs | ||
| # amip_1850_allF | x | 30 | | ||
| # amip_1850_aeroF | x | 30 | | ||
| # TODO: only return Native for some experiments... | ||
|
There was a problem hiding this comment. https://e3sm.org/data/get-e3sm-data/released-e3sm-data/v1-1-deg-data-cmip6/ shows CMIP6 and Native for piControl, historical, 1pctCO2, abrupt-4xCO2, amip, but only Native for abrupt-4xCO2-ext300yrs, amip_1850_allF, amip_1850_aeroF. |
||
| esgf_native = f"`Native <https://esgf-node.llnl.gov/search/e3sm/?model_version={model_version[1:]}_0&experiment={experiment}&ensemble_member=ens{ensemble_num}>`_" | ||
| if experiment == 'hist-all-xGHG-xaer': | ||
| experiment_id = 'hist-nat' | ||
| else: | ||
| experiment_id = experiment | ||
| esgf_cmip = f"`CMIP <https://esgf-node.llnl.gov/search/cmip6/?source_id={source_id}&experiment_id={experiment_id}&variant_label=r{ensemble_num}i1p1f1>`_" | ||
| if link_type == "cmip": | ||
| esgf = esgf_cmip | ||
| elif link_type == "native": | ||
| esgf = esgf_native | ||
| else: | ||
| esgf = esgf_cmip + ', ' + esgf_native | ||
| else: | ||
| esgf = "" | ||
| return esgf | ||
|
|
@@ -82,16 +105,19 @@ def __init__(self, simulation_dict): | |
| self.experiment = simulation_dict["experiment"] | ||
|
|
||
| self.ensemble_num = simulation_dict["ensemble_num"] | ||
| self.link_type = simulation_dict["link_type"] | ||
|
|
||
| if "hpss_path" in simulation_dict: | ||
| hpss_path = simulation_dict["hpss_path"] | ||
|
There was a problem hiding this comment. If the HPSS path doesn't match our expected form, we're going to just have to get it directly from the csv. |
||
| else: | ||
| hpss_path = f"/home/projects/e3sm/www/{self.group}/E3SM{self.model_version}/{self.resolution}/{self.simulation_name}" | ||
| self.data_size, self.hpss = get_data_size_and_hpss(hpss_path) | ||
|
|
||
| if self.resolution == "NARRM": | ||
| source_id = f"E3SM-{self.model_version[1:]}-0-{self.resolution}" | ||
| else: | ||
| source_id = f"E3SM-{self.model_version[1:]}-0" | ||
| self.esgf = get_esgf(source_id, self.model_version, self.experiment, self.ensemble_num, self.link_type) | ||
|
|
||
| self.run_script_original = get_run_script_original(self.model_version, self.simulation_name) | ||
| self.run_script_reproduction = get_run_script_reproduction(self.model_version, self.simulation_name) | ||
|
|
@@ -137,6 +163,45 @@ def append(self, group): | |
| self.groups.update([(group.name, group)]) | ||
|
|
||
| # Construct simulations ########################################### | ||
| def parse_hpss_path(simulation_dictionary): | ||
| hpss_path = simulation_dictionary["hpss_path"] | ||
| model_version = simulation_dictionary["model_version"] | ||
| group = simulation_dictionary["group"] | ||
| subdir = os.path.basename(hpss_path) | ||
| simulation_dictionary["simulation_name"] = subdir | ||
| if (model_version == "v1" ) and (group == "WaterCycle"): | ||
| # Example: 20191019.DECKv1b_P3_SSP5-8.5-GHG.ne30_oEC.cori-knl | ||
| match_object = re.search("[0-9]{8}\.([^_]*)_(.*)\..*\.(.*)", subdir) | ||
| if match_object: | ||
| category = match_object.group(1) | ||
| experiment = match_object.group(2) | ||
| machine = match_object.group(3) | ||
| if category.endswith("v1b"): | ||
| # Remove "v1b" | ||
| category = category[:-3] | ||
| if subdir == "20190722.DECKv1b_abrupt4xCO2.ne30_oEC3.compy": | ||
| experiment = "abrupt-4xCO2-ext300yrs" | ||
| if experiment.startswith("H"): | ||
| if experiment.endswith("hist-GHG"): | ||
| experiment = "damip_historical_hist_GHG" | ||
| else: | ||
| experiment = "historical" | ||
| elif experiment.startswith("A"): | ||
| if experiment.endswith("allF"): | ||
| experiment = "amip_1850_allF" | ||
| elif experiment.endswith("aeroF"): | ||
| experiment = "amip_1850_aeroF" | ||
| else: | ||
| experiment = "amip" | ||
| elif experiment.startswith("P"): | ||
| if experiment.endswith("GHG"): | ||
| experiment = "damip_ssp5_8_5_ghg" | ||
| else: | ||
| experiment = "projection" | ||
| simulation_dictionary["category"] = category | ||
| simulation_dictionary["experiment"] = experiment | ||
| simulation_dictionary["machine"] = machine | ||
|
|
||
| def read_simulations(csv_file): | ||
| # model_version > group > resolution > category > simulation_name, | ||
| versions: OrderedDict[str: ModelVersion] = OrderedDict() | ||
|
|
@@ -153,6 +218,10 @@ def read_simulations(csv_file): | |
| for i in range(len(header)): | ||
| label = header[i] | ||
| simulation_dict[label] = row[i].strip() | ||
| if "hpss_path" in simulation_dict: | ||
| parse_hpss_path(simulation_dict) | ||
| if "cmip_only" in simulation_dict: | ||
| simulation_dict["link_type"] = "cmip" | ||
| model_version_name = simulation_dict["model_version"] | ||
| group_name = simulation_dict["group"] | ||
| resolution_name = simulation_dict["resolution"] | ||
|
|
@@ -227,7 +296,7 @@ def construct_pages(csv_file, model_version, group_name): | |
| resolutions, | ||
| ["Simulation", "Data Size (TB)", "ESGF Links", "HPSS Path"], | ||
| f"../docs/source/{model_version}/{group_name}/simulation_data/simulation_table.rst", | ||
| [65, 15, 400, 90] | ||
| ) | ||
| generate_table( | ||
| f"{model_version} {group_name} reproduction table", | ||
|
|
@@ -238,4 +307,5 @@ def construct_pages(csv_file, model_version, group_name): | |
| ) | ||
|
|
||
| if __name__ == "__main__": | ||
| #construct_pages("simulations.csv", "v2", "WaterCycle") | ||
| construct_pages("simulations_v1_water_cycle.csv", "v1", "WaterCycle") | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| model_version, group, resolution, hpss_path, checksum, ensemble_num, link_type, | ||
| v1, WaterCycle, LR, /home/g/golaz/2018/E3SM_simulations/repaired/20180129.DECKv1b_piControl.ne30_oEC.edison, , 1, , | ||
| v1, WaterCycle, LR, /home/g/golaz/2018/E3SM_simulations/repaired/20180215.DECKv1b_abrupt4xCO2.ne30_oEC.edison, , 1, , | ||
| v1, WaterCycle, LR, /home/g/golaz/2019/E3SM_simulations/20190722.DECKv1b_abrupt4xCO2.ne30_oEC3.compy, , 1, , | ||
| v1, WaterCycle, LR, /home/g/golaz/2018/E3SM_simulations/repaired/20180215.DECKv1b_1pctCO2.ne30_oEC.edison, , 1, , | ||
| v1, WaterCycle, LR, /home/g/golaz/2018/E3SM_simulations/repaired/20180215.DECKv1b_H1.ne30_oEC.edison, , 1, , | ||
| v1, WaterCycle, LR, /home/g/golaz/2018/E3SM_simulations/repaired/20180220.DECKv1b_H2.ne30_oEC.edison, , 2, , | ||
| v1, WaterCycle, LR, /home/z/zshaheen/2018/E3SM_simulations/repaired/20180302.DECKv1b_H3.ne30_oEC.edison, , 3, , | ||
| v1, WaterCycle, LR, /home/t/tang30/2018/E3SM_simulations/20180305.DECKv1b_H4.ne30_oEC.edison, , 4, , | ||
| v1, WaterCycle, LR, /home/g/golaz/2018/E3SM_simulations/20180307.DECKv1b_H5.ne30_oEC.edison, , 5, , | ||
| v1, WaterCycle, LR, /home/t/tang30/2018/E3SM_simulations/20180316.DECKv1b_A1.ne30_oEC.edison, , 1, , | ||
| v1, WaterCycle, LR, /home/t/tang30/2018/E3SM_simulations/20180622.DECKv1b_A2.ne30_oEC.edison, , 2, , | ||
| v1, WaterCycle, LR, /home/t/tang30/2018/E3SM_simulations/20180716.DECKv1b_A3.ne30_oEC.edison, , 3, , | ||
| v1, WaterCycle, LR, /home/t/tang30/2018/E3SM_simulations/20180508.DECKv1b_A1_1850allF.ne30_oEC.edison, , , , | ||
| v1, WaterCycle, LR, /home/t/tang30/2018/E3SM_simulations/20180622.DECKv1b_A2_1850allF.ne30_oEC.edison, , , , | ||
| v1, WaterCycle, LR, /home/t/tang30/2018/E3SM_simulations/20180716.DECKv1b_A3_1850allF.ne30_oEC.edison, , , , | ||
| v1, WaterCycle, LR, /home/t/tang30/2018/E3SM_simulations/20180508.DECKv1b_A1_1850aeroF.ne30_oEC.edison, , , , | ||
| v1, WaterCycle, LR, /home/t/tang30/2018/E3SM_simulations/20180622.DECKv1b_A2_1850aeroF.ne30_oEC.edison, , , , | ||
| v1, WaterCycle, LR, /home/t/tang30/2018/E3SM_simulations/20180716.DECKv1b_A3_1850aeroF.ne30_oEC.edison, , , , | ||
| v1, WaterCycle, LR, /home/t/tang30/2019/E3SM_simulations/20190404.DECKv1b_H1_hist-GHG.ne30_oEC.edison, , , , | ||
| v1, WaterCycle, LR, /home/t/tang30/2019/E3SM_simulations/20190404.DECKv1b_H2_hist-GHG.ne30_oEC.edison, , , , | ||
| v1, WaterCycle, LR, /home/t/tang30/2019/E3SM_simulations/20190404.DECKv1b_H3_hist-GHG.ne30_oEC.edison, , , , | ||
| v1, WaterCycle, LR, /home/t/tang30/2019/E3SM_simulations/20190807.DECKv1b_P1_SSP5-8.5.ne30_oEC.cori-knl, , , , | ||
| v1, WaterCycle, LR, /home/t/tang30/2019/E3SM_simulations/20190821.DECKv1b_P2_SSP5-8.5.ne30_oEC.cori-knl, , , , | ||
| v1, WaterCycle, LR, /home/t/tang30/2019/E3SM_simulations/20190821.DECKv1b_P3_SSP5-8.5.ne30_oEC.cori-knl, , , , | ||
| v1, WaterCycle, LR, /home/t/tang30/2019/E3SM_simulations/20190821.DECKv1b_P4_SSP5-8.5.ne30_oEC.cori-knl, , , , | ||
| v1, WaterCycle, LR, /home/t/tang30/2019/E3SM_simulations/20190821.DECKv1b_P5_SSP5-8.5.ne30_oEC.cori-knl, , , , | ||
| v1, WaterCycle, LR, /home/t/tang30/2019/E3SM_simulations/20191019.DECKv1b_P1_SSP5-8.5-GHG.ne30_oEC.cori-knl, , , , | ||
| v1, WaterCycle, LR, /home/t/tang30/2019/E3SM_simulations/20191019.DECKv1b_P2_SSP5-8.5-GHG.ne30_oEC.cori-knl, , , , | ||
| v1, WaterCycle, LR, /home/t/tang30/2019/E3SM_simulations/20191019.DECKv1b_P3_SSP5-8.5-GHG.ne30_oEC.cori-knl, , , , |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The query parameters for the ESGF links differ between v1 and v2.