diff --git a/docs/research-software/mitgcm.md b/docs/research-software/mitgcm.md index ac8600e2f..f299d72bf 100644 --- a/docs/research-software/mitgcm.md +++ b/docs/research-software/mitgcm.md @@ -18,26 +18,18 @@ flow of both the atmosphere and ocean. MITgcm is not available via a module on ARCHER2 as users will build their own executables specific to the problem they are working on. -However, we do provide an optfile which will allow `genmake2` to create -Makefiles which will work on ARCHER2. - -!!! note - The processes to build MITgcm on the ARCHER2 4-cabinet system and full - system are slightly different. Please make sure you use the commands - for the correct system below. You can obtain the MITgcm source code from the developers by cloning from the GitHub repository with the command git clone https://github.com/MITgcm/MITgcm.git -You should then copy the ARCHER2 optfile into the MITgcm directories. You may use the files at the locations below for the 4-cabinet and full systems. +You should then copy the ARCHER2 optfile into the MITgcm directories. +!!! warning + A current ARCHER2 optfile is not available at the present time. Please + contact `support@archer2.ac.uk` for help. -```bash -cp /work/n02/shared/mjmn02/ECCOv4/cases/cce/cce1/scripts/dev_linux_amd64_cray_archer2 MITgcm/tools/build_options/ -``` - You should also set the following environment variables. `MITGCM_ROOTDIR` is used to locate the source code and should point to the top MITgcm directory. Optionally, adding the MITgcm tools directory @@ -63,7 +55,7 @@ running genmake2 -help -Finally, you may then build your executable by running +Finally, you may then build your executable by running make depend make @@ -87,7 +79,7 @@ each for up to one hour. #SBATCH --cpus-per-task=1 # Replace [budget code] below with your project code (e.g. t01) -#SBATCH --account=[budget code] +#SBATCH --account=[budget code] #SBATCH --partition=standard #SBATCH --qos=standard @@ -143,7 +135,7 @@ This can also sometimes lead to performance increases. #SBATCH --cpus-per-task=4 # Replace [budget code] below with your project code (e.g. t01) -#SBATCH --account=[budget code] +#SBATCH --account=[budget code] #SBATCH --partition=standard #SBATCH --qos=standard @@ -170,7 +162,7 @@ those requested in the job submission script. ## Reproducing the ECCO version 4 (release 4) state estimate on ARCHER2 -The ECCO version 4 state estimate (ECCOv4-r4) is an observationally-constrained numerical solution produced by the ECCO group at JPL. If you would like to reproduce the state estimate on ARCHER2 in order to create customised runs and experiments, follow the instructions below. They have been slightly modified from the JPL instructions for ARCHER2. +The ECCO version 4 state estimate (ECCOv4-r4) is an observationally-constrained numerical solution produced by the ECCO group at JPL. If you would like to reproduce the state estimate on ARCHER2 in order to create customised runs and experiments, follow the instructions below. They have been slightly modified from the JPL instructions for ARCHER2. For more information, see the ECCOv4-r4 website @@ -180,11 +172,11 @@ First, navigate to your directory on the ``/work`` filesystem in order to get ac mkdir MYECCO cd MYECCO - -In order to reproduce ECCOv4-r4, we need a specific checkpoint of the MITgcm source code. + +In order to reproduce ECCOv4-r4, we need a specific checkpoint of the MITgcm source code. git clone https://github.com/MITgcm/MITgcm.git -b checkpoint66g - + Next, get the ECCOv4-r4 specific code from GitHub: cd MITgcm @@ -193,7 +185,7 @@ Next, get the ECCOv4-r4 specific code from GitHub: git clone https://github.com/ECCO-GROUP/ECCO-v4-Configurations.git mv ECCO-v4-Configurations/ECCOv4\ Release\ 4/code . rm -rf ECCO-v4-Configurations - + ### Get the ECCOv4-r4 forcing files The surface forcing and other input files that are too large to be stored on GitHub are available via NASA data servers. In total, these files are about 200 GB in size. You must register for an Earthdata account and connect to a WebDAV server in order to access these files. For more detailed instructions, read the help page . @@ -205,9 +197,9 @@ Next, acquire your WebDAV credentials: (second Now, you can use wget to download the required forcing and input files: wget -r --no-parent --user YOURUSERNAME --ask-password https://ecco.jpl.nasa.gov/drive/files/Version4/Release4/input_forcing - wget -r --no-parent --user YOURUSERNAME --ask-password https://ecco.jpl.nasa.gov/drive/files/Version4/Release4/input_init + wget -r --no-parent --user YOURUSERNAME --ask-password https://ecco.jpl.nasa.gov/drive/files/Version4/Release4/input_init wget -r --no-parent --user YOURUSERNAME --ask-password https://ecco.jpl.nasa.gov/drive/files/Version4/Release4/input_ecco - + After using `wget`, you will notice that the `input*` directories are, by default, several levels deep in the directory structure. Use the `mv` command to move the `input*` directories to the directory where you executed the `wget` command. Specifically, ``` @@ -235,14 +227,14 @@ If you haven't already, set your environment variables: export MITGCM_ROOTDIR=../../../../MITgcm export PATH=$MITGCM_ROOTDIR/tools:$PATH export MITGCM_OPT=$MITGCM_ROOTDIR/tools/build_options/dev_linux_amd64_cray_archer2 - + Next, compile the executable: genmake2 -mods ../code -mpi -optfile $MITGCM_OPT make depend make - -Once you have compiled the model, you will have the mitgcmuv executable for ECCOv4-r4. + +Once you have compiled the model, you will have the mitgcmuv executable for ECCOv4-r4. #### Create run directory and link files @@ -250,7 +242,7 @@ In order to run the model, you need to create a run directory and link/copy the mkdir run cd run - + # link the data files ln -s ../input_init/NAMELIST/* . ln -s ../input_init/error_weight/ctrl_weight/* . @@ -261,11 +253,11 @@ In order to run the model, you need to create a run directory and link/copy the ln -s ../input_forcing/eccov4r4* . python mkdir_subdir_diags.py - + # manually copy the mitgcmuv executable cp -p ../build/mitgcmuv . -For a short test run, edit the ``nTimeSteps`` variable in the file ``data``. Comment out the default value and uncomment the line reading ``nTimeSteps=8``. This is a useful test to make sure that the model can at least start up. +For a short test run, edit the ``nTimeSteps`` variable in the file ``data``. Comment out the default value and uncomment the line reading ``nTimeSteps=8``. This is a useful test to make sure that the model can at least start up. To run on ARCHER2, submit a batch script to the Slurm scheduler. Here is an example submission script: @@ -280,7 +272,7 @@ To run on ARCHER2, submit a batch script to the Slurm scheduler. Here is an exam #SBATCH --cpus-per-task=1 # Replace [budget code] below with your project code (e.g. t01) -#SBATCH --account=[budget code] +#SBATCH --account=[budget code] #SBATCH --partition=standard #SBATCH --qos=standard @@ -298,15 +290,15 @@ export SRUN_CPUS_PER_TASK=$SLURM_CPUS_PER_TASK srun --distribution=block:block --hint=nomultithread ./mitgcmuv ``` -This configuration uses 96 MPI processes at 12 MPI processes per node. Once the run has finished, in order to check that the run has successfully completed, check the end of one of the standard output files. +This configuration uses 96 MPI processes at 12 MPI processes per node. Once the run has finished, in order to check that the run has successfully completed, check the end of one of the standard output files. tail STDOUT.0000 - -It should read + +It should read PROGRAM MAIN: Execution ended Normally - -The files named `STDOUT.*` contain diagnostic information that you can use to check your results. As a first pass, check the printed statistics for any clear signs of trouble (e.g. NaN values, extremely large values). + +The files named `STDOUT.*` contain diagnostic information that you can use to check your results. As a first pass, check the printed statistics for any clear signs of trouble (e.g. NaN values, extremely large values). #### ECCOv4-r4 in adjoint mode @@ -318,19 +310,19 @@ If you have access to the commercial TAF software produced by