Merge pull request #10 from EPCCed/data-generation

Data generation

docs/data-generation.md

@@ -30,18 +30,24 @@ Following the structure given in the [general data generation](ML_training.md) c
     cmake --build build --target hasegawa-wakatani
     ```
 
-    Before simulating the training data, a burn-in run must be conducted at the desired resolution. For an example of this, see [fine_init.sh](files/data-generation/fine_init.sh). Edit `<account>` on line 9 and `x01` in lines containing paths to match your `$WORK` and desired `/scratch` locations and submit via `sbatch fine_init.sh`.
+    Before simulating the training data, a burn-in run must be conducted at the desired resolution. For an example of this, see [fine_init.sh](../files/data-generation/fine_init.sh). Edit `<account>` on line 9 and `x01` in lines containing paths to match your `$WORK` and desired `/scratch` locations and submit via `sbatch fine_init.sh`.
 
-    Following that, we run a number of sequentially trajectories to generate fine-grained ground-truth data. See [fine_trajectories.sh](files/data-generation/fine_trajectories.sh)
+    Following that, we run a number of sequentially trajectories to generate fine-grained ground-truth data. See [fine_trajectories.sh](../files/data-generation/fine_trajectories.sh)
 
     The initial simulation produces "restart files", `/scratch/space1/x01/data/my-scratch-data/initial/data/BOUT.restart.*.nc` from which a simulation can be continued. Those, as well as the input file (`/scratch/space1/x01/data/my-scratch-data/initial/data/BOUT.inp` should be placed in `/scratch/space1/x01/data/my-scratch-data/0`.
 
     Edit line 17 (`for TRAJ_INDEX in {1..10}`) to give the desired number of trajectories. Edit `<account>` on line 9 and `x01` in lines containing paths for your project and submit via `sbatch fine_trajectories.sh`.
 
 3. Coarsen selected simulation snapshots.
 
-    Fine-grained data must be coarsened to match the desired coarse-grained resolution. This can be done via interpolation for a general solution. Files in [files/coarsening](files/coarsening) perform this task. Submit `submit-resize.sh` via `sbatch submit-resize.sh`.
+    Fine-grained data must be coarsened to match the desired coarse-grained resolution. This can be done via interpolation for a general solution. Files in [files/coarsening](../files/coarsening) perform this task. Submit `submit-resize.sh` via `sbatch submit-resize.sh`.
+
+    _Note: this operates on one trajectory at a time and will therefore need to be repeated for each trajectory run in step 2.
 
 4. Single-timestep coarse simulations.
 
-
+    With the previous step having extracted fine-grained data for each time step (and each trajectory for which it was repeated), we now need to run a single-timestep coarse-grained simulation. To do this, see [files/coarse_simulations](../files/coarse_simulations/). Submitting [run_coarse_sims.sh](../files/coarse_simulations/run_coarse_sims.sh) will run a single step simulation for each coarsened timestep created in the previous step.
+
+Subsequent steps: calculating the error; reformatting data for ingestion into TensorFlow; and model training are covered in [ML model training implementation](training_implementation.md).
+
+

docs/workflow.md

@@ -5,8 +5,7 @@ The system needs to have all the tools and packages (in suitable versions) insta
 The example workflow described here does not require a pre-trained ML model, we are using a placeholder model that alwyas returns zeroes to showcase the framework, and the script is provided here. Obviously, any other model can be exported in the desired format and used in the workflow.
 
 [< Back](./)
-
-## Export the ML model
+<!-- ## Export the ML model
 
 Activate the conda environment with SmartSim (see Cirrus example to make sure it has all relevant packages)
 ```
@@ -29,7 +28,7 @@ You can now test the zero model:
 ```
 python zero_model_test.py
 ```
-This script launches a database and uploads the zero model. It generates a random tensor and uses it as input for the model inference, which should return a tensor of the same dimensions filled with zero. The output gets printed on screen so that one can easily verify the content of the returned tensor.
+This script launches a database and uploads the zero model. It generates a random tensor and uses it as input for the model inference, which should return a tensor of the same dimensions filled with zero. The output gets printed on screen so that one can easily verify the content of the returned tensor. -->
 
 ## Compile Hasegawa Wakatani with SmartRedis
 

files/coarse_simulations/coarse_BOUT.inp

@@ -0,0 +1,53 @@
+#
+# Hasegawa-Wakatani test case
+#
+
+timestep = 0.026   # Output timestep
+nout = 1        # Number of output steps
+restart = true
+append = false
+
+MYG = 0  # No need for Y communications
+
+periodicX = true  # Domain is periodic in X
+
+[mesh]
+
+nx = 260  # Note 4 guard cells in X
+ny = 1
+nz = 256  # Periodic, so no guard cells in Z
+
+dx = 0.1
+dy = 1.0
+dz = 0.1
+
+[hw]
+
+alpha = 1.0    # Adiabaticity (~ conductivity)
+kappa = 2.0    # Density gradient drive
+Dvort = 0.005   # Vorticity diffusion
+Dn = 0.005   # Density diffusion
+
+modified = true
+
+bracket = 2  # 0 = std, 1 = simple, 2 = arakawa
+
+[solver]
+type = rk4
+adaptive = false
+timestep = 0.026
+
+[all]
+#scale = 0.0
+bndry_all = dirichlet_o2
+
+[n]
+#bndry_all = neumann_o2
+
+[vort]
+#bndry_all = dirichlet_o2
+
+#scale = 0.1  # Fluctuation amplitude
+#function = mixmode(2*pi*x) * mixmode(z) # Fluctuation function
+
+

files/coarse_simulations/run_coarse_sims.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+#SBATCH --nodes=1
+#SBATCH --ntasks=1
+# #SBATCH --exclusive
+#SBATCH --time=08:00:00
+#SBATCH --partition=standard
+#SBATCH --qos=standard
+#SBATCH --account=<account>
+
+eval "$(/work/x01/x01/$USER/miniconda3/bin/conda shell.bash hook)"
+conda activate boutsmartsim
+
+module load intel-20.4/mpi
+module load intel-20.4/compilers
+module load fftw/3.3.10-intel20.4-impi20.4
+module load netcdf-parallel/4.9.2-intel20-impi20
+
+ID_TRAJ=1
+BASE_PATH=/scratch/space1/x01/data/my-scratch-data
+
+executable=/work/x01/x01/$USER/my-hw/build/hasegawa-wakatani
+
+for i in {0..1000}
+do
+  coarse_TRAJ_PATH=${BASE_PATH}/${i}/coarse
+  coarse_TRAJ_SIM_PATH=${BASE_PATH}/${i}/coarse_sim
+  mkdir -p $coarse_TRAJ_SIM_PATH
+  cp ${coarse_TRAJ_PATH}/BOUT.restart.* $coarse_TRAJ_SIM_PATH
+  cp coarse_BOUT.inp $coarse_TRAJ_SIM_PATH/BOUT.inp
+
+  cd $coarse_TRAJ_SIM_PATH
+
+  srun --nodes=1 --ntasks=1 --job-name=ss_job_0 --distribution=block:block \
+    $executable -d . \
+    restart=true append=false \
+    solver:type=rk4 solver:adaptive=false solver:timestep=0.026 \
+    nout=1 timestep=0.026 mesh:nx=260 mesh:nz=256 mesh:dx=0.1 mesh:dz=0.1 -d .
+done