Trouble creating mapping files for surface datasets at ultra high resolution

[ekluzek]

MPAS Atmosphere have some ultra high resolution grid that we want to run at: 7.5 km and 3.75 km. I made an initial go at running mkmapdata.sh in ctsm5.1.dev069 on cheyenne for the 7.5km grid at...

/glade/scratch/duda/7.5km/x1.10485762.scrip.nc

For a 12-hour submission with the normal batch job (using the --hires option to mkmapdata.sh) I only successfully created the first two of the eight mapping files needed. This means I'll either need to do multiple long submissions to get all of the maps, or I need to increase the number of processors.

In the past I had trouble increasing the number of processors, but using eight processors seemed to be robust to make all the maps at all resolutions. So we've stuck at always using 8 over 4-nodes. Memory has been an issue and using 4 large memory nodes on cheyenne was required.

[ekluzek]

I also suspect I will NOT be able to run the 1km grid at this resolution. So we'll run with "--fast" and then use a global average value for STD_ELEV when running CTSM.

[ekluzek]

OK, with cheyenne back up, I was able to create the last 5 mapping files using 16 processors over 8 nodes. I didn't even try the 1km file. Some of those grids were higher resolution than the previous files and took about 8.4 hours of wallclock. I'm trying to make the surface dataset now, we'll see how that step goes.

[ekluzek]

I was able to make the surface dataset, although I needed to use large memory nodes when using the "--hires" option.

I was able to make the 1km file when I sent just it with 16 processors over 8 nodes. It completed in about 2 hours. So possibly we should up the number of processors we use for mkmapdata. However, when I tried using the 1km file for the 7.5km case it failed in trying to create the surface dataset since there were fractions on the 1km mapping file that were greater than 1.0 (something like 1.17).

[ekluzek]

PR #1663 takes care of this issue. So I'm closing as a wontfix.