Extreme slowdown in MET executables built with GNU compilers vs Intel

[mkavulich]

The UFS SRW App is using MET builds on Hera via spack-stack. On Hera specifically there are builds of both Intel and GNU software stacks, so some tests are run using GNU and others with Intel to ensure good coverage of different configurations. We recently noticed that, for MET 11.1 (it may affect other versions too, this is just the only version we have built right now), the Intel-compiled MET tools (specifically Intel 2021.5) are significantly faster than the GNU-compiled MET tools (specifically, GNU 9.2.0). I understand that due to optimization differences you would usually expect GNU to be a bit slower, but here we are seeing a nearly 2-times slowdown for most tools, and for GenEnsProd specifically it is 5-6 times slower! This seems like a way bigger performance hit than I would expect from just compiler differences.

I am curious if this behavior is known, or is perhaps some problem specific to our particular configuration. Just a brief look at the build logs (on Hera: /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/upp-addon-env/log.install.met) indicates pretty standard compile flags (no -O0 or anything like that), but I was not the person who built this stack and I have not dug into it deeply. It would be nice to find a solution because some of our fairly simple ensemble verification tests are taking hours to run with GNU, and we'd rather not just stop running these tests. It would also be great if there were a GNU build by the MET team we could test out to see if we observe the same behavior; I know there are shared builds available in /contrib/met/ but as far as I can tell those are all built with Intel.

Let me know if there's any more details I should provide. Thanks!

[mkavulich]

Just to follow-up with more information: I was also able to test a version of MET 11.1.1 built with GNU 13.3 compilers, and observed nearly identical slowdowns as were observed with GNU 9.2, so I don't believe this to be a version-specific problem.

[jprestop]

HI @mkavulich. Thank you for bringing this to our attention. As you noted we would usually expect GNU to be a bit slower, but I think the slowdown you observed is interesting. I was not aware of this behavior, and I don't believe others on the team are either.

You are correct that the builds available in /config/met are all built with Intel. I'd be happy to install a version in /contrib built with GNU. Would you like me to install version 11.1.1? Is there a specific version of GNU you'd like me to use - 13.2 or 9.2?

[mkavulich]

Yes, 11.1.1 would be best since that's the latest version we have tested with the SRW App. I think for consistency sake we should try GNU 9.2 first since that's a platform-provided compiler (/apps/modules/modulefiles/gnu/9.2.0).

Thanks, I appreciate your response, and let me know if there's any more information I can provide to help!

[jprestop]

Hi @mkavulich.

I have compiled MET-11.1.1 with GNU 9.2.0. To use this version, you can load:

module load gnu/9.2.0
module use /contrib/met/modulefiles
module load met/11.1.1_gnu

Please give that a try and let us know how it goes.

[mkavulich]

Thanks Julie, I was able to confirm that your GNU build also has the same slow execution we were observing. I haven't investigated the other tools, but here are the timings for my equivalent GenEnsProd tests:

Intel

took 0:03:50

11/12 15:34:46.989Z metplus.64c89632 INFO: Processing forecast lead 0 hours
11/12 15:34:46.999Z metplus.64c89632 INFO: COMMAND: /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/upp-addon-env/install/intel/2021.5.0/met-11.1.0-xlt7nqu/bin/gen_ens_prod -v 2 -ens /scratch1/BMC/hmtb/kavulich/UFS/workdir/test_develop/2024-11-02/intel/expt_dirs/MET_ensemble_verification_only_vx_time_lag/2021050500/stage/ADPSFC/file_lists.20241112153446/20210505000000_0_gen_ens_prod.txt -out /scratch1/BMC/hmtb/kavulich/UFS/workdir/test_develop/2024-11-02/intel/expt_dirs/MET_ensemble_verification_only_vx_time_lag/2021050500/metprd/GenEnsProd/gen_ens_prod_FV3_GFS_v15p2_CONUS_25km_ADPSFC_NDAS_000000L_20210505_000000V.nc -config /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/upp-addon-env/install/intel/2021.5.0/metplus-5.1.0-eznr7ze/parm/met_config/GenEnsProdConfig_wrapped
11/12 15:38:37.216Z metplus.64c89632 INFO: Finished running /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/upp-addon-env/install/intel/2021.5.0/met-11.1.0-xlt7nqu/bin/gen_ens_prod - took 0:03:50.196937

Full log file located at /scratch1/BMC/hmtb/kavulich/UFS/workdir/test_develop/2024-11-02/intel/expt_dirs/MET_ensemble_verification_only_vx_time_lag/log/metplus.log.GenEnsProd_ADPSFC

GNU

took 0:22:23

11/21 22:24:56.796Z metplus.44f5c996 INFO: Processing forecast lead 0 hours
11/21 22:24:56.805Z metplus.44f5c996 INFO: COMMAND: /contrib/met/11.1.1_gnu/bin/gen_ens_prod -v 2 -ens /scratch1/BMC/hmtb/kavulich/UFS/workdir/test_develop/2024-11-02/gnu/expt_dirs/met_gnu_jprestop_test/MET_ensemble_verification_only_vx_time_lag/2021050500/stage/ADPSFC/file_lists.20241121222456/20210505000000_0_gen_ens_prod.txt -out /scratch1/BMC/hmtb/kavulich/UFS/workdir/test_develop/2024-11-02/gnu/expt_dirs/met_gnu_jprestop_test/MET_ensemble_verification_only_vx_time_lag/2021050500/metprd/GenEnsProd/gen_ens_prod_FV3_GFS_v15p2_CONUS_25km_ADPSFC_NDAS_000000L_20210505_000000V.nc -config /contrib/METplus/METplus-5.1.0/parm/met_config/GenEnsProdConfig_wrapped
11/21 22:47:20.413Z metplus.44f5c996 INFO: Finished running /contrib/met/11.1.1_gnu/bin/gen_ens_prod - took 0:22:23.591071

Full log file located at /scratch1/BMC/hmtb/kavulich/UFS/workdir/test_develop/2024-11-02/gnu/expt_dirs/met_gnu_jprestop_test/MET_ensemble_verification_only_vx_time_lag/log/metplus.log.GenEnsProd_ADPSFC

[jprestop]

Hi @mkavulich. I hope you had a nice holiday! Thank you so much for this information. That is quite a difference, unfortunately. I'm unsure how/when we'll be able to proceed with looking into these differences. For now, if possible, it would be great if you're able to switch your runs to using the Intel compiled MET version so you can experience only those faster run times.