add 3D aerosol species mixing ratio

[mingxuanwupnnl]

Add aerosol species mixing ratio, inlcuding:

mmrbc, mmroa, mmrsoa, mmrdust, mmrss, mmrso4

[chengzhuzhang]

Posting discussion with @mingxuanwupnnl to here, two issues were raised during testing this PR.

1. According to Mingxuan, I’m trying to output mmrbc/dust etc at model hybrid levels. Although I put ps and PS in the list, it still gave the following error info:

^[[1;31;47m!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: You are trying to write variable "mmrdust" in association
! with variable "ps" (table AERmon), but you you need to
! write the associated variable first in order to
! initialize the file and dimensions.
!
!!!!!!!!!!!!!!!!!!!!!!!!!^[[0m
 
 
^[[2;31;47mC Traceback:
! In function: cmor_set_refvar
! called from: cmor_write
! ^[[0m
 
^[[1;31;47m!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Could not find variable: 'ps' (table: AERmon) in file of
! associated variable: 'mmrdust'
!
!!!!!!!!!!!!!!!!!!!!!!!!!^[[0m

2. By processing variables one at a time, the program runs okay. However, when provide a list of variables for e3sm_to_cmip, it gave error as follows:
   a process in the process pool was terminated abruptly

The script used is attached:

#!/bin/bash
 
caseid=20200919.DECKv1b_H1_FrozenUS.ne30_oEC.cori-knl
 
start=1970
end=2014
ypf=45
 
#e2c_path=/p/user_pub/e3sm/zhang40/e3sm_to_cmip_data
model_data=/pscratch/sd/m/mwu1/archive/$caseid/archive/atm/hist
 
result_dir=/pscratch/sd/m/mwu1/E3SM_proc/$caseid
 
rgr_dir=${result_dir}/rgr
rgr_dir_vert=${result_dir}/rgr_vert
rgr_dir_vert_plev=${result_dir}/rgr_vert_plev
native_dir=${result_dir}/native
 
map_file=/pscratch/sd/m/mwu1/E3SM_proc/map_ne30np4_to_cmip6_180x360_aave.20181001.nc
# path to CMIP6 tables directory
tables_path=/pscratch/sd/m/mwu1/E3SM_proc/cmip6-cmor-tables/Tables
# path to CMOR case metadata
metadata_path=/pscratch/sd/m/mwu1/E3SM_proc/e3sm_to_cmip/e3sm_to_cmip/resources/default_metadata.json
 
 
# note: space is not accepted in nco var list
 
## atm monthly h0 
input_path=${model_data}
flags='-7 --dfl_lvl=1 --no_cll_msr'
raw_var_list="PHIS,hyam,hybm,hyai,hybi,PS,Mass_bc,Mass_dst,Mass_ncl,Mass_pom,Mass_so4,Mass_soa"
 
# atm 2D variables
cmip_var_list="ps, mmrbc, mmrdust, mmroa, mmrsoa, mmrss, mmrso4"
 
ncclimo -P cam -j 1 --map=${map_file} --start=$start --end=$end --ypf=$ypf --split -c $caseid -o ${native_dir} -O ${rgr_dir} -v ${raw_var_list} -i ${input_path} ${flags}
 
# CMORIZE Atmosphere monthly variables: 2D and model level 3D variables (CLOUD,CLDICE,CLDLIQ)
e3sm_to_cmip -i ${rgr_dir} -o $result_dir  -v ${cmip_var_list} -t $tables_path -u ${metadata_path}
#
 
exit

[chengzhuzhang]

I tried to look it up the error:"a process in the process pool was terminated abruptly". It seem to suggest there is problem spawning subprocesses with processpoolexecutor from script: https://github.com/E3SM-Project/e3sm_to_cmip/blob/master/e3sm_to_cmip/__main__.py
@tomvothecoder or @TonyB9000 does this error look familiar? [we can look at this after the holiday]

[tomvothecoder]

I tried to look it up the error:"a process in the process pool was terminated abruptly". It seem to suggest there is problem spawning subprocesses with processpoolexecutor from script: master/e3sm_to_cmip/main.py @tomvothecoder or @TonyB9000 does this error look familiar? [we can look at this after the holiday]

I don't remember if I ran into the "a process in the process pool was terminated abruptly" error in the past or not.

Here are a few possible reasons why this error is raised:

1. Running out of disk space (source)
2. Future pool code is unguarded -- (not inside if __name__) (source) -- I think this is unlikely in e3sm_to_cmip
3. Running ProcessPoolExecutor code in an interactive console is not supported (source)

I'll be on holiday and vacation the rest of the week and traveling all of next week.

[TonyB9000]

The error message "a process in the process pool was terminated abruptly" does not appear to be output by e2c directly, It would be good to have context (the surrounding e2c info/error messages) to better understand where the error occurs. Likewise, the first set of errors are output by the CMOR module.

I think the expertise of Charlie or Xylar might be more familiar with these sort of error messages. I don't think I've run into them before.

I'm guessing the the process-pool termination is a conseguence of the earlier CMOR errors, I note many of these module errors tend to return "0" (All is A-OK) to the calling process, which then tries valiantly to continue but fails.

[chengzhuzhang]

@mingxuanwupnnl I did some more testing, it does look like a memory issue. If running the e3sm_to_cmp command with an additional flag--serial, it should bypass the "a process in the process pool was terminated abruptly" problem.

[tomvothecoder]

@mingxuanwupnnl I did some more testing, it does look like a memory issue. If running the e3sm_to_cmp command with an additional flag--serial, it should bypass the "a process in the process pool was terminated abruptly" problem.

I would run a debugger to find the line that crashes with "a process in the process pool was terminated abruptly".

A random thought, but maybe this multiprocessing issue is similar to e3sm_diags where the scheduler in the package and the one used by Xarray and Dask with .load() are conflicting (comment). The difference is that e3sm_diags uses dask.bag and dask's multiprocessing scheduler, while e3sm_to_cmip uses the Python multiprocessing.Pool.

[mingxuanwupnnl]

@mingxuanwupnnl I did some more testing, it does look like a memory issue. If running the e3sm_to_cmp command with an additional flag--serial, it should bypass the "a process in the process pool was terminated abruptly" problem.

Hi @chengzhuzhang Thank you for bringing this up. I was distracted by other tasks at that time. I wonder how we can proceed. For this PR, how about we just do for 3D aerosol mass mixing ratio? I remember that If I just do two vars a time, it works. If there are three or more vars to be converted, then it will crash. I don't have ideas on debug this issue related to memory. Or we can leave it for now and do 2D variables first.

[chengzhuzhang]

@mingxuanwupnnl thank you for following up this PR. Sorry that we just get to revive this after so long. I think your PR is good as-is, the performance bottleneck seems to be around with the e3sm_to_cmip code base but was not reported. You can keep update this PR for additional aerosol variables, or if it is preferred, you could start a new PR for new additions.

[chengzhuzhang]

@mingxuanwupnnl Thank you. I will review and test when NERSC is back on line.