process_codes-20200609.1600.tar

AA_README_AA0000640017062300020020000000455413667731036013307 0ustar  bartoletti1climate
We can now employ the archive/.cfg/Archive_Map to location and "ls" all files from all archives
for all experiments and dataset types, by appropriate pattern.

The archive assessment intends to determine, for each experiment[ensemble] and dataset type,
the count of matching files, and that the "first" and "last" file found in each matched set
are indeed from the same "set" (have the same name structure) and have rational simulated
date ranges (e.g. 1850_01 to 2014_12).

Files returned have the general structure:

	tar_path/structured_name.sim_date.nc

Because we are using the Archive Map as our key, we know that the "tar_paths" will match for
any first and last file found.  But we need to ensure that "structured_name" is a match, and
that the first and last "sim_dates" are reasonable. This presents two problems:

  1.  The beginning of the structured_name may have a generation-date that varies across the
      dataset, and must be trimmed off if we are to compare the datafile names for matches.

  2.  The structured_name has "dot-delimited" components, and the number of components will
      vary across dataset types, and even for the same dataset type across experiments. This
      makes locating the "sim-year" field often specific to each dataset, and we have created
      a table
		/p/user_pub/e3sm/archive/.cfg/Filename_SimDate_Locator

      that identifies the dot-delimited field for sim-year, given experiment and dataset type


For each "Campaign,Model,Experiment,Ensemble,DatasetType,Count,First,Last", we want to output

either
	Campaign,Model,Experiment,Ensemble,DatasetType,MATCHED,Count,firstDate,LastDate,years_total

or	Campaign,Model,Experiment,Ensemble,DatasetType,NOMATCH,Count,firstStructuredName,LastStructuredName

Anywhere a NO_MATCH occurs, we must modify the Archive_Map to provide more specificity to the entry.


THE PROCESS:

	1.  Ensure the file "/p/user_pub/e3sm/archive/.cfg/Filename_SimDate_Locator" has an entry
	    for each experiment dataset, indicating the "dot"-delimited position of the sim-date
	    value in each filename.  This should have been done as part of the ArchivePathMapper.

	2.  Issue

		archive_assess_count_first_last.sh  any_subset_of_Archive_Map_lines

	    If output contains spurious NO_MATCH indications, but only due to embedded machine
	    name in the file names, you can manually set the flag "trust_match=1" in the script.


AA_README_APM0000640017062300020020000000671413667731036013443 0ustar  bartoletti1climate
BIG SELF_DEFINED TASK:

	Create a utility that will peruse EVERY archived directory (Experiment-Ensemble), to extract ALL tarred filepaths, to record the tar_paths to the actual files intended per (publish-able) dataset.
	e.g., for DECK,
		(atm nat mon) should return "archive/cam/hist/"  (for which the file_extraction_pattern *.cam.h0* will work beautifully)
	but, for HRv1,
		(atm nat mon) would return "run-0006-01-01-180907--0046-01-01-190111/"		(for years 0006-0045)
				       and "run/"						(for years 0046-0055)

	BOTH of the latter being still inadequate for a simple pattern like *cam.h0*, because those paths also archived other stuff matching *cam.h0*, so upon human inspection, we discover we need

			"run-0006-01-01-180907--0046-01-01-190111/theta"
		and	"run/theta"

	in order that the realm/freq pattern *cam.h0* will finally isolate the desired files.


THE GOAL:
	For any required publication of a [ CAMPAIGN MODEL EXPERIMENT ENSEMBLE (realm gridtype freq) ], we will have a table of (one or more) entries

		CAMPAIGN,MODEL,EXPERIMENT,ENSEMBLE,realm_gridtype_freq,archivePath,tarfilePath+extractionPattern

PREREQS:

	Ensure that archive/.cfg file "Standard_Datatype_Extraction_Patterns" are complete with respect to required publications


The Working Process:

1.  Issue ./archive_path_mapper_stage1.sh file_of_archive_specifications

	The "file_of_archive_specifications" can be the entire Archive_Locator (/p/user_pub/e3sm/archive/.cfg/Archive_Locator)
	or any file containng a subset of those lines, indicating the archives to be processed.

	Output will fill directory "PathsFound" with files of the form
		C:M:E:E:DSTITLE
	e.g.
		BGC:1_1:hist-BCRC:ens1:atm_nat_6hr

	Each file will contain all filenames found in archive using patterns found by the "tail" patterns in the_SDEP.

2.  Issue ./archive_path_mapper_stage2.sh 
	This will examine each file in PathsFound, trim away all lines that begin with known inappropriate path patterns
	and for the paths that remain, list the first and last files found for those patterns.
	Output wil be the file "headset12_lists_first_last", which will contain hundreds of entries of the form

		DECK:1_0:abrupt-4xCO2:ens1:lnd_nat_mon
		    HEADF:archive/lnd/hist/20180215.DECKv1b_abrupt4xCO2.ne30_oEC.edison.clm2.h0.0001-01.nc
		    HEADL:archive/lnd/hist/20180215.DECKv1b_abrupt4xCO2.ne30_oEC.edison.clm2.h0.0156-10.nc
		    HEADF:run/20180215.DECKv1b_abrupt4xCO2.ne30_oEC.edison.clm2.h0.0155-12.nc
		    HEADL:run/20180215.DECKv1b_abrupt4xCO2.ne30_oEC.edison.clm2.h0.0155-12.nc

	One must edit this file MANUALLY to determine which paths to the "First and Last" basenames is the correct one.
	In the case above, this would be "archive/lnd/hist", and since the "lnd_nat_mon" search pattern is *clm2.h0*,
	we need to convert that 5-line entry into

		DECK:1_0:abrupt-4xCO2:ens1:lnd_nat_mon:archive/lnd/hist/*clm2.h0*

	When such editing is completed, the finished file should be named "archive_dataset_map_prelim"

3.  Issue ./archive_path_mapper_stage3.sh
	This will take the output from stage 3, "archive_dataset_map_prelim", together with the Archive_Locator,
	to insert the appropriate archive_path into the final listing of the Archive_Map.


EXTRA_CREDIT:

        4.  Ensure the file "/p/user_pub/e3sm/archive/.cfg/Filename_SimDate_Locator" has an entry
            for each experiment dataset, indicating the "dot"-delimited position of the sim-date
            value in each filename.  Use the file "headset12_lists_first_last" to obtain the information.


AA_README_PS0000640017062300020020000000313613667731036013343 0ustar  bartoletti1climate Publication Status Page Preparation Report

The routine:

	pub_stat_page_prep.sh pub_dir_pathhead	(typically, /p/user_pub/work/E3SM/)

This will find the leaf dirs under the publication directories, and output lines of the form

	ModVer,Resolution,Ens,Title,Status,Grid,Years,Filecount,Dataset_ID,Dataset_First_File,Dataset_Publication_Path,Notes,Title_Expanded

Explicitly:

	ModVer:			A component of the path
	Resolution:		A component of the path
	Ens			A component of the path
	Title:			(RealmCode Grid Freq) derived from path elements.
	Status:			(placeholder for publication status)
	Grid:			A component of the path
	Years:			(placeholder for dataset years covered)
	Filecount:		From live directory examination
	Dataset_ID		Derived from path
	Dataset_First_File:	From live directory examination
	Dataset_Publication_Path:  Derived from path
	Notes:			(placeholder)
	Title_Expanded:		(placeholder)

For ALL model version experiments, ensembles, dataset_types and publication versions.

By converting this CSV output into an Excel document, the Excel rows can be copy/pasted directly into the associate confluence tables.

All that is missing by this operation is the ablity to determine the actual publication status for each dataset.

Formerly, this was obtained by calling "esglist_datasets" with the status flag, on aims3.

	Before running esglist_datasets:

		(activate the esgf-pub environment)
		source /usr/local/conda/bin/activate esgf-pub

		(if 72 hour auth cert has timed out.  have your esgf-node password ready)
		myproxy-logon -s esgf-node.llnl.gov -l <your_esgf_user_name> -t 72 -o ~/.globus/certificate-file

archive_assess_count_first_last.sh0000750017062300020020000000542613667731036020526 0ustar  bartoletti1climate#!/bin/bash

# Usage:  archive_assess_count_first_last.sh archive_map(all or part)
# For the entire archive, use /p/user_pub/e3sm/archive/.cfg/Archive_Map
# For a subset, create a file with selected lines, and give its full path

PURPOSE="1.  Find and List all Default Model Outputs from all Experiment/Ensembles by Holodeck of each archive index.db file"

# format:  Campaign,Model,Experiment,Ensemble,DatasetType,ArchivePath,DatatypeExtractionPattern,Notes
# Full Archive Map file:  /p/user_pub/e3sm/archive/.cfg/Archive_Map

archive_map_file=$1
simdate_locator=/p/user_pub/e3sm/archive/.cfg/Filename_SimDate_Locator

The_Holodeck=/p/user_pub/e3sm/bartoletti1/Pub_Status/ArchiveAssess/Holodeck


zstash_version=`zstash version`
if [ $zstash_version != "v0.4.1" ]; then
	echo "ABORTING:  zstash version is not 0.4.1 or is unavailable"
	exit 1
fi

trust_match=1

startTime=`date +%s`

IFS=$'\n'

cd $The_Holodeck

for aline in `cat $archive_map_file`; do

	apath=`echo $aline | cut -f6 -d,`
	if [ $apath == "NAV" ]; then
		continue
	fi

	basetag=`echo $aline | cut -f1-5 -d,`
	pattern=`echo $aline | cut -f7 -d,`


	rm -rf $The_Holodeck/*
	mkdir $The_Holodeck/zstash

	ln -s $apath/index.db $The_Holodeck/zstash/index.db
	
	zstash ls --hpss=none $pattern > /tmp/zstash_manifest

	filecount=`cat /tmp/zstash_manifest | wc -l`
	if [ $filecount -eq 0 ]; then
		echo "$basetag,0,(pattern=$pattern)"
		continue
	fi

	# SHOULD sort on simdate field here. Not easy, because the
	# tarpath could make the indexing difficult.  Or not. 
	init_file=`cat /tmp/zstash_manifest | head -1`
	last_file=`cat /tmp/zstash_manifest | tail -1`

	# trim tar-path from first and last
	filebase1=`echo $init_file | rev | cut -f1 -d/ | rev`
	filebase2=`echo $last_file | rev | cut -f1 -d/ | rev`

	simdfield=`grep $basetag: $simdate_locator | cut -f2 -d:`
	prevfield=$((simdfield - 1))
	
	sim_date1=`echo $filebase1 | cut -f$simdfield -d.`
	filetest1=`echo $filebase1 | cut -f1-$prevfield -d.`
	sim_date2=`echo $filebase2 | cut -f$simdfield -d.`
	filetest2=`echo $filebase2 | cut -f1-$prevfield -d.`

	# last part eliminates variable leading run_dates
	filecore1=`echo $filetest1 | cut -f2- -d.`
	filecore2=`echo $filetest2 | cut -f2- -d.`

	if [ $trust_match -eq 0 ]; then
		if [ $filecore1 != $filecore2 ]; then
			echo "$basetag,NO_MATCH,$filecount,$init_file,$last_file"
			echo "                        $filecore1"
			echo "                        $filecore2"
			continue
		fi
	fi

	y1=`echo $sim_date1 | cut -c1-4`
	y2=`echo $sim_date2 | cut -c1-4`
	m2=`echo $sim_date2 | cut -c6-7`
	y1=$((10#$y1))
	y2=$((10#$y2))
	yspan=$((y2 - y1 + 1))
	if [ $m2 == "01" ]; then
		yspan=$((y2 - y1))
	fi
	echo "$basetag,MATCHED,$filecount,$filetest1,$sim_date1,$sim_date2,$yspan"

done

finalTime=`date +%s`

et=$(($finalTime-$startTime))

echo "Done.  ET = $et"


exit

archive_holodeck_setup_ls-only.sh0000750017062300020020000000136613667731036020247 0ustar  bartoletti1climate#!/bin/bash

# echo "DEBUG:  argcount = $#"

instructions="full path to a leaf archive directory"

zstash_version=`zstash version`
if [ $zstash_version != "v0.4.1" ]; then
	echo "ABORTING:  zstash version is not 0.4.1 or is unavailable"
	exit 1
fi

fulldir=$1

The_Holodeck=/p/user_pub/e3sm/bartoletti1/Pub_Status/ArchiveAssess/Holodeck

IFS=$'\n'

# PLAN:  populate the subordinate "zstash" subdirectory with simlinks to the appropriate tarfiles and index.db file.

echo "Clearing the Holodeck"
echo " "
rm -rf $The_Holodeck/*

zstash_dir=$The_Holodeck/zstash
mkdir $zstash_dir

echo "We will set up holodeck zstash ls for directory $fulldir" 

ln -s $fulldir/index.db $zstash_dir/index.db

echo "Holodeck prepared for entry. Enjoy your simulation!"

exit

archive_holodeck_setup.sh0000750017062300020020000000157213667731036016571 0ustar  bartoletti1climate#!/bin/bash

# echo "DEBUG:  argcount = $#"

instructions="full path to a leaf archive directory"

zstash_version=`zstash version`
if [ $zstash_version != "v0.4.1" ]; then
	echo "ABORTING:  zstash version is not 0.4.1 or is unavailable"
	exit 1
fi

fulldir=$1

The_Holodeck=/p/user_pub/e3sm/bartoletti1/Pub_Status/ArchiveAssess/Holodeck

IFS=$'\n'

# PLAN:  populate the subordinate "zstash" subdirectory with simlinks to the appropriate tarfiles and index.db file.

echo "Clearing the Holodeck"
echo " "
rm -rf $The_Holodeck/*

zstash_dir=$The_Holodeck/zstash
mkdir $zstash_dir

echo "We will setup holodeck zstash for directory $fulldir" 

# major kludge
rm -rf $The_Holodeck/archive

for targ in `ls $fulldir`; do
	# touch nothing
	# echo "ln -s $fulldir/$targ zstash/$targ"
	ln -s $fulldir/$targ $zstash_dir/$targ
done

echo "Holodeck prepared for entry. Enjoy your simulation!"

exit

archive_path_mapper_stage1.sh0000750017062300020020000000346313667731036017326 0ustar  bartoletti1climate#!/bin/bash

# Usage:  archive_path_mapper_stage1.sh achive_locations_file
# For the entire archive, use /p/user_pub/e3sm/archive/.cfg/Archive_Locator
# For a subset, create a file with selected lines, and give its full path

PURPOSE="1.  Find and List all Default Model Outputs from all Experiment/Ensembles by Holodeck of each archive index.db file"

# format:  Campaign,Model,Experiment,Ensemble,ArchivePath,
# Full Archve Location file:  /p/user_pub/e3sm/archive/.cfg/Archive_Locator

archive_loc_file=$1

the_SDEP=/p/user_pub/e3sm/archive/.cfg/Standard_Datatype_Extraction_Patterns

WorkDir=/p/user_pub/e3sm/bartoletti1/Pub_Status/ArchivePathMapper/
The_Holodeck=$WorkDir/Holodeck


zstash_version=`zstash version`
if [ $zstash_version != "v0.4.1" ]; then
	echo "ABORTING:  zstash version is not 0.4.1 or is unavailable"
	exit 1
fi

mv $WorkDir/PathsFound/* $WorkDir/PrevPathsFound/

startTime=`date +%s`

IFS=$'\n'

cd $The_Holodeck

# Archive_Locator Entry processed

ALE=0

for aline in `cat ../$archive_loc_file`; do

	apath=`echo $aline | cut -f5 -d,`
	if [ $apath == "NAV" ]; then
		continue
	fi

	ALE=$(($ALE + 1))

	basetag=`echo $aline | cut -f1-4 -d, | tr , :`

	# echo "ALE:BaseTag = $ALE:$basetag"

	rm -rf $The_Holodeck/*
	mkdir $The_Holodeck/zstash

	ln -s $apath/index.db $The_Holodeck/zstash/index.db
	
	for fpatspec in `cat $the_SDEP`; do
		dstitle=`echo $fpatspec | cut -f1 -d, | tr ' ' _`
		dspattern=`echo $fpatspec | cut -f2 -d,`
		if [ $dspattern == "ignore" ]; then
			continue
		fi
		outfile="$ALE:$basetag:$dstitle"
		# echo "        $outfile"
		zstash ls --hpss=none $dspattern > ../PathsFound/$outfile	
		if [ -s ../PathsFound/$outfile ]; then
			continue
		fi
		echo "EMPTY: $outfile"
		rm ../PathsFound/$outfile
	done

done

finalTime=`date +%s`

et=$(($finalTime-$startTime))

echo "Done.  ET = $et"


exit

archive_path_mapper_stage2.sh0000750017062300020020000000147313667731036017326 0ustar  bartoletti1climate#!/bin/bash

# for each named file in PathsFound

flist=`ls PathsFound`

rm -f headset12_lists_first_last

for afile in `ls PathsFound`; do
	echo $afile >> headset12_lists_first_last
	# the list of grep -v exclusions here may need future additions
	headlist=`cat PathsFound/$afile | cut -c1-12 | sort | uniq | grep -v "^rest/" | grep -v "^post/" | grep -v "^test" | grep -v "^init" | grep -v "^run/try" | grep -v "^run/bench" | grep -v "^old/run" | grep -v "^pp/remap" | grep -v "^a-prime" | grep -v "^lnd_rerun" | grep -v "^atm/ncdiff" | grep -v "^archive/rest"`
	for ahead in $headlist; do
		samp=`grep ^$ahead PathsFound/$afile | sort | head -1`
		echo "    HEADF:$samp" >> headset12_lists_first_last
		samp=`grep ^$ahead PathsFound/$afile | sort | tail -1`
		echo "    HEADL:$samp" >> headset12_lists_first_last
	done
done
archive_path_mapper_stage3.sh0000750017062300020020000000130113667731036017315 0ustar  bartoletti1climate#!/bin/bash

infile=archive_dataset_map_prelim

archive_loc=/p/user_pub/e3sm/archive/.cfg/Archive_Locator

for aline in `cat $infile`; do
	pureline=`echo $aline | cut -f2- -d:`
	headpart=`echo $pureline | cut -f1-4 -d:`
	ds_title=`echo $pureline | cut -f5 -d:`
	epattern=`echo $pureline | cut -f6 -d:`

	seek=`echo $headpart | tr : ,`
	found=`cat $archive_loc | grep $seek`

	lc=`echo $found | wc -l`
	if [ $lc -eq 0 ]; then
		echo "ERROR: no archive location found for $seek"
		continue
	fi
	if [ $lc -gt 1 ]; then
		echo "WARNING: multiple archive locations found for $seek"
	fi
	for apath in $found; do
		the_path=`echo $apath | cut -f5 -d,`
		echo "$seek,$ds_title,$the_path,$epattern"
	done
done

	

archive_status_prep.sh0000750017062300020020000001105213667731036016124 0ustar  bartoletti1climate#!/bin/bash

# Usage:  archive_status_prep.sh archive_map(all or part)
#
# For the entire archive, use /p/user_pub/e3sm/archive/.cfg/Archive_Map
# For a subset, create a file with selected lines, and give its full path
# format:  Campaign,Model,Experiment,Ensemble,DatasetType,ArchivePath,DatatypeExtractionPattern,Notes
# Full Archive Map file:  /p/user_pub/e3sm/archive/.cfg/Archive_Map


# PURPOSE:  From input Archive Map
#  campaign, model, experiment, ens, title, archive_path, tar_extract_pattern
#
#  Produce a listing
# 	model, resolution, ens, Title, status, grid, years, filecount, datasetID, datasetFirstFile, DatasetPubPath,Notes, titleExpanded, misc
#
#  THe following table outlines the source of each item
#
#	model			Direct from Archive Map
#	resolution		use "$default_res" below
#	ensemble		Direct from Archive Map
#	title			extract from Archive Map "DatasetType"
#	status			[HC] "arch"
#	grid			[HC] "native"
#	years			[HC] ""
#	filecount		Generated by inspection
#	datasetID		Constructed from all of these values and sources
#	firstFile		Generated by inspection
#	pubPath			Constructed from datasetID
#	notes			[HC] "model-output"
#	titleExpanded		reit title
#	misc			[HC] ""

#  A datasetID needs
#	E3SM.<model>.<experiment>.<resolution>.<realm>.<grid>.model_output.<freq>.<ens>

archive_map_file=$1
simdate_locator=/p/user_pub/e3sm/archive/.cfg/Filename_SimDate_Locator

The_Holodeck=/p/user_pub/e3sm/bartoletti1/Pub_Status/ArchiveAssess/Holodeck


zstash_version=`zstash version`
if [ $zstash_version != "v0.4.1" ]; then
	echo "ABORTING:  zstash version is not 0.4.1 or is unavailable"
	exit 1
fi

default_res="1deg_atm_60-30km_ocean"

startTime=`date +%s`

IFS=$'\n'

cd $The_Holodeck

for aline in `cat $archive_map_file`; do

	model=`echo $aline | cut -f2 -d,`
	experiment=`echo $aline | cut -f3 -d,`
	resolution=$default_res
	ens=`echo $aline | cut -f4 -d,`
	title=`echo $aline | cut -f5 -d, | tr _ ' '` # may need tr _ ' '
	realmcode=`echo $title | cut -f1 -d' '`
	freq=`echo $title | cut -f3 -d' '`
	pstatus="arch"
	grid="native"
	years=""
	notes="model-output"
	otype=$notes
	misc=""

	# translate realmcode to realm
	if [ $realmcode == "atm" ]; then
		realm="atmos"
	elif [ $realmcode == "lnd" ]; then
		realm="land"
	elif [ $realmcode == "ocn" ]; then
		realm="ocean"
	elif [ $realmcode == "river" ]; then
		realm="river"
	elif [ $realmcode == "sea-ice" ]; then
		realm="sea-ice"
	else realm="UNK_$realmcode"
	fi

	datasetID="E3SM.$model.$experiment.$resolution.$realm.$grid.$otype.$freq.$ens"
	pubPath=`echo $datasetID | tr . /`

	# echo "DEBUG: dsID=$datasetID"
	# continue

	# still need to formulate filecount, firstFile, (titleExpanded)

	apath=`echo $aline | cut -f6 -d,`
	if [ $apath == "NAV" ]; then
		continue
	fi

	basetag=`echo $aline | cut -f1-5 -d,`
	pattern=`echo $aline | cut -f7 -d,`

	# echo "DEBUG:  Basetag=$basetag"

	rm -rf $The_Holodeck/*
	mkdir $The_Holodeck/zstash

	ln -s $apath/index.db $The_Holodeck/zstash/index.db
	
	zstash ls --hpss=none $pattern > /tmp/zstash_manifest

	filecount=`cat /tmp/zstash_manifest | wc -l`

	init_file=`cat /tmp/zstash_manifest | head -1`
	last_file=`cat /tmp/zstash_manifest | tail -1`

	# trim tar-path from first and last
	filebase1=`echo $init_file | rev | cut -f1 -d/ | rev`
	filebase2=`echo $last_file | rev | cut -f1 -d/ | rev`

	simdfield=`grep $basetag: $simdate_locator | cut -f2 -d:`
	prevfield=$((simdfield - 1))
	
	sim_date1=`echo $filebase1 | cut -f$simdfield -d.`
	filetest1=`echo $filebase1 | cut -f1-$prevfield -d.`
	sim_date2=`echo $filebase2 | cut -f$simdfield -d.`
	filetest2=`echo $filebase2 | cut -f1-$prevfield -d.`

	# last part eliminates variable leading run_dates
	filecore1=`echo $filetest1 | cut -f2- -d.`
	filecore2=`echo $filetest2 | cut -f2- -d.`

	if [ $filecore1 != $filecore2 ]; then
		# echo "$basetag,NO_MATCH,$filecount,$init_file,$last_file"
		echo "DEBUG:                  $filecore1"
		echo "DEBUG:                  $filecore2"
		years="NAV"
		
	else
		y1=`echo $sim_date1 | cut -c1-4`
		y2=`echo $sim_date2 | cut -c1-4`
		m2=`echo $sim_date2 | cut -c6-7`
		y1=$((10#$y1))
		y2=$((10#$y2))
		yspan=$((y2 - y1 + 1))
		if [ $m2 == "01" ]; then
			yspan=$((y2 - y1))
		fi
		# echo "$basetag,MATCHED,$filecount,$filetest1,$sim_date1,$sim_date2,$yspan"
		years=$yspan
	fi

	# 	model, resolution, ens, Title, status, grid, years, filecount, datasetID, datasetFirstFile, DatasetPubPath,Notes, titleExpanded, misc
	echo "$model,$resolution,$ens,$title,$pstatus,$grid,$years,$filecount,$datasetID,$filebase1,$pubPath,$otype,$title,$misc"

done

finalTime=`date +%s`

et=$(($finalTime-$startTime))

echo "Done.  ET = $et"


exit

archive_versus_pub_compare.sh0000750017062300020020000000331713667731036017463 0ustar  bartoletti1climate#!/bin/bash

arc_list=$1
pub_list=$2

trisect=/p/user_pub/e3sm/bartoletti1/abin/trisect

# using
#   A.  the ArchiveAssess output from "archive_assess_count_first_last.sh"
#   B.  the assessment report from sproket
#   (properly limited to a single common experiment)
#   do
#	1.  Extract "experiment/ensemble and datasetType" from each
#	2.  Use "trisect" to find the "A-only", "B-only", and Both.

# arc_list:  Campaign,Model,Experiment,Ensemble,r_g_f,MATCHED,files,canonFile,beginDate,finalDate,years
# pub_list:  years,files,datasetID,firstFile
#
#   where datasetID = Project.Model.Experiment.Resolution.Realm.Grid.outType.Freq.Ensemble.version

# So . . . Generate A_Key and B_Key lists

rm -f /tmp/A_list
rm -f /tmp/B_list

for aline in `cat $arc_list`; do

	A_model=`echo $aline | cut -f2 -d,`
	A_Exper=`echo $aline | cut -f3 -d,`
	A_Ensem=`echo $aline | cut -f4 -d,`
	A_r_g_f=`echo $aline | cut -f5 -d,`

	echo $A_model,$A_Exper,$A_Ensem,$A_r_g_f >> /tmp/A_list
done

for aline in `cat $pub_list`; do

	ds_ID=`echo $aline | cut -f3 -d,`
	B_model=`echo $ds_ID | cut -f2 -d.`
	B_exper=`echo $ds_ID | cut -f3 -d.`
	B_ensem=`echo $ds_ID | cut -f9 -d.`

	realm=`echo $ds_ID | cut -f5 -d.`
	gridv=`echo $ds_ID | cut -f6 -d.`
	freqv=`echo $ds_ID | cut -f8 -d.`

	if [ $realm == "atmos" ]; then
		r_code="atm"
	elif [ $realm == "land" ]; then
		r_code="lnd"
	elif [ $realm == "ocean" ]; then
		r_code="ocn"
	else
		r_code=$realm
	fi

	if [ $gridv == "native" ]; then
		g_code="nat"
	else
		g_code="reg"
	fi

	B_r_g_f="${r_code}_${g_code}_$freqv"

	echo $B_model,$B_exper,$B_ensem,$B_r_g_f >> /tmp/B_list
done

sort /tmp/A_list | uniq > /tmp/sA_list	
sort /tmp/B_list | uniq > /tmp/sB_list	

$trisect -f1 /tmp/sA_list -f2 /tmp/sB_list

exit 0
Automation_via_Holodeck0000640017062300020020000001636413667731036016241 0ustar  bartoletti1climateThe Nearly Automated Dataset Extraction and (Pre)Publication Process

TL;DR:
	IMPORTANT:  Due to values hard-coded into the scripts, one must invoke this "automated" process
		for each combination of (realm,freq_set,resolution,dataset_version)

	1.  Create a file "the_target_archives" consisting of lines selected from the file

		/p/user_pub/e3sm/archive/.cfg/Archive_Locator

		The lines listed must cover all cases (experiment,ensembles) for all intended datasets 

	2.  Edit the "publication_staging_control_script.sh" to set the values for these two variables

		dstype_static="atm nat"	(EXAMPLE)
		dstype_freq_list="mon day 6hr_snap day_cosp" (EXAMPLE)

		As we are extracting from archives, the first variable will always be "<realm> nat".
		The second can be any space-separated set of frequency-codes.
		See the file: /p/user_pub/e3sm/archive/.cfg/Standard_Datatype_Extraction_Patterns for options.

	3.  Edit the "" script to set the values for these two variables

		resolution=1deg_atm_60-30km_ocean
		pubvers=v1

	4.  Invoke

		nohup ./publication_staging_control_script.sh the_target_archives &

		A log file "Holodeck_Process_Log-<date_time>" will be produced to track progress.

	If datasets for multiple realms is required, you will need to repeat Step 2 to reflect a different realm,
	and re-invoke Step 3.


BACKGROUND:


E3SM Publication Staging is now operating under a new premise:  All data archives are now local and amenable to zstash extraction.

I have crafted a working - if fragile - automated extraction regime that can access multiple experiments and cases, locate their archive directories and file extraction patterns, create the faceted publication destination directories, and successfully extract and move the dataset files to the intended publication directories.  I would caution that the process is not yet entirely generalized, and should be reimplemented at some point in a fixed python application, once the dust settles on a stable procedure.

The following provides background and detailed explanation of the process.  As I write this, it is already staging all of the DECK cam.h[1-4] datasets, for all experiments previously published.

BACKGROUND:  File Directory Structure and Configurations

The LLNL E3SM Archives presently have the following location and structure:

	/p/user_pub/e3sm/archive/
				1_0/
					DECK/
						(case_leaf)/tarfiles + index.db
						(case_leaf)/tarfiles + index.db
						...
					HiResMIP/
						(case_leaf)/tarfiles + index.db
						(case_leaf)/tarfiles + index.db
						...
				1_1/
					BGC/
						(ditto)
				1_1_ECA/
					BGC/
						(ditto)

	Note that "case_leaf" is a directory name uniquly identifying an experiment and ensemble.
	EXAMPLE: 20180622.DECKv1b_A2_1850aeroF.ne30_oEC.edison  (here, A2 = AMIP ens2)
	These directory names exactly match the author-supplied NERSC HPSS leaf directory names.
	This practice facilitates point-to-point Globus transfers.

	Ideally, the "case-leaf" directory should contain nothing but the tar files and the zstash database file "index.db".  In practice, it may contain other material, but no further directory traversal should be required to locate the available tar files.

In order to provide the opportunity for various "staging" dataset checks or modifications, the files are extracted to 

	/p/user_pub/e3sm/staging/pre_pub/(facets)/(files.nc)

where (facets)/(files.nc) matches the structure of the eventual publication directories.

We can extract and move the dataset files to publication using zstash, while leaving the archive intact and clean, by employing a "holodeck" of symlinks. (Credit to Sterling Baldwin for schooling me on the virtues of symlinks. I merely take credit for the use of the term "holodeck" in this context.)

With any empty directory serving as the "holodeck", the following depicts the sequence of operations leading to extraction and movement of dataset files to their intended publication directory:

1. Begin with empty directory, call this the $HolodeckPath

	/path/to/holodeck/

2. Create a zstash subdirectory

	/path/to/holodeck/
			zstash/

3. Having located the appropriate "case-leaf" archive directory, form a list of the contained tar file names (and index.db) and their full paths, and loop over each to create in the zstash subdirectory the necessary symlinks

	/path/to/holodeck/
			zstash/
				tarfilename --> symlink to full tarfile path
				tarfilename --> symlink to full tarfile path
				...
				index.db --> symlink to full index.db path

4. With the holodeck as the working directory, and the necessary datafile extraction pattern in hand, one can issue the zstash extract command to obtain

	/path/to/holodeck/
			stored/path/to/
				datafile
				datafile
				...
				datafile

			zstash/
				tarfilename --> symlink to full tarfile path
				tarfilename --> symlink to full tarfile path
				...
				index.db --> symlink to full index.db path

5. Having previously constructed the staging/prepub/(facets)/ destination path (targ_pub_dir="$prepubroot/$modelver/$exp_name/$midpath/$freq/$ensemble/$pubvers") and ensured it is empty (may be overridden with "overwriteFlag=1", and cognizant of the extracted tarpath-head (e.g. "stored/path/to/"), one can now issue

	mv stored/path/to/* pub_dest_path
	chmod 644 pub_dest_path/*

   leaving you

	/path/to/holodeck/
			stored/path/to/
				(empty)

			zstash/
				tarfilename --> symlink to full tarfile path
				tarfilename --> symlink to full tarfile path
				...
				index.db --> symlink to full index.db path

6. At this point, the dataset has been fully staged for pre-publication inspection and possible modifications.

   If there are additional datasets to publish from this same archive "case-leaf", one need only issue

	rm -rf $HolodeckPath/stored/

   and repeat steps 4 and 5 with another datafile_extraction_pattern, with working directory the $HolodeckPath with existing zstash symlinks.  

   If instead there are other cases for which datasets exist to be published, issue

	rm -rf $HolodeckPath/*

   and repeat from step 2 with a new case-leaf and associated datafile_extraction_patterns



IN SUMMARY:

With the "holodeck" as the working directory, a script given the desired (model,campaign,case,ensemble,datasetSpec) will need to

	1. Construct and test for existence and population of the target pre-publication directory.
		- Skip if the directory exists and is non-empty, unless the script overwriteFlag=1.
	2. Access the Archive_Map to obtain the information regarding the archive source data
		- Ensure exactly 1 line returned from CMEE_archive_map and CMEE_archive_map_augment
	3. populate the subordinate "zstash" subdirectory with simlinks to the appropriate tarfiles and index.db file.
		- Ensure holodeck contains only empty zstash subdirectory
	4. execute the zstash extraction command, with the appropriate file-selection pattern.
	5. create the necessary target publication directory
	6. move the extracted files to their publication directory
	7. clean the holodeck by removing all zstash symlinks and non-zstash directories

STEP 0:

	must contruct and test $pub_head_dir/$thisModel/$thisExp/$path_mid/$freq/$ens/v1
	where
		pub_head_dir=/p/user_pub/work/E3SM
		thisModel (supplied)
		thisExp   (supplied)
		path_mid  (supplied)	e.g. 1deg_atm_60-30km_ocean/atmos/native/model-output
		freq      (supplied)
		ens       (supplied)
		ver       (supplied)
dirperms.sh0000750017062300020020000000014413667731036013677 0ustar  bartoletti1climate#!/bin/bash

dirlist=$1

for aline in `cat $dirlist`; do
	chmod 755 $aline
	chmod 644 $aline/*
done
holodeck_stage_publication.sh0000750017062300020020000000774413667731036017433 0ustar  bartoletti1climate#!/bin/bash

# EDIT THIS VARIABLE!
# resolution=0_25deg_atm_18-6km_ocean
resolution=1deg_atm_60-30km_ocean
pubvers=v1
overwriteFlag=1


Datatype_Patterns=/p/user_pub/e3sm/archive/.cfg/Standard_Datatype_Extraction_Patterns
The_Holodeck=/p/user_pub/e3sm/bartoletti1/Pub_Work/1_Refactor/Holodeck

Archive_Map=/p/user_pub/e3sm/archive/.cfg/Archive_Map
prepubroot=/p/user_pub/e3sm/staging/prepub
pubroot=/p/user_pub/work/E3SM

# echo "DEBUG:  argcount = $#"

if [ $# -ne 2 ]; then
	echo " "
	echo "    Usage:  holodeck_stage_publication.sh cmee_am_selection_line datasetspec"
	echo "            (a line selected from archive/.cfg/Archive_Locator)"
	echo "            Give datasetspec as \"realm grid freq\", as in \"atm nat mon\", in quotations."
	echo "            See the file /p/user_pub/e3sm/archive/.cfg/Standard_Datatype_Extraction_Patterns"
	echo " "
	exit 0
fi

# echo "DEBUG: DOLLAR_1 = $1"
# echo "DEBUG: DOLLAR_2 = $2"


zstash_version=`zstash version`
if [ $zstash_version != "v0.4.1" ]; then
	echo "ABORTING:  zstash version is not 0.4.1 or is unavailable"
	exit 1
fi

IFS=$'\n'


AL_selection_line=$1

AL_path=`echo $AL_selection_line | cut -f5 -d,`
AL_key=`echo $AL_selection_line | cut -f1-4 -d,`

ds_spec=$2	# realm-code nat freq_code, e.g. "atm nat mon", from Standard_Datatype_Extraction_Patterns
ds_key=`echo $ds_spec | tr ' ' _`

AM_key="$AL_key,$ds_key"
# echo "Produced AM_key: $AM_key"

# Determine realm for  midpath
realm_code=`echo $ds_spec | cut -f1 -d' '`

if [ $realm_code == "atm" ]; then
	realm="atmos"
elif [ $realm_code == "lnd" ]; then
	realm="land"
elif [ $realm_code == "ocn" ]; then
	realm="ocean"
elif [ $realm_code == "river" ]; then
	realm="river"
elif [ $realm_code == "sea-ice" ]; then
	realm="sea-ice"
else
	echo "ERROR: unrecognized realm code: $realm_code"
	exit 1
fi


midpath=$resolution/$realm/native/model-output

AM_list=`grep $AM_key $Archive_Map | sort`

listcount=`echo $AM_list | wc -l`

echo "Matched $listcount AM lines"
# may activate multiple Archive_Map lines, must loop.

cd $The_Holodeck

for am_line in $AM_list; do

	# echo "DEBUG: am_line = $am_line"

	campaign=`echo $am_line | cut -f1 -d,`
	modelver=`echo $am_line | cut -f2 -d,`
	exp_name=`echo $am_line | cut -f3 -d,`
	ensemble=`echo $am_line | cut -f4 -d,`
	arch_dir=`echo $am_line | cut -f6 -d,`
	tar_patt=`echo $am_line | cut -f7 -d,`

	# PLAN Step 1: test and exit if publication directory already exists.

	freq=`echo $ds_spec | cut -f3 -d' '`
	targ_pub_dir="$prepubroot/$modelver/$exp_name/$midpath/$freq/$ensemble/$pubvers"

	# echo "DEBUG: freqword = $freq"

	if [ $overwriteFlag -eq 0 ]; then
		if [ -d $targ_pub_dir ]; then
			fc=`ls $targ_pub_dir | wc -l`
			if [ $fc -gt 0 ]; then
				echo "ABORT: PUB DIR EXISTS ($fc files) $targ_pub_dir"
				echo " "
				continue
			fi
		fi
	fi

	# PLAN Step 2: Use the supplied Archive_Locator line to obtain the information regarding the archive source data

	echo "Conducting zstash extract for directory $arch_dir with dataset extraction pattern $tar_patt"
	echo "Target Publication Dir:  $targ_pub_dir"

	# continue



	# Real Stuff Below

	# PLAN Step 3:  populate the subordinate "zstash" subdirectory with simlinks to the appropriate tarfiles and index.db file.
	#    - Ensure holodeck contains only empty zstash subdirectory

	echo "Cleaning the Holodeck"
	echo " "
	rm -rf $The_Holodeck/*

	zstash_dir=$The_Holodeck/zstash
	mkdir $zstash_dir

	for targ in `ls $arch_dir`; do
		# touch nothing
		# echo "ln -s $arch_dir/$targ zstash/$targ"
		ln -s $arch_dir/$targ $zstash_dir/$targ
	done

	echo "zstash extract --hpss=none $tar_patt"
	zstash extract --hpss=none $tar_patt
	exitcode=$?

	if [ $exitcode -ne 0 ]; then
		echo "ERROR:  zstash returned exitcode $exitcode"
		exit $exitcode
	fi


	echo "MKDIR: mkdir -p $targ_pub_dir"
	mkdir -p $targ_pub_dir
	chmod 755 $targ_pub_dir

	echo "mv $The_Holodeck/$file_extract_pattern $targ_pub_dir"
	mv $The_Holodeck/$tar_patt $targ_pub_dir
	chmod 644 $targ_pub_dir/*
	echo " "

	echo "Cleaning the Holodeck"
	echo " "
	rm -rf $The_Holodeck/*
done

exit 0
list_pub_dirs.sh0000750017062300020020000000140113667731036014711 0ustar  bartoletti1climate#!/bin/bash

mod_exp_camp=/p/user_pub/e3sm/bartoletti1/Model_Experiment_Campaign

pub_head_dir=/p/user_pub/work/E3SM

path_mid=1deg_atm_60-30km_ocean/atmos/native/model-output

for campaign in BGC DECK; do

	model_experiments=`grep $campaign $mod_exp_camp | cut -f1,2 -d,`
 
	for mod_exp in $model_experiments; do

		thisModel=`echo $mod_exp | cut -f1 -d,`
		thisExp=`echo $mod_exp | cut -f2 -d,`

		# find all ensembles
		ensembles=`find $pub_head_dir -type d | grep $thisModel | grep "/$thisExp/" | grep "/ens" | cut -f13 -d/ | sort | uniq`
		
		for ens in $ensembles; do

			for freq in day 6hr_ave 6hr_snap 3hr; do
				# echo "($campaign) $exp $freq $ens"
				echo "($campaign) $pub_head_dir/$thisModel/$thisExp/$path_mid/$freq/$ens/v1"
			done
		done
	done
done


make_mapfile.sh0000750017062300020020000000067113667731036014471 0ustar  bartoletti1climate#!/bin/bash

# FORMAT
# esgmapfile make -i /path/to/ini/directory --max-processes <some_number_of_cores> --project <e3sm/cmip6> --outdir /output/location <path_to_data_directory>

dataset_fullpath=$1

echo "make_mapfile: processing: $dataset_fullpath"
esgmapfile make --debug -i /p/user_pub/e3sm/staging/ini_std/ --max-processes 20 --project e3sm --outdir /p/user_pub/e3sm/staging/mapfiles/mapfiles_auto_publish $dataset_fullpath

exit $?


MANIFEST0000640017062300020020000000275013667731036012652 0ustar  bartoletti1climateROOTDIR:/p/user_pub/e3sm/bartoletti1

alpha/

TARGET:alpha/MANIFEST

PUB_WORK/

TARGET:Pub_Work/1_Refactor/Automation_via_Holodeck
TARGET:Pub_Work/1_Refactor/holodeck_stage_publication.sh
TARGET:Pub_Work/1_Refactor/list_pub_dirs.sh
TARGET:Pub_Work/1_Refactor/publication_staging_control_script.sh
TARGET:Pub_Work/2_Mapwork/make_mapfile.sh
TARGET:Pub_Work/2_Mapwork/multi_mapfile_generate.sh
TARGET:Pub_Work/2_Mapwork/dirperms.sh
TARGET:Pub_Work/3_Publish/publish-all-stages-autodelete.sh
TARGET:Pub_Work/3_Publish/publish-all-stages.sh
TARGET:Pub_Work/3_Publish/publish-azb-v2.sh
TARGET:Pub_Work/3_Publish/publish-db-ingest-only.sh
TARGET:Pub_Work/3_Publish/publish-publish-only.sh
TARGET:Pub_Work/3_Publish/publish-thredds-n-publish-only.sh

PUB_STATUS/

TARGET:Pub_Status/ArchivePathMapper/AA_README_APM
TARGET:Pub_Status/ArchivePathMapper/archive_path_mapper_stage1.sh
TARGET:Pub_Status/ArchivePathMapper/archive_path_mapper_stage2.sh
TARGET:Pub_Status/ArchivePathMapper/archive_path_mapper_stage3.sh
TARGET:Pub_Status/ArchiveAssess/AA_README_AA
TARGET:Pub_Status/ArchiveAssess/archive_assess_count_first_last.sh
TARGET:Pub_Status/ArchiveAssess/archive_status_prep.sh
TARGET:Pub_Status/ArchiveAssess/archive_holodeck_setup_ls-only.sh
TARGET:Pub_Status/ArchiveAssess/archive_holodeck_setup.sh
TARGET:Pub_Status/sproket/process_sproket_output.sh
TARGET:Pub_Status/sproket/sproket_search.sh
TARGET:Pub_Status/AA_README_PS
TARGET:Pub_Status/archive_versus_pub_compare.sh
TARGET:Pub_Status/pub-stat-page-prep.sh


multi_mapfile_generate.sh0000750017062300020020000000176613667731036016566 0ustar  bartoletti1climate#!/bin/bash

workpath=/p/user_pub/e3sm/bartoletti1/Pub_Work/2_Mapwork

if [ $# -ne 1 ]; then
	echo "Usage:  multi_mapfile_Generate.sh map_status_control_file"
	exit 1
fi

map_status_file=$1

datasets=`grep -v DONE $map_status_file | grep -v HOLD`

startTime=`date +%s`

ts=`date +%Y%m%d.%H%M%S`
rlog=mfg_runlog-$ts

setcount=0;

for dataset in $datasets; do
	ts=`date +%Y%m%d.%H%M%S`
	ds_tm1=`date +%s`
	echo "STATUS:$ts:INPROCESS: dataset $dataset" >> $rlog 2>&1
	$workpath/make_mapfile.sh $dataset >> $rlog 2>&1
	ds_tm2=`date +%s`
	ds_et=$(($ds_tm2 - $ds_tm1))

	ts=`date +%Y%m%d.%H%M%S`
	if [ $? -eq 0 ]; then
		echo "STATUS:$ts:COMPLETED: dataset $dataset" >> $rlog 2>&1
	else
		echo "STATUS:$ts:FAILURE:   dataset $dataset (exit code $?)" >> $rlog 2>&1
	fi
	echo "STATUS: dataset ET = $ds_et" >> $rlog 2>&1
	setcount=$(($setcount + 1))
done

finalTime=`date +%s`
et=$(($finalTime - $startTime))

echo " " >> $rlog 2>&1
echo "Processed $setcount datasets." >> $rlog 2>&1
echo "Elapsed Time: $et" >> $rlog 2>&1
process_sproket_output.sh0000750017062300020020000000272213667731036016723 0ustar  bartoletti1climate#!/bin/bash

rm -f ESGF_assessment_report

for aline in `cat E3SM_datafile_paths_leaf`; do
	filecount=`grep $aline E3SM_datafile_paths_full | wc -l`
	firstfile=`grep $aline E3SM_datafile_paths_full | rev | cut -f1 -d/ | rev | sort | uniq | head -1`
	finalfile=`grep $aline E3SM_datafile_paths_full | rev | cut -f1 -d/ | rev | sort | uniq | tail -1`
	datasetID=`echo $aline | tr / .`

	# handle variable sim-date year counts

	if [[ $firstfile =~ -*([0-9]{4}-[0-9]{2}) ]]; then

		# process YYYY-MM
		yearmo1=`echo ${BASH_REMATCH[1]}`
		if [[ $finalfile =~ -*([0-9]{4}-[0-9]{2}) ]]; then
			yearmo2=`echo ${BASH_REMATCH[1]}`
		else
			echo "NOMATCH,$filecount,$datasetID,$firstfile" >> ESGF_assessment_report
			continue
		fi

		y1=`echo $yearmo1 | cut -c1-4`
		y2=`echo $yearmo2 | cut -c1-4`
		m2=`echo $yearmo2 | cut -c6-7`

		if [ $m2 == "01" ]; then
			yspan=$((y2 - y1))
		else
			yspan=$((y2 - y1 + 1))
		fi

		echo "$yspan,$filecount,$datasetID,$firstfile" >> ESGF_assessment_report
		

	elif [[ $firstfile =~ -*([0-9]{6}_[0-9]{6}) ]]; then

		# process YYYYMM-YYYYMM first file only
		ym_ym=`echo ${BASH_REMATCH[1]}`

		y1=`echo $ym_ym | cut -c1-4`
		y2=`echo $ym_ym | cut -c8-11`
		m2=`echo $ym_ym | cut -c12-13`

		if [ $m2 == "01" ]; then
			yspan=$((y2 - y1))
		else
			yspan=$((y2 - y1 + 1))
		fi

		echo "$yspan,$filecount,$datasetID,$firstfile" >> ESGF_assessment_report
	else
		echo "NOMATCH,$filecount,$datasetID,$firstfile" >> ESGF_assessment_report
		continue
	fi

done


publication_staging_control_script.sh0000750017062300020020000000353113667731036021226 0ustar  bartoletti1climate#!/bin/bash

# EDIT THESE AS NEEDED
dstype_static="atm nat"
dstype_freq_list="day_cosp"


holodeck_stager="/p/user_pub/e3sm/bartoletti1/Pub_Work/1_Refactor/holodeck_stage_publication.sh"


rm -f nohup.out

echo " "
echo "# Publication Staging Controller #"
echo " "

if [ $# -eq 0 ]; then
	echo "usage: (this_script) file_of_Archive_Locator_selectedlines"
	exit
fi

if [ $1 == "help" ]; then
	echo "usage: (this_script) file_of_Archive_Locator_selectedlines"
	exit
fi

# file of Archive_Locator lines
AL_selected=$1


startTime=`date +%s`
ts=`date +%Y%m%d.%H%M%S`
holodeck_log=/p/user_pub/e3sm/bartoletti1/Pub_Work/1_Refactor/Holodeck_Process_Log-$ts

# Here, we prepare to process one or more Archive_Locator lines, even when multiple lines may refer to 
# different parts of a single dataset

casecount=0;

for AL_line in `cat $AL_selected`; do

	echo " " >> $holodeck_log 2>&1
	echo "$ts:Publication Staging Controller: Archive_Locator_line = $AL_line" >> $holodeck_log 2>&1
	echo " "

	datasets=0;
	# For the current case (Experiment / Ensemble), we are prepared only to pull one or more
	# frequencies for a single realm (e.g.  cam.h1, h2, h3 ...)

	ts=`date +%Y%m%d.%H%M%S`
	echo "$ts:Processing case spec: $AL_line" >> $holodeck_log 2>&1
	echo " "
	for freq in $dstype_freq_list; do
		argslist=()
		argslist[0]=$AL_line
		dstype="$dstype_static $freq"
		argslist[1]=$dstype
		ts=`date +%Y%m%d.%H%M%S`
		echo "$ts: Calling holodeck_stager with $AL_line \"$dstype\"" >> $holodeck_log 2>&1
		$holodeck_stager "${argslist[@]}" >> $holodeck_log 2>&1
		datasets=$(($datasets + 1))
	done
	ts=`date +%Y%m%d.%H%M%S`
	echo "$ts:Completion case spec: $AL_line ($datasets datasets)" >> $holodeck_log 2>&1
	casecount=$(($casecount + 1))
done

finalTime=`date +%s`
et=$(($finalTime - $startTime))

echo "Elapsed time: $et seconds.  ($casecount cases processed)" >> $holodeck_log


exit
	
publish-all-stages-autodelete.sh0000750017062300020020000001064713667731036017714 0ustar  bartoletti1climate#!/bin/bash

if [ $# -ne 1 ]; then
        echo "Please enter a descriptive job_name for tracking purposes"
        exit 1
fi

tagname=`echo $1 | tr ' ' _`


utcStart=`date +%s`
ts=`date +%Y%m%d.%H%M%S`

rlog="rlog-$ts-$tagname"

logdir=/p/user_pub/e3sm/bartoletti1/Pub_Work/3_Publish/runlogs
mapfiles_stage1=/p/user_pub/e3sm/staging/mapfiles/stage_1_db_ingest
mapfiles_stage2=/p/user_pub/e3sm/staging/mapfiles/stage_2_thredds_ingest
mapfiles_stage3=/p/user_pub/e3sm/staging/mapfiles/stage_3_publish
mapfiles_aborted=/p/user_pub/e3sm/staging/mapfiles/mapfiles_hold
mapfiles_archive=/p/user_pub/e3sm/staging/mapfiles/mapfiles_archive
ini_dir=/p/user_pub/e3sm/staging/ini_std
project=e3sm
email_recip=bartoletti1@llnl.gov

mapfiles=( `ls $mapfiles_stage1 | grep -v .part` )
mapfile_count=${#mapfiles[@]}


ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: mapfile ingestion to database ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
for mapfile in "${mapfiles[@]}"
do
	#check for deletion needed

	telltale=`echo $mapfile | rev | cut -f1-2 -d. | rev`
	vnum=`echo $telltale | cut -f1 -d. | cut -c2-`
	if [ $vnum -gt 1 ]; then
		vpre=$(($vnum - 1))
		echo "version is $vnum, deleting version $vpre"
		newtail="#$vpre"
		echo $mapfile > tmp_mystring
		sedcmd=`echo "s/.$telltale/$newtail/g"`
		sed -i $sedcmd tmp_mystring
		to_delete=`cat tmp_mystring`
		echo "$ts: INPROCESS: Calling esgunpublish for $to_delete" >> $logdir/$rlog 2>&1
		esgunpublish --project e3sm --database-delete --delete $to_delete >> $logdir/$rlog 2>&1
		ts=`date +%Y%m%d.%H%M%S`
		echo "$ts: COMPLETED: Deletion of dataset $to_delete" >> $logdir/$rlog 2>&1
	fi


    esgpublish -i $ini_dir --project $project --map $mapfiles_stage1/$mapfile --replace --commit-every 100  --no-thredds-reinit >> $logdir/$rlog 2>&1
    retcode=$?
    ts=`date +%Y%m%d.%H%M%S`
    if [ $retcode != 0 ] ; then
        echo "$ts: -- Failed to ingest $mapfile into the database" >> $logdir/$rlog 2>&1
	mv $mapfiles_stage1/$mapfile $mapfiles_aborted/$mapfile
        continue
    fi
    echo "$ts: -- Successfully ingested $mapfile into the database" >> $logdir/$rlog 2>&1
    mv $mapfiles_stage1/$mapfile $mapfiles_stage2/$mapfile
done

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile ingestion to database ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

mapfiles=( `ls $mapfiles_stage2` )
mapfile_count=${#mapfiles[@]}

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: mapfile ingestion to thredds ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfiles_stage2/$mapfile --service fileservice --noscan --thredds  --no-thredds-reinit >> $logdir/$rlog 2>&1
    retcode=$?
    ts=`date +%Y%m%d.%H%M%S`
    if [ $retcode != 0 ] ; then
        echo "$ts: -- Failed to ingest $mapfile into thredds" >> $logdir/$rlog 2>&1
	mv $mapfiles_stage2/$mapfile $mapfiles_aborted/$mapfile
	continue
    fi
    echo "$ts: -- Successfully ingested $mapfile into thredds" >> $logdir/$rlog 2>&1
    mv $mapfiles_stage2/$mapfile $mapfiles_stage3/$mapfile
done

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile ingestion to thredds ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: calling --thredds-reinit ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

esgpublish --project $project --thredds-reinit >> $logdir/$rlog 2>&1

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: thredds-reinit ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

mapfiles=( `ls $mapfiles_stage3` )
mapfile_count=${#mapfiles[@]}

echo "$ts: INPROCESS: mapfile publication ($mapfile_count mapfiles)" >> $logdir/$rlog
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfiles_stage3/$mapfile --service fileservice --noscan --publish >> $logdir/$rlog 2>&1
    retcode=$?
    ts=`date +%Y%m%d.%H%M%S`
    if [ $retcode != 0 ] ; then
        echo "$ts: -- Failed to publish $mapfile" >> $logdir/$rlog 2>&1
	mv $mapfiles_stage3/$mapfile $mapfiles_aborted/$mapfile
        continue
    fi
    echo Successfully published $mapfile >> $logdir/$rlog
    mv $mapfiles_stage3/$mapfile $mapfiles_archive/$mapfile
done



utcFinal=`date +%s`
elapsed=$(($utcFinal - $utcStart))

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile publication ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
echo "$ts: Elapsed time:  $elapsed seconds" >> $logdir/$rlog 2>&1

echo "All done ($tagname).  Elapsed time:  $elapsed seconds" | sendmail $email_recip

publish-all-stages.sh0000750017062300020020000000741413667731036015561 0ustar  bartoletti1climate#!/bin/bash

if [ $# -ne 1 ]; then
        echo "Please enter a descriptive job_name for tracking purposes"
        exit 1
fi

tagname=`echo $1 | tr ' ' _`


utcStart=`date +%s`
ts=`date +%Y%m%d.%H%M%S`

rlog="rlog-$ts-$tagname"

logdir=/p/user_pub/e3sm/bartoletti1/Pub_Work/3_Publish/runlogs
mapfiles_stage1=/p/user_pub/e3sm/staging/mapfiles/stage_1_db_ingest
mapfiles_stage2=/p/user_pub/e3sm/staging/mapfiles/stage_2_thredds_ingest
mapfiles_stage3=/p/user_pub/e3sm/staging/mapfiles/stage_3_publish
mapfiles_aborted=/p/user_pub/e3sm/staging/mapfiles/mapfiles_hold
mapfiles_archive=/p/user_pub/e3sm/staging/mapfiles/mapfiles_archive
ini_dir=/p/user_pub/e3sm/staging/ini_std
project=e3sm
email_recip=bartoletti1@llnl.gov

mapfiles=( `ls $mapfiles_stage1 | grep -v .part` )
mapfile_count=${#mapfiles[@]}

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: mapfile ingestion to database ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfiles_stage1/$mapfile --replace --commit-every 100  --no-thredds-reinit >> $logdir/$rlog 2>&1
    retcode=$?
    ts=`date +%Y%m%d.%H%M%S`
    if [ $retcode != 0 ] ; then
        echo "$ts: -- Failed to ingest $mapfile into the database" >> $logdir/$rlog 2>&1
	mv $mapfiles_stage1/$mapfile $mapfiles_aborted/$mapfile
        continue
    fi
    echo "$ts: -- Successfully ingested $mapfile into the database" >> $logdir/$rlog 2>&1
    mv $mapfiles_stage1/$mapfile $mapfiles_stage2/$mapfile
done

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile ingestion to database ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

mapfiles=( `ls $mapfiles_stage2` )
mapfile_count=${#mapfiles[@]}

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: mapfile ingestion to thredds ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfiles_stage2/$mapfile --service fileservice --noscan --thredds  --no-thredds-reinit >> $logdir/$rlog 2>&1
    retcode=$?
    ts=`date +%Y%m%d.%H%M%S`
    if [ $retcode != 0 ] ; then
        echo "$ts: -- Failed to ingest $mapfile into thredds" >> $logdir/$rlog 2>&1
	mv $mapfiles_stage2/$mapfile $mapfiles_aborted/$mapfile
	continue
    fi
    echo "$ts: -- Successfully ingested $mapfile into thredds" >> $logdir/$rlog 2>&1
    mv $mapfiles_stage2/$mapfile $mapfiles_stage3/$mapfile
done

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile ingestion to thredds ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: calling --thredds-reinit ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

esgpublish --project $project --thredds-reinit >> $logdir/$rlog 2>&1

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: thredds-reinit ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

mapfiles=( `ls $mapfiles_stage3` )
mapfile_count=${#mapfiles[@]}

echo "$ts: INPROCESS: mapfile publication ($mapfile_count mapfiles)" >> $logdir/$rlog
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfiles_stage3/$mapfile --service fileservice --noscan --publish >> $logdir/$rlog 2>&1
    retcode=$?
    ts=`date +%Y%m%d.%H%M%S`
    if [ $retcode != 0 ] ; then
        echo "$ts: -- Failed to publish $mapfile" >> $logdir/$rlog 2>&1
	mv $mapfiles_stage3/$mapfile $mapfiles_aborted/$mapfile
        continue
    fi
    echo Successfully published $mapfile >> $logdir/$rlog
    mv $mapfiles_stage3/$mapfile $mapfiles_archive/$mapfile
done



utcFinal=`date +%s`
elapsed=$(($utcFinal - $utcStart))

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile publication ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
echo "$ts: Elapsed time:  $elapsed seconds" >> $logdir/$rlog 2>&1

echo "All done ($tagname).  Elapsed time:  $elapsed seconds" | sendmail $email_recip

publish-azb-v2.sh0000750017062300020020000000344413667731036014625 0ustar  bartoletti1climate#!/bin/bash

utcStart=`date +%s`

mapfile_dir=/p/user_pub/e3sm/staging/mapfiles
mapfile_done=/p/user_pub/e3sm/staging/mapfiles_temp_done
project=e3sm
email_recip=bartoletti1@llnl.gov

ini_dir=/p/user_pub/e3sm/staging/ini_old

exit_on_any_fail=1

mapfile_count=`ls $mapfile_dir | wc -l`
mapfiles_published=0

mapfiles=( `ls $mapfile_dir` )

for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfile_dir/$mapfile --commit-every 100  --no-thredds-reinit
    if [ $? != 0 ] ; then
        echo Failed to engest $mapfile into the database
	if [ $exit_on_any_fail ]; then
            exit 1
        else
            continue
        fi
    fi
    echo Successfully engested $mapfile into the database

    esgpublish -i $ini_dir --project $project --map $mapfile_dir/$mapfile --service fileservice --noscan --thredds  --no-thredds-reinit
    if [ $? != 0 ] ; then
        echo Failed to engest $mapfile into thredds
	if [ $exit_on_any_fail ]; then
            exit 1
        else
            continue
        fi
    fi
    echo Successfully engested $mapfile into thredds

    esgpublish --project $project --thredds-reinit

    esgpublish -i $ini_dir --project $project --map $mapfile_dir/$mapfile --service fileservice --noscan --publish
    if [ $? != 0 ] ; then
        echo Failed to publish $mapfile
	if [ $exit_on_any_fail ]; then
            exit 1
        else
            continue
        fi
    fi
    echo Successfully published $mapfile

    mapfiles_published=$(($mapfiles_published + 1))

    mv -v $mapfile_dir/$mapfile $mapfile_done/

done

esgpublish --project $project --thredds-reinit

utcFinal=`date +%s`
elapsed=$(($utcFinal - $utcStart))

echo "All done.  Published $mapfiles_published of $mapfile_count mapfiles.  Elapsed time:  $elapsed seconds" | sendmail $email_recip

publish-db-ingest-only.sh0000750017062300020020000000201013667731036016343 0ustar  bartoletti1climate#!/bin/bash

utcStart=`date +%s`

tag="BGC_CTC_raw - DB Ingest-Only"
mapfile_dir=/p/user_pub/e3sm/staging/mapfiles/mapfiles_bgc_raw
mapfile_done=/p/user_pub/e3sm/staging/mapfiles/mapfiles_archive/BGC/mapfiles_bgc_raw
project=e3sm
email_recip=bartoletti1@llnl.gov

ini_dir=/p/user_pub/e3sm/staging/ini_std

mapfiles=( `ls $mapfile_dir` )
mapfile_count=${#mapfiles[@]}

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: mapfile ingestion to database ($mapfile_count mapfiles)"
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfile_dir/$mapfile --commit-every 100  --no-thredds-reinit
    if [ $? != 0 ] ; then
        echo Failed to engest $mapfile into the database
        exit 1
    fi
    echo Successfully engested $mapfile into the database
done
ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile ingestion to database ($mapfile_count mapfiles)"

utcFinal=`date +%s`
elapsed=$(($utcFinal - $utcStart))

echo "All done ($tag).  Elapsed time:  $elapsed seconds" | sendmail $email_recip

publish-publish-only.sh0000750017062300020020000000323313667731036016145 0ustar  bartoletti1climate#!/bin/bash

if [ $# -ne 1 ]; then
	echo "Please enter a descriptive job_name for tracking purposes"
	exit 1
fi

tag=`echo $1 | tr ' ' _`


utcStart=`date +%s`
ts=`date +%Y%m%d.%H%M%S`

rlog="rlog-$ts-$tag"

logdir=/p/user_pub/e3sm/bartoletti1/Pub_Work/3_Publish/runlogs
mapfile_indir=/p/user_pub/e3sm/staging/mapfiles/stage_3_publish
mapfile_exdir=/p/user_pub/e3sm/staging/mapfiles/mapfiles_archive/BGC/mapfiles_bgc_raw
ini_dir=/p/user_pub/e3sm/staging/ini_std
project=e3sm
email_recip=bartoletti1@llnl.gov


mapfiles=( `ls $mapfile_indir` )
mapfile_count=${#mapfiles[@]}

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: calling --thredds-reinit ($mapfile_count mapfiles)" > $logdir/$rlog 2>&1

esgpublish --project $project --thredds-reinit >> $logdir/$rlog 2>&1

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: thredds-reinit ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
echo "$ts: INPROCESS: mapfile publication ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfile_indir/$mapfile --service fileservice --noscan --publish >> $logdir/$rlog 2>&1
    if [ $? != 0 ] ; then
        echo Failed to publish $mapfile >> $logdir/$rlog 2>&1
        exit 1
    fi
    echo Successfully published $mapfile >> $logdir/$rlog 2>&1
    mv -v $mapfile_indir/$mapfile $mapfile_exdir/
done

utcFinal=`date +%s`
elapsed=$(($utcFinal - $utcStart))

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile publication ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
echo "$ts: Elapsed time:  $elapsed seconds" >> $logdir/$rlog 2>&1

echo "All done ($tag).  Elapsed time:  $elapsed seconds" | sendmail $email_recip

publish-thredds-n-publish-only.sh0000750017062300020020000000306713667731036020040 0ustar  bartoletti1climate#!/bin/bash

utcStart=`date +%s`

tag=BGC_CTC_raw
mapfile_dir=/p/user_pub/e3sm/staging/mapfiles/mapfiles_bgc_raw
mapfile_done=/p/user_pub/e3sm/staging/mapfiles/mapfiles_archive/BGC/mapfiles_bgc_raw
project=e3sm
email_recip=bartoletti1@llnl.gov

ini_dir=/p/user_pub/e3sm/staging/ini_std

mapfiles=( `ls $mapfile_dir` )
mapfile_count=${#mapfiles[@]}

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: mapfile ingestion to thredds ($mapfile_count mapfiles)"
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfile_dir/$mapfile --service fileservice --noscan --thredds  --no-thredds-reinit
    if [ $? != 0 ] ; then
        echo Failed to engest $mapfile into thredds
        exit 1
    fi
    echo Successfully engested $mapfile into thredds
done
ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile ingestion to thredds ($mapfile_count mapfiles)"

esgpublish --project $project --thredds-reinit

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: mapfile publication ($mapfile_count mapfiles)"
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfile_dir/$mapfile --service fileservice --noscan --publish
    if [ $? != 0 ] ; then
        echo Failed to publish $mapfile
        exit 1
    fi
    echo Successfully published $mapfile
    mv -v $mapfile_dir/$mapfile $mapfile_done/
done
ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile publication ($mapfile_count mapfiles)"

utcFinal=`date +%s`
elapsed=$(($utcFinal - $utcStart))

echo "All done ($tag).  Elapsed time:  $elapsed seconds" | sendmail $email_recip

pub-stat-page-prep.sh0000750017062300020020000000421113667731036015466 0ustar  bartoletti1climate#!/bin/bash

# 
pathhead=$1

catalogeur=/p/user_pub/e3sm/bartoletti1/Pub_Status/catalog_E3SM_PUB.sh

pretag=`date +%s`

for ens in ens1 ens2 ens3 ens4 ens5; do
	for v in v1 v2 v3; do
		$catalogeur $pathhead $ens.$v $pretag
	done
done

genlist=`ls | grep $pretag`
onefile=`ls | grep $pretag | head -1`

newtag=`echo $onefile | cut -f2-3 -d-`
now_utc=`date -u +%Y%m%d.%H%M`
count_pathlist_v="$newtag-$now_utc"

cat ${pretag}* | sort | uniq > $count_pathlist_v

rm ${pretag}*


# Now for part 2:

inputform="count,pubpath"
datasetID="Project.ModelVer.Experiment.Resolution[.F2010_hi_low].Realm.Grid.Outtype.Freq.Ens.vn"

now_utc=`date -u +%Y%m%d.%H%M`

outfile=task-stat-list3-$now_utc

echo "ModVer,Resolution,Ens,Title,Status,Grid,Years,Filecount,Dataset_ID,Dataset_First_File,Dataset_Publication_Path,Notes,Title_Expanded" > $outfile

for aline in `cat $count_pathlist_v`; do

	fcount=`echo $aline | cut -f1 -d,`
	apath=`echo $aline | cut -f2 -d,`
	# avers=`echo $aline | cut -f3 -d,`

	ffile=`ls $apath | head -1`

	# NOTE: requires (pub) env to access ncdump
	# tperiod=`ncdump -h $apath/$ffile | grep "time_period_freq" | cut -f2 -d=`

	dsid=`echo $apath | tr / . | cut -f5- -d.`

	modver=`echo $dsid | cut -f2 -d.`
	experiment=`echo $dsid | cut -f3 -d.`

	if [[ $experiment =~ "F2010" ]]; then
		resstr=`echo $dsid | cut -f4 -d.`
		realm=`echo $dsid | cut -f6 -d.`
		grid=`echo $dsid | cut -f7 -d.`
		outtype=`echo $dsid | cut -f8 -d.`
		freq=`echo $dsid | cut -f9 -d.`
		ens=`echo $dsid | cut -f10 -d.`
	else
		resstr=`echo $dsid | cut -f4 -d.`
		realm=`echo $dsid | cut -f5 -d.`
		grid=`echo $dsid | cut -f6 -d.`
		outtype=`echo $dsid | cut -f7 -d.`
		freq=`echo $dsid | cut -f8 -d.`
		ens=`echo $dsid | cut -f9 -d.`
	fi


	if [ $realm == "atmos" ]; then
		realmcode="atm"
	elif [ $realm == "land" ]; then
		realmcode="lnd"
	elif [ $realm == "ocean" ]; then
		realmcode="ocn"
	elif [ $realm == "sea-ice" ]; then
		realmcode="sice"
	else	realmcode="misc"
	fi

	gridcode="nat"
	if [ $grid != "native" ]; then
		gridcode="reg"
	fi

	title="$realmcode $gridcode $freq"



	echo "$modver,$resstr,$ens,$title,published,$grid,,$fcount,$dsid,$ffile,$apath,$outtype," >> $outfile

done


sproket_search.sh0000750017062300020020000000126413667731036015072 0ustar  bartoletti1climate#!/bin/bash

sprok=/p/user_pub/e3sm/bartoletti1/abin/sproket-linux-0.2.13
config=/p/user_pub/e3sm/bartoletti1/abin/.sproket_config.json

# to obtain ALL file names
# $sprok -config $config -urls.only -no.download -no.verify -y > E3SM_datafiles
# to obtain the datasetIDs
$sprok -config $config -values.for dataset_id_template_ -no.download -no.verify -y > E3SM_datasets

exit 0








$sprok -config $config -field.keys > E3SM_fields

IFS=$'\n'

rm -f E3SM_field_values

for afield in `cat E3SM_Fields_useful`; do
	echo " " >> E3SM_field_values
	echo "FIELD_$afield" >> E3SM_field_values
	echo " " >> E3SM_field_values
	$sprok -config $config -values.for $afield >> E3SM_field_values
done