process_codes-20200604.2155.tar

AA_README_AA0000640017062300020020000000363313666266707013313 0ustar  bartoletti1climate
We can now employ the archive/.cfg/Archive_Map to location and "ls" all files from all archives
for all experiments and dataset types, by appropriate pattern.

The archive assessment intends to determine, for each experiment[ensemble] and dataset type,
the count of matching files, and that the "first" and "last" file found in each matched set
are indeed from the same "set" (have the same name structure) and have rational simulated
date ranges (e.g. 1850_01 to 2014_12).

Files returned have the general structure:

	tar_path/structured_name.sim_date.nc

Because we are using the Archive Map as our key, we know that the "tar_paths" will match for
any first and last file found.  But we need to ensure that "structured_name" is a match, and
that the first and last "sim_dates" are reasonable. This presents two problems:

  1.  The beginning of the structured_name may have a generation-date that varies across the
      dataset, and must be trimmed off if we are to compare the datafile names for matches.

  2.  The structured_name has "dot-delimited" components, and the number of components will
      vary across dataset types, and even for the same dataset type across experiments. This
      makes locating the "sim-year" field often specific to each dataset, and we have created
      a table
		/p/user_pub/e3sm/archive/.cfg/Filename_SimDate_Locator

      that identifies the dot-delimited field for sim-year, given experiment and dataset type


For each "Campaign,Model,Experiment,Ensemble,DatasetType,Count,First,Last", we want to output

either
	Campaign,Model,Experiment,Ensemble,DatasetType,MATCHED,Count,firstDate,LastDate,years_total

or	Campaign,Model,Experiment,Ensemble,DatasetType,NOMATCH,Count,firstStructuredName,LastStructuredName

Anywhere a NO_MATCH occurs, we must modify the Archive_Map to provide more specificity to the entry.


THE PROCESS:

	archive_assess_count_first_last.sh  any_subset_of_Archive_Map_lines


AA_README_APM0000640017062300020020000000630313666266707013444 0ustar  bartoletti1climate
BIG SELF_DEFINED TASK:

	Create a utility that will peruse EVERY archived directory (Experiment-Ensemble), to extract ALL tarred filepaths, to record the tar_paths to the actual files intended per (publish-able) dataset.
	e.g., for DECK,
		(atm nat mon) should return "archive/cam/hist/"  (for which the file_extraction_pattern *.cam.h0* will work beautifully)
	but, for HRv1,
		(atm nat mon) would return "run-0006-01-01-180907--0046-01-01-190111/"		(for years 0006-0045)
				       and "run/"						(for years 0046-0055)

	BOTH of the latter being still inadequate for a simple pattern like *cam.h0*, because those paths also archived other stuff matching *cam.h0*, so upon human inspection, we discover we need

			"run-0006-01-01-180907--0046-01-01-190111/theta"
		and	"run/theta"

	in order that the realm/freq pattern *cam.h0* will finally isolate the desired files.


THE GOAL:
	For any required publication of a [ CAMPAIGN MODEL EXPERIMENT ENSEMBLE (realm gridtype freq) ], we will have a table of (one or more) entries

		CAMPAIGN,MODEL,EXPERIMENT,ENSEMBLE,realm_gridtype_freq,archivePath,tarfilePath+extractionPattern

PREREQS:

	Ensure that archive/.cfg file "Standard_Datatype_Extraction_Patterns" are complete with respect to required publications


The Working Process:

1.  Ensure that the script "archive_path_mapper_stage1.sh" is given the desired file of archive specifications.
    This is usually

	/p/user_pub/e3sm/archive/.cfg/Archive_Locator  (the entire archive)

	(Any file containing just a subset may also be supplied)

2.  Issue ./archive_path_mapper_stage1.sh file_of_archive_specifications
	Output will fill directory "PathsFound" with files of the form
		C:M:E:E:DSTITLE
	e.g.
		BGC:1_1:hist-BCRC:ens1:atm_nat_6hr

	Each file will contain all filenames found in archive using patterns found by the "tail" patterns in the_SDEP.

3.  Issue ./archive_path_mapper_stage2.sh 
	This will examine each file in PathsFound, trim away all lines that begin with known inappropriate path patterns
	and for the paths that remain, list the first and last files found for those patterns.
	Output wil be the file "headset12_lists_first_last", which will contain hundreds of entries of the form

		DECK:1_0:abrupt-4xCO2:ens1:lnd_nat_mon
		    HEADF:archive/lnd/hist/20180215.DECKv1b_abrupt4xCO2.ne30_oEC.edison.clm2.h0.0001-01.nc
		    HEADL:archive/lnd/hist/20180215.DECKv1b_abrupt4xCO2.ne30_oEC.edison.clm2.h0.0156-10.nc
		    HEADF:run/20180215.DECKv1b_abrupt4xCO2.ne30_oEC.edison.clm2.h0.0155-12.nc
		    HEADL:run/20180215.DECKv1b_abrupt4xCO2.ne30_oEC.edison.clm2.h0.0155-12.nc

	One must edit this file MANUALLY to determine which paths to the "First and Last" basenames is the correct one.
	In the case above, this would be "archive/lnd/hist", and since the "lnd_nat_mon" search pattern is *clm2.h0*,
	we need to convert that 5-line entry into

		DECK:1_0:abrupt-4xCO2:ens1:lnd_nat_mon:archive/lnd/hist/*clm2.h0*

	When such editing is completed, the finished file should be named "archive_dataset_map_prelim"

4.  Issue ./archive_path_mapper_stage3.sh
	This will take the output from stage 3, "archive_dataset_map_prelim", together with the Archive_Locator,
	to insert the appropriate archive_path into the final listing of the Archive_Map.




archive_assess_count_first_last.sh0000750017062300020020000000502413666266707020527 0ustar  bartoletti1climate#!/bin/bash

# Usage:  archive_assess_count_first_last.sh archive_map(all or part)
# For the entire archive, use /p/user_pub/e3sm/archive/.cfg/Archive_Map
# For a subset, create a file with selected lines, and give its full path

PURPOSE="1.  Find and List all Default Model Outputs from all Experiment/Ensembles by Holodeck of each archive index.db file"

# format:  Campaign,Model,Experiment,Ensemble,DatasetType,ArchivePath,DatatypeExtractionPattern,Notes
# Full Archive Map file:  /p/user_pub/e3sm/archive/.cfg/Archive_Map

archive_map_file=$1
simdate_locator=/p/user_pub/e3sm/archive/.cfg/Filename_SimDate_Locator

The_Holodeck=/p/user_pub/e3sm/bartoletti1/Pub_Status/ArchiveAssess/Holodeck


zstash_version=`zstash version`
if [ $zstash_version != "v0.4.1" ]; then
	echo "ABORTING:  zstash version is not 0.4.1 or is unavailable"
	exit 1
fi


startTime=`date +%s`

IFS=$'\n'

cd $The_Holodeck

for aline in `cat $archive_map_file`; do

	apath=`echo $aline | cut -f6 -d,`
	if [ $apath == "NAV" ]; then
		continue
	fi

	basetag=`echo $aline | cut -f1-5 -d,`
	pattern=`echo $aline | cut -f7 -d,`


	rm -rf $The_Holodeck/*
	mkdir $The_Holodeck/zstash

	ln -s $apath/index.db $The_Holodeck/zstash/index.db
	
	zstash ls --hpss=none $pattern > /tmp/zstash_manifest

	filecount=`cat /tmp/zstash_manifest | wc -l`
	if [ $filecount -eq 0 ]; then
		echo "$basetag,0,(pattern=$pattern)"
		continue
	fi

	init_file=`cat /tmp/zstash_manifest | head -1`
	last_file=`cat /tmp/zstash_manifest | tail -1`

	# trim tar-path from first and last
	filebase1=`echo $init_file | rev | cut -f1 -d/ | rev`
	filebase2=`echo $last_file | rev | cut -f1 -d/ | rev`

	simdfield=`grep $basetag: $simdate_locator | cut -f2 -d:`
	prevfield=$((simdfield - 1))
	
	sim_date1=`echo $filebase1 | cut -f$simdfield -d.`
	filetest1=`echo $filebase1 | cut -f1-$prevfield -d.`
	sim_date2=`echo $filebase2 | cut -f$simdfield -d.`
	filetest2=`echo $filebase2 | cut -f1-$prevfield -d.`

	# last part eliminates variable leading run_dates
	filecore1=`echo $filetest1 | cut -f2- -d.`
	filecore2=`echo $filetest2 | cut -f2- -d.`

	if [ $filecore1 != $filecore2 ]; then
		echo "$basetag,NO_MATCH,$filecount,$init_file,$last_file"
		echo "                        $filecore1"
		echo "                        $filecore2"
	else
		y1=`echo $sim_date1 | cut -c1-4`
		y2=`echo $sim_date2 | cut -c1-4`
		y1=$((10#$y1))
		y2=$((10#$y2))
		yspan=$((y2 - y1 + 1))
		echo "$basetag,MATCHED,$filecount,$filetest1,$sim_date1,$sim_date2,$yspan"
	fi

done

finalTime=`date +%s`

et=$(($finalTime-$startTime))

echo "Done.  ET = $et"


exit

archive_holodeck_setup_ls-only.sh0000750017062300020020000000136613666266707020256 0ustar  bartoletti1climate#!/bin/bash

# echo "DEBUG:  argcount = $#"

instructions="full path to a leaf archive directory"

zstash_version=`zstash version`
if [ $zstash_version != "v0.4.1" ]; then
	echo "ABORTING:  zstash version is not 0.4.1 or is unavailable"
	exit 1
fi

fulldir=$1

The_Holodeck=/p/user_pub/e3sm/bartoletti1/Pub_Status/ArchiveAssess/Holodeck

IFS=$'\n'

# PLAN:  populate the subordinate "zstash" subdirectory with simlinks to the appropriate tarfiles and index.db file.

echo "Clearing the Holodeck"
echo " "
rm -rf $The_Holodeck/*

zstash_dir=$The_Holodeck/zstash
mkdir $zstash_dir

echo "We will set up holodeck zstash ls for directory $fulldir" 

ln -s $fulldir/index.db $zstash_dir/index.db

echo "Holodeck prepared for entry. Enjoy your simulation!"

exit

archive_holodeck_setup.sh0000750017062300020020000000157213666266707016600 0ustar  bartoletti1climate#!/bin/bash

# echo "DEBUG:  argcount = $#"

instructions="full path to a leaf archive directory"

zstash_version=`zstash version`
if [ $zstash_version != "v0.4.1" ]; then
	echo "ABORTING:  zstash version is not 0.4.1 or is unavailable"
	exit 1
fi

fulldir=$1

The_Holodeck=/p/user_pub/e3sm/bartoletti1/Pub_Status/ArchiveAssess/Holodeck

IFS=$'\n'

# PLAN:  populate the subordinate "zstash" subdirectory with simlinks to the appropriate tarfiles and index.db file.

echo "Clearing the Holodeck"
echo " "
rm -rf $The_Holodeck/*

zstash_dir=$The_Holodeck/zstash
mkdir $zstash_dir

echo "We will setup holodeck zstash for directory $fulldir" 

# major kludge
rm -rf $The_Holodeck/archive

for targ in `ls $fulldir`; do
	# touch nothing
	# echo "ln -s $fulldir/$targ zstash/$targ"
	ln -s $fulldir/$targ $zstash_dir/$targ
done

echo "Holodeck prepared for entry. Enjoy your simulation!"

exit

archive_path_mapper_stage1.sh0000750017062300020020000000346313666266707017335 0ustar  bartoletti1climate#!/bin/bash

# Usage:  archive_path_mapper_stage1.sh achive_locations_file
# For the entire archive, use /p/user_pub/e3sm/archive/.cfg/Archive_Locator
# For a subset, create a file with selected lines, and give its full path

PURPOSE="1.  Find and List all Default Model Outputs from all Experiment/Ensembles by Holodeck of each archive index.db file"

# format:  Campaign,Model,Experiment,Ensemble,ArchivePath,
# Full Archve Location file:  /p/user_pub/e3sm/archive/.cfg/Archive_Locator

archive_loc_file=$1

the_SDEP=/p/user_pub/e3sm/archive/.cfg/Standard_Datatype_Extraction_Patterns

WorkDir=/p/user_pub/e3sm/bartoletti1/Pub_Status/ArchivePathMapper/
The_Holodeck=$WorkDir/Holodeck


zstash_version=`zstash version`
if [ $zstash_version != "v0.4.1" ]; then
	echo "ABORTING:  zstash version is not 0.4.1 or is unavailable"
	exit 1
fi

mv $WorkDir/PathsFound/* $WorkDir/PrevPathsFound/

startTime=`date +%s`

IFS=$'\n'

cd $The_Holodeck

# Archive_Locator Entry processed

ALE=0

for aline in `cat ../$archive_loc_file`; do

	apath=`echo $aline | cut -f5 -d,`
	if [ $apath == "NAV" ]; then
		continue
	fi

	ALE=$(($ALE + 1))

	basetag=`echo $aline | cut -f1-4 -d, | tr , :`

	# echo "ALE:BaseTag = $ALE:$basetag"

	rm -rf $The_Holodeck/*
	mkdir $The_Holodeck/zstash

	ln -s $apath/index.db $The_Holodeck/zstash/index.db
	
	for fpatspec in `cat $the_SDEP`; do
		dstitle=`echo $fpatspec | cut -f1 -d, | tr ' ' _`
		dspattern=`echo $fpatspec | cut -f2 -d,`
		if [ $dspattern == "ignore" ]; then
			continue
		fi
		outfile="$ALE:$basetag:$dstitle"
		# echo "        $outfile"
		zstash ls --hpss=none $dspattern > ../PathsFound/$outfile	
		if [ -s ../PathsFound/$outfile ]; then
			continue
		fi
		echo "EMPTY: $outfile"
		rm ../PathsFound/$outfile
	done

done

finalTime=`date +%s`

et=$(($finalTime-$startTime))

echo "Done.  ET = $et"


exit

archive_path_mapper_stage2.sh0000750017062300020020000000147313666266707017335 0ustar  bartoletti1climate#!/bin/bash

# for each named file in PathsFound

flist=`ls PathsFound`

rm -f headset12_lists_first_last

for afile in `ls PathsFound`; do
	echo $afile >> headset12_lists_first_last
	# the list of grep -v exclusions here may need future additions
	headlist=`cat PathsFound/$afile | cut -c1-12 | sort | uniq | grep -v "^rest/" | grep -v "^post/" | grep -v "^test" | grep -v "^init" | grep -v "^run/try" | grep -v "^run/bench" | grep -v "^old/run" | grep -v "^pp/remap" | grep -v "^a-prime" | grep -v "^lnd_rerun" | grep -v "^atm/ncdiff" | grep -v "^archive/rest"`
	for ahead in $headlist; do
		samp=`grep ^$ahead PathsFound/$afile | sort | head -1`
		echo "    HEADF:$samp" >> headset12_lists_first_last
		samp=`grep ^$ahead PathsFound/$afile | sort | tail -1`
		echo "    HEADL:$samp" >> headset12_lists_first_last
	done
done
archive_path_mapper_stage3.sh0000750017062300020020000000130113666266707017324 0ustar  bartoletti1climate#!/bin/bash

infile=archive_dataset_map_prelim

archive_loc=/p/user_pub/e3sm/archive/.cfg/Archive_Locator

for aline in `cat $infile`; do
	pureline=`echo $aline | cut -f2- -d:`
	headpart=`echo $pureline | cut -f1-4 -d:`
	ds_title=`echo $pureline | cut -f5 -d:`
	epattern=`echo $pureline | cut -f6 -d:`

	seek=`echo $headpart | tr : ,`
	found=`cat $archive_loc | grep $seek`

	lc=`echo $found | wc -l`
	if [ $lc -eq 0 ]; then
		echo "ERROR: no archive location found for $seek"
		continue
	fi
	if [ $lc -gt 1 ]; then
		echo "WARNING: multiple archive locations found for $seek"
	fi
	for apath in $found; do
		the_path=`echo $apath | cut -f5 -d,`
		echo "$seek,$ds_title,$the_path,$epattern"
	done
done

	

archive_status_prep.sh0000750017062300020020000001105213666266707016133 0ustar  bartoletti1climate#!/bin/bash

# Usage:  archive_status_prep.sh archive_map(all or part)
#
# For the entire archive, use /p/user_pub/e3sm/archive/.cfg/Archive_Map
# For a subset, create a file with selected lines, and give its full path
# format:  Campaign,Model,Experiment,Ensemble,DatasetType,ArchivePath,DatatypeExtractionPattern,Notes
# Full Archive Map file:  /p/user_pub/e3sm/archive/.cfg/Archive_Map


# PURPOSE:  From input Archive Map
#  campaign, model, experiment, ens, title, archive_path, tar_extract_pattern
#
#  Produce a listing
# 	model, resolution, ens, Title, status, grid, years, filecount, datasetID, datasetFirstFile, DatasetPubPath,Notes, titleExpanded, misc
#
#  THe following table outlines the source of each item
#
#	model			Direct from Archive Map
#	resolution		use "$default_res" below
#	ensemble		Direct from Archive Map
#	title			extract from Archive Map "DatasetType"
#	status			[HC] "arch"
#	grid			[HC] "native"
#	years			[HC] ""
#	filecount		Generated by inspection
#	datasetID		Constructed from all of these values and sources
#	firstFile		Generated by inspection
#	pubPath			Constructed from datasetID
#	notes			[HC] "model-output"
#	titleExpanded		reit title
#	misc			[HC] ""

#  A datasetID needs
#	E3SM.<model>.<experiment>.<resolution>.<realm>.<grid>.model_output.<freq>.<ens>

archive_map_file=$1
simdate_locator=/p/user_pub/e3sm/archive/.cfg/Filename_SimDate_Locator

The_Holodeck=/p/user_pub/e3sm/bartoletti1/Pub_Status/ArchiveAssess/Holodeck


zstash_version=`zstash version`
if [ $zstash_version != "v0.4.1" ]; then
	echo "ABORTING:  zstash version is not 0.4.1 or is unavailable"
	exit 1
fi

default_res="1deg_atm_60-30km_ocean"

startTime=`date +%s`

IFS=$'\n'

cd $The_Holodeck

for aline in `cat $archive_map_file`; do

	model=`echo $aline | cut -f2 -d,`
	experiment=`echo $aline | cut -f3 -d,`
	resolution=$default_res
	ens=`echo $aline | cut -f4 -d,`
	title=`echo $aline | cut -f5 -d, | tr _ ' '` # may need tr _ ' '
	realmcode=`echo $title | cut -f1 -d' '`
	freq=`echo $title | cut -f3 -d' '`
	pstatus="arch"
	grid="native"
	years=""
	notes="model-output"
	otype=$notes
	misc=""

	# translate realmcode to realm
	if [ $realmcode == "atm" ]; then
		realm="atmos"
	elif [ $realmcode == "lnd" ]; then
		realm="land"
	elif [ $realmcode == "ocn" ]; then
		realm="ocean"
	elif [ $realmcode == "river" ]; then
		realm="river"
	elif [ $realmcode == "sea-ice" ]; then
		realm="sea-ice"
	else realm="UNK_$realmcode"
	fi

	datasetID="E3SM.$model.$experiment.$resolution.$realm.$grid.$otype.$freq.$ens"
	pubPath=`echo $datasetID | tr . /`

	# echo "DEBUG: dsID=$datasetID"
	# continue

	# still need to formulate filecount, firstFile, (titleExpanded)

	apath=`echo $aline | cut -f6 -d,`
	if [ $apath == "NAV" ]; then
		continue
	fi

	basetag=`echo $aline | cut -f1-5 -d,`
	pattern=`echo $aline | cut -f7 -d,`

	# echo "DEBUG:  Basetag=$basetag"

	rm -rf $The_Holodeck/*
	mkdir $The_Holodeck/zstash

	ln -s $apath/index.db $The_Holodeck/zstash/index.db
	
	zstash ls --hpss=none $pattern > /tmp/zstash_manifest

	filecount=`cat /tmp/zstash_manifest | wc -l`

	init_file=`cat /tmp/zstash_manifest | head -1`
	last_file=`cat /tmp/zstash_manifest | tail -1`

	# trim tar-path from first and last
	filebase1=`echo $init_file | rev | cut -f1 -d/ | rev`
	filebase2=`echo $last_file | rev | cut -f1 -d/ | rev`

	simdfield=`grep $basetag: $simdate_locator | cut -f2 -d:`
	prevfield=$((simdfield - 1))
	
	sim_date1=`echo $filebase1 | cut -f$simdfield -d.`
	filetest1=`echo $filebase1 | cut -f1-$prevfield -d.`
	sim_date2=`echo $filebase2 | cut -f$simdfield -d.`
	filetest2=`echo $filebase2 | cut -f1-$prevfield -d.`

	# last part eliminates variable leading run_dates
	filecore1=`echo $filetest1 | cut -f2- -d.`
	filecore2=`echo $filetest2 | cut -f2- -d.`

	if [ $filecore1 != $filecore2 ]; then
		# echo "$basetag,NO_MATCH,$filecount,$init_file,$last_file"
		echo "DEBUG:                  $filecore1"
		echo "DEBUG:                  $filecore2"
		years="NAV"
		
	else
		y1=`echo $sim_date1 | cut -c1-4`
		y2=`echo $sim_date2 | cut -c1-4`
		m2=`echo $sim_date2 | cut -c6-7`
		y1=$((10#$y1))
		y2=$((10#$y2))
		yspan=$((y2 - y1 + 1))
		if [ $m2 == "01" ]; then
			yspan=$((y2 - y1))
		fi
		# echo "$basetag,MATCHED,$filecount,$filetest1,$sim_date1,$sim_date2,$yspan"
		years=$yspan
	fi

	# 	model, resolution, ens, Title, status, grid, years, filecount, datasetID, datasetFirstFile, DatasetPubPath,Notes, titleExpanded, misc
	echo "$model,$resolution,$ens,$title,$pstatus,$grid,$years,$filecount,$datasetID,$filebase1,$pubPath,$otype,$title,$misc"

done

finalTime=`date +%s`

et=$(($finalTime-$startTime))

echo "Done.  ET = $et"


exit

Automation_via_Holodeck0000640017062300020020000001636413666266706016247 0ustar  bartoletti1climateThe Nearly Automated Dataset Extraction and (Pre)Publication Process

TL;DR:
	IMPORTANT:  Due to values hard-coded into the scripts, one must invoke this "automated" process
		for each combination of (realm,freq_set,resolution,dataset_version)

	1.  Create a file "the_target_archives" consisting of lines selected from the file

		/p/user_pub/e3sm/archive/.cfg/Archive_Locator

		The lines listed must cover all cases (experiment,ensembles) for all intended datasets 

	2.  Edit the "publication_staging_control_script.sh" to set the values for these two variables

		dstype_static="atm nat"	(EXAMPLE)
		dstype_freq_list="mon day 6hr_snap day_cosp" (EXAMPLE)

		As we are extracting from archives, the first variable will always be "<realm> nat".
		The second can be any space-separated set of frequency-codes.
		See the file: /p/user_pub/e3sm/archive/.cfg/Standard_Datatype_Extraction_Patterns for options.

	3.  Edit the "" script to set the values for these two variables

		resolution=1deg_atm_60-30km_ocean
		pubvers=v1

	4.  Invoke

		nohup ./publication_staging_control_script.sh the_target_archives &

		A log file "Holodeck_Process_Log-<date_time>" will be produced to track progress.

	If datasets for multiple realms is required, you will need to repeat Step 2 to reflect a different realm,
	and re-invoke Step 3.


BACKGROUND:


E3SM Publication Staging is now operating under a new premise:  All data archives are now local and amenable to zstash extraction.

I have crafted a working - if fragile - automated extraction regime that can access multiple experiments and cases, locate their archive directories and file extraction patterns, create the faceted publication destination directories, and successfully extract and move the dataset files to the intended publication directories.  I would caution that the process is not yet entirely generalized, and should be reimplemented at some point in a fixed python application, once the dust settles on a stable procedure.

The following provides background and detailed explanation of the process.  As I write this, it is already staging all of the DECK cam.h[1-4] datasets, for all experiments previously published.

BACKGROUND:  File Directory Structure and Configurations

The LLNL E3SM Archives presently have the following location and structure:

	/p/user_pub/e3sm/archive/
				1_0/
					DECK/
						(case_leaf)/tarfiles + index.db
						(case_leaf)/tarfiles + index.db
						...
					HiResMIP/
						(case_leaf)/tarfiles + index.db
						(case_leaf)/tarfiles + index.db
						...
				1_1/
					BGC/
						(ditto)
				1_1_ECA/
					BGC/
						(ditto)

	Note that "case_leaf" is a directory name uniquly identifying an experiment and ensemble.
	EXAMPLE: 20180622.DECKv1b_A2_1850aeroF.ne30_oEC.edison  (here, A2 = AMIP ens2)
	These directory names exactly match the author-supplied NERSC HPSS leaf directory names.
	This practice facilitates point-to-point Globus transfers.

	Ideally, the "case-leaf" directory should contain nothing but the tar files and the zstash database file "index.db".  In practice, it may contain other material, but no further directory traversal should be required to locate the available tar files.

In order to provide the opportunity for various "staging" dataset checks or modifications, the files are extracted to 

	/p/user_pub/e3sm/staging/pre_pub/(facets)/(files.nc)

where (facets)/(files.nc) matches the structure of the eventual publication directories.

We can extract and move the dataset files to publication using zstash, while leaving the archive intact and clean, by employing a "holodeck" of symlinks. (Credit to Sterling Baldwin for schooling me on the virtues of symlinks. I merely take credit for the use of the term "holodeck" in this context.)

With any empty directory serving as the "holodeck", the following depicts the sequence of operations leading to extraction and movement of dataset files to their intended publication directory:

1. Begin with empty directory, call this the $HolodeckPath

	/path/to/holodeck/

2. Create a zstash subdirectory

	/path/to/holodeck/
			zstash/

3. Having located the appropriate "case-leaf" archive directory, form a list of the contained tar file names (and index.db) and their full paths, and loop over each to create in the zstash subdirectory the necessary symlinks

	/path/to/holodeck/
			zstash/
				tarfilename --> symlink to full tarfile path
				tarfilename --> symlink to full tarfile path
				...
				index.db --> symlink to full index.db path

4. With the holodeck as the working directory, and the necessary datafile extraction pattern in hand, one can issue the zstash extract command to obtain

	/path/to/holodeck/
			stored/path/to/
				datafile
				datafile
				...
				datafile

			zstash/
				tarfilename --> symlink to full tarfile path
				tarfilename --> symlink to full tarfile path
				...
				index.db --> symlink to full index.db path

5. Having previously constructed the staging/prepub/(facets)/ destination path (targ_pub_dir="$prepubroot/$modelver/$exp_name/$midpath/$freq/$ensemble/$pubvers") and ensured it is empty (may be overridden with "overwriteFlag=1", and cognizant of the extracted tarpath-head (e.g. "stored/path/to/"), one can now issue

	mv stored/path/to/* pub_dest_path
	chmod 644 pub_dest_path/*

   leaving you

	/path/to/holodeck/
			stored/path/to/
				(empty)

			zstash/
				tarfilename --> symlink to full tarfile path
				tarfilename --> symlink to full tarfile path
				...
				index.db --> symlink to full index.db path

6. At this point, the dataset has been fully staged for pre-publication inspection and possible modifications.

   If there are additional datasets to publish from this same archive "case-leaf", one need only issue

	rm -rf $HolodeckPath/stored/

   and repeat steps 4 and 5 with another datafile_extraction_pattern, with working directory the $HolodeckPath with existing zstash symlinks.  

   If instead there are other cases for which datasets exist to be published, issue

	rm -rf $HolodeckPath/*

   and repeat from step 2 with a new case-leaf and associated datafile_extraction_patterns



IN SUMMARY:

With the "holodeck" as the working directory, a script given the desired (model,campaign,case,ensemble,datasetSpec) will need to

	1. Construct and test for existence and population of the target pre-publication directory.
		- Skip if the directory exists and is non-empty, unless the script overwriteFlag=1.
	2. Access the Archive_Map to obtain the information regarding the archive source data
		- Ensure exactly 1 line returned from CMEE_archive_map and CMEE_archive_map_augment
	3. populate the subordinate "zstash" subdirectory with simlinks to the appropriate tarfiles and index.db file.
		- Ensure holodeck contains only empty zstash subdirectory
	4. execute the zstash extraction command, with the appropriate file-selection pattern.
	5. create the necessary target publication directory
	6. move the extracted files to their publication directory
	7. clean the holodeck by removing all zstash symlinks and non-zstash directories

STEP 0:

	must contruct and test $pub_head_dir/$thisModel/$thisExp/$path_mid/$freq/$ens/v1
	where
		pub_head_dir=/p/user_pub/work/E3SM
		thisModel (supplied)
		thisExp   (supplied)
		path_mid  (supplied)	e.g. 1deg_atm_60-30km_ocean/atmos/native/model-output
		freq      (supplied)
		ens       (supplied)
		ver       (supplied)
dirperms.sh0000750017062300020020000000014413666266707013706 0ustar  bartoletti1climate#!/bin/bash

dirlist=$1

for aline in `cat $dirlist`; do
	chmod 755 $aline
	chmod 644 $aline/*
done
holodeck_stage_publication.sh0000750017062300020020000000774413666266706017441 0ustar  bartoletti1climate#!/bin/bash

# EDIT THIS VARIABLE!
# resolution=0_25deg_atm_18-6km_ocean
resolution=1deg_atm_60-30km_ocean
pubvers=v1
overwriteFlag=1


Datatype_Patterns=/p/user_pub/e3sm/archive/.cfg/Standard_Datatype_Extraction_Patterns
The_Holodeck=/p/user_pub/e3sm/bartoletti1/Pub_Work/1_Refactor/Holodeck

Archive_Map=/p/user_pub/e3sm/archive/.cfg/Archive_Map
prepubroot=/p/user_pub/e3sm/staging/prepub
pubroot=/p/user_pub/work/E3SM

# echo "DEBUG:  argcount = $#"

if [ $# -ne 2 ]; then
	echo " "
	echo "    Usage:  holodeck_stage_publication.sh cmee_am_selection_line datasetspec"
	echo "            (a line selected from archive/.cfg/Archive_Locator)"
	echo "            Give datasetspec as \"realm grid freq\", as in \"atm nat mon\", in quotations."
	echo "            See the file /p/user_pub/e3sm/archive/.cfg/Standard_Datatype_Extraction_Patterns"
	echo " "
	exit 0
fi

# echo "DEBUG: DOLLAR_1 = $1"
# echo "DEBUG: DOLLAR_2 = $2"


zstash_version=`zstash version`
if [ $zstash_version != "v0.4.1" ]; then
	echo "ABORTING:  zstash version is not 0.4.1 or is unavailable"
	exit 1
fi

IFS=$'\n'


AL_selection_line=$1

AL_path=`echo $AL_selection_line | cut -f5 -d,`
AL_key=`echo $AL_selection_line | cut -f1-4 -d,`

ds_spec=$2	# realm-code nat freq_code, e.g. "atm nat mon", from Standard_Datatype_Extraction_Patterns
ds_key=`echo $ds_spec | tr ' ' _`

AM_key="$AL_key,$ds_key"
# echo "Produced AM_key: $AM_key"

# Determine realm for  midpath
realm_code=`echo $ds_spec | cut -f1 -d' '`

if [ $realm_code == "atm" ]; then
	realm="atmos"
elif [ $realm_code == "lnd" ]; then
	realm="land"
elif [ $realm_code == "ocn" ]; then
	realm="ocean"
elif [ $realm_code == "river" ]; then
	realm="river"
elif [ $realm_code == "sea-ice" ]; then
	realm="sea-ice"
else
	echo "ERROR: unrecognized realm code: $realm_code"
	exit 1
fi


midpath=$resolution/$realm/native/model-output

AM_list=`grep $AM_key $Archive_Map | sort`

listcount=`echo $AM_list | wc -l`

echo "Matched $listcount AM lines"
# may activate multiple Archive_Map lines, must loop.

cd $The_Holodeck

for am_line in $AM_list; do

	# echo "DEBUG: am_line = $am_line"

	campaign=`echo $am_line | cut -f1 -d,`
	modelver=`echo $am_line | cut -f2 -d,`
	exp_name=`echo $am_line | cut -f3 -d,`
	ensemble=`echo $am_line | cut -f4 -d,`
	arch_dir=`echo $am_line | cut -f6 -d,`
	tar_patt=`echo $am_line | cut -f7 -d,`

	# PLAN Step 1: test and exit if publication directory already exists.

	freq=`echo $ds_spec | cut -f3 -d' '`
	targ_pub_dir="$prepubroot/$modelver/$exp_name/$midpath/$freq/$ensemble/$pubvers"

	# echo "DEBUG: freqword = $freq"

	if [ $overwriteFlag -eq 0 ]; then
		if [ -d $targ_pub_dir ]; then
			fc=`ls $targ_pub_dir | wc -l`
			if [ $fc -gt 0 ]; then
				echo "ABORT: PUB DIR EXISTS ($fc files) $targ_pub_dir"
				echo " "
				continue
			fi
		fi
	fi

	# PLAN Step 2: Use the supplied Archive_Locator line to obtain the information regarding the archive source data

	echo "Conducting zstash extract for directory $arch_dir with dataset extraction pattern $tar_patt"
	echo "Target Publication Dir:  $targ_pub_dir"

	# continue



	# Real Stuff Below

	# PLAN Step 3:  populate the subordinate "zstash" subdirectory with simlinks to the appropriate tarfiles and index.db file.
	#    - Ensure holodeck contains only empty zstash subdirectory

	echo "Cleaning the Holodeck"
	echo " "
	rm -rf $The_Holodeck/*

	zstash_dir=$The_Holodeck/zstash
	mkdir $zstash_dir

	for targ in `ls $arch_dir`; do
		# touch nothing
		# echo "ln -s $arch_dir/$targ zstash/$targ"
		ln -s $arch_dir/$targ $zstash_dir/$targ
	done

	echo "zstash extract --hpss=none $tar_patt"
	zstash extract --hpss=none $tar_patt
	exitcode=$?

	if [ $exitcode -ne 0 ]; then
		echo "ERROR:  zstash returned exitcode $exitcode"
		exit $exitcode
	fi


	echo "MKDIR: mkdir -p $targ_pub_dir"
	mkdir -p $targ_pub_dir
	chmod 755 $targ_pub_dir

	echo "mv $The_Holodeck/$file_extract_pattern $targ_pub_dir"
	mv $The_Holodeck/$tar_patt $targ_pub_dir
	chmod 644 $targ_pub_dir/*
	echo " "

	echo "Cleaning the Holodeck"
	echo " "
	rm -rf $The_Holodeck/*
done

exit 0
list_pub_dirs.sh0000750017062300020020000000140113666266706014717 0ustar  bartoletti1climate#!/bin/bash

mod_exp_camp=/p/user_pub/e3sm/bartoletti1/Model_Experiment_Campaign

pub_head_dir=/p/user_pub/work/E3SM

path_mid=1deg_atm_60-30km_ocean/atmos/native/model-output

for campaign in BGC DECK; do

	model_experiments=`grep $campaign $mod_exp_camp | cut -f1,2 -d,`
 
	for mod_exp in $model_experiments; do

		thisModel=`echo $mod_exp | cut -f1 -d,`
		thisExp=`echo $mod_exp | cut -f2 -d,`

		# find all ensembles
		ensembles=`find $pub_head_dir -type d | grep $thisModel | grep "/$thisExp/" | grep "/ens" | cut -f13 -d/ | sort | uniq`
		
		for ens in $ensembles; do

			for freq in day 6hr_ave 6hr_snap 3hr; do
				# echo "($campaign) $exp $freq $ens"
				echo "($campaign) $pub_head_dir/$thisModel/$thisExp/$path_mid/$freq/$ens/v1"
			done
		done
	done
done


make_mapfile.sh0000750017062300020020000000067113666266707014500 0ustar  bartoletti1climate#!/bin/bash

# FORMAT
# esgmapfile make -i /path/to/ini/directory --max-processes <some_number_of_cores> --project <e3sm/cmip6> --outdir /output/location <path_to_data_directory>

dataset_fullpath=$1

echo "make_mapfile: processing: $dataset_fullpath"
esgmapfile make --debug -i /p/user_pub/e3sm/staging/ini_std/ --max-processes 20 --project e3sm --outdir /p/user_pub/e3sm/staging/mapfiles/mapfiles_auto_publish $dataset_fullpath

exit $?


multi_mapfile_generate.sh0000750017062300020020000000176613666266707016575 0ustar  bartoletti1climate#!/bin/bash

workpath=/p/user_pub/e3sm/bartoletti1/Pub_Work/2_Mapwork

if [ $# -ne 1 ]; then
	echo "Usage:  multi_mapfile_Generate.sh map_status_control_file"
	exit 1
fi

map_status_file=$1

datasets=`grep -v DONE $map_status_file | grep -v HOLD`

startTime=`date +%s`

ts=`date +%Y%m%d.%H%M%S`
rlog=mfg_runlog-$ts

setcount=0;

for dataset in $datasets; do
	ts=`date +%Y%m%d.%H%M%S`
	ds_tm1=`date +%s`
	echo "STATUS:$ts:INPROCESS: dataset $dataset" >> $rlog 2>&1
	$workpath/make_mapfile.sh $dataset >> $rlog 2>&1
	ds_tm2=`date +%s`
	ds_et=$(($ds_tm2 - $ds_tm1))

	ts=`date +%Y%m%d.%H%M%S`
	if [ $? -eq 0 ]; then
		echo "STATUS:$ts:COMPLETED: dataset $dataset" >> $rlog 2>&1
	else
		echo "STATUS:$ts:FAILURE:   dataset $dataset (exit code $?)" >> $rlog 2>&1
	fi
	echo "STATUS: dataset ET = $ds_et" >> $rlog 2>&1
	setcount=$(($setcount + 1))
done

finalTime=`date +%s`
et=$(($finalTime - $startTime))

echo " " >> $rlog 2>&1
echo "Processed $setcount datasets." >> $rlog 2>&1
echo "Elapsed Time: $et" >> $rlog 2>&1
publication_staging_control_script.sh0000750017062300020020000000353113666266707021235 0ustar  bartoletti1climate#!/bin/bash

# EDIT THESE AS NEEDED
dstype_static="atm nat"
dstype_freq_list="day_cosp"


holodeck_stager="/p/user_pub/e3sm/bartoletti1/Pub_Work/1_Refactor/holodeck_stage_publication.sh"


rm -f nohup.out

echo " "
echo "# Publication Staging Controller #"
echo " "

if [ $# -eq 0 ]; then
	echo "usage: (this_script) file_of_Archive_Locator_selectedlines"
	exit
fi

if [ $1 == "help" ]; then
	echo "usage: (this_script) file_of_Archive_Locator_selectedlines"
	exit
fi

# file of Archive_Locator lines
AL_selected=$1


startTime=`date +%s`
ts=`date +%Y%m%d.%H%M%S`
holodeck_log=/p/user_pub/e3sm/bartoletti1/Pub_Work/1_Refactor/Holodeck_Process_Log-$ts

# Here, we prepare to process one or more Archive_Locator lines, even when multiple lines may refer to 
# different parts of a single dataset

casecount=0;

for AL_line in `cat $AL_selected`; do

	echo " " >> $holodeck_log 2>&1
	echo "$ts:Publication Staging Controller: Archive_Locator_line = $AL_line" >> $holodeck_log 2>&1
	echo " "

	datasets=0;
	# For the current case (Experiment / Ensemble), we are prepared only to pull one or more
	# frequencies for a single realm (e.g.  cam.h1, h2, h3 ...)

	ts=`date +%Y%m%d.%H%M%S`
	echo "$ts:Processing case spec: $AL_line" >> $holodeck_log 2>&1
	echo " "
	for freq in $dstype_freq_list; do
		argslist=()
		argslist[0]=$AL_line
		dstype="$dstype_static $freq"
		argslist[1]=$dstype
		ts=`date +%Y%m%d.%H%M%S`
		echo "$ts: Calling holodeck_stager with $AL_line \"$dstype\"" >> $holodeck_log 2>&1
		$holodeck_stager "${argslist[@]}" >> $holodeck_log 2>&1
		datasets=$(($datasets + 1))
	done
	ts=`date +%Y%m%d.%H%M%S`
	echo "$ts:Completion case spec: $AL_line ($datasets datasets)" >> $holodeck_log 2>&1
	casecount=$(($casecount + 1))
done

finalTime=`date +%s`
et=$(($finalTime - $startTime))

echo "Elapsed time: $et seconds.  ($casecount cases processed)" >> $holodeck_log


exit
	
publish-all-stages-autodelete.sh0000750017062300020020000001064713666266707017723 0ustar  bartoletti1climate#!/bin/bash

if [ $# -ne 1 ]; then
        echo "Please enter a descriptive job_name for tracking purposes"
        exit 1
fi

tagname=`echo $1 | tr ' ' _`


utcStart=`date +%s`
ts=`date +%Y%m%d.%H%M%S`

rlog="rlog-$ts-$tagname"

logdir=/p/user_pub/e3sm/bartoletti1/Pub_Work/3_Publish/runlogs
mapfiles_stage1=/p/user_pub/e3sm/staging/mapfiles/stage_1_db_ingest
mapfiles_stage2=/p/user_pub/e3sm/staging/mapfiles/stage_2_thredds_ingest
mapfiles_stage3=/p/user_pub/e3sm/staging/mapfiles/stage_3_publish
mapfiles_aborted=/p/user_pub/e3sm/staging/mapfiles/mapfiles_hold
mapfiles_archive=/p/user_pub/e3sm/staging/mapfiles/mapfiles_archive
ini_dir=/p/user_pub/e3sm/staging/ini_std
project=e3sm
email_recip=bartoletti1@llnl.gov

mapfiles=( `ls $mapfiles_stage1 | grep -v .part` )
mapfile_count=${#mapfiles[@]}


ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: mapfile ingestion to database ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
for mapfile in "${mapfiles[@]}"
do
	#check for deletion needed

	telltale=`echo $mapfile | rev | cut -f1-2 -d. | rev`
	vnum=`echo $telltale | cut -f1 -d. | cut -c2-`
	if [ $vnum -gt 1 ]; then
		vpre=$(($vnum - 1))
		echo "version is $vnum, deleting version $vpre"
		newtail="#$vpre"
		echo $mapfile > tmp_mystring
		sedcmd=`echo "s/.$telltale/$newtail/g"`
		sed -i $sedcmd tmp_mystring
		to_delete=`cat tmp_mystring`
		echo "$ts: INPROCESS: Calling esgunpublish for $to_delete" >> $logdir/$rlog 2>&1
		esgunpublish --project e3sm --database-delete --delete $to_delete >> $logdir/$rlog 2>&1
		ts=`date +%Y%m%d.%H%M%S`
		echo "$ts: COMPLETED: Deletion of dataset $to_delete" >> $logdir/$rlog 2>&1
	fi


    esgpublish -i $ini_dir --project $project --map $mapfiles_stage1/$mapfile --replace --commit-every 100  --no-thredds-reinit >> $logdir/$rlog 2>&1
    retcode=$?
    ts=`date +%Y%m%d.%H%M%S`
    if [ $retcode != 0 ] ; then
        echo "$ts: -- Failed to ingest $mapfile into the database" >> $logdir/$rlog 2>&1
	mv $mapfiles_stage1/$mapfile $mapfiles_aborted/$mapfile
        continue
    fi
    echo "$ts: -- Successfully ingested $mapfile into the database" >> $logdir/$rlog 2>&1
    mv $mapfiles_stage1/$mapfile $mapfiles_stage2/$mapfile
done

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile ingestion to database ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

mapfiles=( `ls $mapfiles_stage2` )
mapfile_count=${#mapfiles[@]}

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: mapfile ingestion to thredds ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfiles_stage2/$mapfile --service fileservice --noscan --thredds  --no-thredds-reinit >> $logdir/$rlog 2>&1
    retcode=$?
    ts=`date +%Y%m%d.%H%M%S`
    if [ $retcode != 0 ] ; then
        echo "$ts: -- Failed to ingest $mapfile into thredds" >> $logdir/$rlog 2>&1
	mv $mapfiles_stage2/$mapfile $mapfiles_aborted/$mapfile
	continue
    fi
    echo "$ts: -- Successfully ingested $mapfile into thredds" >> $logdir/$rlog 2>&1
    mv $mapfiles_stage2/$mapfile $mapfiles_stage3/$mapfile
done

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile ingestion to thredds ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: calling --thredds-reinit ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

esgpublish --project $project --thredds-reinit >> $logdir/$rlog 2>&1

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: thredds-reinit ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

mapfiles=( `ls $mapfiles_stage3` )
mapfile_count=${#mapfiles[@]}

echo "$ts: INPROCESS: mapfile publication ($mapfile_count mapfiles)" >> $logdir/$rlog
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfiles_stage3/$mapfile --service fileservice --noscan --publish >> $logdir/$rlog 2>&1
    retcode=$?
    ts=`date +%Y%m%d.%H%M%S`
    if [ $retcode != 0 ] ; then
        echo "$ts: -- Failed to publish $mapfile" >> $logdir/$rlog 2>&1
	mv $mapfiles_stage3/$mapfile $mapfiles_aborted/$mapfile
        continue
    fi
    echo Successfully published $mapfile >> $logdir/$rlog
    mv $mapfiles_stage3/$mapfile $mapfiles_archive/$mapfile
done



utcFinal=`date +%s`
elapsed=$(($utcFinal - $utcStart))

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile publication ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
echo "$ts: Elapsed time:  $elapsed seconds" >> $logdir/$rlog 2>&1

echo "All done ($tagname).  Elapsed time:  $elapsed seconds" | sendmail $email_recip

publish-all-stages.sh0000750017062300020020000000741413666266707015570 0ustar  bartoletti1climate#!/bin/bash

if [ $# -ne 1 ]; then
        echo "Please enter a descriptive job_name for tracking purposes"
        exit 1
fi

tagname=`echo $1 | tr ' ' _`


utcStart=`date +%s`
ts=`date +%Y%m%d.%H%M%S`

rlog="rlog-$ts-$tagname"

logdir=/p/user_pub/e3sm/bartoletti1/Pub_Work/3_Publish/runlogs
mapfiles_stage1=/p/user_pub/e3sm/staging/mapfiles/stage_1_db_ingest
mapfiles_stage2=/p/user_pub/e3sm/staging/mapfiles/stage_2_thredds_ingest
mapfiles_stage3=/p/user_pub/e3sm/staging/mapfiles/stage_3_publish
mapfiles_aborted=/p/user_pub/e3sm/staging/mapfiles/mapfiles_hold
mapfiles_archive=/p/user_pub/e3sm/staging/mapfiles/mapfiles_archive
ini_dir=/p/user_pub/e3sm/staging/ini_std
project=e3sm
email_recip=bartoletti1@llnl.gov

mapfiles=( `ls $mapfiles_stage1 | grep -v .part` )
mapfile_count=${#mapfiles[@]}

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: mapfile ingestion to database ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfiles_stage1/$mapfile --replace --commit-every 100  --no-thredds-reinit >> $logdir/$rlog 2>&1
    retcode=$?
    ts=`date +%Y%m%d.%H%M%S`
    if [ $retcode != 0 ] ; then
        echo "$ts: -- Failed to ingest $mapfile into the database" >> $logdir/$rlog 2>&1
	mv $mapfiles_stage1/$mapfile $mapfiles_aborted/$mapfile
        continue
    fi
    echo "$ts: -- Successfully ingested $mapfile into the database" >> $logdir/$rlog 2>&1
    mv $mapfiles_stage1/$mapfile $mapfiles_stage2/$mapfile
done

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile ingestion to database ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

mapfiles=( `ls $mapfiles_stage2` )
mapfile_count=${#mapfiles[@]}

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: mapfile ingestion to thredds ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfiles_stage2/$mapfile --service fileservice --noscan --thredds  --no-thredds-reinit >> $logdir/$rlog 2>&1
    retcode=$?
    ts=`date +%Y%m%d.%H%M%S`
    if [ $retcode != 0 ] ; then
        echo "$ts: -- Failed to ingest $mapfile into thredds" >> $logdir/$rlog 2>&1
	mv $mapfiles_stage2/$mapfile $mapfiles_aborted/$mapfile
	continue
    fi
    echo "$ts: -- Successfully ingested $mapfile into thredds" >> $logdir/$rlog 2>&1
    mv $mapfiles_stage2/$mapfile $mapfiles_stage3/$mapfile
done

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile ingestion to thredds ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: calling --thredds-reinit ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

esgpublish --project $project --thredds-reinit >> $logdir/$rlog 2>&1

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: thredds-reinit ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

mapfiles=( `ls $mapfiles_stage3` )
mapfile_count=${#mapfiles[@]}

echo "$ts: INPROCESS: mapfile publication ($mapfile_count mapfiles)" >> $logdir/$rlog
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfiles_stage3/$mapfile --service fileservice --noscan --publish >> $logdir/$rlog 2>&1
    retcode=$?
    ts=`date +%Y%m%d.%H%M%S`
    if [ $retcode != 0 ] ; then
        echo "$ts: -- Failed to publish $mapfile" >> $logdir/$rlog 2>&1
	mv $mapfiles_stage3/$mapfile $mapfiles_aborted/$mapfile
        continue
    fi
    echo Successfully published $mapfile >> $logdir/$rlog
    mv $mapfiles_stage3/$mapfile $mapfiles_archive/$mapfile
done



utcFinal=`date +%s`
elapsed=$(($utcFinal - $utcStart))

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile publication ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
echo "$ts: Elapsed time:  $elapsed seconds" >> $logdir/$rlog 2>&1

echo "All done ($tagname).  Elapsed time:  $elapsed seconds" | sendmail $email_recip

publish-azb-v2.sh0000750017062300020020000000344413666266707014634 0ustar  bartoletti1climate#!/bin/bash

utcStart=`date +%s`

mapfile_dir=/p/user_pub/e3sm/staging/mapfiles
mapfile_done=/p/user_pub/e3sm/staging/mapfiles_temp_done
project=e3sm
email_recip=bartoletti1@llnl.gov

ini_dir=/p/user_pub/e3sm/staging/ini_old

exit_on_any_fail=1

mapfile_count=`ls $mapfile_dir | wc -l`
mapfiles_published=0

mapfiles=( `ls $mapfile_dir` )

for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfile_dir/$mapfile --commit-every 100  --no-thredds-reinit
    if [ $? != 0 ] ; then
        echo Failed to engest $mapfile into the database
	if [ $exit_on_any_fail ]; then
            exit 1
        else
            continue
        fi
    fi
    echo Successfully engested $mapfile into the database

    esgpublish -i $ini_dir --project $project --map $mapfile_dir/$mapfile --service fileservice --noscan --thredds  --no-thredds-reinit
    if [ $? != 0 ] ; then
        echo Failed to engest $mapfile into thredds
	if [ $exit_on_any_fail ]; then
            exit 1
        else
            continue
        fi
    fi
    echo Successfully engested $mapfile into thredds

    esgpublish --project $project --thredds-reinit

    esgpublish -i $ini_dir --project $project --map $mapfile_dir/$mapfile --service fileservice --noscan --publish
    if [ $? != 0 ] ; then
        echo Failed to publish $mapfile
	if [ $exit_on_any_fail ]; then
            exit 1
        else
            continue
        fi
    fi
    echo Successfully published $mapfile

    mapfiles_published=$(($mapfiles_published + 1))

    mv -v $mapfile_dir/$mapfile $mapfile_done/

done

esgpublish --project $project --thredds-reinit

utcFinal=`date +%s`
elapsed=$(($utcFinal - $utcStart))

echo "All done.  Published $mapfiles_published of $mapfile_count mapfiles.  Elapsed time:  $elapsed seconds" | sendmail $email_recip

publish-db-ingest-only.sh0000750017062300020020000000201013666266707016352 0ustar  bartoletti1climate#!/bin/bash

utcStart=`date +%s`

tag="BGC_CTC_raw - DB Ingest-Only"
mapfile_dir=/p/user_pub/e3sm/staging/mapfiles/mapfiles_bgc_raw
mapfile_done=/p/user_pub/e3sm/staging/mapfiles/mapfiles_archive/BGC/mapfiles_bgc_raw
project=e3sm
email_recip=bartoletti1@llnl.gov

ini_dir=/p/user_pub/e3sm/staging/ini_std

mapfiles=( `ls $mapfile_dir` )
mapfile_count=${#mapfiles[@]}

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: mapfile ingestion to database ($mapfile_count mapfiles)"
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfile_dir/$mapfile --commit-every 100  --no-thredds-reinit
    if [ $? != 0 ] ; then
        echo Failed to engest $mapfile into the database
        exit 1
    fi
    echo Successfully engested $mapfile into the database
done
ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile ingestion to database ($mapfile_count mapfiles)"

utcFinal=`date +%s`
elapsed=$(($utcFinal - $utcStart))

echo "All done ($tag).  Elapsed time:  $elapsed seconds" | sendmail $email_recip

publish-publish-only.sh0000750017062300020020000000323313666266707016154 0ustar  bartoletti1climate#!/bin/bash

if [ $# -ne 1 ]; then
	echo "Please enter a descriptive job_name for tracking purposes"
	exit 1
fi

tag=`echo $1 | tr ' ' _`


utcStart=`date +%s`
ts=`date +%Y%m%d.%H%M%S`

rlog="rlog-$ts-$tag"

logdir=/p/user_pub/e3sm/bartoletti1/Pub_Work/3_Publish/runlogs
mapfile_indir=/p/user_pub/e3sm/staging/mapfiles/stage_3_publish
mapfile_exdir=/p/user_pub/e3sm/staging/mapfiles/mapfiles_archive/BGC/mapfiles_bgc_raw
ini_dir=/p/user_pub/e3sm/staging/ini_std
project=e3sm
email_recip=bartoletti1@llnl.gov


mapfiles=( `ls $mapfile_indir` )
mapfile_count=${#mapfiles[@]}

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: calling --thredds-reinit ($mapfile_count mapfiles)" > $logdir/$rlog 2>&1

esgpublish --project $project --thredds-reinit >> $logdir/$rlog 2>&1

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: thredds-reinit ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
echo "$ts: INPROCESS: mapfile publication ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfile_indir/$mapfile --service fileservice --noscan --publish >> $logdir/$rlog 2>&1
    if [ $? != 0 ] ; then
        echo Failed to publish $mapfile >> $logdir/$rlog 2>&1
        exit 1
    fi
    echo Successfully published $mapfile >> $logdir/$rlog 2>&1
    mv -v $mapfile_indir/$mapfile $mapfile_exdir/
done

utcFinal=`date +%s`
elapsed=$(($utcFinal - $utcStart))

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile publication ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
echo "$ts: Elapsed time:  $elapsed seconds" >> $logdir/$rlog 2>&1

echo "All done ($tag).  Elapsed time:  $elapsed seconds" | sendmail $email_recip

publish-thredds-n-publish-only.sh0000750017062300020020000000306713666266707020047 0ustar  bartoletti1climate#!/bin/bash

utcStart=`date +%s`

tag=BGC_CTC_raw
mapfile_dir=/p/user_pub/e3sm/staging/mapfiles/mapfiles_bgc_raw
mapfile_done=/p/user_pub/e3sm/staging/mapfiles/mapfiles_archive/BGC/mapfiles_bgc_raw
project=e3sm
email_recip=bartoletti1@llnl.gov

ini_dir=/p/user_pub/e3sm/staging/ini_std

mapfiles=( `ls $mapfile_dir` )
mapfile_count=${#mapfiles[@]}

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: mapfile ingestion to thredds ($mapfile_count mapfiles)"
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfile_dir/$mapfile --service fileservice --noscan --thredds  --no-thredds-reinit
    if [ $? != 0 ] ; then
        echo Failed to engest $mapfile into thredds
        exit 1
    fi
    echo Successfully engested $mapfile into thredds
done
ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile ingestion to thredds ($mapfile_count mapfiles)"

esgpublish --project $project --thredds-reinit

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: mapfile publication ($mapfile_count mapfiles)"
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfile_dir/$mapfile --service fileservice --noscan --publish
    if [ $? != 0 ] ; then
        echo Failed to publish $mapfile
        exit 1
    fi
    echo Successfully published $mapfile
    mv -v $mapfile_dir/$mapfile $mapfile_done/
done
ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile publication ($mapfile_count mapfiles)"

utcFinal=`date +%s`
elapsed=$(($utcFinal - $utcStart))

echo "All done ($tag).  Elapsed time:  $elapsed seconds" | sendmail $email_recip