process_codes-20200727.2026.tar

AA_Automation_via_Holodeck0000640017062300020020000001642113707634135016571 0ustar  bartoletti1climateThe Nearly Automated Dataset Extraction and (Pre)Publication Process

TL;DR:
	IMPORTANT:  Due to values hard-coded into the scripts, one must invoke this "automated" process
		for each combination of (realm,freq_set,resolution,dataset_version)

	1.  Create a file "the_target_archives" consisting of lines selected from the file

		/p/user_pub/e3sm/archive/.cfg/Archive_Locator

		The lines listed must cover all cases (experiment,ensembles) for all intended datasets 

	2.  Edit the "publication_staging_control_script.sh" to set the values for these two variables

		dstype_static="atm nat"	(EXAMPLE)
		dstype_freq_list="mon day 6hr_snap day_cosp" (EXAMPLE)

		As we are extracting from archives, the first variable will always be "<realm> nat".
		The second can be any space-separated set of frequency-codes.
		See the file: /p/user_pub/e3sm/archive/.cfg/Standard_Datatype_Extraction_Patterns for options.

	3.  Edit the "holodeck_stage_publication.sh" script to set the values for these two variables

		resolution=1deg_atm_60-30km_ocean
		pubvers=v1

	4.  Invoke

		nohup ./publication_staging_control_script.sh the_target_archives &

		A log file "Holodeck_Process_Log-<date_time>" will be produced to track progress.

	If datasets for multiple realms is required, you will need to repeat Step 2 to reflect a different realm,
	and re-invoke Step 3.


BACKGROUND:


E3SM Publication Staging is now operating under a new premise:  All data archives are now local and amenable to zstash extraction.

I have crafted a working - if fragile - automated extraction regime that can access multiple experiments and cases, locate their archive directories and file extraction patterns, create the faceted publication destination directories, and successfully extract and move the dataset files to the intended publication directories.  I would caution that the process is not yet entirely generalized, and should be reimplemented at some point in a fixed python application, once the dust settles on a stable procedure.

The following provides background and detailed explanation of the process.  As I write this, it is already staging all of the DECK cam.h[1-4] datasets, for all experiments previously published.

BACKGROUND:  File Directory Structure and Configurations

The LLNL E3SM Archives presently have the following location and structure:

	/p/user_pub/e3sm/archive/
				1_0/
					DECK/
						(case_leaf)/tarfiles + index.db
						(case_leaf)/tarfiles + index.db
						...
					HiResMIP/
						(case_leaf)/tarfiles + index.db
						(case_leaf)/tarfiles + index.db
						...
				1_1/
					BGC/
						(ditto)
				1_1_ECA/
					BGC/
						(ditto)

	Note that "case_leaf" is a directory name uniquly identifying an experiment and ensemble.
	EXAMPLE: 20180622.DECKv1b_A2_1850aeroF.ne30_oEC.edison  (here, A2 = AMIP ens2)
	These directory names exactly match the author-supplied NERSC HPSS leaf directory names.
	This practice facilitates point-to-point Globus transfers.

	Ideally, the "case-leaf" directory should contain nothing but the tar files and the zstash database file "index.db".  In practice, it may contain other material, but no further directory traversal should be required to locate the available tar files.

In order to provide the opportunity for various "staging" dataset checks or modifications, the files are extracted to 

	/p/user_pub/e3sm/staging/pre_pub/(facets)/(files.nc)

where (facets)/(files.nc) matches the structure of the eventual publication directories.

We can extract and move the dataset files to publication using zstash, while leaving the archive intact and clean, by employing a "holodeck" of symlinks. (Credit to Sterling Baldwin for schooling me on the virtues of symlinks. I merely take credit for the use of the term "holodeck" in this context.)

With any empty directory serving as the "holodeck", the following depicts the sequence of operations leading to extraction and movement of dataset files to their intended publication directory:

1. Begin with empty directory, call this the $HolodeckPath

	/path/to/holodeck/

2. Create a zstash subdirectory

	/path/to/holodeck/
			zstash/

3. Having located the appropriate "case-leaf" archive directory, form a list of the contained tar file names (and index.db) and their full paths, and loop over each to create in the zstash subdirectory the necessary symlinks

	/path/to/holodeck/
			zstash/
				tarfilename --> symlink to full tarfile path
				tarfilename --> symlink to full tarfile path
				...
				index.db --> symlink to full index.db path

4. With the holodeck as the working directory, and the necessary datafile extraction pattern in hand, one can issue the zstash extract command to obtain

	/path/to/holodeck/
			stored/path/to/
				datafile
				datafile
				...
				datafile

			zstash/
				tarfilename --> symlink to full tarfile path
				tarfilename --> symlink to full tarfile path
				...
				index.db --> symlink to full index.db path

5. Having previously constructed the staging/prepub/(facets)/ destination path (targ_pub_dir="$prepubroot/$modelver/$exp_name/$midpath/$freq/$ensemble/$pubvers") and ensured it is empty (may be overridden with "overwriteFlag=1", and cognizant of the extracted tarpath-head (e.g. "stored/path/to/"), one can now issue

	mv stored/path/to/* pub_dest_path
	chmod 644 pub_dest_path/*

   leaving you

	/path/to/holodeck/
			stored/path/to/
				(empty)

			zstash/
				tarfilename --> symlink to full tarfile path
				tarfilename --> symlink to full tarfile path
				...
				index.db --> symlink to full index.db path

6. At this point, the dataset has been fully staged for pre-publication inspection and possible modifications.

   If there are additional datasets to publish from this same archive "case-leaf", one need only issue

	rm -rf $HolodeckPath/stored/

   and repeat steps 4 and 5 with another datafile_extraction_pattern, with working directory the $HolodeckPath with existing zstash symlinks.  

   If instead there are other cases for which datasets exist to be published, issue

	rm -rf $HolodeckPath/*

   and repeat from step 2 with a new case-leaf and associated datafile_extraction_patterns



IN SUMMARY:

With the "holodeck" as the working directory, a script given the desired (model,campaign,case,ensemble,datasetSpec) will need to

	1. Construct and test for existence and population of the target pre-publication directory.
		- Skip if the directory exists and is non-empty, unless the script overwriteFlag=1.
	2. Access the Archive_Map to obtain the information regarding the archive source data
		- Ensure exactly 1 line returned from CMEE_archive_map and CMEE_archive_map_augment
	3. populate the subordinate "zstash" subdirectory with simlinks to the appropriate tarfiles and index.db file.
		- Ensure holodeck contains only empty zstash subdirectory
	4. execute the zstash extraction command, with the appropriate file-selection pattern.
	5. create the necessary target publication directory
	6. move the extracted files to their publication directory
	7. clean the holodeck by removing all zstash symlinks and non-zstash directories

STEP 0:

	must contruct and test $pub_head_dir/$thisModel/$thisExp/$path_mid/$freq/$ens/v1
	where
		pub_head_dir=/p/user_pub/work/E3SM
		thisModel (supplied)
		thisExp   (supplied)
		path_mid  (supplied)	e.g. 1deg_atm_60-30km_ocean/atmos/native/model-output
		freq      (supplied)
		ens       (supplied)
		ver       (supplied)
AA_README_AA0000640017062300020020000000455413707634136013305 0ustar  bartoletti1climate
We can now employ the archive/.cfg/Archive_Map to location and "ls" all files from all archives
for all experiments and dataset types, by appropriate pattern.

The archive assessment intends to determine, for each experiment[ensemble] and dataset type,
the count of matching files, and that the "first" and "last" file found in each matched set
are indeed from the same "set" (have the same name structure) and have rational simulated
date ranges (e.g. 1850_01 to 2014_12).

Files returned have the general structure:

	tar_path/structured_name.sim_date.nc

Because we are using the Archive Map as our key, we know that the "tar_paths" will match for
any first and last file found.  But we need to ensure that "structured_name" is a match, and
that the first and last "sim_dates" are reasonable. This presents two problems:

  1.  The beginning of the structured_name may have a generation-date that varies across the
      dataset, and must be trimmed off if we are to compare the datafile names for matches.

  2.  The structured_name has "dot-delimited" components, and the number of components will
      vary across dataset types, and even for the same dataset type across experiments. This
      makes locating the "sim-year" field often specific to each dataset, and we have created
      a table
		/p/user_pub/e3sm/archive/.cfg/Filename_SimDate_Locator

      that identifies the dot-delimited field for sim-year, given experiment and dataset type


For each "Campaign,Model,Experiment,Ensemble,DatasetType,Count,First,Last", we want to output

either
	Campaign,Model,Experiment,Ensemble,DatasetType,MATCHED,Count,firstDate,LastDate,years_total

or	Campaign,Model,Experiment,Ensemble,DatasetType,NOMATCH,Count,firstStructuredName,LastStructuredName

Anywhere a NO_MATCH occurs, we must modify the Archive_Map to provide more specificity to the entry.


THE PROCESS:

	1.  Ensure the file "/p/user_pub/e3sm/archive/.cfg/Filename_SimDate_Locator" has an entry
	    for each experiment dataset, indicating the "dot"-delimited position of the sim-date
	    value in each filename.  This should have been done as part of the ArchivePathMapper.

	2.  Issue

		archive_assess_count_first_last.sh  any_subset_of_Archive_Map_lines

	    If output contains spurious NO_MATCH indications, but only due to embedded machine
	    name in the file names, you can manually set the flag "trust_match=1" in the script.


AA_README_APM0000640017062300020020000000706613707634136013442 0ustar  bartoletti1climate
BIG SELF_DEFINED TASK:

	Create a utility that will peruse EVERY archived directory (Experiment-Ensemble), to extract ALL tarred filepaths, to record the tar_paths to the actual files intended per (publish-able) dataset.
	e.g., for DECK,
		(atm nat mon) should return "archive/cam/hist/"  (for which the file_extraction_pattern *.cam.h0* will work beautifully)
	but, for HRv1,
		(atm nat mon) would return "run-0006-01-01-180907--0046-01-01-190111/"		(for years 0006-0045)
				       and "run/"						(for years 0046-0055)

	BOTH of the latter being still inadequate for a simple pattern like *cam.h0*, because those paths also archived other stuff matching *cam.h0*, so upon human inspection, we discover we need

			"run-0006-01-01-180907--0046-01-01-190111/theta"
		and	"run/theta"

	in order that the realm/freq pattern *cam.h0* will finally isolate the desired files.


THE GOAL:
	For any required publication of a [ CAMPAIGN MODEL EXPERIMENT ENSEMBLE (realm gridtype freq) ], we will have a table of (one or more) entries

		CAMPAIGN,MODEL,EXPERIMENT,ENSEMBLE,realm_gridtype_freq,archivePath,tarfilePath+extractionPattern

PREREQS:

	Ensure that archive/.cfg file "Standard_Datatype_Extraction_Patterns" are complete with respect to required publications


The Working Process:

0.  Upon arrival of new Archives, ensure that /p/user_pub/e3sm/archive/.cfg/Archive_Locator is updated.

1.  Issue ./archive_path_mapper_stage1.sh file_of_archive_specifications

	The "file_of_archive_specifications" can be the entire Archive_Locator (/p/user_pub/e3sm/archive/.cfg/Archive_Locator)
	or any file containing a subset of those lines, indicating the archives to be processed.

	Output will fill directory "PathsFound" with files of the form
		C:M:E:E:DSTITLE
	e.g.
		BGC:1_1:hist-BCRC:ens1:atm_nat_6hr

	Each file will contain all filenames found in archive using patterns found by the "tail" patterns in the_SDEP.

2.  Issue ./archive_path_mapper_stage2.sh 
	This will examine each file in PathsFound, trim away all lines that begin with known inappropriate path patterns
	and for the paths that remain, list the first and last files found for those patterns.
	Output wil be the file "headset12_lists_first_last", which will contain hundreds of entries of the form

		DECK:1_0:abrupt-4xCO2:ens1:lnd_nat_mon
		    HEADF:archive/lnd/hist/20180215.DECKv1b_abrupt4xCO2.ne30_oEC.edison.clm2.h0.0001-01.nc
		    HEADL:archive/lnd/hist/20180215.DECKv1b_abrupt4xCO2.ne30_oEC.edison.clm2.h0.0156-10.nc
		    HEADF:run/20180215.DECKv1b_abrupt4xCO2.ne30_oEC.edison.clm2.h0.0155-12.nc
		    HEADL:run/20180215.DECKv1b_abrupt4xCO2.ne30_oEC.edison.clm2.h0.0155-12.nc

	One must edit this file MANUALLY to determine which paths to the "First and Last" basenames is the correct one.
	In the case above, this would be "archive/lnd/hist", and since the "lnd_nat_mon" search pattern is *clm2.h0*,
	we need to convert that 5-line entry into

		DECK:1_0:abrupt-4xCO2:ens1:lnd_nat_mon:archive/lnd/hist/*clm2.h0*

	When such editing is completed, the finished file should be named "archive_dataset_map_prelim"

3.  Issue ./archive_path_mapper_stage3.sh
	This will take the output from stage 3, "archive_dataset_map_prelim", together with the Archive_Locator,
	to insert the appropriate archive_path into the final listing of the Archive_Map.


EXTRA_CREDIT:

        4.  Ensure the file "/p/user_pub/e3sm/archive/.cfg/Filename_SimDate_Locator" has an entry
            for each experiment dataset, indicating the "dot"-delimited position of the sim-date
            value in each filename.  Use the file "headset12_lists_first_last" to obtain the information.


AA_README_Extraction0000640017062300020020000000564013707634135015140 0ustar  bartoletti1climateThe tools within Pub_Work/1_Refactor begin with properly extracting desired dataset files from archive,
whether for immediate push to publication or for push to "Staging" where possible regridding or other
data cleanup may be applied.

NOTE:  One or more lines selected from the Archive_Map (/p/user_pub/e3sm/archive/.cfg/Archive_Map)
	will be required in a temporary file as an input parameter.  Beware that some datasets are
	spread across two different archives, and both lines may be required in order to extract
	a complete dataset.

A.  To list or extract files withough the creation of the full-faceted publication directories,
    a simple script is supplied:

	extract_dataset_files_to.sh  file_with_one_line_from_Archive_Map [destination_directory]

    If no destination directory is given, only the list of files that would have been extracted are
    provided (streamed to stdout).  Otherwise, the files are extracted to the indicated destination.

B.  If one or more datasets are to be extracted for publication or staging, with the consequent
    construction of faceted publications (or staging) directories, the script

	publication_staging_control_script.sh  configfile  file_of_lines_from_Archive_Map

    will automate most of that process.  In particular, you can publish/pre-publish multiple
    dataset frequencies in a single pass.  The necessary steps are

    1.  Create the config file.

    IMPORTANT:  Due to values hard-coded into the jobset_config file, one must invoke this "automated" process
    for each combination of (realm,freq_set,resolution,dataset_version).  The jobset_config file must
    contain:

	dstype_static=<type>    (where <type> examples are "atm nat", "ocn nat", "lnd nat", etc)
	dstype_freq_list=<list> (where <list> examples are "mon", "day 6hr 6hr_snap", etc)
	resolution=<res>        (where res is one of 1deg_atm_60-30km_ocean or 0_25deg_atm_18-6km_ocean)
	pubvers=<ver>           (where ver is one of v1, v2, etc)
	overwriteFlag=<0|1>     (Boolean, allows adding files to a non-empty destination directory)

	As we are extracting from archives, the first variable will always be "<realm> nat".
	The second can be any space-separated set of frequency-codes.
	See the file: /p/user_pub/e3sm/archive/.cfg/Standard_Datatype_Extraction_Patterns for options.


    2.  Create a file "the_target_archives" consisting of lines selected from the file

	/p/user_pub/e3sm/archive/.cfg/Archive_Locator

	The lines listed must cover all cases (experiment,ensembles) for all intended datasets 
	that are consistent with the values in the config file.


    3.  Invoke

	nohup ./publication_staging_control_script.sh the_target_archives &

	A log file "Holodeck_Process_Log-<date_time>" will be produced to track progress.

    If datasets for multiple realms or resolutions is required, you will need to repeat Steps 1,2,3 
    to reflect a different realm or resolution, and likewise select an archive list appropriately.








AA_README_MW0000640017062300020020000000202213707634136013333 0ustar  bartoletti1climateTo create mapfiles, ensure

	conda activate pub

In order to facilitate the generation of mapfiles for multiple data sets, the script

	multi_mapfile_generate.sh

is provided.  It takes as input a control file containing lines of one of three forms:

	/full_path_to_directory_of_dataset_files	
	HOLD:/full_path_to_directory_of_dataset_files	
	DONE:/full_path_to_directory_of_dataset_files	

Only the lines that do not begin with "HOLD:" or "DONE:" are processed.  This provides
an easy way to conduct piecemeal processing for very many datasets, allowing the file
to list datasets that are not yet ready for mapfile generation, as well as to maintain
a record of mapfiles already processed.

The lines to be processed are passed, one at a time, to the core script

	make_mapfile.sh

which serves to call "esgmapfile -make -i path_to_ini_files -outdir . . ."  hiding the
many details that tend to remain (near) constant.

This is merely a stopgap measure towards a fuller, data-driven automation and state
maintenance for mapfile generation.

AA_README_PS0000640017062300020020000000313613707634136013341 0ustar  bartoletti1climate Publication Status Page Preparation Report

The routine:

	pub_stat_page_prep.sh pub_dir_pathhead	(typically, /p/user_pub/work/E3SM/)

This will find the leaf dirs under the publication directories, and output lines of the form

	ModVer,Resolution,Ens,Title,Status,Grid,Years,Filecount,Dataset_ID,Dataset_First_File,Dataset_Publication_Path,Notes,Title_Expanded

Explicitly:

	ModVer:			A component of the path
	Resolution:		A component of the path
	Ens			A component of the path
	Title:			(RealmCode Grid Freq) derived from path elements.
	Status:			(placeholder for publication status)
	Grid:			A component of the path
	Years:			(placeholder for dataset years covered)
	Filecount:		From live directory examination
	Dataset_ID		Derived from path
	Dataset_First_File:	From live directory examination
	Dataset_Publication_Path:  Derived from path
	Notes:			(placeholder)
	Title_Expanded:		(placeholder)

For ALL model version experiments, ensembles, dataset_types and publication versions.

By converting this CSV output into an Excel document, the Excel rows can be copy/pasted directly into the associate confluence tables.

All that is missing by this operation is the ablity to determine the actual publication status for each dataset.

Formerly, this was obtained by calling "esglist_datasets" with the status flag, on aims3.

	Before running esglist_datasets:

		(activate the esgf-pub environment)
		source /usr/local/conda/bin/activate esgf-pub

		(if 72 hour auth cert has timed out.  have your esgf-node password ready)
		myproxy-logon -s esgf-node.llnl.gov -l <your_esgf_user_name> -t 72 -o ~/.globus/certificate-file

all_dataset_status.py0000640017062300020020000001452113707634136015750 0ustar  bartoletti1climateimport os, sys, argparse, re

'''
The Big Idea:  Using the Archive_Map and the (sproket-based) ESGF publication report,
produce from each the list of "canonical" experiment/dataset "keys" where

	akey = ( <model>, <experiment>, <ensemble> )
and
	dkey = <realm_grid_freq>

Use these to index a dictionary structure "dataset_status[akey][dkey] = ( A: bool, W: bool, P: bool )
with the Boolean values set according to the presence or absence of the key-pair in either of the
two input files (archive map, esgf publication report)
'''


# INPUT FILES
arch_map='/p/user_pub/e3sm/archive/.cfg/Archive_Map'
esgf_pr='/p/user_pub/e3sm/bartoletti1/Pub_Status/sproket/ESGF_publication_report'

# split the Archive_Map into a list of records, each record a list of fields
#   Campaign,Model,Experiment,Ensemble,DatasetType,ArchivePath,DatatypeTarExtractionPattern,Notes
with open(arch_map) as f:
    contents = f.read().split('\n')
am_list = [ aline.split(',') for aline in contents if aline[:-1] ]

# create a sorted list of unique dataset types
dstype_list = [ arch_rec[4] for arch_rec in am_list ]
dstype_list = list(set(dstype_list))
dstype_list.sort()

# create a dictionary keyed by 'arch_loc_key' = tuple (Model,Experiment,Ensemble) with value the SET of archive path(s)
# (Not really needed here, but nice to have, and we can reuse its keys for the dataset_status table)
arch_loc_dict = {}
for _ in am_list:
    arch_loc_dict[ tuple([_[1],_[2],_[3]]) ] = set()
for _ in am_list:
    arch_loc_dict[ tuple([_[1],_[2],_[3]]) ] |= {_[5]}  # add another archive path to the set - may be more than one.

# create a dictionary keyed by 'arch_loc_key', each value a dictionary keyed by dstype, value a Boolean dictionary (A,W,P)
# For each (model,experiment,ensemble), create a dataset_type entry for every allowable dataset-type
# initialize "Archive,Warehouse,Published" to ALL FALSE.
dataset_status = {}
for akey in arch_loc_dict:
    dataset_status[akey] = { dstype: { 'A': False, 'W': False, 'P': False } for dstype in dstype_list }

# for each record in the ArchiveMap, set the corresponding [archive_key][dataset_key]['A'] = True
for _ in am_list:
    akey = tuple([_[1],_[2],_[3]])
    dkey = _[4]
    dataset_status[akey][dkey]['A'] = True

# extract the datasetID from each published dataset listed in the ESGF_publication_report
# obtain the key 'Model,Experiment,Ensemble'

with open(esgf_pr) as f:
    contents = f.read().split('\n')

esgf_pr_list = [ aline.split(',') for aline in contents if aline[:-1] and not aline[0] == 'NOMATCH' ]
#esgf_pr_list = [ aline.split(',') for aline in contents if aline[:-1] ]
 
#### BEGIN rationalizing archive and publication experiment-case names, and dataset-type names ####
# dsid = proj.model.experiment.resolution[.tuning].realm.grid.outtype.freq.ens.ver

# call to specialize a publication experiment name to an archive experiment name
def specialize_expname(expn,reso,tune):
    if expn == 'F2010plus4k':
        expn = 'F2010-plus4k'
    if expn[0:5] == 'F2010' or expn == '1950-Control':
        if reso[0:4] == '1deg' and tune == 'highres':
            expn = expn + '-LRtunedHR'
        else:
            expn = expn + '-HR'
    return expn

def get_dsid_arch_key( dsid ):
    comps=dsid.split('.')
    expname = specialize_expname(comps[2],comps[3],comps[4])
    return comps[1],expname,comps[-2]

def get_dsid_type_key( dsid ):
    comps=dsid.split('.')
    realm = comps[-6]
    gridv = comps[-5]
    otype = comps[-4]
    freq = comps[-3]

    if realm == 'atmos':
        realm = 'atm'
    elif realm == 'land':
        realm = 'lnd'
    elif realm == 'ocean':
        realm = 'ocn'

    if gridv == 'native':
        grid = 'nat'
    elif otype == 'climo':
        grid = 'climo'
    elif otype == 'monClim':
        grid = 'climo'
        freq = 'mon'
    elif otype == 'seasonClim':
        grid = 'climo'
        freq = 'season'
    elif otype == 'time-series':
        grid = 'reg'
        freq = 'ts-' + freq
    else:
        grid = 'reg'
    return '_'.join([realm,grid,freq])

#### COMPLETED rationalizing archive and publication experiment-case names, and dataset-type names ####

# DEBUG/INFO:  Report when any "published" dataset is not an archived dataset
'''
for id in esgf_pr_list:
    akey = get_dsid_arch_key( id[2] )

    if akey in dataset_status:
        print(f' key { akey } in dataset_status')
    else:
        print(f' key { akey } is NEW')
'''
# sys.exit(0)

# Update 'P' status of this (possibly new) dataset_type in the (possibly new) datasetID_key in the dataset_status table
# using the esgf_pr_list (sproket-based publication report)

for id in esgf_pr_list:
    akey = get_dsid_arch_key( id[2] )
    dkey = get_dsid_type_key( id[2] )

    if not akey in dataset_status:  # first time we've encountered this experiment/case
        dataset_status[akey] = { dstype: { 'A': False, 'W': False, 'P': True } for dstype in dstype_list }
    elif not dkey in dataset_status[akey]:  # first time we've encountered this dataset type for this experiment/case
        dataset_status[akey][dkey] = { 'A': False, 'W': False, 'P': True }
    else:  # just set Publication to True
        dataset_status[akey][dkey]['P'] = True

def dataset_print_csv( akey, dkey ):
    print(f'{akey[0]},{akey[1]},{akey[2]},{dkey}')

'''
# print all combos (model,experiment,ensemble,dataset_type)

for akey in dataset_status:
    # print(f'{ akey }:')
    for dkey in dataset_status[akey]:
        # print(f'    { dkey } : { dataset_status[akey][dkey] }')
        dataset_print_csv(akey,dkey)
'''

for akey in dataset_status:
    for dkey in dataset_status[akey]:
        if dataset_status[akey][dkey]['A'] and dataset_status[akey][dkey]['P']:
            print('A_P,',end ='')
            dataset_print_csv(akey,dkey)

for akey in dataset_status:
    for dkey in dataset_status[akey]:
        if dataset_status[akey][dkey]['A'] and not dataset_status[akey][dkey]['P']:
            print('A_notP,',end ='')
            dataset_print_csv(akey,dkey)

for akey in dataset_status:
    for dkey in dataset_status[akey]:
        if dataset_status[akey][dkey]['P'] and not dataset_status[akey][dkey]['A']:
            print('notA_P,',end ='')
            dataset_print_csv(akey,dkey)

for akey in dataset_status:
    for dkey in dataset_status[akey]:
        if not dataset_status[akey][dkey]['A'] and not dataset_status[akey][dkey]['P']:
            print('notA_notP,',end ='')
            dataset_print_csv(akey,dkey)

sys.exit(0)


archive_assess_count_first_last.sh0000750017062300020020000000726013707634136020522 0ustar  bartoletti1climate#!/bin/bash

# Usage:  archive_assess_count_first_last.sh archive_map(all or part) [trust_match]
# For the entire archive, use /p/user_pub/e3sm/archive/.cfg/Archive_Map
# For a subset, create a file with selected lines, and give its full path

# if "trust_match" is absent, and the canonical part of the first/last filenames (used to extract sim-dates)
# do NOT match, The output will not calculate "years" for that dataset, and the output will contain lines
# NO_MATCH: <filenames> in order to allow manual inspection.  If the inspection indicates that the files
# belong to different datasets, or are otherwise inappropriately paired, then there are errors in the
# Archive_Map that need to be addressed (the file-matching patterns are insufficiently specific.)
# If the inspection indicates that the match failures are due to "harmless" differences (different
# machine-names, or different generation-dates due to restarts) then re-run this script with
# "trust_match" (as indicated in "Usage" above) to skip the first/last filename check.

PURPOSE="1.  Find and List all Default Model Outputs from all Experiment/Ensembles by Holodeck of each archive index.db file"

# format:  Campaign,Model,Experiment,Ensemble,DatasetType,ArchivePath,DatatypeExtractionPattern,Notes
# Full Archive Map file:  /p/user_pub/e3sm/archive/.cfg/Archive_Map

archive_map_file=$1
simdate_locator=/p/user_pub/e3sm/archive/.cfg/Filename_SimDate_Locator

The_Holodeck=/p/user_pub/e3sm/bartoletti1/Pub_Status/ArchiveAssess/Holodeck


zstash_version=`zstash version`
if [ $zstash_version != "v0.4.1" ]; then
	echo "ABORTING:  zstash version is not 0.4.1 or is unavailable"
	exit 1
fi

trust_match=0
if [ $# -eq 2 ]; then
        if [ $2 == "trust_match" ]; then
                trust_match=1
        fi
fi


startTime=`date +%s`

IFS=$'\n'

cd $The_Holodeck

for aline in `cat $archive_map_file`; do

	apath=`echo $aline | cut -f6 -d,`
	if [ $apath == "NAV" ]; then
		continue
	fi

	basetag=`echo $aline | cut -f1-5 -d,`
	pattern=`echo $aline | cut -f7 -d,`


	rm -rf $The_Holodeck/*
	mkdir $The_Holodeck/zstash

	ln -s $apath/index.db $The_Holodeck/zstash/index.db
	
	zstash ls --hpss=none $pattern > /tmp/zstash_manifest

	filecount=`cat /tmp/zstash_manifest | wc -l`
	if [ $filecount -eq 0 ]; then
		echo "$basetag,0,(pattern=$pattern)"
		continue
	fi

	# SHOULD sort on simdate field here. Not easy, because the
	# tarpath could make the indexing difficult.  Or not. 
	init_file=`cat /tmp/zstash_manifest | head -1`
	last_file=`cat /tmp/zstash_manifest | tail -1`

	# trim tar-path from first and last
	filebase1=`echo $init_file | rev | cut -f1 -d/ | rev`
	filebase2=`echo $last_file | rev | cut -f1 -d/ | rev`

	simdfield=`grep $basetag: $simdate_locator | cut -f2 -d:`
	prevfield=$((simdfield - 1))
	
	sim_date1=`echo $filebase1 | cut -f$simdfield -d.`
	filetest1=`echo $filebase1 | cut -f1-$prevfield -d.`
	sim_date2=`echo $filebase2 | cut -f$simdfield -d.`
	filetest2=`echo $filebase2 | cut -f1-$prevfield -d.`

	# last part eliminates variable leading run_dates
	filecore1=`echo $filetest1 | cut -f2- -d.`
	filecore2=`echo $filetest2 | cut -f2- -d.`

	if [ $trust_match -eq 0 ]; then
		if [ $filecore1 != $filecore2 ]; then
			echo "$basetag,NO_MATCH,$filecount,$init_file,$last_file"
			echo "NO_MATCH:               $filecore1"
			echo "NO_MATCH:               $filecore2"
			continue
		fi
	fi

	y1=`echo $sim_date1 | cut -c1-4`
	y2=`echo $sim_date2 | cut -c1-4`
	m2=`echo $sim_date2 | cut -c6-7`
	y1=$((10#$y1))
	y2=$((10#$y2))
	yspan=$((y2 - y1 + 1))
	if [ $m2 == "01" ]; then
		yspan=$((y2 - y1))
	fi
	echo "$basetag,MATCHED,$filecount,$filetest1,$sim_date1,$sim_date2,$yspan"

done

finalTime=`date +%s`

et=$(($finalTime-$startTime))

echo "Done.  ET = $et"


exit

archive_holodeck_setup_ls-only.sh0000750017062300020020000000153213707634136020240 0ustar  bartoletti1climate#!/bin/bash

# echo "DEBUG:  argcount = $#"

instructions="full path to a leaf archive directory"

zstash_version=`zstash version`
if [ $zstash_version != "v0.4.1" ]; then
	echo "ABORTING:  zstash version is not 0.4.1 or is unavailable"
	exit 1
fi

fulldir=$1

if [ ! -d $fulldir ]; then
	echo "ABORTING:  cannot find archive directory: $fulldir"
	exit 1
fi

	
The_Holodeck=/p/user_pub/e3sm/bartoletti1/Pub_Status/ArchiveAssess/Holodeck

IFS=$'\n'

# PLAN:  populate the subordinate "zstash" subdirectory with simlinks to the appropriate tarfiles and index.db file.

echo "Clearing the Holodeck"
echo " "
rm -rf $The_Holodeck/*

zstash_dir=$The_Holodeck/zstash
mkdir $zstash_dir

echo "We will set up holodeck zstash ls for directory $fulldir" 

ln -s $fulldir/index.db $zstash_dir/index.db

echo "Holodeck prepared for entry. Enjoy your simulation!"

exit

archive_holodeck_setup.sh0000750017062300020020000000203213707634136016557 0ustar  bartoletti1climate#!/bin/bash

# echo "DEBUG:  argcount = $#"

#  USAGE:  archive_holodeck_setup.sh full_path_to_a_leaf_archive_directory"

# EDIT THIS LINE for your own Holodeck
The_Holodeck=/p/user_pub/e3sm/bartoletti1/Pub_Status/ArchiveAssess/Holodeck

zstash_version=`zstash version`
if [ $zstash_version != "v0.4.1" ]; then
	echo "ABORTING:  zstash version is not 0.4.1 or is unavailable"
	exit 1
fi

fulldir=$1

if [ ! -d $fulldir ]; then
	echo "ABORTING:  cannot find archive directory: $fulldir"
	exit 1
fi

IFS=$'\n'

# PLAN:  populate the subordinate "zstash" subdirectory with simlinks to the appropriate tarfiles and index.db file.

echo "Clearing the Holodeck"
echo " "
rm -rf $The_Holodeck/*

zstash_dir=$The_Holodeck/zstash
mkdir $zstash_dir

echo "We will setup holodeck zstash for directory $fulldir" 

# major kludge
rm -rf $The_Holodeck/archive

for targ in `ls $fulldir`; do
	# touch nothing
	# echo "ln -s $fulldir/$targ zstash/$targ"
	ln -s $fulldir/$targ $zstash_dir/$targ
done

echo "Holodeck prepared for entry. Enjoy your simulation!"

exit

archive_path_mapper_stage1.sh0000750017062300020020000000346313707634136017324 0ustar  bartoletti1climate#!/bin/bash

# Usage:  archive_path_mapper_stage1.sh achive_locations_file
# For the entire archive, use /p/user_pub/e3sm/archive/.cfg/Archive_Locator
# For a subset, create a file with selected lines, and give its full path

PURPOSE="1.  Find and List all Default Model Outputs from all Experiment/Ensembles by Holodeck of each archive index.db file"

# format:  Campaign,Model,Experiment,Ensemble,ArchivePath,
# Full Archve Location file:  /p/user_pub/e3sm/archive/.cfg/Archive_Locator

archive_loc_file=$1

the_SDEP=/p/user_pub/e3sm/archive/.cfg/Standard_Datatype_Extraction_Patterns

WorkDir=/p/user_pub/e3sm/bartoletti1/Pub_Status/ArchivePathMapper/
The_Holodeck=$WorkDir/Holodeck


zstash_version=`zstash version`
if [ $zstash_version != "v0.4.1" ]; then
	echo "ABORTING:  zstash version is not 0.4.1 or is unavailable"
	exit 1
fi

mv $WorkDir/PathsFound/* $WorkDir/PrevPathsFound/

startTime=`date +%s`

IFS=$'\n'

cd $The_Holodeck

# Archive_Locator Entry processed

ALE=0

for aline in `cat ../$archive_loc_file`; do

	apath=`echo $aline | cut -f5 -d,`
	if [ $apath == "NAV" ]; then
		continue
	fi

	ALE=$(($ALE + 1))

	basetag=`echo $aline | cut -f1-4 -d, | tr , :`

	# echo "ALE:BaseTag = $ALE:$basetag"

	rm -rf $The_Holodeck/*
	mkdir $The_Holodeck/zstash

	ln -s $apath/index.db $The_Holodeck/zstash/index.db
	
	for fpatspec in `cat $the_SDEP`; do
		dstitle=`echo $fpatspec | cut -f1 -d, | tr ' ' _`
		dspattern=`echo $fpatspec | cut -f2 -d,`
		if [ $dspattern == "ignore" ]; then
			continue
		fi
		outfile="$ALE:$basetag:$dstitle"
		# echo "        $outfile"
		zstash ls --hpss=none $dspattern > ../PathsFound/$outfile	
		if [ -s ../PathsFound/$outfile ]; then
			continue
		fi
		echo "EMPTY: $outfile"
		rm ../PathsFound/$outfile
	done

done

finalTime=`date +%s`

et=$(($finalTime-$startTime))

echo "Done.  ET = $et"


exit

archive_path_mapper_stage2.sh0000750017062300020020000000152713707634136017324 0ustar  bartoletti1climate#!/bin/bash

# for each named file in PathsFound

flist=`ls PathsFound`

rm -f headset_list_first_last

for afile in `ls PathsFound`; do
	echo $afile >> headset12_lists_first_last
	# the list of grep -v exclusions here may need future additions
	headlist=`cat PathsFound/$afile | cut -c1-20 | sort | uniq | grep -v "^rest/" | grep -v "^post/" | grep -v "^test" | grep -v "^init" | grep -v "^run/try" | grep -v "^run/bench" | grep -v "^old/run" | grep -v "^pp/remap" | grep -v "^a-prime" | grep -v "^lnd_rerun" | grep -v "^atm/ncdiff" | grep -v "^archive/rest" | grep -v "fullD" | grep -v "photic"`
	for ahead in $headlist; do
		samp=`grep ^$ahead PathsFound/$afile | sort | head -1`
		echo "    HEADF:$samp" >> headset_list_first_last
		samp=`grep ^$ahead PathsFound/$afile | sort | tail -1`
		echo "    HEADL:$samp" >> headset_list_first_last
	done
done
archive_path_mapper_stage3.sh0000750017062300020020000000130113707634136017313 0ustar  bartoletti1climate#!/bin/bash

infile=archive_dataset_map_prelim

archive_loc=/p/user_pub/e3sm/archive/.cfg/Archive_Locator

for aline in `cat $infile`; do
	pureline=`echo $aline | cut -f2- -d:`
	headpart=`echo $pureline | cut -f1-4 -d:`
	ds_title=`echo $pureline | cut -f5 -d:`
	epattern=`echo $pureline | cut -f6 -d:`

	seek=`echo $headpart | tr : ,`
	found=`cat $archive_loc | grep $seek`

	lc=`echo $found | wc -l`
	if [ $lc -eq 0 ]; then
		echo "ERROR: no archive location found for $seek"
		continue
	fi
	if [ $lc -gt 1 ]; then
		echo "WARNING: multiple archive locations found for $seek"
	fi
	for apath in $found; do
		the_path=`echo $apath | cut -f5 -d,`
		echo "$seek,$ds_title,$the_path,$epattern"
	done
done

	

archive_status_prep.sh0000750017062300020020000001344413707634136016131 0ustar  bartoletti1climate#!/bin/bash

# Usage:  archive_status_prep.sh archive_map(all or part) [trust_match]
#
# For the entire archive, use /p/user_pub/e3sm/archive/.cfg/Archive_Map
# For a subset, create a file with selected lines, and give its full path
# format:  Campaign,Model,Experiment,Ensemble,DatasetType,ArchivePath,DatatypeExtractionPattern,Notes
# Full Archive Map file:  /p/user_pub/e3sm/archive/.cfg/Archive_Map
#
# if "trust_match" is absent, and the canonical part of the first/last filenames (used to extract sim-dates)
# do NOT match, The output will not calculate "years" for that dataset, and the output will contain lines
# NO_MATCH: <filenames> in order to allow manual inspection.  If the inspection indicates that the files
# belong to different datasets, or are otherwise inappropriately paired, then there are errors in the 
# Archive_Map that need to be addressed (the file-matching patterns are insufficiently specific.)
# If the inspection indicates that the match failures are due to "harmless" differences (different
# machine-names, or different generation-dates due to restarts) then re-run this script with
# "trust_match" (as indicated in "Usage" above) to skip the first/last filename check.


# PURPOSE:  From input Archive Map
#  campaign, model, experiment, ens, title, archive_path, tar_extract_pattern
#
#  Produce a listing
# 	model, resolution, ens, Title, status, grid, years, filecount, datasetID, datasetFirstFile, DatasetPubPath,Notes, titleExpanded, misc
#
#  THe following table outlines the source of each item
#
#	model			Direct from Archive Map
#	resolution		use "$default_res" below
#	ensemble		Direct from Archive Map
#	title			extract from Archive Map "DatasetType"
#	status			[HC] "arch"
#	grid			[HC] "native"
#	years			[HC] ""
#	filecount		Generated by inspection
#	datasetID		Constructed from all of these values and sources
#	firstFile		Generated by inspection
#	pubPath			Constructed from datasetID
#	notes			[HC] "model-output"
#	titleExpanded		reit title
#	misc			[HC] ""

#  A datasetID needs
#	E3SM.<model>.<experiment>.<resolution>.<realm>.<grid>.model_output.<freq>.<ens>

archive_map_file=$1

trust_match=0
if [ $# -eq 2 ]; then
	if [ $2 == "trust_match" ]; then
		trust_match=1
	fi
fi

simdate_locator=/p/user_pub/e3sm/archive/.cfg/Filename_SimDate_Locator

The_Holodeck=/p/user_pub/e3sm/bartoletti1/Pub_Status/ArchiveAssess/Holodeck


zstash_version=`zstash version`
if [ $zstash_version != "v0.4.1" ]; then
	echo "ABORTING:  zstash version is not 0.4.1 or is unavailable"
	exit 1
fi

default_res="1deg_atm_60-30km_ocean"

startTime=`date +%s`

IFS=$'\n'

cd $The_Holodeck

for aline in `cat $archive_map_file`; do

	model=`echo $aline | cut -f2 -d,`
	experiment=`echo $aline | cut -f3 -d,`
	resolution=$default_res
	ens=`echo $aline | cut -f4 -d,`
	title=`echo $aline | cut -f5 -d, | tr _ ' '` # may need tr _ ' '
	realmcode=`echo $title | cut -f1 -d' '`
	freq=`echo $title | cut -f3 -d' '`
	pstatus="arch"
	grid="native"
	years=""
	notes="model-output"
	otype=$notes
	misc=""

	# translate realmcode to realm
	if [ $realmcode == "atm" ]; then
		realm="atmos"
	elif [ $realmcode == "lnd" ]; then
		realm="land"
	elif [ $realmcode == "ocn" ]; then
		realm="ocean"
	elif [ $realmcode == "river" ]; then
		realm="river"
	elif [ $realmcode == "sea-ice" ]; then
		realm="sea-ice"
	else realm="UNK_$realmcode"
	fi

	datasetID="E3SM.$model.$experiment.$resolution.$realm.$grid.$otype.$freq.$ens"
	pubPath=`echo $datasetID | tr . /`

	# echo "DEBUG: dsID=$datasetID"
	# continue

	# still need to formulate filecount, firstFile, (titleExpanded)

	apath=`echo $aline | cut -f6 -d,`
	if [ $apath == "NAV" ]; then
		continue
	fi

	basetag=`echo $aline | cut -f1-5 -d,`
	pattern=`echo $aline | cut -f7 -d,`

	# echo "DEBUG:  Basetag=$basetag"

	rm -rf $The_Holodeck/*
	mkdir $The_Holodeck/zstash

	ln -s $apath/index.db $The_Holodeck/zstash/index.db
	
	zstash ls --hpss=none $pattern > /tmp/zstash_manifest

	filecount=`cat /tmp/zstash_manifest | wc -l`

	init_file=`cat /tmp/zstash_manifest | head -1`
	last_file=`cat /tmp/zstash_manifest | tail -1`

	# trim tar-path from first and last
	filebase1=`echo $init_file | rev | cut -f1 -d/ | rev`
	filebase2=`echo $last_file | rev | cut -f1 -d/ | rev`

	# echo "DEBUG: filebase1=$filebase1"
	# echo "DEBUG: filebase2=$filebase2"

	simdfield=`grep $basetag: $simdate_locator | cut -f2 -d:`
	prevfield=$((simdfield - 1))
	
	sim_date1=`echo $filebase1 | cut -f$simdfield -d.`
	filetest1=`echo $filebase1 | cut -f1-$prevfield -d.`
	sim_date2=`echo $filebase2 | cut -f$simdfield -d.`
	filetest2=`echo $filebase2 | cut -f1-$prevfield -d.`

	# echo "DEBUG: simdfield=$simdfield"
	# echo "DEBUG: sim_date1=$sim_date1"
	# echo "DEBUG: sim_date2=$sim_date2"
	# last part eliminates variable leading run_dates
	filecore1=`echo $filetest1 | cut -f2- -d.`
	filecore2=`echo $filetest2 | cut -f2- -d.`

	if [ $trust_match -eq 0 ]; then

		if [ $filecore1 != $filecore2 ]; then
			# echo "$basetag,NO_MATCH,$filecount,$init_file,$last_file"
			echo "NO_MATCH:                  $filecore1"
			echo "NO_MATCH:                  $filecore2"
			years="NAV"
			echo "$model,$resolution,$ens,$title,$pstatus,$grid,$years,$filecount,$datasetID,$filebase1,$pubPath,$otype,$title,$misc"
			continue;
		fi
	fi
			
	y1=`echo $sim_date1 | cut -c1-4`
	y2=`echo $sim_date2 | cut -c1-4`
	m2=`echo $sim_date2 | cut -c6-7`
	y1=$((10#$y1))
	y2=$((10#$y2))
	yspan=$((y2 - y1 + 1))
	if [ $m2 == "01" ]; then
		yspan=$((y2 - y1))
	fi
	# echo "$basetag,MATCHED,$filecount,$filetest1,$sim_date1,$sim_date2,$yspan"
	years=$yspan

	# 	model, resolution, ens, Title, status, grid, years, filecount, datasetID, datasetFirstFile, DatasetPubPath,Notes, titleExpanded, misc
	echo "$model,$resolution,$ens,$title,$pstatus,$grid,$years,$filecount,$datasetID,$filebase1,$pubPath,$otype,$title,$misc"

done

finalTime=`date +%s`

et=$(($finalTime-$startTime))

echo "Done.  ET = $et"


exit

archive_versus_pub_compare.sh0000750017062300020020000000352613707634136017463 0ustar  bartoletti1climate#!/bin/bash

arc_list=$1
pub_list=$2

trisect=/p/user_pub/e3sm/bartoletti1/abin/trisect

# using
#   A.  the ArchiveAssess output from "archive_assess_count_first_last.sh"
#   B.  the assessment report from sproket
#   (properly limited to a single common experiment)
#   do
#	1.  Extract "experiment/ensemble and datasetType" from each
#	2.  Use "trisect" to find the "A-only", "B-only", and Both.

# arc_list:  Campaign,Model,Experiment,Ensemble,r_g_f,MATCHED,files,canonFile,beginDate,finalDate,years
# pub_list:  years,files,datasetID,firstFile
#
#   where datasetID = Project.Model.Experiment.Resolution.Realm.Grid.outType.Freq.Ensemble.version

# So . . . Generate A_Key and B_Key lists

rm -f /tmp/A_list
rm -f /tmp/B_list

for aline in `cat $arc_list`; do

	A_model=`echo $aline | cut -f2 -d,`
	A_Exper=`echo $aline | cut -f3 -d,`
	A_Ensem=`echo $aline | cut -f4 -d,`
	A_r_g_f=`echo $aline | cut -f5 -d,`

	echo $A_model,$A_Exper,$A_Ensem,$A_r_g_f >> /tmp/A_list
done

for aline in `cat $pub_list`; do

	ds_ID=`echo $aline | cut -f3 -d,`
	B_model=`echo $ds_ID | cut -f2 -d.`
	B_exper=`echo $ds_ID | cut -f3 -d.`
	B_ensem=`echo $ds_ID | cut -f9 -d.`

	realm=`echo $ds_ID | cut -f5 -d.`
	gridv=`echo $ds_ID | cut -f6 -d.`
	freqv=`echo $ds_ID | cut -f8 -d.`

	if [ $realm == "atmos" ]; then
		r_code="atm"
	elif [ $realm == "land" ]; then
		r_code="lnd"
	elif [ $realm == "ocean" ]; then
		r_code="ocn"
	else
		r_code=$realm
	fi

	if [ $gridv == "native" ]; then
		g_code="nat"
	else
		g_code="reg"
	fi

	B_r_g_f="${r_code}_${g_code}_$freqv"

	echo $B_model,$B_exper,$B_ensem,$B_r_g_f >> /tmp/B_list
done

sort /tmp/A_list | uniq > /tmp/sA_list	
sort /tmp/B_list | uniq > /tmp/sB_list	

$trisect -f1 /tmp/sA_list -f2 /tmp/sB_list

echo "ARCHIVE_ONLY:"
cat f1-only
echo " "
echo "ESGF_PUBLISH_ONLY:"
cat f2-only
echo " "
echo "In ARCHIVE and ESGF_PUBLISH"
cat fboth

exit 0
dirperms.sh0000750017062300020020000000014413707634136013675 0ustar  bartoletti1climate#!/bin/bash

dirlist=$1

for aline in `cat $dirlist`; do
	chmod 755 $aline
	chmod 644 $aline/*
done
extract_dataset_files_to.sh0000750017062300020020000000433713707634135017122 0ustar  bartoletti1climate#!/bin/bash

# USAGE:  extract_dataset_files_to.sh  file_with_one_line_from_archive_map [dest_dir]
#
# This script simply accepts a file with a line from the Archive map and a path to a destination directory, and 
# populates that directory with the indicated dataset files.  The destination directory must already exist.
# if no "dest_dir" is supplied, only the list of files that would have been extracted is produced
#
# NOTE: As this script will "cd" to the Holodeck, you should supply full paths to the files and directories,
# or else execute from the Holodeck with Holodeck-relative paths

spec=`cat $1 | head -1`
dest=$2

list_only=1

if [ $# -gt 1 ]; then
	dest=$2
	list_only=0

	if [ ! -d $dest ]; then
		echo "ERROR:  Expected destination directory not found: $dest"
		exit 1;
	fi
fi

The_Holodeck=/p/user_pub/e3sm/bartoletti1/Pub_Work/1_Refactor/Holodeck

Archive_Map=/p/user_pub/e3sm/archive/.cfg/Archive_Map


zstash_version=`zstash version`
if [ $zstash_version != "v0.4.1" ]; then
	echo "ABORTING:  zstash version is not 0.4.1 or is unavailable"
	exit 1
fi

IFS=$'\n'

campaign=`echo $spec | cut -f1 -d,`
modelver=`echo $spec | cut -f2 -d,`
exp_name=`echo $spec | cut -f3 -d,`
ensemble=`echo $spec | cut -f4 -d,`
arch_dir=`echo $spec | cut -f6 -d,`
tar_patt=`echo $spec | cut -f7 -d,`

#OVERRIDE
tar_patt=${tar_patt}000*

echo "Cleaning the Holodeck"
echo " "
rm -rf $The_Holodeck/*

zstash_dir=$The_Holodeck/zstash
mkdir $zstash_dir

# create Holodeck symlinks to archive tar-files
for targ in `ls $arch_dir`; do
	# touch nothing
	# echo "ln -s $arch_dir/$targ zstash/$targ"
	ln -s $arch_dir/$targ $zstash_dir/$targ
done

cd $The_Holodeck

if [ $list_only -eq 1 ]; then

	echo "zstash ls --hpss=none $tar_patt"
	zstash ls --hpss=none $tar_patt
	exitcode=$?

	if [ $exitcode -ne 0 ]; then
		echo "ERROR:  zstash returned exitcode $exitcode"
		exit $exitcode
	fi

	exit 0
fi

echo "zstash extract --hpss=none $tar_patt"
zstash extract --hpss=none $tar_patt
exitcode=$?

if [ $exitcode -ne 0 ]; then
	echo "ERROR:  zstash returned exitcode $exitcode"
	exit $exitcode
fi

echo "mv $The_Holodeck/$file_extract_pattern $dest"
mv $The_Holodeck/$tar_patt $dest
chmod 644 $dest/*.nc


echo " "
echo "Thank you for using the Zstash Holodeck(tm)"
echo " "

exit 0

holodeck_stage_publication.sh0000750017062300020020000001074013707634135017416 0ustar  bartoletti1climate#!/bin/bash

# Obtain values for
#   resolution		(e.g. 1deg_atm_60-30km_ocean, 0_25deg_atm_18-6km_ocean, etc)
#   pubvers		(e.g. v1, v2, etc)
#   overwriteFlag	(Boolean)
source $1

Datatype_Patterns=/p/user_pub/e3sm/archive/.cfg/Standard_Datatype_Extraction_Patterns
The_Holodeck=/p/user_pub/e3sm/bartoletti1/Pub_Work/1_Refactor/Holodeck

Archive_Map=/p/user_pub/e3sm/archive/.cfg/Archive_Map
prepubroot=/p/user_pub/e3sm/staging/prepub
pubroot=/p/user_pub/work/E3SM

# echo "DEBUG:  argcount = $#"

if [ $# -ne 3 ]; then
	echo " "
	echo "    Usage:  holodeck_stage_publication.sh jobset_config AL_selection_line datasetspec"
	echo " "
	echo "            The jobset_config file must contain lines:"
	echo "                resolution=<res>        (where res is one of 1deg_atm_60-30km_ocean or 0_25deg_atm_18-6km_ocean)"
	echo "                pubvers=<ver>           (where ver is one of v1, v2, etc)"
	echo "                overwriteFlag=<0|1>     (Boolean, allows adding files to a non-empty destination directory)"
	echo "            AL_selection_line is a line selected from archive/.cfg/Archive_Locator."
	echo "            Give datasetspec as \"realm grid freq\", as in \"atm nat mon\", in quotations."
	echo "            See the file /p/user_pub/e3sm/archive/.cfg/Standard_Datatype_Extraction_Patterns"
	echo " "
	exit 0
fi

# echo "DEBUG: DOLLAR_1 = $1"
# echo "DEBUG: DOLLAR_2 = $2"


zstash_version=`zstash version`
if [ $zstash_version != "v0.4.1" ]; then
	echo "ABORTING:  zstash version is not 0.4.1 or is unavailable"
	exit 1
fi

IFS=$'\n'


AL_selection_line=$r21

AL_path=`echo $AL_selection_line | cut -f5 -d,`
AL_key=`echo $AL_selection_line | cut -f1-4 -d,`

ds_spec=$3	# realm-code nat freq_code, e.g. "atm nat mon", from Standard_Datatype_Extraction_Patterns
ds_key=`echo $ds_spec | tr ' ' _`

AM_key="$AL_key,$ds_key"
# echo "Produced AM_key: $AM_key"

# Determine realm for  midpath
realm_code=`echo $ds_spec | cut -f1 -d' '`

if [ $realm_code == "atm" ]; then
	realm="atmos"
elif [ $realm_code == "lnd" ]; then
	realm="land"
elif [ $realm_code == "ocn" ]; then
	realm="ocean"
elif [ $realm_code == "river" ]; then
	realm="river"
elif [ $realm_code == "sea-ice" ]; then
	realm="sea-ice"
else
	echo "ERROR: unrecognized realm code: $realm_code"
	exit 1
fi


midpath=$resolution/$realm/native/model-output

AM_list=`grep $AM_key $Archive_Map | sort`

listcount=`echo $AM_list | wc -l`

echo "Matched $listcount AM lines"
# may activate multiple Archive_Map lines, must loop.

cd $The_Holodeck

for am_line in $AM_list; do

	# echo "DEBUG: am_line = $am_line"

	campaign=`echo $am_line | cut -f1 -d,`
	modelver=`echo $am_line | cut -f2 -d,`
	exp_name=`echo $am_line | cut -f3 -d,`
	ensemble=`echo $am_line | cut -f4 -d,`
	arch_dir=`echo $am_line | cut -f6 -d,`
	tar_patt=`echo $am_line | cut -f7 -d,`

	# PLAN Step 1: test and exit if publication directory already exists.

	freq=`echo $ds_spec | cut -f3 -d' '`
	targ_pub_dir="$prepubroot/$modelver/$exp_name/$midpath/$freq/$ensemble/$pubvers"

	# echo "DEBUG: freqword = $freq"

	if [ $overwriteFlag -eq 0 ]; then
		if [ -d $targ_pub_dir ]; then
			fc=`ls $targ_pub_dir | wc -l`
			if [ $fc -gt 0 ]; then
				echo "ABORT: PUB DIR EXISTS ($fc files) $targ_pub_dir"
				echo " "
				continue
			fi
		fi
	fi

	# PLAN Step 2: Use the supplied Archive_Locator line to obtain the information regarding the archive source data

	echo "Conducting zstash extract for directory $arch_dir with dataset extraction pattern $tar_patt"
	echo "Target Publication Dir:  $targ_pub_dir"

	# continue



	# Real Stuff Below

	# PLAN Step 3:  populate the subordinate "zstash" subdirectory with simlinks to the appropriate tarfiles and index.db file.
	#    - Ensure holodeck contains only empty zstash subdirectory

	echo "Cleaning the Holodeck"
	echo " "
	rm -rf $The_Holodeck/*

	zstash_dir=$The_Holodeck/zstash
	mkdir $zstash_dir

	for targ in `ls $arch_dir`; do
		# touch nothing
		# echo "ln -s $arch_dir/$targ zstash/$targ"
		ln -s $arch_dir/$targ $zstash_dir/$targ
	done

	# Conduct extraction via the Holodeck

	echo "zstash extract --hpss=none $tar_patt"
	zstash extract --hpss=none $tar_patt
	exitcode=$?

	if [ $exitcode -ne 0 ]; then
		echo "ERROR:  zstash returned exitcode $exitcode"
		exit $exitcode
	fi


	echo "MKDIR: mkdir -p $targ_pub_dir"
	mkdir -p $targ_pub_dir
	chmod 755 $targ_pub_dir

	echo "mv $The_Holodeck/$file_extract_pattern $targ_pub_dir"
	mv $The_Holodeck/$tar_patt $targ_pub_dir
	chmod 644 $targ_pub_dir/*
	echo " "

	echo "Cleaning the Holodeck"
	echo " "
	rm -rf $The_Holodeck/*
done

exit 0
make_mapfile.sh0000750017062300020020000000067113707634136014467 0ustar  bartoletti1climate#!/bin/bash

# FORMAT
# esgmapfile make -i /path/to/ini/directory --max-processes <some_number_of_cores> --project <e3sm/cmip6> --outdir /output/location <path_to_data_directory>

dataset_fullpath=$1

echo "make_mapfile: processing: $dataset_fullpath"
esgmapfile make --debug -i /p/user_pub/e3sm/staging/ini_std/ --max-processes 20 --project e3sm --outdir /p/user_pub/e3sm/staging/mapfiles/mapfiles_auto_publish $dataset_fullpath

exit $?


MANIFEST0000640017062300020020000000316513707634135012650 0ustar  bartoletti1climateROOTDIR:/p/user_pub/e3sm/bartoletti1

alpha/

TARGET:alpha/MANIFEST

PUB_WORK/

TARGET:Pub_Work/1_Refactor/AA_Automation_via_Holodeck
TARGET:Pub_Work/1_Refactor/AA_README_Extraction
TARGET:Pub_Work/1_Refactor/extract_dataset_files_to.sh
TARGET:Pub_Work/1_Refactor/holodeck_stage_publication.sh
TARGET:Pub_Work/1_Refactor/publication_staging_control_script.sh
TARGET:Pub_Work/2_Mapwork/AA_README_MW
TARGET:Pub_Work/2_Mapwork/make_mapfile.sh
TARGET:Pub_Work/2_Mapwork/multi_mapfile_generate.sh
TARGET:Pub_Work/2_Mapwork/dirperms.sh
TARGET:Pub_Work/3_Publish/publish-all-stages-autodelete.sh
TARGET:Pub_Work/3_Publish/publish-all-stages.sh
TARGET:Pub_Work/3_Publish/publish-azb-v2.sh
TARGET:Pub_Work/3_Publish/publish-db-ingest-only.sh
TARGET:Pub_Work/3_Publish/publish-publish-only.sh
TARGET:Pub_Work/3_Publish/publish-thredds-n-publish-only.sh

PUB_STATUS/

TARGET:Pub_Status/all_dataset_status.py
TARGET:Pub_Status/ArchivePathMapper/AA_README_APM
TARGET:Pub_Status/ArchivePathMapper/archive_path_mapper_stage1.sh
TARGET:Pub_Status/ArchivePathMapper/archive_path_mapper_stage2.sh
TARGET:Pub_Status/ArchivePathMapper/archive_path_mapper_stage3.sh
TARGET:Pub_Status/ArchiveAssess/AA_README_AA
TARGET:Pub_Status/ArchiveAssess/archive_assess_count_first_last.sh
TARGET:Pub_Status/ArchiveAssess/archive_status_prep.sh
TARGET:Pub_Status/ArchiveAssess/archive_holodeck_setup_ls-only.sh
TARGET:Pub_Status/ArchiveAssess/archive_holodeck_setup.sh
TARGET:Pub_Status/sproket/process_sproket_output.sh
TARGET:Pub_Status/sproket/sproket_search.sh
TARGET:Pub_Status/AA_README_PS
TARGET:Pub_Status/archive_versus_pub_compare.sh
TARGET:Pub_Status/pub-stat-page-prep.sh


multi_mapfile_generate.sh0000750017062300020020000000176613707634136016564 0ustar  bartoletti1climate#!/bin/bash

workpath=/p/user_pub/e3sm/bartoletti1/Pub_Work/2_Mapwork

if [ $# -ne 1 ]; then
	echo "Usage:  multi_mapfile_Generate.sh map_status_control_file"
	exit 1
fi

map_status_file=$1

datasets=`grep -v DONE $map_status_file | grep -v HOLD`

startTime=`date +%s`

ts=`date +%Y%m%d.%H%M%S`
rlog=mfg_runlog-$ts

setcount=0;

for dataset in $datasets; do
	ts=`date +%Y%m%d.%H%M%S`
	ds_tm1=`date +%s`
	echo "STATUS:$ts:INPROCESS: dataset $dataset" >> $rlog 2>&1
	$workpath/make_mapfile.sh $dataset >> $rlog 2>&1
	ds_tm2=`date +%s`
	ds_et=$(($ds_tm2 - $ds_tm1))

	ts=`date +%Y%m%d.%H%M%S`
	if [ $? -eq 0 ]; then
		echo "STATUS:$ts:COMPLETED: dataset $dataset" >> $rlog 2>&1
	else
		echo "STATUS:$ts:FAILURE:   dataset $dataset (exit code $?)" >> $rlog 2>&1
	fi
	echo "STATUS: dataset ET = $ds_et" >> $rlog 2>&1
	setcount=$(($setcount + 1))
done

finalTime=`date +%s`
et=$(($finalTime - $startTime))

echo " " >> $rlog 2>&1
echo "Processed $setcount datasets." >> $rlog 2>&1
echo "Elapsed Time: $et" >> $rlog 2>&1
process_sproket_output.sh0000750017062300020020000000272713707634136016726 0ustar  bartoletti1climate#!/bin/bash

rm -f ESGF_publication_report

for aline in `cat E3SM_datafile_paths_leaf`; do
	filecount=`grep $aline E3SM_datafile_paths_full | wc -l`
	firstfile=`grep $aline E3SM_datafile_paths_full | rev | cut -f1 -d/ | rev | sort | uniq | head -1`
	finalfile=`grep $aline E3SM_datafile_paths_full | rev | cut -f1 -d/ | rev | sort | uniq | tail -1`
	datasetID=`echo $aline | tr / .`

	# handle variable sim-date year counts

	if [[ $firstfile =~ -*([0-9]{4}-[0-9]{2}) ]]; then

		# process YYYY-MM
		yearmo1=`echo ${BASH_REMATCH[1]}`
		if [[ $finalfile =~ -*([0-9]{4}-[0-9]{2}) ]]; then
			yearmo2=`echo ${BASH_REMATCH[1]}`
		else
			echo "NOMATCH,$filecount,$datasetID,$firstfile" >> ESGF_publication_report
			continue
		fi

		y1=`echo $yearmo1 | cut -c1-4`
		y2=`echo $yearmo2 | cut -c1-4`
		m2=`echo $yearmo2 | cut -c6-7`

		if [ $m2 == "01" ]; then
			yspan=$((y2 - y1))
		else
			yspan=$((y2 - y1 + 1))
		fi

		echo "$yspan,$filecount,$datasetID,$firstfile" >> ESGF_publication_report
		

	elif [[ $firstfile =~ -*([0-9]{6}_[0-9]{6}) ]]; then

		# process YYYYMM-YYYYMM first file only
		ym_ym=`echo ${BASH_REMATCH[1]}`

		y1=`echo $ym_ym | cut -c1-4`
		y2=`echo $ym_ym | cut -c8-11`
		m2=`echo $ym_ym | cut -c12-13`

		if [ $m2 == "01" ]; then
			yspan=$((y2 - y1))
		else
			yspan=$((y2 - y1 + 1))
		fi

		echo "$yspan,$filecount,$datasetID,$firstfile" >> ESGF_publication_report
	else
		echo "NOMATCH,$filecount,$datasetID,$firstfile" >> ESGF_publication_report
		continue
	fi

done


publication_staging_control_script.sh0000750017062300020020000000471313707634136021227 0ustar  bartoletti1climate#!/bin/bash


holodeck_stager="/p/user_pub/e3sm/bartoletti1/Pub_Work/1_Refactor/holodeck_stage_publication.sh"

USAGE=$(cat <<-END
usage: (this_script) jobset_configfile file_of_Archive_Locator_selectedlines
   The jobset_config file must contain lines:
       dstype_static=<type>    (where <type> examples are "atm nat", "ocn nat", "lnd reg", etc)
       dstype_freq_list=<list> (where <list> examples are "mon", "day 6hr 6hr_snap", etc)
       resolution=<res>        (where res is one of 1deg_atm_60-30km_ocean or 0_25deg_atm_18-6km_ocean)
       pubvers=<ver>           (where ver is one of v1, v2, etc)
       overwriteFlag=<0|1>     (Boolean, allows adding files to a non-empty destination directory)
END
)

# echo "$USAGE"
# exit

rm -f nohup.out

echo " "
echo "# Publication Staging Controller #"
echo " "

if [ $# -eq 0 ]; then
	echo "$USAGE"
	exit
fi

if [ $1 == "help" ]; then
	echo "$USAGE"
	exit
fi

# obtain values for 
#   dstype_static	(e.g.  "atm nat", "ocn reg", etc)
#   dstype_freq_list	(e.g.  "mon", "mon day 6hr 3hr", "6hr_snap day_cosp", etc)
#   
#   
#   
source $1


# file of Archive_Locator lines
AL_selected=$2


startTime=`date +%s`
ts=`date +%Y%m%d.%H%M%S`
holodeck_log=/p/user_pub/e3sm/bartoletti1/Pub_Work/1_Refactor/Holodeck_Process_Log-$ts

# Here, we prepare to process one or more Archive_Locator lines, even when multiple lines may refer to 
# different parts of a single dataset

casecount=0;

for AL_line in `cat $AL_selected`; do

	echo " " >> $holodeck_log 2>&1
	echo "$ts:Publication Staging Controller: Archive_Locator_line = $AL_line" >> $holodeck_log 2>&1
	echo " "

	datasets=0;
	# For the current case (Experiment / Ensemble), we are prepared only to pull one or more
	# frequencies for a single realm (e.g.  cam.h1, h2, h3 ...)

	ts=`date +%Y%m%d.%H%M%S`
	echo "$ts:Processing case spec: $AL_line" >> $holodeck_log 2>&1
	echo " "
	for freq in $dstype_freq_list; do
		argslist=()
		argslist[0]=$AL_line
		dstype="$dstype_static $freq"
		argslist[1]=$dstype
		ts=`date +%Y%m%d.%H%M%S`
		echo "$ts: Calling holodeck_stager with $AL_line \"$dstype\"" >> $holodeck_log 2>&1
		$holodeck_stager "${argslist[@]}" >> $holodeck_log 2>&1
		datasets=$(($datasets + 1))
	done
	ts=`date +%Y%m%d.%H%M%S`
	echo "$ts:Completion case spec: $AL_line ($datasets datasets)" >> $holodeck_log 2>&1
	casecount=$(($casecount + 1))
done

finalTime=`date +%s`
et=$(($finalTime - $startTime))

echo "Elapsed time: $et seconds.  ($casecount cases processed)" >> $holodeck_log


exit
	
publish-all-stages-autodelete.sh0000750017062300020020000001064713707634136017712 0ustar  bartoletti1climate#!/bin/bash

if [ $# -ne 1 ]; then
        echo "Please enter a descriptive job_name for tracking purposes"
        exit 1
fi

tagname=`echo $1 | tr ' ' _`


utcStart=`date +%s`
ts=`date +%Y%m%d.%H%M%S`

rlog="rlog-$ts-$tagname"

logdir=/p/user_pub/e3sm/bartoletti1/Pub_Work/3_Publish/runlogs
mapfiles_stage1=/p/user_pub/e3sm/staging/mapfiles/stage_1_db_ingest
mapfiles_stage2=/p/user_pub/e3sm/staging/mapfiles/stage_2_thredds_ingest
mapfiles_stage3=/p/user_pub/e3sm/staging/mapfiles/stage_3_publish
mapfiles_aborted=/p/user_pub/e3sm/staging/mapfiles/mapfiles_hold
mapfiles_archive=/p/user_pub/e3sm/staging/mapfiles/mapfiles_archive
ini_dir=/p/user_pub/e3sm/staging/ini_std
project=e3sm
email_recip=bartoletti1@llnl.gov

mapfiles=( `ls $mapfiles_stage1 | grep -v .part` )
mapfile_count=${#mapfiles[@]}


ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: mapfile ingestion to database ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
for mapfile in "${mapfiles[@]}"
do
	#check for deletion needed

	telltale=`echo $mapfile | rev | cut -f1-2 -d. | rev`
	vnum=`echo $telltale | cut -f1 -d. | cut -c2-`
	if [ $vnum -gt 1 ]; then
		vpre=$(($vnum - 1))
		echo "version is $vnum, deleting version $vpre"
		newtail="#$vpre"
		echo $mapfile > tmp_mystring
		sedcmd=`echo "s/.$telltale/$newtail/g"`
		sed -i $sedcmd tmp_mystring
		to_delete=`cat tmp_mystring`
		echo "$ts: INPROCESS: Calling esgunpublish for $to_delete" >> $logdir/$rlog 2>&1
		esgunpublish --project e3sm --database-delete --delete $to_delete >> $logdir/$rlog 2>&1
		ts=`date +%Y%m%d.%H%M%S`
		echo "$ts: COMPLETED: Deletion of dataset $to_delete" >> $logdir/$rlog 2>&1
	fi


    esgpublish -i $ini_dir --project $project --map $mapfiles_stage1/$mapfile --replace --commit-every 100  --no-thredds-reinit >> $logdir/$rlog 2>&1
    retcode=$?
    ts=`date +%Y%m%d.%H%M%S`
    if [ $retcode != 0 ] ; then
        echo "$ts: -- Failed to ingest $mapfile into the database" >> $logdir/$rlog 2>&1
	mv $mapfiles_stage1/$mapfile $mapfiles_aborted/$mapfile
        continue
    fi
    echo "$ts: -- Successfully ingested $mapfile into the database" >> $logdir/$rlog 2>&1
    mv $mapfiles_stage1/$mapfile $mapfiles_stage2/$mapfile
done

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile ingestion to database ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

mapfiles=( `ls $mapfiles_stage2` )
mapfile_count=${#mapfiles[@]}

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: mapfile ingestion to thredds ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfiles_stage2/$mapfile --service fileservice --noscan --thredds  --no-thredds-reinit >> $logdir/$rlog 2>&1
    retcode=$?
    ts=`date +%Y%m%d.%H%M%S`
    if [ $retcode != 0 ] ; then
        echo "$ts: -- Failed to ingest $mapfile into thredds" >> $logdir/$rlog 2>&1
	mv $mapfiles_stage2/$mapfile $mapfiles_aborted/$mapfile
	continue
    fi
    echo "$ts: -- Successfully ingested $mapfile into thredds" >> $logdir/$rlog 2>&1
    mv $mapfiles_stage2/$mapfile $mapfiles_stage3/$mapfile
done

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile ingestion to thredds ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: calling --thredds-reinit ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

esgpublish --project $project --thredds-reinit >> $logdir/$rlog 2>&1

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: thredds-reinit ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

mapfiles=( `ls $mapfiles_stage3` )
mapfile_count=${#mapfiles[@]}

echo "$ts: INPROCESS: mapfile publication ($mapfile_count mapfiles)" >> $logdir/$rlog
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfiles_stage3/$mapfile --service fileservice --noscan --publish >> $logdir/$rlog 2>&1
    retcode=$?
    ts=`date +%Y%m%d.%H%M%S`
    if [ $retcode != 0 ] ; then
        echo "$ts: -- Failed to publish $mapfile" >> $logdir/$rlog 2>&1
	mv $mapfiles_stage3/$mapfile $mapfiles_aborted/$mapfile
        continue
    fi
    echo Successfully published $mapfile >> $logdir/$rlog
    mv $mapfiles_stage3/$mapfile $mapfiles_archive/$mapfile
done



utcFinal=`date +%s`
elapsed=$(($utcFinal - $utcStart))

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile publication ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
echo "$ts: Elapsed time:  $elapsed seconds" >> $logdir/$rlog 2>&1

echo "All done ($tagname).  Elapsed time:  $elapsed seconds" | sendmail $email_recip

publish-all-stages.sh0000750017062300020020000000741413707634136015557 0ustar  bartoletti1climate#!/bin/bash

if [ $# -ne 1 ]; then
        echo "Please enter a descriptive job_name for tracking purposes"
        exit 1
fi

tagname=`echo $1 | tr ' ' _`


utcStart=`date +%s`
ts=`date +%Y%m%d.%H%M%S`

rlog="rlog-$ts-$tagname"

logdir=/p/user_pub/e3sm/bartoletti1/Pub_Work/3_Publish/runlogs
mapfiles_stage1=/p/user_pub/e3sm/staging/mapfiles/stage_1_db_ingest
mapfiles_stage2=/p/user_pub/e3sm/staging/mapfiles/stage_2_thredds_ingest
mapfiles_stage3=/p/user_pub/e3sm/staging/mapfiles/stage_3_publish
mapfiles_aborted=/p/user_pub/e3sm/staging/mapfiles/mapfiles_hold
mapfiles_archive=/p/user_pub/e3sm/staging/mapfiles/mapfiles_archive
ini_dir=/p/user_pub/e3sm/staging/ini_std
project=e3sm
email_recip=bartoletti1@llnl.gov

mapfiles=( `ls $mapfiles_stage1 | grep -v .part` )
mapfile_count=${#mapfiles[@]}

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: mapfile ingestion to database ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfiles_stage1/$mapfile --replace --commit-every 100  --no-thredds-reinit >> $logdir/$rlog 2>&1
    retcode=$?
    ts=`date +%Y%m%d.%H%M%S`
    if [ $retcode != 0 ] ; then
        echo "$ts: -- Failed to ingest $mapfile into the database" >> $logdir/$rlog 2>&1
	mv $mapfiles_stage1/$mapfile $mapfiles_aborted/$mapfile
        continue
    fi
    echo "$ts: -- Successfully ingested $mapfile into the database" >> $logdir/$rlog 2>&1
    mv $mapfiles_stage1/$mapfile $mapfiles_stage2/$mapfile
done

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile ingestion to database ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

mapfiles=( `ls $mapfiles_stage2` )
mapfile_count=${#mapfiles[@]}

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: mapfile ingestion to thredds ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfiles_stage2/$mapfile --service fileservice --noscan --thredds  --no-thredds-reinit >> $logdir/$rlog 2>&1
    retcode=$?
    ts=`date +%Y%m%d.%H%M%S`
    if [ $retcode != 0 ] ; then
        echo "$ts: -- Failed to ingest $mapfile into thredds" >> $logdir/$rlog 2>&1
	mv $mapfiles_stage2/$mapfile $mapfiles_aborted/$mapfile
	continue
    fi
    echo "$ts: -- Successfully ingested $mapfile into thredds" >> $logdir/$rlog 2>&1
    mv $mapfiles_stage2/$mapfile $mapfiles_stage3/$mapfile
done

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile ingestion to thredds ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: calling --thredds-reinit ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

esgpublish --project $project --thredds-reinit >> $logdir/$rlog 2>&1

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: thredds-reinit ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1

mapfiles=( `ls $mapfiles_stage3` )
mapfile_count=${#mapfiles[@]}

echo "$ts: INPROCESS: mapfile publication ($mapfile_count mapfiles)" >> $logdir/$rlog
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfiles_stage3/$mapfile --service fileservice --noscan --publish >> $logdir/$rlog 2>&1
    retcode=$?
    ts=`date +%Y%m%d.%H%M%S`
    if [ $retcode != 0 ] ; then
        echo "$ts: -- Failed to publish $mapfile" >> $logdir/$rlog 2>&1
	mv $mapfiles_stage3/$mapfile $mapfiles_aborted/$mapfile
        continue
    fi
    echo Successfully published $mapfile >> $logdir/$rlog
    mv $mapfiles_stage3/$mapfile $mapfiles_archive/$mapfile
done



utcFinal=`date +%s`
elapsed=$(($utcFinal - $utcStart))

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile publication ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
echo "$ts: Elapsed time:  $elapsed seconds" >> $logdir/$rlog 2>&1

echo "All done ($tagname).  Elapsed time:  $elapsed seconds" | sendmail $email_recip

publish-azb-v2.sh0000750017062300020020000000344413707634136014623 0ustar  bartoletti1climate#!/bin/bash

utcStart=`date +%s`

mapfile_dir=/p/user_pub/e3sm/staging/mapfiles
mapfile_done=/p/user_pub/e3sm/staging/mapfiles_temp_done
project=e3sm
email_recip=bartoletti1@llnl.gov

ini_dir=/p/user_pub/e3sm/staging/ini_old

exit_on_any_fail=1

mapfile_count=`ls $mapfile_dir | wc -l`
mapfiles_published=0

mapfiles=( `ls $mapfile_dir` )

for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfile_dir/$mapfile --commit-every 100  --no-thredds-reinit
    if [ $? != 0 ] ; then
        echo Failed to engest $mapfile into the database
	if [ $exit_on_any_fail ]; then
            exit 1
        else
            continue
        fi
    fi
    echo Successfully engested $mapfile into the database

    esgpublish -i $ini_dir --project $project --map $mapfile_dir/$mapfile --service fileservice --noscan --thredds  --no-thredds-reinit
    if [ $? != 0 ] ; then
        echo Failed to engest $mapfile into thredds
	if [ $exit_on_any_fail ]; then
            exit 1
        else
            continue
        fi
    fi
    echo Successfully engested $mapfile into thredds

    esgpublish --project $project --thredds-reinit

    esgpublish -i $ini_dir --project $project --map $mapfile_dir/$mapfile --service fileservice --noscan --publish
    if [ $? != 0 ] ; then
        echo Failed to publish $mapfile
	if [ $exit_on_any_fail ]; then
            exit 1
        else
            continue
        fi
    fi
    echo Successfully published $mapfile

    mapfiles_published=$(($mapfiles_published + 1))

    mv -v $mapfile_dir/$mapfile $mapfile_done/

done

esgpublish --project $project --thredds-reinit

utcFinal=`date +%s`
elapsed=$(($utcFinal - $utcStart))

echo "All done.  Published $mapfiles_published of $mapfile_count mapfiles.  Elapsed time:  $elapsed seconds" | sendmail $email_recip

publish-db-ingest-only.sh0000750017062300020020000000201013707634136016341 0ustar  bartoletti1climate#!/bin/bash

utcStart=`date +%s`

tag="BGC_CTC_raw - DB Ingest-Only"
mapfile_dir=/p/user_pub/e3sm/staging/mapfiles/mapfiles_bgc_raw
mapfile_done=/p/user_pub/e3sm/staging/mapfiles/mapfiles_archive/BGC/mapfiles_bgc_raw
project=e3sm
email_recip=bartoletti1@llnl.gov

ini_dir=/p/user_pub/e3sm/staging/ini_std

mapfiles=( `ls $mapfile_dir` )
mapfile_count=${#mapfiles[@]}

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: mapfile ingestion to database ($mapfile_count mapfiles)"
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfile_dir/$mapfile --commit-every 100  --no-thredds-reinit
    if [ $? != 0 ] ; then
        echo Failed to engest $mapfile into the database
        exit 1
    fi
    echo Successfully engested $mapfile into the database
done
ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile ingestion to database ($mapfile_count mapfiles)"

utcFinal=`date +%s`
elapsed=$(($utcFinal - $utcStart))

echo "All done ($tag).  Elapsed time:  $elapsed seconds" | sendmail $email_recip

publish-publish-only.sh0000750017062300020020000000323313707634136016143 0ustar  bartoletti1climate#!/bin/bash

if [ $# -ne 1 ]; then
	echo "Please enter a descriptive job_name for tracking purposes"
	exit 1
fi

tag=`echo $1 | tr ' ' _`


utcStart=`date +%s`
ts=`date +%Y%m%d.%H%M%S`

rlog="rlog-$ts-$tag"

logdir=/p/user_pub/e3sm/bartoletti1/Pub_Work/3_Publish/runlogs
mapfile_indir=/p/user_pub/e3sm/staging/mapfiles/stage_3_publish
mapfile_exdir=/p/user_pub/e3sm/staging/mapfiles/mapfiles_archive/BGC/mapfiles_bgc_raw
ini_dir=/p/user_pub/e3sm/staging/ini_std
project=e3sm
email_recip=bartoletti1@llnl.gov


mapfiles=( `ls $mapfile_indir` )
mapfile_count=${#mapfiles[@]}

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: calling --thredds-reinit ($mapfile_count mapfiles)" > $logdir/$rlog 2>&1

esgpublish --project $project --thredds-reinit >> $logdir/$rlog 2>&1

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: thredds-reinit ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
echo "$ts: INPROCESS: mapfile publication ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfile_indir/$mapfile --service fileservice --noscan --publish >> $logdir/$rlog 2>&1
    if [ $? != 0 ] ; then
        echo Failed to publish $mapfile >> $logdir/$rlog 2>&1
        exit 1
    fi
    echo Successfully published $mapfile >> $logdir/$rlog 2>&1
    mv -v $mapfile_indir/$mapfile $mapfile_exdir/
done

utcFinal=`date +%s`
elapsed=$(($utcFinal - $utcStart))

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile publication ($mapfile_count mapfiles)" >> $logdir/$rlog 2>&1
echo "$ts: Elapsed time:  $elapsed seconds" >> $logdir/$rlog 2>&1

echo "All done ($tag).  Elapsed time:  $elapsed seconds" | sendmail $email_recip

publish-thredds-n-publish-only.sh0000750017062300020020000000306713707634136020036 0ustar  bartoletti1climate#!/bin/bash

utcStart=`date +%s`

tag=BGC_CTC_raw
mapfile_dir=/p/user_pub/e3sm/staging/mapfiles/mapfiles_bgc_raw
mapfile_done=/p/user_pub/e3sm/staging/mapfiles/mapfiles_archive/BGC/mapfiles_bgc_raw
project=e3sm
email_recip=bartoletti1@llnl.gov

ini_dir=/p/user_pub/e3sm/staging/ini_std

mapfiles=( `ls $mapfile_dir` )
mapfile_count=${#mapfiles[@]}

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: mapfile ingestion to thredds ($mapfile_count mapfiles)"
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfile_dir/$mapfile --service fileservice --noscan --thredds  --no-thredds-reinit
    if [ $? != 0 ] ; then
        echo Failed to engest $mapfile into thredds
        exit 1
    fi
    echo Successfully engested $mapfile into thredds
done
ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile ingestion to thredds ($mapfile_count mapfiles)"

esgpublish --project $project --thredds-reinit

ts=`date +%Y%m%d.%H%M%S`
echo "$ts: INPROCESS: mapfile publication ($mapfile_count mapfiles)"
for mapfile in "${mapfiles[@]}"
do
    esgpublish -i $ini_dir --project $project --map $mapfile_dir/$mapfile --service fileservice --noscan --publish
    if [ $? != 0 ] ; then
        echo Failed to publish $mapfile
        exit 1
    fi
    echo Successfully published $mapfile
    mv -v $mapfile_dir/$mapfile $mapfile_done/
done
ts=`date +%Y%m%d.%H%M%S`
echo "$ts: COMPLETED: mapfile publication ($mapfile_count mapfiles)"

utcFinal=`date +%s`
elapsed=$(($utcFinal - $utcStart))

echo "All done ($tag).  Elapsed time:  $elapsed seconds" | sendmail $email_recip

pub-stat-page-prep.sh0000750017062300020020000000421113707634136015464 0ustar  bartoletti1climate#!/bin/bash

# 
pathhead=$1

catalogeur=/p/user_pub/e3sm/bartoletti1/Pub_Status/catalog_E3SM_PUB.sh

pretag=`date +%s`

for ens in ens1 ens2 ens3 ens4 ens5; do
	for v in v1 v2 v3; do
		$catalogeur $pathhead $ens.$v $pretag
	done
done

genlist=`ls | grep $pretag`
onefile=`ls | grep $pretag | head -1`

newtag=`echo $onefile | cut -f2-3 -d-`
now_utc=`date -u +%Y%m%d.%H%M`
count_pathlist_v="$newtag-$now_utc"

cat ${pretag}* | sort | uniq > $count_pathlist_v

rm ${pretag}*


# Now for part 2:

inputform="count,pubpath"
datasetID="Project.ModelVer.Experiment.Resolution[.F2010_hi_low].Realm.Grid.Outtype.Freq.Ens.vn"

now_utc=`date -u +%Y%m%d.%H%M`

outfile=task-stat-list3-$now_utc

echo "ModVer,Resolution,Ens,Title,Status,Grid,Years,Filecount,Dataset_ID,Dataset_First_File,Dataset_Publication_Path,Notes,Title_Expanded" > $outfile

for aline in `cat $count_pathlist_v`; do

	fcount=`echo $aline | cut -f1 -d,`
	apath=`echo $aline | cut -f2 -d,`
	# avers=`echo $aline | cut -f3 -d,`

	ffile=`ls $apath | head -1`

	# NOTE: requires (pub) env to access ncdump
	# tperiod=`ncdump -h $apath/$ffile | grep "time_period_freq" | cut -f2 -d=`

	dsid=`echo $apath | tr / . | cut -f5- -d.`

	modver=`echo $dsid | cut -f2 -d.`
	experiment=`echo $dsid | cut -f3 -d.`

	if [[ $experiment =~ "F2010" ]]; then
		resstr=`echo $dsid | cut -f4 -d.`
		realm=`echo $dsid | cut -f6 -d.`
		grid=`echo $dsid | cut -f7 -d.`
		outtype=`echo $dsid | cut -f8 -d.`
		freq=`echo $dsid | cut -f9 -d.`
		ens=`echo $dsid | cut -f10 -d.`
	else
		resstr=`echo $dsid | cut -f4 -d.`
		realm=`echo $dsid | cut -f5 -d.`
		grid=`echo $dsid | cut -f6 -d.`
		outtype=`echo $dsid | cut -f7 -d.`
		freq=`echo $dsid | cut -f8 -d.`
		ens=`echo $dsid | cut -f9 -d.`
	fi


	if [ $realm == "atmos" ]; then
		realmcode="atm"
	elif [ $realm == "land" ]; then
		realmcode="lnd"
	elif [ $realm == "ocean" ]; then
		realmcode="ocn"
	elif [ $realm == "sea-ice" ]; then
		realmcode="sice"
	else	realmcode="misc"
	fi

	gridcode="nat"
	if [ $grid != "native" ]; then
		gridcode="reg"
	fi

	title="$realmcode $gridcode $freq"



	echo "$modver,$resstr,$ens,$title,published,$grid,,$fcount,$dsid,$ffile,$apath,$outtype," >> $outfile

done


sproket_search.sh0000750017062300020020000000126413707634136015070 0ustar  bartoletti1climate#!/bin/bash

sprok=/p/user_pub/e3sm/bartoletti1/abin/sproket-linux-0.2.13
config=/p/user_pub/e3sm/bartoletti1/abin/.sproket_config.json

# to obtain ALL file names
# $sprok -config $config -urls.only -no.download -no.verify -y > E3SM_datafiles
# to obtain the datasetIDs
$sprok -config $config -values.for dataset_id_template_ -no.download -no.verify -y > E3SM_datasets

exit 0








$sprok -config $config -field.keys > E3SM_fields

IFS=$'\n'

rm -f E3SM_field_values

for afield in `cat E3SM_Fields_useful`; do
	echo " " >> E3SM_field_values
	echo "FIELD_$afield" >> E3SM_field_values
	echo " " >> E3SM_field_values
	$sprok -config $config -values.for $afield >> E3SM_field_values
done