[WIP] Stage 091 (rucioDatasets) #79
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This script gets the NDJSON lines of tasks metadata from Oracle2JSON.py and convert 'input' and 'output' strings of datasets to arrays, adding 'bytes' from Rucio.
- removed unused 'remove_tid()' function as we mustn't remove _tid part from dataset names
- 'input' datasets are out of our scope, so only 'output' dataset types was left.
- added 'deleted' boolean value:
dataset considered to be deleted if it wasn't found in Rucio
- the NULL value of dataset 'bytes' is set to -1
Added: - comments in datasets_processing.py script - README file - input / output examples files
- primary_input - ctag - output_formats
- index name changed from 'prodsys' to 'prodsys_rucio_ami', as we're going to store parameters from these 3 storages - added type 'task' as PARENT - added type 'output_datasets' as CHILD of 'task' new fields was added: - primary_input (prodSys) - ctag (prodSys) - output_formats (prodSys) - cross_section_ref (AMI) - cross_section (AMI) - get_filt_eff (AMI) - k-factor (AMI) 'input' datasets was removed and substituted to 'primary_input'
Is was decided to substitute array of 'input' dataset to single 'primary_input' string.
This was referenced Nov 22, 2017
mgolosova
changed the title
mgolosova
changed the title
4 tasks
mgolosova
added a commit
that referenced
this pull request
Nov 28, 2017
Stage 019 transformation Now it can accept data not only from Stage 016, but also from 091 (see #79) -- or any other stage, that produces data in the required format.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add information about datasets from Rucio to the ora2es chain (009-019-069).