Sample YAML generated from spreadsheets #5
Replies: 10 comments 16 replies
-
|
Here is the json for CG0373: {
"id": "abcffc09-b50f-46a2-bea0-0baba8d3e3c1",
"created": "2021-11-06T09:44:41Z",
"changed": "2023-03-15T16:53:41Z",
"creator": {
"id": "dd0f9aa3-68f9-4825-84a4-86c8303daaff"
},
"content": "changed: '2023-03-15T16:53:41Z'\ncontent: null\ncreated: '2021-11-06T09:44:41Z'\ncreator:\n id: dd0f9aa3-68f9-4825-84a4-86c8303daaff\nid: abcffc09-b50f-46a2-bea0-0baba8d3e3c1\njson:\n Authorities:\n - Organization: CDISC\n Standards:\n - Name: SDTMIG\n References:\n - Citations:\n - Cited_Guidance: There is a one-to-one correspondence between a domain dataset\n and it's Supplemental Qualifier dataset by creating one SUPPQUAL for each\n domain dataset. The single SUPPQUAL dataset option that was introduced\n in SDTMIG v3.1.1 is now deprecated. The set of Supplemental Qualifiers\n for each domain is included in a separate dataset with the name SUPP--\n where '--' denotes the source domain which the supplemental Qualifiers\n relate back to.\n Document: IG v3.2\n Item: ''\n Section: 8.4.2\n Origin: SDTM and SDTMIG Conformance Rules\n Rule_Identifier:\n Id: CG0373\n Version: '1'\n Version: '2.0'\n Version: '3.2'\n - Name: SDTMIG\n References:\n - Citations:\n - Cited_Guidance: There is a one-to-one correspondence between a domain dataset\n and its Supplemental Qualifier dataset. The single SUPPQUAL dataset option\n that was introduced in SDTMIG v3.1 was deprecated. The set of Supplemental\n Qualifiers for each domain is included in a separate dataset with the\n name SUPP-- where \"--\" denotes the source domain which the Supplemental\n Qualifiers relate back to.\n Document: IG v3.3\n Item: ''\n Section: 8.4.2\n Origin: SDTM and SDTMIG Conformance Rules\n Rule_Identifier:\n Id: CG0373\n Version: '1'\n Version: '2.0'\n Version: '3.3'\n - Name: SDTMIG\n References:\n - Citations:\n - Cited_Guidance: \"There is a one-to-one correspondence between a domain dataset\\\n \\ and it's Supplemental Qualifier dataset by creating one SUPPQUAL for\\\n \\ each domain dataset. The single SUPPQUAL dataset option that was introduced\\\n \\ in SDTMIG v3.1.1 is now deprecated. The set of Supplemental Qualifiers\\\n \\ for each domain is included in a separate dataset with the name SUPP--\\\n \\ where \\u201C--\\u201C denotes the source domain which the supplemental\\\n \\ Qualifiers relate back to.\"\n Document: IG v3.2\n Item: ''\n Section: 8.4.2\n Origin: SDTM and SDTMIG Conformance Rules\n Rule_Identifier:\n Id: CG0373\n Version: '1'\n Version: '2.0'\n Version: '3.4'\n Check:\n Check: null\n Core:\n Id: CDISC.SDTMIG.CG0373\n Status: Draft\n Version: '1'\n Description: Trigger error when Dataset present in study with DOMAIN not equal to\n SUPP--.RDOMAIN\n Executability: null\n Outcome:\n Message: Dataset present in study with DOMAIN not equal to SUPP--.RDOMAIN\n Rule_Type: Record Data\n Scope:\n Classes:\n Include:\n - ALL\n Domains:\n Include:\n - ALL\n Sensitivity: Record\n",
"json": {
"Check": {
"Check": null
},
"Core": {
"Id": "CDISC.SDTMIG.CG0373",
"Version": "1",
"Status": "Draft"
},
"Description": "Trigger error when Dataset present in study with DOMAIN not equal to SUPP--.RDOMAIN",
"Outcome": {
"Message": "Dataset present in study with DOMAIN not equal to SUPP--.RDOMAIN"
},
"Rule_Type": "Record Data",
"Sensitivity": "Record",
"Authorities": [
{
"Organization": "CDISC",
"Standards": [
{
"Name": "SDTMIG",
"Version": "3.2",
"References": [
{
"Origin": "SDTM and SDTMIG Conformance Rules",
"Rule_Identifier": {
"Id": "CG0373",
"Version": "1"
},
"Version": "2.0",
"Citations": [
{
"Cited_Guidance": "There is a one-to-one correspondence between a domain dataset and it's Supplemental Qualifier dataset by creating one SUPPQUAL for each domain dataset. The single SUPPQUAL dataset option that was introduced in SDTMIG v3.1.1 is now deprecated. The set of Supplemental Qualifiers for each domain is included in a separate dataset with the name SUPP-- where '--' denotes the source domain which the supplemental Qualifiers relate back to.",
"Document": "IG v3.2",
"Item": "",
"Section": "8.4.2"
}
]
}
]
},
{
"Name": "SDTMIG",
"Version": "3.3",
"References": [
{
"Origin": "SDTM and SDTMIG Conformance Rules",
"Rule_Identifier": {
"Id": "CG0373",
"Version": "1"
},
"Version": "2.0",
"Citations": [
{
"Cited_Guidance": "There is a one-to-one correspondence between a domain dataset and its Supplemental Qualifier dataset. The single SUPPQUAL dataset option that was introduced in SDTMIG v3.1 was deprecated. The set of Supplemental Qualifiers for each domain is included in a separate dataset with the name SUPP-- where \"--\" denotes the source domain which the Supplemental Qualifiers relate back to.",
"Document": "IG v3.3",
"Item": "",
"Section": "8.4.2"
}
]
}
]
},
{
"Name": "SDTMIG",
"Version": "3.4",
"References": [
{
"Origin": "SDTM and SDTMIG Conformance Rules",
"Rule_Identifier": {
"Id": "CG0373",
"Version": "1"
},
"Version": "2.0",
"Citations": [
{
"Cited_Guidance": "There is a one-to-one correspondence between a domain dataset and it's Supplemental Qualifier dataset by creating one SUPPQUAL for each domain dataset. The single SUPPQUAL dataset option that was introduced in SDTMIG v3.1.1 is now deprecated. The set of Supplemental Qualifiers for each domain is included in a separate dataset with the name SUPP-- where \u201c--\u201c denotes the source domain which the supplemental Qualifiers relate back to.",
"Document": "IG v3.2",
"Item": "",
"Section": "8.4.2"
}
]
}
]
}
]
}
],
"Scope": {
"Classes": {
"Include": [
"ALL"
]
},
"Domains": {
"Include": [
"ALL"
]
}
},
"Executability": null
}
} |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
@htu these look great. Some comments about your samples:
|
Beta Was this translation helpful? Give feedback.
-
|
@gerrycampion Thanks for providing those points. I will work on them. Could you elaborate a little on this one: "The property names in the yaml file should not have underscores. The property names in the json file should have underscores." Or give an example? |
Beta Was this translation helpful? Give feedback.
-
|
@htu Have a look at Rule Type (Rule_Type) and Output Variables (Output_Variables) in the yaml, the json.content, and the json.json below. Check:
all:
- name: --TPTREF
operator: empty
- name: --ELTM
operator: non_empty
Core:
Id: CORE-000028
Version: '1'
Status: Published
Description: Raise an error when --TPTREF is empty, but --ELTM is not empty
Outcome:
Message: --TPTREF is empty and --ELTM is not empty
Output Variables:
- --TPTREF
- --ELTM
Rule Type: Record Data{
"id": "8e4ce5f6-5d7d-420c-887d-26c09b89b793",
"changed": "2022-12-02T16:10:42.162Z",
"content": "Check:\n all:\n - name: --TPTREF\n operator: empty\n - name: --ELTM\n operator: non_empty\nCore:\n Id: CORE-000028\n Version: '1'\n Status: Published\nDescription: Raise an error when --TPTREF is empty, but --ELTM is not empty\nOutcome:\n Message: --TPTREF is empty and --ELTM is not empty\n Output Variables:\n - --TPTREF\n - --ELTM\nRule Type: Record Data\nSensitivity: Record\nAuthorities:\n - Organization: CDISC\n Standards:\n - Name: SDTMIG\n Version: '3.4'\n References:\n - Origin: SDTM and SDTMIG Conformance Rules\n Rule Identifier:\n Id: CG0008\n Version: '1'\n Version: '2.0'\n Citations:\n - Cited Guidance: The interval of time between a planned time point\n and a fixed reference point, represented in a standardized character\n format. The fixed reference point is in --TPTREF.\n Document: SDTM Model v2.0\n Item: --ELTM\n Section: Timing\n - Name: SDTMIG\n Version: '3.3'\n References:\n - Origin: SDTM and SDTMIG Conformance Rules\n Rule Identifier:\n Id: CG0008\n Version: '1'\n Version: '2.0'\n Citations:\n - Cited Guidance: Planned Elapsed time relative to a planned fixed reference\n (--TPTREF) such as \"Previous Dose\" or \"Previous Meal\". This variable\n is useful where there are repetitive measures. Not a clock time\n or a date/time variable, but an interval. Format=ISO8601\n Document: SDTM Model v1.7\n Item: Table 2.2.5, --ELTM\n Section: 2.2.5\n - Name: SDTMIG\n Version: '3.2'\n References:\n - Origin: SDTM and SDTMIG Conformance Rules\n Rule Identifier:\n Id: CG0008\n Version: '1'\n Version: '2.0'\n Citations:\n - Cited Guidance: Planned Elapsed time in ISO 8601 character format\n relative to a planned fixed reference (--TPTREF) such as \"Previous\n Dose\" or \"Previous Meal\".\n Document: SDTM Model v1.4\n Item: Table 2.2.5, --ELTM\n Section: 2.2.5\nScope:\n Classes:\n Include:\n - ALL\n Domains:\n Include:\n - ALL\n",
"created": "2021-09-29T22:40:39Z",
"creator": {
"id": "e7bbd139-9a5c-4b86-859e-e4854e7724fe"
},
"json": {
"Check": {
"all": [
{
"name": "--TPTREF",
"operator": "empty"
},
{
"name": "--ELTM",
"operator": "non_empty"
}
]
},
"Core": {
"Id": "CORE-000028",
"Version": "1",
"Status": "Published"
},
"Description": "Raise an error when --TPTREF is empty, but --ELTM is not empty",
"Outcome": {
"Message": "--TPTREF is empty and --ELTM is not empty",
"Output_Variables": [
"--TPTREF",
"--ELTM"
]
},
"Rule_Type": "Record Data"
} |
Beta Was this translation helpful? Give feedback.
-
|
Mappings from Class to Classes: Include: Mappings from Class, Domain to Classes: Include: Mappings from Class to Domains: NOT (***) in Class or Domain should be mapped to Exclude: instead of Include: No Mapping - RELATIONSHIP |
Beta Was this translation helpful? Give feedback.
-
|
It seems that CG0365~CG0367 follows the mapping that you suggested. These three are published. Here are the status for those records: |
Beta Was this translation helpful? Give feedback.
-
|
It seems that there is no consistency in terms of interpreting the mapping. Please see CG0367 (published): |
Beta Was this translation helpful? Give feedback.
-
|
@htu CG0367 contains a bug. |
Beta Was this translation helpful? Give feedback.
-
|
FYI, I went through and fixed the SPECIAL-PURPOSE hyphens in prod |
Beta Was this translation helpful? Give feedback.
-
|
For 367, I see you were talking about Special Purpose vs AP. I don't know why the rule was written this way. It should be written using Class: ALL, Domain: AP-- |
Beta Was this translation helpful? Give feedback.
-
|
For reference, the CT list that has the class names: |
Beta Was this translation helpful? Give feedback.
-
|
@gerrycampion could you copy the prod to the dev so that I can grab the changes that you made? Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
@htu I copied these |
Beta Was this translation helpful? Give feedback.
-
|
I tested a few on existing rules and it merges citations correctly and passed the rule test. Here is CG0180: |
Beta Was this translation helpful? Give feedback.
-
# Variable: LBORNRLO
# Condition: LBORRES ^= continuous measurement
# Rule: LBORNRLO = null
Core:
Id: CDISC.SDTMIG.CG0180
Version: '1'
Status: Draft
Description: Raise an error when LBORRES ^= continuous measurement and LBORNRLO is
not empty.
Outcome:
Message: LBORRES is contnot inuous measurement and LBORNRLO not equal to null
Rule Type: Record Data
Sensitivity: Record
Authorities:
- Organization: CDISC
Standards:
- Name: SDTMIG
Version: '3.2'
References:
- Origin: SDTM and SDTMIG Conformance Rules
Rule Identifier:
Id: CG0180
Version: '1'
Version: '2.0'
Citations:
- Cited Guidance: Lower end of reference range for continuous measurements
in original units. Should be populated only for continuous results.
Document: IG v3.2
Item: Specification
Section: '6.3'
- Name: SDTMIG
Version: '3.3'
References:
- Origin: SDTM and SDTMIG Conformance Rules
Rule Identifier:
Id: CG0180
Version: '1'
Version: '2.0'
Citations:
- Cited Guidance: Lower end of reference range for continuous measurements
in original units. Should be populated only for continuous results.
Document: IG v3.3
Item: Specification
Section: 6.3.6
- Name: SDTMIG
Version: '3.4'
References:
- Origin: SDTM and SDTMIG Conformance Rules
Rule Identifier:
Id: CG0180
Version: '1'
Version: '2.0'
Citations:
- Cited Guidance: Lower end of reference range for continuous measurements
for LBSTRESC/LBSTRESN in standardized units. Should be populated
only for continuous results.
Document: SDTMIG v3.4
Item: Specification
Section: 6.3.5.6
Scope:
Classes:
Include:
- FINDINGS
Domains:
Include:
- LB
Executability: Partially Executable
Check:
all:
- name: LBORRES
operator: non_empty
- name: LBORNRLO
operator: non_empty |
Beta Was this translation helpful? Give feedback.
-
|
Here is a new one: # Variable: GEN
# Condition: Variable Core Status = Required
# Rule: Variable present in dataset and ^= null
Core:
Id: CDISC.SDTMIG.CG0014
Version: '1'
Status: Draft
Description: Trigger error when Variable Core Status not equal to Required and Variable
present not in dataset and is null
Outcome:
Message: Variable Core Status not equal to Required and Variable present not in
dataset and is null
Rule Type: Record Data
Sensitivity: Variable
Authorities:
- Organization: CDISC
Standards:
- Name: SDTMIG
Version: '3.2'
References:
- Origin: SDTM and SDTMIG Conformance Rules
Rule Identifier:
Id: CG0014
Version: '1'
Version: '2.0'
Citations:
- Cited Guidance: Required variables must always be included in the
dataset and cannot be null for any record.
Document: IG v3.2
Section: 4.1.1.5
- Name: SDTMIG
Version: '3.3'
References:
- Origin: SDTM and SDTMIG Conformance Rules
Rule Identifier:
Id: CG0014
Version: '1'
Version: '2.0'
Citations:
- Cited Guidance: Required variables must always be included in the
dataset and cannot be null for any record.
Document: IG v3.3
Section: 4.1.5
- Name: SDTMIG
Version: '3.4'
References:
- Origin: SDTM and SDTMIG Conformance Rules
Rule Identifier:
Id: CG0014
Version: '1'
Version: '2.0'
Citations:
- Cited Guidance: Required variables must always be included in the
dataset and cannot be null for any record.
Document: IG v3.4
Section: 4.1.5
Scope:
Classes:
Include:
- ALL
Domains:
Include:
- ALL
Executability: Partially Executable
Check:
all: []
|
Beta Was this translation helpful? Give feedback.
-
|
This is a non executable CG0196 after running through the rule builder program: # Variable: PEORRES
# Condition: PEORRES ^= null and no abnormal findings
# Rule: PEORRES = 'NORMAL'
Core:
Id: CDISC.SDTMIG.CG0196 (NOT EXECUTABLE)
Version: '1'
Status: Draft
Description: Trigger error when PEORRES is null and no abnormal fnot indnot ings and
PEORRES not equal to 'NORMAL'
Outcome:
Message: PEORRES is null and no abnormal fnot indnot ings and PEORRES not equal
to 'NORMAL'
Rule Type: Record Data
Sensitivity: Record
Authorities:
- Organization: CDISC
Standards:
- Name: SDTMIG
Version: '3.2'
References:
- Origin: SDTM and SDTMIG Conformance Rules
Rule Identifier:
Id: CG0196
Version: '1'
Version: '2.0'
Citations:
- Cited Guidance: Text description of any abnormal findings. If the
examination was completed and there were no abnormal findings, the
value should be NORMAL.
Document: IG v3.2
Section: '6.3'
- Name: SDTMIG
Version: '3.3'
References:
- Origin: SDTM and SDTMIG Conformance Rules
Rule Identifier:
Id: CG0196
Version: '1'
Version: '2.0'
Citations:
- Cited Guidance: Text description of any abnormal findings. If the
examination was completed and there were no abnormal findings, the
value should be NORMAL.
Document: IG v3.3
Section: 6.3.12
- Name: SDTMIG
Version: '3.4'
References:
- Origin: SDTM and SDTMIG Conformance Rules
Rule Identifier:
Id: CG0196
Version: '1'
Version: '2.0'
Citations:
- Cited Guidance: If the examination was completed and there were no
abnormal findings, the value should be "NORMAL".
Document: IG v3.4
Item: Specification
Section: 6.3.8
Scope:
Classes:
Include:
- FINDINGS
Domains:
Include:
- PE
Executability: Partially Executable
Check:
all: [] |
Beta Was this translation helpful? Give feedback.
-
|
@htu These look really good. 2 things:
|
Beta Was this translation helpful? Give feedback.
-
|
How do we determine Rule Type and Sensitivity? I am trying to look into the condition in the rule definition spreadsheet to make the assessment. If it is not the right approach, I can simply leave them null. |
Beta Was this translation helpful? Give feedback.
-
|
These are both useful, but I think they should be outside of the scope of this initial assignment due to their complexity. I know the team is looking to improve the consistency of the description and messages, but I would add this as a separate issue ticket in github. I don't know of a straightforward way to determine the Rule Type and Sensitivity based on the spreadsheet. If you share your logic and it is straightforward, we can keep it, but if they aren't 100% accurate, it will probably be better to leave them null so that the authors know it is something that needs to be addressed instead of assuming it is correct. |
Beta Was this translation helpful? Give feedback.
-
|
Here is the logic that I am trying to use to determine the rule type: import re
import os
import pandas as pd
from rulebuilder.echo_msg import echo_msg
from rulebuilder.read_rules import read_rules
# from rulebuilder.get_existing_rule import get_existing_rule
def get_rtype(rule_data, exist_rule_data: dict = {}):
"""
===============
get_rtype
===============
This method builds the json.Rule_Type element in the core rule.
Parameters:
-----------
rule_data: dataframe
a data frame containing all the records for a rule. It can be obtained from
read_rules and select the records from the rule definition data frame.
existing_rule_data: dict
a data frame containing all the records for a rule that already developed. It
can be read from the existing rule folder using get_existing_rule.
returns
-------
r_str: a string for json.Rule_Type
possible rule type:
- Dataset Contents Check against Define XML and Library Metadata
- Dataset Metadata Check
- Dataset Metadata Check against Define XML
- Define-XML
- Domain Presence Check
- Record Data
- Value Level Metadata Check against Define XML
- Variable Metadata Check
- Variable Metadata Check against Define XML
Raises
------
ValueError
None
"""
v_prg = __name__
v_stp = 1.0
v_msg = "Setting parameters..."
echo_msg(v_prg, v_stp, v_msg, 3)
# if not rule_data.empty:
# return None
v_stp = 1.1
v_msg = "Input parameter rule_data is empty."
if rule_data.empty:
echo_msg(v_prg, v_stp, v_msg, 0)
return None
r_str = exist_rule_data.get("json", {}).get("Rule_Type")
if r_str is None:
r_condition = rule_data.iloc[0]["Condition"]
# r_rule = rule_data.iloc[0]["Rule"]
if r_condition is None:
v_stp = 1.2
v_msg = "No Condition is found in the rule definition."
echo_msg(v_prg, v_stp, v_msg, 0)
return None
r_str = "Record Data"
pattern = r"^(study|dataset)"
# Use the re.search() method to search for the pattern in the input string
match = re.search(pattern, r_condition, re.IGNORECASE)
# Check if a match was found
if match:
v_stp = 2.1
# Convert the matched keyword to lowercase and capitalize the first letter
r_str = match.group(1).lower().capitalize()
v_msg = "Found a match."
echo_msg(v_prg, v_stp, v_msg, 4)
else:
# No match found
v_stp = 2.2
v_msg = "No match found from '" + r_condition + "'"
echo_msg(v_prg, v_stp, v_msg, 4)
return r_str |
Beta Was this translation helpful? Give feedback.
-
|
Here is the logic for sensitivity: def get_sensitivity(rule_data, exist_rule_data: dict = {}):
"""
===============
get_sensitivity
===============
This method builds the json.Sensitivity element in the core rule.
Parameters:
-----------
rule_data: dataframe
a data frame containng all the records for a rule. It can be obtained from
read_rules and select the records from the rule definition data frame.
existing_rule_data: dict
a data frame containng all the records for a rule that already developed. It
can be read from the existing rule folder using get_existing_rule.
returns
-------
r_str: a string for json.Sensitivity
possible sensitivities:
- Domain
- Dataset
- Study
- Record
- Variable
- Term
Raises
------
ValueError
None
"""
if rule_data.empty:
return None
r_str = exist_rule_data.get("json", {}).get("Sensitivity")
if r_str is None:
r_condition = rule_data.iloc[0]["Condition"]
# r_rule = rule_data.iloc[0]["Rule"]
r_str = "Record"
if r_condition is not None:
pattern = r"^(study|dataset|domain|variable|term)"
# Use the re.search() method to search for the pattern in the input string
match = re.search(pattern, r_condition, re.IGNORECASE)
# Check if a match was found
if match:
# Convert the matched keyword to lowercase and capitalize the first letter
r_str = match.group(1).lower().capitalize()
else:
# No match found
print(f"No match found from {r_condition}")
return r_str |
Beta Was this translation helpful? Give feedback.
-
|
@htu, this makes sense, but because these aren't always 100% accurate indications of rule type and sensitivity, i think we should keep those properties unpopulated. |
Beta Was this translation helpful? Give feedback.
-
@htu has provided the following samples for review
CG0001
CG0373
Beta Was this translation helpful? Give feedback.
All reactions