Clinical-Evidence-to-Support-O&O

[karafecho]

This discussion thread is intended to promote discussion and poll TCDC members about compelling and useful approaches for incorporating clinical evidence into O&O. Please comment and/or add new suggestions. We'll take an informal poll after sufficient discussion has taken place.

Possibilities include the following:

1. One-hop clinical support for relative safety of candidate drugs

• Use clinical edge attributes to determine if a candidate drug is actually administered to patients in the real world, regardless of indication
• Example: if a clinical edge exists to support an identified relationship between disease X and drug Y, then there is evidence that at least one healthcare provider believed the drug was safe enough to be prescribed for a patient(s), regardless of the indication; if multiple clinical edges exist, especially in data derived from multiple healthcare systems, then that increases the evidence in support of the safety of a drug

2. One-hop clinical support for inferred answers: threshold for the “known” and “unknown”

• Use clinical edge attributes to filter out both “known” and “novel” answers
• Example: high co-occurrence counts (or low P values or high observed-expected frequency ratios) could be used to set a threshold and filter results for “known” answers; likewise, low co-occurrence counts (or high P values or low observed-expected frequency ratios) could be used to set a threshold for potentially “novel” answers
  - From Sharat: this is essentially the same as Sharat/Qian's suggestion 6d.

3. One-hop clinical support for weighting literature-based scores: weighting clinical evidence vs research literature evidence

• Use clinical edge attributes to weight research literature evidence
• Example: P values (or specified α) could be factored into PubMed-based scores
  - From Sharat: this could really help with O&O because confidence levels are so important for everything, and literature evidence does not have it

4. One-hop clinical support for adding confidence to text-mined results and vice-versa

• Use clinical evidence to increase/decrease confidence in text-mined results (e.g., SemMedDB)
• Example: co-occurrence counts and/or P values (or specified α) and/or contemporaneous classification could be used to rank text-mined results

5. Clinical support for grouping answers: RWE

• Use clinical predicate(s) to group answers
• Example: add the RWE clinical predicate (and any others) as a ‘decoration’ to the meta-KG so ARAs can select clinical KPs for grouping answers
  - From Sharat: yes certainly, a good control

[karafecho]

Additional thoughts from Sharat and Qian:

6. Given a disease/condition, what all drugs are prescribed for it? And same question in reverse.
a. Any associations with side effects?
b. Any indications of efficacy? (we realize a and b will take some implementation)
c. Are these drugs given off target (or does an ARA has to infer that?) (Non-clinical KPs can provide such information, then retrieve data from clinical KPs to verify the efficacy/SE for these off target drugs. This can be a good use case for drug repurposing.)
d. See if you can determine a novelty factor by going past the “well known” drugs or “well known” diseases. (EHR risk KP (from multiomics) also develops risk predictions, which might also be used for novelty detection, such as, leveraging the predicate of associated_with_increased_likelihood_of.)

The reason for Question 1 is this. A very important factor for a drug researcher is the mechanism or pathway. If they can get the Pathway (eg MyD88 signalling) between drug to disease from any ARS/ARA result, or their own expertise, then they will want to query which other diseases does that Pathway MyD88 affect (Question 12 in Sandrine’s list, the next priority). Then they will want to know whether there are already existing (or in-trial) drugs for that other disease, which may use the same Pathway (MyD88) – and see how effective they are. This can help with the ordering of that result.

7. Can above answers be filtered by “recently approved” drugs? Drugs in clinical trial?
(Is multiomics KP ready with CT data? Can FDA orphan drug designations be considered as well?)

8. Given a disease, what are groups/clusters of drugs and diseases?
a. This can help with understanding what all symptoms/co-morbid diseases/drug combos have to be considered in a basic “what drugs treat this disease” query. (This can help us expand the search space to infer new associations as a creative mode. You could do clustering via standard AI methods)

[CaseyTa]

Regarding Kara's original list:

For #⁠2, using observed-expected frequency ratio or p-values (as opposed to raw co-occurrence) could work well for this, but we may have to change our paradigm on how we're returning results. Currently, COHD's TRAPI endpoint is only returning an edge when the pair of concepts have a significant association. But to label the unknowns, we may want to return info on all relevant edges, including the non-significant ones, so that we can distinguish between the edges "with sufficient power/sample size but no significant association" vs edges "without sufficient power/sample size and no significant association". This may then require a different predicate (e.g., has_clinical_data or something) outside of the associated with hierarchy,

For #⁠4, I'm currently not sure how we can specifically say that we would like to increase the confidence of an edge. I think each edge currently just has a one-dimensional score, which I guess could include many possible factors (e.g., confidence, relevance, effect size, novelty, etc). I've mentioned in the past that I think it would be very useful for Translator to be able to score these dimensions separately, but that's never gotten much traction.

Can someone please provide a concrete example for #⁠5? I don't yet see how this is very distinct from the combination of #⁠1 and #⁠2.

Regarding Sharat and Qian's list:

For #⁠6, COHD doesn't have any direct measure of efficacy, but perhaps relative frequency can be used as a proxy, i.e., hopefully the relative frequency which patients with disease X gets prescribed drug Y correlates with the efficacy of drug Y. However, this would have to be filtered to only include drugs associated with the disease (or known to treat the disease), otherwise you'd just get the most generally common drugs ranked at the top.

For #⁠8, OpenPredict has disease-disease and drug-drug similarity calculations based on cosine similarities of embeddings that could be useful here. COHD disease-disease associations would be more reflective of comorbidities.

[karafecho]

[FYI: I just renumbered the suggestions above, so that we can more easily refer to them.]

Re 2: ICEES KG would be able to support this, and it sounds like COHD would too, with a (minor?) adjustment in the alpha value for significance. I don't think we'd need to change the predicate, as the edge attributes will allow the user to draw his/her own conclusions.

Re 4: This was my attempt to capture your suggestion, Casey. Please feel free to revise the description. I think you may be correct about the scoring, but that would be something for O&O to consider, I think.

Re 5: I suggested this (actually, I think it was Greg's idea originally) as a simple approach for selecting associations that are found in the real world. For example, selecting edges on drug X and disease Y associations but restricted to those edges identified by way of real-world evidence (i.e., contributed by clinical KPs).

Re 6: The clinical KPs can provide associations between diagnosed diseases and prescribed drugs, with caveats of course. However, things like side-effects (very hard to define), efficacy (I think you mean effectiveness), and off-label use (very hard to identify) probably are a bit out of scope. That said, as you note, "novelty" or distinguishing the "known" from the "unknown" probably is within scope. Perhaps of interest, ICEES KG exposes data on adverse outcomes (e.g., ED visits for respiratory issues, liver transplant, ventilation), which may be helpful here, but again, there are caveats.

Re 7: This information would need to be provided by a non-clinical KP.

Re 8: Like COHD, ICEES KG can provide disease-disease and drug-drug associations. Unlike COHD, ICEES KG is cohort-based by design, so the associations would apply to a specific cohort. For example, the ICEES KG asthma instance exposes data on a cohort of roughly 160,000 patients with a diagnosis of asthma or a related common pulmonary disorder (identified by way of a complex algorithm that factors in diagnostic codes, relevant labs, etc). As such, any disease-disease or drug-drug associations that are identified would apply to a specific cohort. This consideration may add specificity to the proposed query. Casey's suggestions are also worth consideration.

[CaseyTa]

Re 2: I agree that the users can dive into the edge attributes to only pull out the significant edges, if that's what they want, but I worry about a couple things. 1) Currently, the way the ARAs are using the clinical KP edges, they expect only significant results to be returned, so if we change our behavior to return all edges that have data, then we could be introducing breaking changes to existing queries. 2) A smaller concern is that if a user only wants the significant edges, a bunch of data will get passed around needlessly. Alternatively to changing the predicate, maybe this type of query works best as an overlay operation rather than lookup.

Re 4: I think you captured it well. I was just voicing some limitations of my own suggestion :)

Side note re 6: TIL the difference between efficacy and effectiveness; thanks, Kara!

[karafecho]

Re (2): I think we need clarification from the ARAs on their expectations and from the Biolink folks on the choice of predicate. For instance, the Biolink definition of associated_with does not specify an alpha value, so there's some flexibility there: https://biolink.github.io/biolink-model/docs/associated_with.html. I also think this will vary by user, as you suggest, so if we let the user set the threshold for returning results, then the expectation and the precise definition of associated_with becomes the user's, not the ARA's or the KP's or the Biolink folks'.

I like the idea of an overlay operation instead of (or in addition to?) a lookup.

I am wondering what others think. If we converge on (2) as an immediately actionable approach for incorporating clinical evidence to support O&O, then we should definitely engage the ARAs and the Biolink folks sooner rather than later as we move toward implementation prior to the February relay meeting.

[karafecho]

From Sui by way of Slack:

Hi Kara - t his is a great discussion and I regret that I cannot partake due to scheduling. I really commend you guys for considering clinical evidence (as was shown in one of Sharat's inaugural O& O slide) as separate, explicit layer to support edge confidence that is based on EDVIDENCE! Much better than "folding it in" , in some obscure manner, into EPC along with other differences based on knowledge bases. This comes so much more close to reality of medical (research) practice, where clinical data, as deposited in clinical KPs, have a weight that dwarfs the confidence derived from any mechanistic/bibliometric considerations extracted from non-clinical KPs...

[karafecho]

From Jared by way of Slack:

IMHO, #3 is potentially the most interesting/useful/valuable to the everyday eventual user of Translator. How to weight literature evidence? One can’t just count up mentions in any old PubMed abstract. That might work for a rare disease, but for HTN and AD among many other common diseases every conceivable (hyperbole, but close) bioentity is co-mentioned in some abstract with that disease. Would be immensely useful to be able to weight these mentions by something other than journal impact factor.

[karafecho]

January 9, 2023 update from Kara by way of Slack:

Eight votes in favor of moving forward with Approach 2, with one vote against. Based on the vote and related discussion, I think we have reached general consensus on moving forward with a manual pilot test of Approach 2 in the context of the MVP1 templated query what drugs may treat disease X? However, we will also continue our discussion on other approaches for incorporating clinical evidence into O&O.

In terms of the disease we should focus on for the pilot test of Approach 2, I'd suggest that we focus on the rare pulmonary diseases that we identified for MVP1 and that we are using to drive development of the TCDC CARA.

• primary ciliary dyskinesia (MONDO:0016575)
• cystic fibrosis (MONDO:0009061)
• idiopathic bronchiectasis (MONDO:0018956)
• lymphangioleiomyomatosis (MONDO:0011705)
• idiopathic pulmonary fibrosis (MONDO:0008345)

[karafecho]

In terms of specific diseases, we have 2 thumbs up votes and 2 no strong opinion votes for cystic fibrosis and asthma. While we certainly don't have sufficient votes to formalize a decision, I suggest that in the interest of time, we move forward with these two diseases for testing.

As such, I ran two UI queries at ui.test.transltr.io, which attaches scores to answers, in addition to evidence (publications).

Cystic fibrosis

UI results: https://ui.test.transltr.io/results?q=81eed809-787d-43d6-a0ab-252a1151d3e3
ARS results: https://ars.test.transltr.io/ars/api/messages/81eed809-787d-43d6-a0ab-252a1151d3e3?trace=y

Asthma

UI results: https://ui.test.transltr.io/results?q=4de7052a-51e8-4967-ab52-9d76ee1bf2c3
ARS results: https://ars.test.transltr.io/ars/api/messages/4de7052a-51e8-4967-ab52-9d76ee1bf2c3?trace=y

How do we retrieve results in a format that is suitable for testing?

If I plug the PK into the ARAX UI, I can see results for each ARA or KP who responded, but I cannot obtain a complete list of results, preferably with scores. If I pull the results directly from the ARS, I can retrieve a JSON file, but like the ARAX UI, this file contains the ARA- and KP-specific PKs, but not a complete list of results, preferably with scores.

Here is a link the spreadsheet I would like to populate for testing.

We will need to brainstorm regarding next steps.