Prefixes for bad wdl's for validate, run or both

[sckott]

via @tefirman in #53 (comment)

My proposal is thus for prefixes:

• should fail womtool, but not cromwell run: badVal
• should not fail womtool, but should fail cromwell run: badRun
• should fail womtool, and should fail cromwell run: badValRun
• should not fail womtool or cromwell run: no prefix

Another different take on this is to nest similar types of wdls within the same dir, and then we know the expectation from the parent dir the wdl's are in. e.g.,

• good
  • helloHostname
  • helloDockerHostname
  • etc
• badVal
  • testWdl
• badRun
  • someWdl
• badValRun
  • badFile

A 3rd option that I kinda like is maintaining the metadata about the various WDLs in a single file, json or yaml, whatever. e.g.,

wdls:
  badFile: 
    - womtool: fail
    - cromwell_run: succeed
  helloHostname: 
    - womtool: succeed
    - cromwell_run: succeed
  parseBatchfile: 
    - womtool: succeed
    - cromwell_run: fail

[sckott]

Thoughts? @seankross @tefirman

[tefirman]

I think I like the first approach best, followed by the second approach, not a huge fan of the third method, feels like it would be easy to forget to add a workflow to the metadata file.

I also "badVal" and "badValRun" are actually the same thing, because if a workflow doesn't validate, there's no way it's going to run successfully. So I think we would just need "badVal" and "badRun".

[sckott]

Thanks @tefirman Sounds good to do 1st one, but I'll wait to see what @seankross says

I also "badVal" and "badValRun" are actually the same thing, because if a workflow doesn't validate, there's no way it's going to run successfully. So I think we would just need "badVal" and "badRun".

Okay, so we'd just have

• should fail womtool, but not cromwell run: badVal
• should not fail womtool, but should fail cromwell run: badRun
• should fail womtool, and should fail cromwell run: badVal
• should not fail womtool or cromwell run: no prefix

And for tests for cromwell run we'd know to assert that a workflow prefixed with badVal should fail on run (in addition to validate)

[seankross]

@sckott I think you meant to say: (@tefirman please double check this for me)

	womtool pass	womtool fail
cromwell pass	[no prefix]	[not possible]
cromwell fail	fail-cromwell	fail-all

[tefirman]

Exactly, WOMtool fail + Cromwell pass is not possible to my knowledge, so maybe it would just be:

• [no prefix]: WOMtool pass, Cromwell pass
• badRun: WOMtool pass, Cromwell fail
• badVal: WOMtool fail, Cromwell fail

[sckott]

Okay, great, so badRun and badVal it is. I'll proceed with this and document it