add script, which set QED dependencies of QED dependencies to the development version in integration tests

[SimeonEhrig]

As soon the PR is merged, other QED projects needs to rerun the CI pipeline to use the function.

fix #36

[SimeonEhrig]

the test PR for this function is here: QEDjl-project/QEDcore.jl#35

[AntonReinhard]

This is good and fixes the issue. I think it's still necessary to keep the pipeline of integration tests that test against released version for merges to main, since those versions will themselves be released and need to work with the other registered packages.
The only exception would be releases to main with required simultaneous releases because of breaking changes.

[SimeonEhrig]

This is good and fixes the issue. I think it's still necessary to keep the pipeline of integration tests that test against released version for merges to main, since those versions will themselves be released and need to work with the other registered packages.
The only exception would be releases to main with required simultaneous releases because of breaking changes.

I disagree because we only guaranty two thinks.

• First if we release a QED package, which has dependency to other QED package, this dependencies are released versions.
• Second, nearly all the time, the dev branches of all QED projects works together for a specific time point.

So the question is, if we test the our projects with the dev and the last release version of the dependencies, what we want to do, if the last release is not working? And also what is the benefit if we know that our current dev branch breaks with the release version of another package but not with the dev version?

[AntonReinhard]

First if we release a QED package, which has dependency to other QED package, this dependencies are released versions.

Do we guarantee this currently? That's what I was suggesting, that we should test merges to main (which will be officially released right after) against the released versions of everything else, not their dev branches. If we do that already then I didn't realize that.

[SimeonEhrig]

Do we guarantee this currently? That's what I was suggesting, that we should test merges to main (which will be officially released right after) against the released versions of everything else, not their dev branches. If we do that already then I didn't realize that.

Yes, because if we release we freeze all dev branches and create the new release version from it. The CI guaranties that all dev branches works together. There is only the exception, if a QED project has no changes and the dev branch is still equal to the last release. Therefore we do not need to do a release.

[SimeonEhrig]

@AntonReinhard How continue with this PR? Uwe will be back in September.

[szabo137]

@SimeonEhrig I know we discussed the freeze of all devs for a full release in a very early stage of the project. However, at this point, I don't understand our conclusion from the past anymore 😅

Here are some thoughts from my side, how the integration tests should work (somewhat the dream state):

If a PR is opened targeting the main branch of package (which is most certainly a release or a hotfix), the integration tests should run for the released versions of the downstream packages. The reason is simple: if the PR is merged, it is the release and we want to guarantee, that all released versions work together properly. If these integration tests fail, two workflows are conceivable:

1. The fix can be done directly on the release branch, which is fine, because all changes there are ported back to the dev anyway.
2. The fix needs to be done downstream, which gives the developer an order of releases, most certainly the dependency graph.

If a PR is opened targeting dev, the integration tests should run for the dev versions of the downstream packages. Running against main, i.e. the released versions, is not necessary, because this will be done during the release pipeline anyways (see above). If these tests break, one needs to proceed with the fixing workflow written in the docs: fixing on a branch of the downstream package and using commit messages indicating the fix to the upstream PR.

These two pipelines are somewhat similar, but the difference is: the first workflow is very strict with fixes to guarantee interoperability of released versions. The second (currently used) workflow has some states, where packages do not work together, e.g. of the downstream fix is not merged yet, which is fine for dev branches. Side note: if for a packages, there are no stages changes, dev and main are in sync, therefore testing against dev is testing against release.

@AntonReinhard @SimeonEhrig what do you think?

[AntonReinhard]

I agree. I think the problem with freezing the dev branches is that we would have to release every package every time we want to release any package. And releasing does take some time, plus it would probably end up bloating the versions unnecessarily.

The fix needs to be done downstream, which gives the developer an order of releases, most certainly the dependency graph.

As you say later, this means we will still need the option to change the branch we test against for merges to main, the same way it currently works with merges to dev.

[szabo137]

As you say later, this means we will still need the option to change the branch we test against for merges to main, the same way it currently works with merges to dev.

Actually no, the tests for PRs targeting main should always test against released versions. In this regard, fails of the tests always cause a suspension of the release till the incompatible/breaking parts are fixed and the PR to main can be merged without any side effects.

[AntonReinhard]

Actually no, the tests for PRs targeting main should always test against released versions. In this regard, fails of the tests always cause a suspension of the release till the incompatible/breaking parts are fixed and the PR to main can be merged without any side effects.

Okay, that just means then that sometimes it will be somewhat difficult to release (as with the QEDcore move), but it should always be possible, you're right.

So then @SimeonEhrig I think all that should be done here is to have two separate pipelines, where for

• merges to main, everything is tested against all other released versions, and for
• merges to dev, everything is tested against all other dev versions
  instead of always testing against dev versions, as it is now.

[SimeonEhrig]

Okay, if the target branch is main test against the released version from the registry, else against the dev branch (we need also a solution, if we merge in another feature branch).

[szabo137]

... (we need also a solution, if we merge in another feature branch).

What do you mean with that? We already have workflow to handle feature branches, or am I wrong? 🤔

[SimeonEhrig]

... (we need also a solution, if we merge in another feature branch).

What do you mean with that? We already have workflow to handle feature branches, or am I wrong? 🤔

I mean, if you want to merge a feature branch in another feature branch and not the dev branch.

[szabo137]

... (we need also a solution, if we merge in another feature branch).

What do you mean with that? We already have workflow to handle feature branches, or am I wrong? 🤔

I mean, if you want to merge a feature branch in another feature branch and not the dev branch.

In the current contribution policy, this could only happen on a fork, not in the main repo. I think for such cases, we don't provide CI support?

[SimeonEhrig]

In the current contribution policy, this could only happen on a fork, not in the main repo. I think for such cases, we don't provide CI support?

It's more a implementation question. Ether I implement

if $TARGET_BRANCH == "main"
  // use registered packages
else
  // use dev version

or

if $TARGET_BRANCH == "main"
  // use registered packages
elif $TARGET_BRANCH == "dev"
  // use dev version
else
  error("unsupported")

[AntonReinhard]

I think the first option makes the most sense. Use registered packages for PRs to main and dev otherwise.

[SimeonEhrig]

The script is used in other repositories to generated integration tests. Therefore I need to open PRs in another repositories to test it.

• test again dev branch: [DO NOT MERGE] Test set dev dep deps QEDcore.jl#35
• test again main branch: [DO NOT MERGE] Test set dev dep deps with target main branch QEDcore.jl#43

[AntonReinhard]

LGTM