[DEVCON-2524] output deploy queue to checkrun summary #772

tlin4194 · 2024-10-28T15:39:03Z

Issue:

Previously, when a revision is queued, we mark the github status as queued but have no other indication in the check run itself that this check run is waiting on something.
Users have to figure out which check run is holding up the queue which is difficult because deploys for a root can happen off any branch. We can figure this out using the temporal UI however it would be a better CX if we could just surface the queue itself in the check run.

Part 1

Add queue info to checkrun summary on creating new check run or unlock signal. Queue info is shown as [commit SHA](link to specific check run)

Part 2

Check run summaries do not get updated in cases where a a Terraform workflow is pending Confirm/Reject checkrun action between Plan and Apply steps, since the deploy queue lock only tracks the "Unlock" checkrun action.

Surfaces deploy queue from queue package to deploy/terraform package. Some complication using Queue structs here due to cyclic dependencies between the two packages.
To avoid restructuring the two packages, instead we will pass the GithubCheckRunCache (github package) over from queue worker to deploy/terraform workflow.
StateReceiver will use GithubCheckRunCache to update checkrun summaries for each revision on the queue when a Confirm/Reject action is pending.

Test Plan

Recreating Unlock action: open 2 PRs, run atlantis apply -f on PR 1, and try to merge PR 2. PR 2 will be locked by confirm reject action on PR 1. Any subsequent queued deploys will show up as queued commits (with corresponding links) in checkrun summary.
Recreating diverged confirm/reject action: run atlantis apply -f on unmerged PR 1 and confirm. Create PR 3 and run atlantis apply -f on unmerged PR 3.
Merged PR 1 deploy will update to to show pending action on PR 3. Links will go directly to the specific run instead of just the commit in case there were multiple forced applies on PR 1.

Improve Test Cases

Fix queue_test and updater_test where assertions in CreateOrUpdate were not getting caught since the function is called within a temporal worker.

tlin4194 · 2024-10-30T14:57:08Z

/ptal #dev-console

…p has waiting actions

server/neptune/workflows/internal/deploy/terraform/state.go

asubrocky · 2024-11-06T20:45:39Z

server/neptune/workflows/internal/deploy/terraform/state.go

+			}
+
+			workflow.GetLogger(ctx).Debug(fmt.Sprintf("Updating action pending summary for deployment id: %s", i.ID.String()))
+			_, err := n.CheckRunCache.CreateOrUpdate(ctx, i.ID.String(), request)


I think this might pose some non-determinism errors, see https://community.temporal.io/t/workflow-determinism/4027, but I could be wrong. If it does, let's add some workflow versioning here

server/neptune/workflows/internal/deploy/terraform/state.go

…gActions

server/neptune/workflows/internal/deploy/terraform/state.go

…Temporal nondeterministic workflow change

… original state

…t apply (once with empty Result, once with Result filled in

tlin4194 · 2024-11-09T01:35:52Z

Current PR is tested on staging, with behavior as expected.
The only issue is that after a deploy pending on confirm/reject action gets unblocked, if the queue is currently locked the state receiver cannot restore the unlock button on the checkruns for queued items. The workaround in the UI is that users will just click on the linked previous checkrun unlock from there.

The actual solution requires exposing queue lock state to state receiver. This would require a significant code restructure since LockState cannot be imported to Terraform package without a cyclic dependency. The following steps are moved to a separate PR.
Steps:

Add lockstate logic to state receiver
Move lockstate to independent package so that lockstate can be imported in state receiver
Split up queue interface into 2 interfaces due to interface bloat, one for deploy worker, one for terraform runner.
Fix test cases

When we update checkruns for queued commits (after the blocking commit gets confirmed or rejected), we need to check if the queue was previously locked by something else and restore the unlock button for each commit. ### Sequence of events without this change: 1. Queue is locked due to diverged commits on latest deployment vs current plan (i.e. if someone previously force applied). All queued commits should have a unlock button. 2. New commit comes in with a pending confirm/reject action. My other PR makes it so that all queued checkruns are updated to reflect that the queue is waiting on this confirm reject action. <img width="1461" alt="Screenshot 2024-11-08 at 3 08 07 PM" src="https://github.com/user-attachments/assets/3daf3dc4-a95a-411c-983c-8ad95f53d2e7"> 4. Confirm/reject commit gets resolved. All queued commit checkruns get updated to the default status, without the unlock button. Currently users would have to click on the commit they want to unlock (first screenshot below), then click on an older checkrun of that commit (linked in blue) to access the unlock button (2nd screenshot below). <img width="877" alt="Screenshot 2024-11-08 at 3 07 08 PM" src="https://github.com/user-attachments/assets/5c86a489-f18f-45cd-b887-583774ccaa0a"> <img width="1243" alt="Screenshot 2024-11-08 at 3 07 17 PM" src="https://github.com/user-attachments/assets/0fa5b694-6350-450a-9155-7a3ca7583b48"> ### After this change: 4. (Previous step 4) All queued commit checkruns get updated back to its original status from before the confirm/reject commit was applied, **including** unlock actions if the queue was previously locked. ### Additional notes: * LockState is migrated out of the queue package into its own `lock` package to avoid cyclic imports with terraform module * bumping max interfacebloat golangci error to allow adding GetLockState() function to queue interface ### Testing 1. Open PR 1. Force apply PR 1, and verify that the force apply is successful. 2. Open PR 2. Merge PR 2 to master, and verify that PR 2 deployment failed and is now locked on PR 1's last commit (there should be an unlock button and link to the locking commit). 3. Open PR 3. Force apply PR 3, and verify that there is a Confirm/Reject action. 4. Verify that PR 2's deployment checkrun on master has an updated summary with link to the pending Confirm/Reject run on PR 3. 5. Reject forced apply on PR 3. 6. Verify that PR 2's deployment checkrun updates again, this time back to the original checkrun summary from step 2. If the queue was previously locked (which it was), there should be an unlock button.

tlin4194 added 3 commits October 25, 2024 17:46

add revisions on queue to checkrun summary

4a5c4a0

update tests with GetQueuedRevisionsSummary queue interface

a986769

fix updater and revision tests to throw errors from child workflow

f9f0431

smonero previously approved these changes Oct 30, 2024

View reviewed changes

tlin4194 dismissed smonero’s stale review via 5926b53 October 30, 2024 16:36

tlin4194 added 5 commits October 30, 2024 15:36

exposing deploy queue to deploy terraform package

f4e8ac5

add commit links

2334947

update checkruns with queue status for Terraform jobs where Apply ste…

fd38bf6

…p has waiting actions

fix deploy_test by checking apply job is not null

43ccc09

provide links to runs in queue instead of commits in queue

cbf4fa5

tlin4194 force-pushed the DEVCON-2524-output-deploy-queue-to-checkrun-summary branch from eb843ba to cbf4fa5 Compare October 31, 2024 19:31

smonero previously approved these changes Nov 6, 2024

View reviewed changes

asubrocky reviewed Nov 6, 2024

View reviewed changes

add comments, replace using reflect to check for empty Apply.Onwaitin…

4cc6c9f

…gActions

tlin4194 dismissed smonero’s stale review via 4cc6c9f November 7, 2024 19:15

asubrocky reviewed Nov 7, 2024

View reviewed changes

server/neptune/workflows/internal/deploy/terraform/state.go Outdated Show resolved Hide resolved

tlin4194 added 3 commits November 7, 2024 16:14

workflow version patch for additional github checkrun updates due to …

1e2c0a8

…Temporal nondeterministic workflow change

bugfix: after confirm action, queued checkruns need to be restored to…

c921b26

… original state

noted TODO: current deploy checkrun is updated twice on confirm/rejec…

d5ef226

…t apply (once with empty Result, once with Result filled in

tlin4194 changed the title ~~Devcon 2524 output deploy queue to checkrun summary~~ [DEVCON-2524] output deploy queue to checkrun summary Nov 8, 2024

tlin4194 force-pushed the DEVCON-2524-output-deploy-queue-to-checkrun-summary branch from 5fbd36d to d5ef226 Compare November 9, 2024 01:27

asubrocky approved these changes Dec 3, 2024

View reviewed changes

tlin4194 merged commit 58ff899 into main Dec 3, 2024
3 checks passed

tlin4194 deleted the DEVCON-2524-output-deploy-queue-to-checkrun-summary branch December 3, 2024 15:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DEVCON-2524] output deploy queue to checkrun summary #772

[DEVCON-2524] output deploy queue to checkrun summary #772

tlin4194 commented Oct 28, 2024 •

edited

Loading

tlin4194 commented Oct 30, 2024

asubrocky Nov 6, 2024

tlin4194 commented Nov 9, 2024 •

edited

Loading

[DEVCON-2524] output deploy queue to checkrun summary #772

[DEVCON-2524] output deploy queue to checkrun summary #772

Conversation

tlin4194 commented Oct 28, 2024 • edited Loading

Issue:

Part 1

Part 2

Test Plan

Improve Test Cases

tlin4194 commented Oct 30, 2024

asubrocky Nov 6, 2024

Choose a reason for hiding this comment

tlin4194 commented Nov 9, 2024 • edited Loading

tlin4194 commented Oct 28, 2024 •

edited

Loading

tlin4194 commented Nov 9, 2024 •

edited

Loading