feat!: DBTP-1506 Update platform helper codebase deploy command to trigger manual deployment pipeline #698

tony-griffin · 2024-12-19T13:46:26Z

Breaking change that will require a coordinated merge with the codebase pipeline work that's happening. Bumped the default Terraform Platform Modules version to 7 for this reason. Version 7 is to be released. Version 6 will be the work on the IP filter spoofing that's still waiting to be merged. Then this PR and John's work on the codebase pipelines will be in version 7.
Required to pull in the terraform code that John is working on for the codebase pipeline and re-deploy my environment
That work provisions an IAM role <application>-<environment>-codebase-pipeline-deploy which is required so that the manual codebase pipeline can assume it and run things in the user’s environment.
Required to update demodjango-deploy/terraform/environments/<environment>/main.tf with a new parameter, pipeline_account_id, that gets passed to the terraform-platform-modules/extensions module

args = {
    application    = "demodjango"
    services       = local.config["extensions"]
    dns_account_id = local.env_config["tony"]["accounts"]["dns"]["id"]
    pipeline_account_id = local.env_config["*"]["accounts"]["deploy"]["id"]
  }

Manual codebase deployment pipeline
Running command:
platform-helper codebase deploy --app demodjango --env tony --codebase application --commit 26628fb
produces the following pipeline execution url:
https://eu-west-2.console.aws.amazon.com/codesuite/codepipeline/pipelines/demodjango-test-manual-release-pipeline/executions/dfbbb4c4-2ca1-4eb5-9909-ecf5c4c5f59f

Checklist:

Title:

Scope included as per conventional commits
Ticket reference included (unless it's a quick out of ticket thing)

Description:

Link to ticket included (unless it's a quick out of ticket thing)
Includes tests (or an explanation for why it doesn't)
If the work includes user interface changes, before and after screenshots included in description
Includes any applicable changes to the documentation in this code base
Includes link(s) to any applicable changes to the documentation in the DBT Platform Documentation (can be to a pull request)

Tasks:

Run the end to end tests for this branch and confirm that they are passing

…ution URL

…l and response object properties

…ntext manager

…ploy-command

codecov-commenter · 2024-12-20T10:35:10Z

❌ 5 Tests Failed:

Tests completed	Failed	Passed	Skipped
665	5	660	3

View the top 3 failed tests by shortest run time

tests/platform_helper/domain/test_codebase.py::test_codebase_deploy_does_not_trigger_pipeline_build_without_confirmation

Stack Traces | 0.005s run time

.../platform_helper/domain/test_codebase.py:339: in test_codebase_deploy_does_not_trigger_pipeline_build_without_confirmation
    codebase.deploy("test-application", "development", "application", "ab1c23d")
.../dbt_platform_helper/domain/codebase.py:164: in deploy
    self.confirm,
E   AttributeError: 'Codebase' object has no attribute 'confirm'

tests/platform_helper/domain/test_codebase.py::test_codebase_deploy_successfully_triggers_a_pipeline_based_deploy

Stack Traces | 0.005s run time

.../platform_helper/domain/test_codebase.py:236: in test_codebase_deploy_successfully_triggers_a_pipeline_based_deploy
    codebase.deploy("test-application", "development", "application", "ab1c23d")
.../dbt_platform_helper/domain/codebase.py:164: in deploy
    self.confirm,
E   AttributeError: 'Codebase' object has no attribute 'confirm'

tests/platform_helper/domain/test_codebase.py::test_codebase_deploy_does_not_trigger_deployment_without_confirmation

Stack Traces | 0.007s run time

.../platform_helper/domain/test_codebase.py:388: in test_codebase_deploy_does_not_trigger_deployment_without_confirmation
    codebase.deploy("test-application", "development", "application", "nonexistent-commit-hash")
.../dbt_platform_helper/domain/codebase.py:164: in deploy
    self.confirm,
E   AttributeError: 'Codebase' object has no attribute 'confirm'

To view more test analytics, go to the Test Analytics Dashboard
📢 Thoughts on this report? Let us know!

chiaramapellimt · 2024-12-20T10:53:15Z

dbt_platform_helper/constants.py

@@ -3,7 +3,7 @@
 # Todo: Can we get rid of this yet?
 PLATFORM_HELPER_VERSION_FILE = ".platform-helper-version"
 # Todo: Move to ???
-DEFAULT_TERRAFORM_PLATFORM_MODULES_VERSION = "5"
+DEFAULT_TERRAFORM_PLATFORM_MODULES_VERSION = "7"


Do we have version 7 of TPM?

@chiaramapellimt , Version 7 is to be released, that's why this is part of a breaking change.
Version 6 will be the work on the IP filter spoofing that's still waiting to be merged.
Then this PR and John's work on the codebase pipelines will be in version 7.

It's so that we don't over burden services with lots of breaking changes at once I believe, this keeps the new logic seperate.

Is it a breaking change for platform-tools when we update the main default version of terraform-platform-modules?

Yes, it should be actually. It will force users with pipelines to use a later breaking version of t-p-m

@antroy-madetech Might be better placed to answer that

antroy-madetech · 2024-12-20T15:21:30Z

dbt_platform_helper/utils/aws.py

@@ -433,6 +433,11 @@ def start_build_extraction(codebuild_client, build_options):
    return response["build"]["arn"]


+def start_pipeline_extraction(codepipeline_client, build_options):


Should this be "start_pipeline_execution"?

@antroy-madetech It could be as yes we are starting the pipeline execution here. But we are also extracting the execution_id and returning it from this function to create the pipeline execution URL

I followed the existing convention as there is a similar method that extracts the arn from a codebuild response.

I'll leave it you whether you think it shoud be changed.

I think change it. The important thing that this is doing is starting the pipeline execution. Perhaps "start_pipeline_and_return_execution_id" or something similar would also work. Or just "start_pipeline" and describe what it returns in the docstring?

Renamed to start_pipeline_and_return_execution_id

antroy-madetech · 2024-12-20T15:22:21Z

tests/platform_helper/domain/test_codebase.py

@@ -397,19 +385,6 @@ def test_codebase_deploy_does_not_trigger_build_with_missing_environment(mock_ap
        )


-def test_codebase_deploy_does_not_trigger_deployment_without_confirmation():


Are we no longer doing confirmation?

This looked like a duplicate test. There is a test for "without_confirmation" here

antroy-madetech

Approved pending the changes suggested

antroy-madetech · 2024-12-20T15:41:01Z

dbt_platform_helper/constants.py

@@ -3,7 +3,7 @@
 # Todo: Can we get rid of this yet?
 PLATFORM_HELPER_VERSION_FILE = ".platform-helper-version"
 # Todo: Move to ???
-DEFAULT_TERRAFORM_PLATFORM_MODULES_VERSION = "5"
+DEFAULT_TERRAFORM_PLATFORM_MODULES_VERSION = "7"


Yes, it should be actually. It will force users with pipelines to use a later breaking version of t-p-m

antroy-madetech · 2024-12-20T15:45:17Z

dbt_platform_helper/utils/aws.py

@@ -433,6 +433,11 @@ def start_build_extraction(codebuild_client, build_options):
    return response["build"]["arn"]


+def start_pipeline_extraction(codepipeline_client, build_options):


I think change it. The important thing that this is doing is starting the pipeline execution. Perhaps "start_pipeline_and_return_execution_id" or something similar would also work. Or just "start_pipeline" and describe what it returns in the docstring?

…ploy-command

tony-griffin added 12 commits December 18, 2024 17:00

Add methods to retrieve pipeline execution ID and build pipeline exec…

8516f49

…ution URL

Update pipeline URL to go to main pipeline execution page

46f60a6

Update deploy command confirmation message

3fd764f

UPdate codebase deployment existing tests to use codepipeline api cal…

e3a7c68

…l and response object properties

Add pipeline_name variable to codepipeline api call

6a0be9a

add test for getting build url for piepline execution

3064809

Update deploy not trigger pipeline test to bring assertions out of co…

af0e53d

…ntext manager

Remove unrequired line

1edb921

Remove unrequired testcocde and test

fc27285

Reformat file using Black

4acaed1

Group commands

a7f3dde

Reformat

b21fa3d

tony-griffin changed the title ~~feat: DBTP-1506 Update platform helper codebase deploy command to trigger manual deployment pipeline~~ feat!: DBTP-1506 Update platform helper codebase deploy command to trigger manual deployment pipeline Dec 20, 2024

tony-griffin and others added 3 commits December 20, 2024 10:27

Merge branch 'main' into DBTP-1506-update-platform-helper-codebase-de…

0795629

…ploy-command

Formatting

8f3c9e0

Bump TPM default version to 7

5b40333

Bump TPM default version in test to 7

9874dc7

tony-griffin marked this pull request as ready for review December 20, 2024 10:47

tony-griffin requested a review from a team as a code owner December 20, 2024 10:47

chiaramapellimt reviewed Dec 20, 2024

View reviewed changes

chiaramapellimt approved these changes Dec 20, 2024

View reviewed changes

antroy-madetech reviewed Dec 20, 2024

View reviewed changes

antroy-madetech approved these changes Dec 20, 2024

View reviewed changes

Rename pipeline start execution method

f6c9496

JohnStainsby mentioned this pull request Dec 24, 2024

feat!: DBTP-1503 - Cross account deploy and manual release pipeline uktrade/terraform-platform-modules#306

Open

7 tasks

tony-griffin added 2 commits January 6, 2025 10:31

Merge branch 'main' into DBTP-1506-update-platform-helper-codebase-de…

ea5c9d6

…ploy-command

Merge branch 'main' into DBTP-1506-update-platform-helper-codebase-de…

f2c1d03

…ploy-command

antroy-madetech closed this Jan 27, 2025

antroy-madetech deleted the DBTP-1506-update-platform-helper-codebase-deploy-command branch January 27, 2025 09:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!: DBTP-1506 Update platform helper codebase deploy command to trigger manual deployment pipeline #698

feat!: DBTP-1506 Update platform helper codebase deploy command to trigger manual deployment pipeline #698

tony-griffin commented Dec 19, 2024 •

edited

Loading

codecov-commenter commented Dec 20, 2024 •

edited

Loading

chiaramapellimt Dec 20, 2024

tony-griffin Dec 20, 2024 •

edited

Loading

a-gleeson Dec 20, 2024

antroy-madetech Dec 20, 2024

tony-griffin Dec 20, 2024

antroy-madetech Dec 20, 2024

tony-griffin Dec 20, 2024 •

edited

Loading

antroy-madetech Dec 20, 2024

tony-griffin Dec 20, 2024 •

edited

Loading

antroy-madetech Dec 20, 2024

tony-griffin Dec 20, 2024

antroy-madetech left a comment

antroy-madetech Dec 20, 2024

antroy-madetech Dec 20, 2024

		@@ -433,6 +433,11 @@ def start_build_extraction(codebuild_client, build_options):
		return response["build"]["arn"]


		def start_pipeline_extraction(codepipeline_client, build_options):

		@@ -397,19 +385,6 @@ def test_codebase_deploy_does_not_trigger_build_with_missing_environment(mock_ap
		)


		def test_codebase_deploy_does_not_trigger_deployment_without_confirmation():

feat!: DBTP-1506 Update platform helper codebase deploy command to trigger manual deployment pipeline #698

feat!: DBTP-1506 Update platform helper codebase deploy command to trigger manual deployment pipeline #698

Conversation

tony-griffin commented Dec 19, 2024 • edited Loading

Checklist:

Title:

Description:

Tasks:

codecov-commenter commented Dec 20, 2024 • edited Loading

❌ 5 Tests Failed:

Choose a reason for hiding this comment

tony-griffin Dec 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tony-griffin Dec 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tony-griffin Dec 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antroy-madetech left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tony-griffin commented Dec 19, 2024 •

edited

Loading

codecov-commenter commented Dec 20, 2024 •

edited

Loading

tony-griffin Dec 20, 2024 •

edited

Loading

tony-griffin Dec 20, 2024 •

edited

Loading

tony-griffin Dec 20, 2024 •

edited

Loading