ROX-20058: Use GCP secret manager + Helm for infra deployments #1015

tommartensen · 2023-09-29T06:22:13Z

Changes

Infra configuration and secrets were moved to GCP secret manager, which supports versioning, streaming of values into helm upgrade.
Infra is now deployed via Helm.
Helm and secrets utils are available through Make targets, which use the scripts in scripts/deploy.
"deployment" & "environment" were simplified to "environment", which can be "development" or "production". Previous usages of "local" deployment have been replaced with check of the .Capabilities API to avoid provisioning non-existent K8s resources, and the testMode value, which can be set to disable telemetry.
The "infra" namespace is now created with Helm instead of explicitly as a K8s resource

Misc

PR GHA workflow: Is now synchronized, ie only one workflow per PR can run
Deployment GHA workflow: The Slack channel ID was fixed
Documentation was updated
Argo Workflow server is now installed through a Helm chart and independent of the infra-server deployment

How I tested my changes

PR workflow (build, PR cluster, e2e tests)
- Port-forward to PR cluster, login through UI on https://localhost:8443
Deployed to dev.infra.rox.systems with the fixed Deployment workflow (notification) and from my machine, following the DEPLOYMENT.md instructions.

Rollout on prod

make configuration-download on master
Replacing ".Values.deployment" with ".Values.environment" in chart/infra-server/configuration/production/{auth0.yaml, oidc.yaml}
make create-consolidated-values

Validate "oidc_yaml" contains ".Values.environment":

yq \
    ".oidc_yaml" \
    "chart/infra-server/configuration/production-values-from-files.yaml" \
| base64 --decode

Remove "auth0_yaml" (migrated to RHSSO), "google_calendar_credentials_json" (functionality removed), "gke__gke_credentials_json" (pre RH migration), all unused in chart.
Switch to tm/helm-charts branch.
ENVIRONMENT=production make secrets-upload
Validate new secret shows updated values: ./scripts/deploy/secrets.sh show production latest, selecting "oidc_yaml" shows the same output as (4).
Validate you're pointing to the correct cluster: kubectl config current-context.
Delete argo and infra namespaces and all secrets in default namespace (except default token). This is required to have a clean slate, otherwise conflicts with Helm unmanaged resources.
ENVIRONMENT=production make install-argo helm-deploy.

ghost · 2023-10-04T08:49:33Z

A single node development cluster (infra-pr-1015) was allocated in production infra for this PR.

CI will attempt to deploy us.gcr.io/stackrox-infra/infra-server:0.8.2-22-gce8e555ae1 to it.

🔌 You can connect to this cluster with:

gcloud container clusters get-credentials infra-pr-1015 --zone us-central1-a --project acs-team-temp-dev

🛠️ And pull infractl from the deployed dev infra-server with:

nohup kubectl -n infra port-forward svc/infra-server-service 8443:8443 &
make pull-infractl-from-dev-server

🚲 You can then use the dev infra instance e.g.:

bin/infractl -k -e localhost:8443 whoami

⚠️ Any clusters that you start using your dev infra instance should have a lifespan shorter then the development cluster instance. Otherwise they will not be destroyed when the dev infra instance ceases to exist when the development cluster is deleted. ⚠️

Further Development

☕ If you make changes, you can commit and push and CI will take care of updating the development cluster.

🚀 If you only modify configuration (chart/infra-server/configuration) or templates (chart/infra-server/{static,templates}), you can get a faster update with:

make install-local

Logs

Logs for the development infra depending on your @redhat.com authuser:

Or:

kubectl -n infra logs -l app=infra-server --tail=1 -f

…lm-charts

gavin-stackrox

Nice! Other than my questions around ManagedCert/Ingress, I think this PR is good to go

gavin-stackrox · 2023-10-11T18:29:21Z

.github/workflows/deploy.yaml


      - name: Notify infra channel about new version
        env:
          SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
        uses: slackapi/slack-github-action@v1.23.0
        with:
-          channel-id: C01H4DC33K3 #acs-infra
+          channel-id: CVANK5K5W #acs-infra


gavin-stackrox · 2023-10-11T18:29:32Z

.github/workflows/deploy.yaml

@@ -1,3 +1,4 @@
+name: Deploy infra


gavin-stackrox · 2023-10-11T18:34:25Z

Makefile

-#############
-## Linting ##
-#############


Ugh. Re-organizing files makes review extra hard. Re-org should be a separate PR IMhO. But not going to hold this PR back for it.

gavin-stackrox · 2023-10-11T18:41:42Z

chart/infra-server/templates/certificate.yaml

@@ -1,4 +1,4 @@
-{{ if ne .Values.deployment "local" -}}
+{{- if .Capabilities.APIVersions.Has "networking.gke.io/v1/ManagedCertificate" -}}


This is going to create the managedcertificate on infra PR instances with dev values. How will this affect the dev instance? If we are not sure, then best to avoid.

The managed certificate will be stuck in Provisioning state for infra PR instances, because the certificate challenge cannot be reached on the {{ .Values.hosts.primary }} or {{ .Values.hosts.secondary }}.

No impact on the dev cluster other than some failed requests from GCP.

Whatabout using .Values.testMode to exclude them? I get that they are probably harmless but given that dependabot can create lots of dev instances I'd prefer to avoid any issues.

OK ce8e555

chart/infra-server/templates/ingress.yaml

wip: secrets + Helm charts

15eb95b

tommartensen self-assigned this Sep 29, 2023

tommartensen added 2 commits September 29, 2023 15:21

fix and stuff

0846847

Merge branch 'master' into tm/helm-charts

2196ee7

tommartensen added 16 commits October 4, 2023 13:38

progress

a1a3b2a

Merge branch 'master' into tm/helm-charts

50caa4e

add secret management script

70e5c12

Merge branch 'tm/helm-charts' of github.com:stackrox/infra into tm/he…

2788b18

…lm-charts

finish stuff for development

b33dcf8

fix stupid typo

ea2ac57

fix cap

cd4eb66

install argo

f3cd6cc

fix bats e2e test target

ba40223

update documentation

750ed49

update

90540ce

fix deployment workflow

22000e6

fix deployment workflow

526e1e5

sequentialize PRs

b05e6dd

send it

58cb41d

add name

cb6a8a4

tommartensen changed the title ~~wip: GCP secrets + Helm charts~~ ROX-... : Use GCP secret manager + Helm for infra deployments Oct 11, 2023

tommartensen changed the title ~~ROX-... : Use GCP secret manager + Helm for infra deployments~~ ROX-20058: Use GCP secret manager + Helm for infra deployments Oct 11, 2023

tommartensen marked this pull request as ready for review October 11, 2023 12:45

tommartensen requested a review from a team as a code owner October 11, 2023 12:45

fix infractl pull

5081f5d

gavin-stackrox suggested changes Oct 11, 2023

View reviewed changes

use correct capabilities lookup

67ab57e

tommartensen requested a review from gavin-stackrox October 12, 2023 13:29

use testmode instead of capabilities

ce8e555

gavin-stackrox approved these changes Oct 13, 2023

View reviewed changes

tommartensen merged commit de64d2f into master Oct 16, 2023
7 checks passed

tommartensen deleted the tm/helm-charts branch October 16, 2023 08:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ROX-20058: Use GCP secret manager + Helm for infra deployments #1015

ROX-20058: Use GCP secret manager + Helm for infra deployments #1015

tommartensen commented Sep 29, 2023 •

edited

Loading

ghost commented Oct 4, 2023 •

edited by ghost

Loading

gavin-stackrox left a comment

gavin-stackrox Oct 11, 2023

gavin-stackrox Oct 11, 2023

gavin-stackrox Oct 11, 2023

gavin-stackrox Oct 11, 2023

tommartensen Oct 12, 2023

gavin-stackrox Oct 12, 2023

tommartensen Oct 13, 2023 •

edited

Loading

		@@ -1,4 +1,4 @@
		{{ if ne .Values.deployment "local" -}}
		{{- if .Capabilities.APIVersions.Has "networking.gke.io/v1/ManagedCertificate" -}}

ROX-20058: Use GCP secret manager + Helm for infra deployments #1015

ROX-20058: Use GCP secret manager + Helm for infra deployments #1015

Conversation

tommartensen commented Sep 29, 2023 • edited Loading

Changes

Misc

How I tested my changes

Rollout on prod

ghost commented Oct 4, 2023 • edited by ghost Loading

Further Development

Logs

gavin-stackrox left a comment

Choose a reason for hiding this comment

gavin-stackrox Oct 11, 2023

Choose a reason for hiding this comment

gavin-stackrox Oct 11, 2023

Choose a reason for hiding this comment

gavin-stackrox Oct 11, 2023

Choose a reason for hiding this comment

gavin-stackrox Oct 11, 2023

Choose a reason for hiding this comment

tommartensen Oct 12, 2023

Choose a reason for hiding this comment

gavin-stackrox Oct 12, 2023

Choose a reason for hiding this comment

tommartensen Oct 13, 2023 • edited Loading

Choose a reason for hiding this comment

tommartensen commented Sep 29, 2023 •

edited

Loading

ghost commented Oct 4, 2023 •

edited by ghost

Loading

tommartensen Oct 13, 2023 •

edited

Loading