Add GitLab CI pipeline files and README

README.md

@@ -2,3 +2,6 @@
 
 This repository holds CI pipeline definitions that are suitable for compiling
 Project Syn cluster catalogs from a Project Syn tenant repository.
+
+Currently, the repository only contains CI pipeline definitions for GitLab.
+See [gitlab/README.md](/gitlab/README.md) for details.

gitlab/README.md

@@ -0,0 +1,109 @@
+# GitLab Commodore compile pipeline
+
+This is a pipeline definition which is suitable to build and push commodore catalogs from a Project Syn tenant repository hosted on a GitLab instance.
+
+Features:
+
+* Show diffs on commits, MRs and master/main
+* Push changes automatically from master/main
+* Override CI job memory limits for individual clusters (if using the GitLab K8s runner)
+
+## Usage
+
+The pipeline can be used in a Project Syn tenant repository as follows:
+
+1. Copy the `gitlab-ci.yml.default` from this directory to `.gitlab-ci.yml` in the tenant repo.
+1. Add the clusters to compile to the `CLUSTERS` in the `.gitlab-ci.yml` in the tenant repo.
+1. Create a project access token for the cluster catalog repository of each cluster listed in `CLUSTERS`.
+   Set the "role" to "Maintainer" and select the `write_repository` scope.
+1. Create a CI/CD variable named `ACCESS_TOKEN_CLUSTERNAME` for each cluster in `CLUSTERS`, where `CLUSTERNAME` is the name of the cluster with `-` replaced by `_`. 
+   Set each variable's value to the corresponding catalog project access token you created before.
+1. Create CI/CD variables `COMMODORE_API_URL` and `COMMODORE_API_TOKEN` which contain the Lieutenant API URL and a suitable API token for the tenant.
+
+> [!NOTE]
+> Project access tokens for catalog repositories are only required for cluster catalog repositories which are hosted on the same GitLab instance as the tenant repo.
+> See below for configuring access to external cluster catalog repositories via SSH.
+
+> [!TIP]
+> If the pipeline needs to clone projects other than the cluster's catalog repo from the local GitLab instance, you need to deactivate the feature _"Limit access to this project"_ in "Settings > CI/CD > Token Access" on those repositories.
+> Alternatively, you can allow access for the job tokens of each tenant repository that needs to access the project.
+
+> [!TIP]
+> Lieutenant supports managing the CI pipeline configuration for "managed" tenant and cluster catalog repositories.
+> See the [Lieutenant documentation](link todo) for details.
+
+### Commodore API Token
+
+To get the `COMMODORE_API_TOKEN`, connect to the Kubernetes cluster hosting your Lieutenant instance and run the following command:
+
+```bash
+TENANT_NAME=t-tenant-id-1234 # Replace with actual tenant id
+kubectl get secret -n lieutenant ${TENANT_NAME} -o go-template='{{.data.token|base64decode}}'
+```
+
+Alternatively, configure the Tenant to manage the `COMMODORE_API_TOKEN` CI/CD variable by adding the following in the `Tenant` resource (for example with `kubectl -n lieutenant edit tenant t-tenant-id-1234`):
+
+```bash
+spec:
+  gitRepoTemplate:
+    ciVariables:
+      - name: COMMODORE_API_TOKEN
+        valueFrom:
+          secretKeyRef:
+            key: token
+            name: t-tenant-id-1234
+        gitlabOptions:
+          masked: true
+```
+
+### External cluster catalog
+
+If the cluster catalog is not hosted on git.vshn.net, you can specify an SSH key which has access to the cluster catalog and the relevant known hosts entry via CI/CD variables on the tenant repo:
+
+1. Create a CI/CD variable named `SSH_PRIVATE_KEY` containing the SSH private key.
+1. Create a CI/CD varaible named `SSH_KNOWN_HOSTS` containing the know hosts entry.
+1. (optional) Create a CI/CD variable named `SSH_CONFIG` containing any required SSH configuration.
+
+### Test new pipeline generation image
+
+The image used to generate the compile and deploy pipelines can be adjusted by setting the following variables.
+
+```yaml
+variables:
+  PIPELINE_GENERATION_IMAGE_NAME: ghcr.io/projectsyn/commodore-compile-pipelines/gitlab
+  PIPELINE_GENERATION_IMAGE_TAG: mytestbranch
+```
+
+## FAQ
+
+### How can the compile pipeline fetch components hosted on the local GitLab instance
+
+The pipeline is configured to use the GitLab CI `CI_JOB_TOKEN` token when fetching repos from the local GitLab instance.
+The `CI_JOB_TOKEN` token has the same permissions to access the API as the user that caused the job to run.
+Therefore, the compile pipeline can access components hosted in all GitLab projects to which that user has access.
+
+
+### Why do pipelines for some MRs fail
+
+One common cause for pipeline failures for MRs is that the GitLab user who created the MR doesn't have access to the cluster catalog repo or another repo hosted on the local GitLab instance.
+To fix the issue:
+
+* Add the MR creator to the cluster catalog repositories as a "Developer" for read-only, or "Maintainer" for read-write access.
+* Add the MR creator to other repositories as a "Developer".
+
+### Configure cpu requests and limits
+
+The following options can be configured as CI/CD variables when the GitLab instance uses a K8s CI runner:
+
+* `CPU_REQUESTS`, which defaults to `800m`
+* `CPU_LIMITS`, which defaults to `2`
+* `MEMORY_LIMITS`, which defaults to `2Gi`
+
+The job generator expects that each of these variables has space-separated entries of the form `c-cluster-id-1234=value` if it's present.
+
+Example:
+
+```yaml
+variables:
+  MEMORY_LIMITS: "c-my-cluster=3Gi c-my-other-cluster=3Gi"
+```

gitlab/commodore-pipeline.yml

@@ -0,0 +1,64 @@
+---
+stages:
+  - lint
+  - build
+  - trigger
+
+variables:
+  PIPELINE_GENERATION_IMAGE_NAME: ghcr.io/projectsyn/commodore-compile-pipelines/gitlab
+  PIPELINE_GENERATION_IMAGE_TAG: latest
+  PIPELINE_GENERATION_IMAGE: ${PIPELINE_GENERATION_IMAGE_NAME}:${PIPELINE_GENERATION_IMAGE_TAG}
+
+yamllint:
+  stage: lint
+  image: ${PIPELINE_GENERATION_IMAGE}
+  variables:
+    YAMLLINT_DEFAULTS: |-
+      extends: relaxed
+      rules:
+        line-length: disable
+        new-line-at-end-of-file: disable
+        trailing-spaces:
+          level: warning
+        octal-values:
+          forbid-implicit-octal: true
+          forbid-explicit-octal: true
+          ignore: |
+            manifests/
+        truthy:
+          allowed-values: ['true', 'false', 'True', 'False', 'TRUE', 'FALSE']
+          check-keys: true
+          ignore: |
+            manifests/
+  script:
+    - step-yamllint
+
+commodore-lint:
+  stage: lint
+  image:
+    name: docker.io/projectsyn/commodore:latest
+    entrypoint: ["/usr/local/bin/entrypoint.sh"]
+  script:
+    - commodore inventory lint .
+
+create-pipelines:
+  stage: build
+  image:
+    name: ${PIPELINE_GENERATION_IMAGE}
+    pull_policy: always
+  script:
+    - step-render-pipeline
+  artifacts:
+    paths:
+      - generated-commodore-compile.yml
+    expire_in: "1 week"
+
+trigger-pipelines:
+  stage: trigger
+  needs:
+    - create-pipelines
+  trigger:
+    include:
+      - artifact: generated-commodore-compile.yml
+        job: create-pipelines
+    strategy: depend

gitlab/gitlab-ci.yml.default

@@ -0,0 +1,6 @@
+include:
+  - https://raw.githubusercontent.com/projectsyn/commodore-compile-pipelines/main/gitlab/commodore-pipeline.yml
+
+variables:
+  # Space separated list of clusters to compile in MRs and push to on the default branch.
+  CLUSTERS: ""