RHSM integration for prefetch task

This adds steps to the prefetch task to detect when a Red Hat subscription activation key is provided. When prefetch is configured for RPM package manager and an acivation key is provided, the pod will be registered with Red Hat's subscription management service so that protected content can be fetched. The input is modified, injecting the entitlement certs. For example, for simple input like 'rpm', the input will first be transformed to:

[
  {
    "type": "rpm",
    "options": {
      "ssl": {
        "client_key": null,
        "client_cert": null,
        "ca_bundle": null,
        "verify": 1
      }
    }
  }
]

After this the entitelement certificate information will be added to
ALL instances of rpm package manager present (in case the input is a JSON array.)

After prefetch the container is unregistered.

pipelines/docker-build-multi-platform-oci-ta/README.md

@@ -135,6 +135,7 @@ This pipeline is pushed as a Tekton bundle to [quay.io](https://quay.io/reposito
 ### prefetch-dependencies-oci-ta:0.1 task parameters
 |name|description|default value|already set by|
 |---|---|---|---|
+|ACTIVATION_KEY| Name of secret which contains subscription activation key| activation-key| |
 |SOURCE_ARTIFACT| The Trusted Artifact URI pointing to the artifact with the application source code.| None| '$(tasks.clone-repository.results.SOURCE_ARTIFACT)'|
 |caTrustConfigMapKey| The name of the key in the ConfigMap that contains the CA bundle data.| ca-bundle.crt| |
 |caTrustConfigMapName| The name of the ConfigMap to read CA bundle data from.| trusted-ca| |

pipelines/docker-build-oci-ta/README.md

@@ -132,6 +132,7 @@ This pipeline is pushed as a Tekton bundle to [quay.io](https://quay.io/reposito
 ### prefetch-dependencies-oci-ta:0.1 task parameters
 |name|description|default value|already set by|
 |---|---|---|---|
+|ACTIVATION_KEY| Name of secret which contains subscription activation key| activation-key| |
 |SOURCE_ARTIFACT| The Trusted Artifact URI pointing to the artifact with the application source code.| None| '$(tasks.clone-repository.results.SOURCE_ARTIFACT)'|
 |caTrustConfigMapKey| The name of the key in the ConfigMap that contains the CA bundle data.| ca-bundle.crt| |
 |caTrustConfigMapName| The name of the ConfigMap to read CA bundle data from.| trusted-ca| |

pipelines/docker-build/README.md

@@ -131,6 +131,7 @@ This pipeline is pushed as a Tekton bundle to [quay.io](https://quay.io/reposito
 ### prefetch-dependencies:0.1 task parameters
 |name|description|default value|already set by|
 |---|---|---|---|
+|ACTIVATION_KEY| Name of secret which contains subscription activation key| activation-key| |
 |caTrustConfigMapKey| The name of the key in the ConfigMap that contains the CA bundle data.| ca-bundle.crt| |
 |caTrustConfigMapName| The name of the ConfigMap to read CA bundle data from.| trusted-ca| |
 |config-file-content| Pass configuration to cachi2. Note this needs to be passed as a YAML-formatted config dump, not as a file path! | | |

pipelines/tekton-bundle-builder/README.md

@@ -66,6 +66,7 @@
 ### prefetch-dependencies:0.1 task parameters
 |name|description|default value|already set by|
 |---|---|---|---|
+|ACTIVATION_KEY| Name of secret which contains subscription activation key| activation-key| |
 |caTrustConfigMapKey| The name of the key in the ConfigMap that contains the CA bundle data.| ca-bundle.crt| |
 |caTrustConfigMapName| The name of the ConfigMap to read CA bundle data from.| trusted-ca| |
 |config-file-content| Pass configuration to cachi2. Note this needs to be passed as a YAML-formatted config dump, not as a file path! | | |

task/prefetch-dependencies-oci-ta/0.1/README.md

@@ -26,6 +26,7 @@ params:
 ## Parameters
 |name|description|default value|required|
 |---|---|---|---|
+|ACTIVATION_KEY|Name of secret which contains subscription activation key|activation-key|false|
 |SOURCE_ARTIFACT|The Trusted Artifact URI pointing to the artifact with the application source code.||true|
 |caTrustConfigMapKey|The name of the key in the ConfigMap that contains the CA bundle data.|ca-bundle.crt|false|
 |caTrustConfigMapName|The name of the ConfigMap to read CA bundle data from.|trusted-ca|false|

task/prefetch-dependencies-oci-ta/0.1/prefetch-dependencies-oci-ta.yaml

@@ -33,6 +33,10 @@ spec:
 
     [available configuration parameters]: https://github.com/containerbuildsystem/cachi2?tab=readme-ov-file#available-configuration-parameters
   params:
+    - name: ACTIVATION_KEY
+      description: Name of secret which contains subscription activation key
+      type: string
+      default: activation-key
     - name: SOURCE_ARTIFACT
       description: The Trusted Artifact URI pointing to the artifact with
         the application source code.
@@ -79,8 +83,16 @@ spec:
         the application source code.
       type: string
   volumes:
+    - name: activation-key
+      secret:
+        optional: true
+        secretName: $(params.ACTIVATION_KEY)
     - name: config
       emptyDir: {}
+    - name: etc-pki-entitlement
+      emptyDir: {}
+    - name: shared
+      emptyDir: {}
     - name: trusted-ca
       configMap:
         items:
@@ -110,6 +122,8 @@ spec:
     volumeMounts:
       - mountPath: /mnt/config
         name: config
+      - mountPath: /shared
+        name: shared
       - mountPath: /var/workdir
         name: workdir
   steps:
@@ -143,15 +157,164 @@ spec:
           # https://github.com/containerbuildsystem/cachi2/issues/577
           yq 'del(.goproxy_url)' <<<"${CONFIG_FILE_CONTENT}" >/mnt/config/config.yaml
         fi
+<<<<<<< HEAD
     - name: prefetch-dependencies
       image: quay.io/redhat-appstudio/cachi2:0.14.0@sha256:3088e307f41894c0654b69bb199a4648ba31721d6173099cf6ca8ff259c0e457
+=======
+    - name: check-prefetch-input
+      image: quay.io/redhat-appstudio/cachi2:0.13.0@sha256:eb34cfe3fea20997eebd8164dc93eedb2fd7a60dc1fb4afcc1b1ff43df9d6667
+      env:
+        - name: INPUT
+          value: $(params.input)
+      script: |
+        if [ -z "${INPUT}" ]; then
+          # Confirm input was provided though it's likely the whole task would be skipped if it wasn't
+          echo "No prefetch will be performed because no input was provided for cachi2 fetch-deps"
+          echo "skip" >/shared/skip
+        fi
+    - name: register-red-hat
+      image: quay.io/redhat-appstudio/cachi2@sha256:eb34cfe3fea20997eebd8164dc93eedb2fd7a60dc1fb4afcc1b1ff43df9d6667
+      results:
+        - name: registered
+          type: string
+      volumeMounts:
+        - mountPath: /activation-key
+          name: activation-key
+      env:
+        - name: INPUT
+          value: $(params.input)
+        - name: ACTIVATION_KEY
+          value: $(params.ACTIVATION_KEY)
+      script: |
+        #!/bin/bash
+        if [ -f /shared/skip ]; then
+          echo "Skipping."
+          exit 0
+        fi
+
+        echo "false" >/shared/registered
+        ACTIVATION_KEY_PATH="/activation-key"
+
+        mkdir -p /shared/rhsm/entitlement
+        mkdir -p /shared/rhsm/consumer
+
+        if [ -e /activation-key/org ]; then
+          cp -r --preserve=mode "$ACTIVATION_KEY_PATH" /tmp/activation-key
+
+          echo "Registering with Red Hat subscription manager."
+          subscription-manager register --org "$(cat /tmp/activation-key/org)" --activationkey "$(cat /tmp/activation-key/activationkey)"
+
+          # copy generated certificates to /shared/rhsm
+          cp /etc/pki/entitlement/*.pem /shared/rhsm/entitlement/
+          cp /etc/pki/consumer/*.pem /shared/rhsm/consumer/
+
+          file="$(find /shared/rhsm/entitlement -regextype egrep -regex '.*[0-9]+\.pem' -printf %f)"
+          echo "file: $file"
+          basename "$file" .pem >/shared/RHSM_ID
+          echo "./RHSM_ID:"
+          cat /shared/RHSM_ID
+
+          # trust the CA used for Red Hat CDN
+          cp /etc/rhsm-host/ca/redhat-uep.pem /shared/rhsm/redhat-uep.pem
+        fi
+    - name: preprocess-input
+      image: quay.io/redhat-appstudio/cachi2@sha256:eb34cfe3fea20997eebd8164dc93eedb2fd7a60dc1fb4afcc1b1ff43df9d6667
+      args:
+        - $(params.input)
+      env:
+        - name: INPUT
+          value: $(params.input)
+        - name: ACTIVATION_KEY
+          value: $(params.ACTIVATION_KEY)
+      script: |
+        #!/bin/python3
+        import json
+        import os
+        import sys
+
+
+        def string_to_json(input: str):
+            if input in ['bundler', 'generic', 'gomod', 'npm', 'pip', 'rpm', 'yarn-classic', 'yarn']:
+                input = '{"type": "%s"}' % input
+                print("json: %s" % input)
+            return input
+
+
+        def json_to_list(input: str):
+            input = json.loads(input)
+            if type(input) is dict:
+                input = [input]
+            return json.dumps(input)
+
+
+        def inject_certs(input: str, rhsm_id: str):
+            input_list: list = json.loads(input)
+
+            cert = ("/shared/rhsm/entitlement/%s.pem" % rhsm_id)
+            key = ("/shared/rhsm/entitlement/%s-key.pem" % rhsm_id)
+            ca_bundle = os.getenv("CA_BUNDLE", None)
+            for pkg_man in input_list:
+                if pkg_man["type"] == "rpm":
+
+                    # preserve verify setting
+                    verify = \
+                        pkg_man.get("options", {}).get("ssl", {}).get("ssl_verify", 1)
+
+                    # preserve other options
+                    options: dict = pkg_man.get('options', {})
+
+                    ssl_options = {
+                        "client_key": key,
+                        "client_cert": cert,
+                        "ca_bundle": ca_bundle,
+                        "ssl_verify": verify}
+
+                    options['ssl'] = ssl_options
+                    pkg_man["options"] = options
+            return (json.dumps(input_list))
+
+
+        def convert_input(input, rhsm_id):
+            input = string_to_json(input)
+            input = json_to_list(input)
+            input = inject_certs(input, rhsm_id)
+            return input
+
+
+        if __name__ == '__main__':
+
+            if os.path.isfile("/shared/skip"):
+                sys.exit()
+
+            rhsm_id = ""
+            input = ""
+
+            try:
+                f = open("/shared/RHSM_ID", "r")
+                rhsm_id = f.read().strip("\n")
+            except FileNotFoundError:
+                print("No RHSM ID found.")
+
+            if rhsm_id == "":
+                input = sys.argv[1]
+            else:
+                print("RHSM Cert ID is: %s" % rhsm_id)
+                print("Called with args: %s" % str(sys.argv))
+                input = convert_input(sys.argv[1], rhsm_id)
+
+            print("Preprocessing result: %s" % input)
+            with open('/shared/rhsm/preprocessed_input', 'w') as f:
+                f.write(input)
+    - name: prefetch-dependencies
+      image: quay.io/redhat-appstudio/cachi2@sha256:eb34cfe3fea20997eebd8164dc93eedb2fd7a60dc1fb4afcc1b1ff43df9d6667
+>>>>>>> 04d5e91c (RHSM integration for prefetch task)
       volumeMounts:
         - mountPath: /mnt/trusted-ca
           name: trusted-ca
           readOnly: true
+        - mountPath: /activation-key
+          name: activation-key
       env:
-        - name: INPUT
-          value: $(params.input)
         - name: DEV_PACKAGE_MANAGERS
           value: $(params.dev-package-managers)
         - name: LOG_LEVEL
@@ -165,9 +328,10 @@ spec:
         - name: WORKSPACE_NETRC_PATH
           value: $(workspaces.netrc.path)
       script: |
-        if [ -z "${INPUT}" ]; then
-          # Confirm input was provided though it's likely the whole task would be skipped if it wasn't
-          echo "No prefetch will be performed because no input was provided for cachi2 fetch-deps"
+        #!/bin/bash
+
+        if [ -f /shared/skip ]; then
+          echo "Skipping."
           exit 0
         fi
 
@@ -183,6 +347,16 @@ spec:
           dev_pacman_flag=""
         fi
 
+        INPUT=$(cat /shared/rhsm/preprocessed_input)
+        export INPUT
+
+        # trust Red Hat CA cert used for Red Hat CDN
+        if [ -f /shared/rhsm/redhat-uep.pem ]; then
+          echo "Adding Red Hat CA certificate to trusted roots."
+          cp /shared/rhsm/redhat-uep.pem /etc/pki/ca-trust/source/anchors/
+          update-ca-trust
+        fi
+
         # Copied from https://github.com/konflux-ci/build-definitions/blob/main/task/git-clone/0.1/git-clone.yaml
         if [ "${WORKSPACE_GIT_AUTH_BOUND}" = "true" ]; then
           if [ -f "${WORKSPACE_GIT_AUTH_PATH}/.git-credentials" ] && [ -f "${WORKSPACE_GIT_AUTH_PATH}/.gitconfig" ]; then
@@ -225,6 +399,18 @@ spec:
 
         cachi2 --log-level="$LOG_LEVEL" inject-files /var/workdir/cachi2/output \
           --for-output-dir=/cachi2/output
+    - name: unregister-rhsm
+      image: quay.io/redhat-appstudio/cachi2@sha256:eb34cfe3fea20997eebd8164dc93eedb2fd7a60dc1fb4afcc1b1ff43df9d6667
+      script: |
+        #!/bin/bash
+        if [ -f /shared/skip ]; then
+          echo "Skipping."
+          exit 0
+        fi
+
+        cp /shared/rhsm/consumer/* /etc/pki/consumer/
+        cp /shared/rhsm/entitlement/* /etc/pki/entitlement/
+        subscription-manager unregister || true
     - name: create-trusted-artifact
       image: quay.io/redhat-appstudio/build-trusted-artifacts:latest@sha256:81c4864dae6bb11595f657be887e205262e70086a05ed16ada827fd6391926ac
       args: