CR Api Gateway - deployment service account missing imagePullSecrets

[pawellegowski89]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

CR api gateway deployment pod can't pull its consul-dataplane docker image. This is because when we deployed deploy CR with kind: Gateway, the correct deployment will be created with the correct pointing to serviceAccount, but unfortunately, this serviceAccount named <gateway-name>-gateway does not contain imagePullSecret in its definition.

Reproduction Steps

1. Docker pull official images and add own tags
2. Docker push official images with own tags to private registry (Azure container registry) which is secured (ImagePullSecret is required)
3. Add secret on k8s - myregistry.azurecr.io-access (kubernetes.io/dockerconfigjson) - is valid
4. Download official consul helm chart 1.5.3 (Consul 1.19.2)
5. Set in values yaml:

global:
  datacenter: mycenter
  name: consul
  image: myregistry.azurecr.io/repo/release/consul:1.19.2
  imageK8S: myregistry.azurecr.io/repo/release/consul-k8s-control-plane:1.5.3
  imageConsulDataplane: myregistry.azurecr.io/repo/release/consul-dataplane:1.5.3
  imagePullSecrets:
    - name: myregistry.azurecr.io-access 

6. Install this chart on k8s
7. Ad own CR for consul Api Gateway

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: api-gateway
  namespace: consul
spec:
  gatewayClassName: consul
  listeners:
  ...

command to find created object with bad definition:

kubectl get serviceAccount my-api-gateway -n ns

Invalid serviceAccount definition:

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    component: api-gateway
    gateway.consul.hashicorp.com/created: "1725890955"
    gateway.consul.hashicorp.com/managed: "true"
    gateway.consul.hashicorp.com/name: int-mesh-gateway
    gateway.consul.hashicorp.com/namespace: test
  name: my-api-gateway
  namespace: test
  ownerReferences:
  - apiVersion: gateway.networking.k8s.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: Gateway
    name: my-api-gateway   

Logs

Logs in 1.19.2 version:

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  14m                   default-scheduler  Successfully assigned spoc/my-api-gateway-58c4f65b9c-82ktg to aks-nodepool1-myVM
  Normal   Pulled     14m                   kubelet            Pulling image "myregistry.azurecr.io/repo/release/consul-k8s-control-plane:1.5.3" 
  Normal   Created    14m                   kubelet            Created container consul-connect-inject-init
  Normal   Started    14m                   kubelet            Started container consul-connect-inject-init
  Normal   Pulling    13m (x4 over 14m)     kubelet            Pulling image "myregistry.azurecr.io/repo/release/consul-dataplane:1.5.3"
  Warning  Failed     13m (x4 over 14m)     kubelet            Failed to pull image "myregistry.azurecr.io/repo/release/consul-dataplane:1.5.3": failed to pull and unpack image "myregistry.azurecr.io/repo/release/consul-dataplane:1.5.3": failed to resolve reference "myregistry.azurecr.io/repo/release/consul-dataplane:1.5.3": failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://myregistry.azurecr.io/oauth2/token?scope=repository%3Arelease%2Fconsul-dataplane%3Apull&service=myregistry.azurecr.io: 401 Unauthorized
  Warning  Failed     13m (x4 over 14m)     kubelet            Error: ErrImagePull
  Warning  Failed     13m (x5 over 14m)     kubelet            Error: ImagePullBackOff
  Normal   BackOff    4m39s (x42 over 14m)  kubelet            Back-off pulling image "myregistry.azurecr.io/repo/release/consul-dataplane:1.5.3"

Expected behavior

The ServiceAccount created by the controller adding objects after detecting CR gateway will contain the correct imagePullSecret defined in the helm chart, in the global section

Environment details

• consul-k8s version: 1.5.3
• values.yaml used to deploy the helm chart:

global:
  datacenter: mycenter
  name: consul
  image: myregistry.azurecr.io/repo/release/consul:1.19.2
  imageK8S: myregistry.azurecr.io/repo/release/consul-k8s-control-plane:1.5.3
  imageConsulDataplane: myregistry.azurecr.io/repo/release/consul-dataplane:1.5.3
  imagePullSecrets:
    - name: myregistry.azurecr.io-access 

Additional Context

I think the definition below is wrong:

https://github.com/hashicorp/consul-k8s/blob/v1.5.3/control-plane/gateways/serviceaccount.go

The ServiceAccount definition is missing the imagePullSecret if it was added in the helm chart under global:

In version 1.19.2 (consul-k8s1.5.3) you fixed the bug only for pulling the consul-k8s-control-plane image, which is needed to init container api gateway pod in #3862 but not for api gateway deployment image - consul-dataplane.

By the way, check this for other CR creation cases where serviceAccount from imagePullSecret is required.

[pawellegowski89]

I installed the latest chart 1.6.1 with consul version 1.20.1 and I continue to have exactly the same problem.

I don't know if the fix isn't working, or if something additional needs to be changed in the values.yaml file for this to finally work properly?

I didn't notice any changes in the definition of serviceAccount.

@nathancoleman please let me know if this change works for you?