jx-admin: Incorrect CustomResourceDefinition api version for for kubernetes 1.22+.

[bhomrich]

During an install onto a new 1.22.10 kubernetes on-prem, jx3 boot pod fails with CRD api version errors.

After following the on-prem install instructions (https://jenkins-x.io/v3/admin/platforms/on-premises/), and getting to the operator install step (jx admin.....)

Logged output is the same as the previously reported issue #10:

saved docs/releases.yaml
saved docs/README.md
jx gitops scheduler
jx gitops hash --pod-spec --kind Deployment -s config-root/namespaces/jx/lighthouse-config/config-cm.yaml -s config-root/namespaces/jx/lighthouse-config/plugins-cm.yaml -d config-root/namespaces/jx/lighthouse
jx gitops label --dir config-root/cluster gitops.jenkins-x.io/pipeline=cluster
jx gitops label --dir config-root/customresourcedefinitions gitops.jenkins-x.io/pipeline=customresourcedefinitions
jx gitops label --dir config-root/namespaces gitops.jenkins-x.io/pipeline=namespaces
jx gitops annotate --dir config-root --selector app=pusher-wave kapp.k14s.io/change-group=apps.jenkins-x.io/pusher-wave
jx gitops annotate --dir config-root --selector app.kubernetes.io/name=ingress-nginx kapp.k14s.io/change-group=apps.jenkins-x.io/ingress-nginx
jx gitops label --dir config-root/cluster --kind=Namespace team=jx
jx gitops annotate --dir config-root/namespaces --kind Deployment --selector app=pusher-wave --invert-selector wave.pusher.com/update-on-config-change=true
using kubectl to apply resources
kubectl apply --force --prune -l=gitops.jenkins-x.io/pipeline=customresourcedefinitions -R -f config-root/customresourcedefinitions
customresourcedefinition.apiextensions.k8s.io/environments.jenkins.io configured
customresourcedefinition.apiextensions.k8s.io/pipelineactivities.jenkins.io configured
customresourcedefinition.apiextensions.k8s.io/releases.jenkins.io configured
customresourcedefinition.apiextensions.k8s.io/sourcerepositories.jenkins.io configured
customresourcedefinition.apiextensions.k8s.io/previews.preview.jenkins.io unchanged
customresourcedefinition.apiextensions.k8s.io/lighthousebreakpoints.lighthouse.jenkins.io unchanged
customresourcedefinition.apiextensions.k8s.io/lighthousejobs.lighthouse.jenkins.io unchanged
customresourcedefinition.apiextensions.k8s.io/externalsecrets.kubernetes-client.io unchanged
customresourcedefinition.apiextensions.k8s.io/clustertasks.tekton.dev configured
customresourcedefinition.apiextensions.k8s.io/conditions.tekton.dev unchanged
customresourcedefinition.apiextensions.k8s.io/pipelineresources.tekton.dev unchanged
customresourcedefinition.apiextensions.k8s.io/pipelineruns.tekton.dev configured
customresourcedefinition.apiextensions.k8s.io/pipelines.tekton.dev configured
customresourcedefinition.apiextensions.k8s.io/runs.tekton.dev configured
customresourcedefinition.apiextensions.k8s.io/taskruns.tekton.dev configured
customresourcedefinition.apiextensions.k8s.io/tasks.tekton.dev configured
unable to recognize "config-root/customresourcedefinitions/kuberhealthy/kuberhealthy/khchecks.comcast.github.io-crd.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "config-root/customresourcedefinitions/kuberhealthy/kuberhealthy/khjobs.comcast.github.io-crd.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
unable to recognize "config-root/customresourcedefinitions/kuberhealthy/kuberhealthy/khstates.comcast.github.io-crd.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
make[1]: *** [versionStream/src/Makefile.mk:289: kubectl-apply] Error 1
make[1]: Leaving directory '/workspace/source'
error: failed to regenerate: failed to regenerate phase 1: failed to run 'make regen-phase-1' command in directory '.', output: ''
make: *** [versionStream/src/Makefile.mk:243: regen-check] Error 1
pod jx-boot-8268e686-0799-42b6-b7ab-5a264e2062ae-96z9b has status Ready

I am a novice in understanding how to get the environment to deploy with changed dependencies (my guess is the current specification of kuberhealthy @ version "64" is playing a role in the issue, but I can't seem to make my install change versions at deployment). If there is a primer on how the install/build process works (where it takes inputs to finally build/deploy resources), I'll happily take a pointer and try to help resolve this issue.

[fabian-kramer]

Hi @bhomrich ,

could you solve the issue? If so, could you please share the solution?
I'm having the same issue on Kubernetes 1.23.x on Ubuntu 20.04

[babadofar]

To fix this you may remove the line that includes kuberhealthy in helmfile.yaml
here: https://github.com/jx3-gitops-repositories/jx3-kubernetes/blob/main/helmfile.yaml#L4

[babadofar]

Here is an overview of how it works https://jenkins-x.io/v3/about/how-it-works/

[CPinhoK]

Issue still occurs in K8s 1.24.10 with the line that includes kuberhealthy in helmfile.yaml removed

unable to recognize "config-root/namespaces/jx/jx-kh-check-health-checks-jx/jx-bot-token-kuberhealthycheck.yaml": no matches for kind "KuberhealthyCheck" in version "comcast.github.io/v1"
unable to recognize "config-root/namespaces/jx/jx-kh-check-health-checks-jx/jx-webhook-events-kuberhealthycheck.yaml": no matches for kind "KuberhealthyCheck" in version "comcast.github.io/v1"
unable to recognize "config-root/namespaces/jx/jx-kh-check-health-checks-jx/jx-webhook-kuberhealthycheck.yaml": no matches for kind "KuberhealthyCheck" in version "comcast.github.io/v1"

[kyleli666]

I tried with newer version kuberhealthy charts, then this problem seems solved, but I still have another issue that tekton cannot strart...(Solved according to tektoncd/pipeline#3367 (comment))

However lighthouse webhook still not working ... :(

[stewartshea]

I opened this PR to help resolve the CRD apiVersion issue
#14

[zwpaper]

if anyone is trying to disable kuberhealthy and meet errors, should look into the below, which also has a kuberhealthy check, and you should also disable those lines.

jx3-kubernetes/helmfiles/jx/helmfile.yaml

Lines 57 to 62 in a08f795

	- chart: jxgh/jx-kh-check
	version: 0.0.78
	name: health-checks-jx
	values:
	- ../../versionStream/charts/jxgh/health-checks-jx/values.yaml.gotmpl
	- jx-values.yaml