Install 4.14.0-0.okd-2023-10-28-073550 failed

[pkqsun]

Hi All,

We want to install latest OKD 4.14 (4.14.0-0.okd-2023-10-28-073550) on OpenStack, but failed.
FCOS: fedora-coreos-38.20231014.3.0-openstack.x86_64.qcow2

When the bootstrap VM is up, the bootkube.service failed.
Before the reboot, release-image.service finished successfully.
The failure logs show:

Nov 04 09:04:42 okd414csf002-bootstrap podman[3656]: 2023-11-04 09:04:42.921975816 +0000 UTC m=+0.101311595 container start c4a480d00d0064c1db1cbd286fd658a1eddb753bf74e3000c2d97816617e26b7 (image=quay.io/openshift/okd@sha256:7a6200e347a1b857e47f2ab0735eb1303af7d796a847d79ef9706f217cd12f5c, name=affectionate_davinci, io.openshift.release.base-image-digest=sha256:068639a8d5ffd7b2e15ee974851875c87b008366fb62a88a32d250ad981c8b47, io.openshift.release=4.14.0-0.okd-2023-10-28-073550)
Nov 04 09:04:42 okd414csf002-bootstrap podman[3656]: 2023-11-04 09:04:42.92294186 +0000 UTC m=+0.102277639 container attach c4a480d00d0064c1db1cbd286fd658a1eddb753bf74e3000c2d97816617e26b7 (image=quay.io/openshift/okd@sha256:7a6200e347a1b857e47f2ab0735eb1303af7d796a847d79ef9706f217cd12f5c, name=affectionate_davinci, io.openshift.release.base-image-digest=sha256:068639a8d5ffd7b2e15ee974851875c87b008366fb62a88a32d250ad981c8b47, io.openshift.release=4.14.0-0.okd-2023-10-28-073550)
Nov 04 09:04:42 okd414csf002-bootstrap podman[3656]: 2023-11-04 09:04:42.850230059 +0000 UTC m=+0.029565849 image pull 8f9b68887e5d3b7ff2ff12832a85cc994c26418d6819f0414328662aea38c7d4 quay.io/openshift/okd@sha256:7a6200e347a1b857e47f2ab0735eb1303af7d796a847d79ef9706f217cd12f5c
Nov 04 09:04:42 okd414csf002-bootstrap affectionate_davinci[3672]: quay.io/openshift/okd-content@sha256:b4f2eac789aebb02dc12bd676f7edb4e3d583bb672129aa496549db6e8cc8fec
Nov 04 09:04:42 okd414csf002-bootstrap podman[3656]: 2023-11-04 09:04:42.953271928 +0000 UTC m=+0.132607718 container died c4a480d00d0064c1db1cbd286fd658a1eddb753bf74e3000c2d97816617e26b7 (image=quay.io/openshift/okd@sha256:7a6200e347a1b857e47f2ab0735eb1303af7d796a847d79ef9706f217cd12f5c, name=affectionate_davinci, io.openshift.release=4.14.0-0.okd-2023-10-28-073550, io.openshift.release.base-image-digest=sha256:068639a8d5ffd7b2e15ee974851875c87b008366fb62a88a32d250ad981c8b47)
Nov 04 09:04:43 okd414csf002-bootstrap podman[3687]: 2023-11-04 09:04:43.00908414 +0000 UTC m=+0.043762779 container remove c4a480d00d0064c1db1cbd286fd658a1eddb753bf74e3000c2d97816617e26b7 (image=quay.io/openshift/okd@sha256:7a6200e347a1b857e47f2ab0735eb1303af7d796a847d79ef9706f217cd12f5c, name=affectionate_davinci, io.openshift.release=4.14.0-0.okd-2023-10-28-073550, io.openshift.release.base-image-digest=sha256:068639a8d5ffd7b2e15ee974851875c87b008366fb62a88a32d250ad981c8b47)
Nov 04 09:04:58 okd414csf002-bootstrap bootkube.sh[3706]: error: unable to read image quay.io/openshift/okd@sha256:7a6200e347a1b857e47f2ab0735eb1303af7d796a847d79ef9706f217cd12f5c: Get "https://quay.io/v2/": context deadline exceeded
Nov 04 09:04:58 okd414csf002-bootstrap systemd[1]: bootkube.service: Main process exited, code=exited, status=1/FAILURE
Nov 04 09:04:58 okd414csf002-bootstrap systemd[1]: bootkube.service: Failed with result 'exit-code'.
Nov 04 09:04:58 okd414csf002-bootstrap systemd[1]: bootkube.service: Consumed 3.342s CPU time.

[root@okd414csf002-bootstrap ~]# crictl images --digests
IMAGE TAG DIGEST IMAGE ID SIZE
quay.io/openshift/okd-content 8def1d5890588 e7202a5c80235 2.36GB
quay.io/openshift/okd 7a6200e347a1b 8f9b68887e5d3 522MB

"Crictl pull" shows the image is in local, but I am not sure what is the error of why " unable to read image quay.io/openshift/okd@sha256:7a6200e347a1b857e47f2ab0735eb1303af7d796a847d79ef9706f217cd12f5c"

Just want to check if anyone else met same issue ?

As a debug, I use the same Openstack, same configuration and same FCOS, tested using OKD 4.13.0-0.okd-2023-07-09-062029 or 4.13.0-0.okd-2023-10-28-065448, the installation all could finish successfully. Just want to bypass the network/registry access issue.

Appreciate for any comments.

[vrutkovs]

Please attach log bundle

[pkqsun]

log bundle attached:
log-bundle-20231105091805.tar.gz…

[vrutkovs]

That didn't work, could you try again? You can also use google drive or any service, as long as it can be downloaded publicly

[pkqsun]

It may related with
OCPBUGS-13094 - [4.14] bootkube.service failed in disconnected network install

We use internal mirror/registry to install OKD 4.14 (also for OKD 4.10 to 4.13, all successfully) and do not have internet access directly.

[pkqsun]

Update:

Hello @vrutkovs.
I removed the log bundle file as I think I found the root cause and the install finished successfully after update install-config.yaml.

For 4.14, after check file /usr/local/bin/bootkube.sh in bootstrap node, I find line:

VERSION="$(oc adm release info -o 'jsonpath={.metadata.version}' "${MIRROR_FLAG}" "${RELEASE_IMAGE_DIGEST}")"
it shows the OKD version info is always fetched by oc adm cmd, that do not use the podman/crio registry mirror I set .
In previous versions, 4.13 and older releases, I also did not set ImageContentSource but set the podman/crio registry mirror via ignition files.

After check file https://github.com/openshift/installer/blob/master/pkg/types/installconfig.go,

       // ImageContentSources lists sources/repositories for the release-image content.
	// The field is deprecated. Please use imageDigestSources.
	// +optional
	DeprecatedImageContentSources []ImageContentSource json:"imageContentSources,omitempty"

	// ImageDigestSources lists sources/repositories for the release-image content.
	// +optional
	ImageDigestSources []ImageDigestSource json:"imageDigestSources,omitempty"

Then I set ImageDigestSource parameter, now the installation goes well and finished successfully.

imageDigestSources:
  - mirrors:
      - quayio.repo.xxx.my.local.mirror/openshift/okd
    source: quay.io/openshift/okd
  - mirrors:
      - quayio.repo.xxx.my.local.mirror/openshift/okd-content
    source: quay.io/openshift/okd-content

Thanks all the same.