Bootstrap doesn't add any of the nodes

[davidtluong]

Describe the bug

Bootstrap times out with no errors and no nodes are added to the cluster.

Master nodes seem to take the ignition file hosted by the bootstrap, since they all restart and boot to CoreOS after hitting the api-int successfully, but I'm not sure if it's actually taking the config considering I can ssh into my bootstraps but get an ssh key denied with the masters.

Version

4.9

How reproducible

Every time

Log bundle

oc get csr --kubeconfig ../install/auth/kubeconfig
No resources found
oc get nodes --kubeconfig ../install/auth/kubeconfig
No resources found

install-config.yaml:

---
apiVersion: v1
baseDomain: psyiyi.com 
compute:
- hyperthreading: Enabled
  name: worker
  replicas: 0
controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3
metadata:
  name: exeggcute
networking:
  clusterNetwork:
  - cidr: 10.100.0.0/14
    hostPrefix: 23
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16 
platform:
  none: {}
pullSecret: '{mypullsecrethere}}}'
sshKey: |
  ssh-rsa ...
  ssh-rsa ...

bootkubelogs.txt

[vrutkovs]

Please attach log bundle

[davidtluong]

log-bundle-20211208094809.tar.gz

[vrutkovs]

I1208 14:28:00.820528 1 api.go:110] Pool master requested by address:"10.0.0.22:51986" User-Agent:"Ignition/2.12.0" Accept-Header: "application/vnd.coreos.ignition+json;version=3.3.0, /;q=0.1"
I1208 14:28:49.736536 1 api.go:110] Pool master requested by address:"10.0.0.22:52562" User-Agent:"Ignition/2.12.0" Accept-Header: "application/vnd.coreos.ignition+json;version=3.3.0, /;q=0.1"
I1208 14:29:35.919655 1 api.go:110] Pool master requested by address:"10.0.0.22:53152" User-Agent:"Ignition/2.12.0" Accept-Header: "application/vnd.coreos.ignition+json;version=3.3.0, /;q=0.1"

Did your masters get valid IPs? It looks like .22 was assigned to all three of them

[davidtluong]

Yes, they all have valid IPs, example, I rebooted master0:

10.0.0.22 is the haproxy serving api-int, api, and *.apps

[fscojr]

Hi, I'm facing the same problem, you managed to solve it ?

[davidtluong]

Yeah, I used fcos 34 by utilizing this stream:
https://github.com/coreos/fedora-coreos-streams/blob/37f9e95f37e60e658248865e6b6bb42869e150a4/streams/stable.json#L118-L134
Since @vrutkovs so kindly pointed out that this is the stream that is tested against via this documentation:
https://github.com/openshift/okd/blob/master/FAQ.md#which-fedora-coreos-should-i-use

[davidtluong]

Also, as you an see from the logs, ssh auth doesn't work for some reason on the masters, it's like it grabbed some kind of wrong master.ign after it hit api-int or it just didn't respect it. Attaching a screenshot to show that it did indeed grab some sort of ignition