Bootstrap failed to complete with VSphere IPI

[faysou]

Describe the bug

okd cluster fails to bootstrap. I'm using a Vsphere IPI method on a bare metal ESXi 7.0.1 home lab. I used to manage to get a cluster installed while using ESXi inside VMWare Workstation on Windows server, but since trying to install okd on the same computer running ESXi directly the installation fails.

Of the three master nodes one manages to get 21 IPs (I see this in VCenter) while the other two nodes manage to get 13 and 14 IPs (a mix of IPv4 and IPv6 IPs). Contrary to when I managed to get successful cluster installations the CPU activity on the bootstrap and master nodes gets flat after some time.
It seems that when one of the master nodes acquires the IP corresponding to the api address it is not reachable.

INFO Creating infrastructure resources...         
INFO Waiting up to 20m0s for the Kubernetes API at https://api.ocp.grafchat.com:6443... 
INFO API v1.20.0-1077+2817867655bb7b-dirty up     
INFO Waiting up to 30m0s for bootstrapping to complete... 
ERROR Attempted to gather ClusterOperator status after installation failure: listing ClusterOperator objects: Get "https://api.ocp.grafchat.com:6443/apis/config.openshift.io/v1/clusteroperators": dial tcp 192.168.1.6:6443: connect: connection refused 
INFO Pulling debug logs from the bootstrap machine 
INFO Bootstrap gather logs captured here "/home/faysou/ocp/log-bundle-20210606232719.tar.gz" 
FATAL Bootstrap failed to complete: failed to wait for bootstrapping to complete: timed out waiting for the condition 

I'm attaching the log bundle.
I'm using fedora 34 as system for installing okd.
I'm using the bind configuration similar to here https://access.redhat.com/articles/5127191 for the dns server (but I only define forward local IPs for api and *.apps and a reverse pointer for api).

I've been working on the installation for about a month and a half and learned many things along the way but I don't manage to get it solved this time, so any help would be greatly appreciated.

Version

4.7.0-0.okd-2021-06-04-191031

Logs

log-bundle-20210606232719.tar.gz

[vrutkovs]

9188:Jun 06 13:27:23 ocp-kx5n5-master-0 hyperkube[1836]: E0606 13:27:23.496089 1836 kubelet_node_status.go:470] Error updating node status, will retry: error getting node "ocp-kx5n5-master-0": Get "https://api-int.ocp.grafchat.com:6443/api/v1/nodes/ocp-kx5n5-master-0?timeout=10s": dial tcp 192.168.1.6:6443: connect: connection refused

Keepalived on .6 is rejecting connections.

Weirdly enough it got configured to use .3:

time="2021-06-06T12:57:14Z" level=info msg="Apply config change" curConfig="{{ocp grafchat.com 192.168.1.6 23 A AAAA 19 192.168.1.7 134 A AAAA 32 3} {0 0 0 [] } 192.168.1.123 localhost ens192 [192.168.1.3 fe80::1%2] {[]} false}"

but nodes are .120-.123

[vrutkovs]

Probably I'm not reading this correctly.

@elmiko could you help me figure out what's going on with coreDNS/keepalived here?

[faysou]

In case it can help 192.168.1.3 is the address of the fedora machine that runs a bind dns server and from which I'm running the openshift-install (using the latest okd version).

I'm attaching my bind configuration files in case it can help.

dns_bind_files.zip

And for DHCP I'm using my internet box with the ip of the dns server as primary dns server (this setup used to work when I had ESXi in VMWare Workstation) :

[faysou]

I've found my problem thanks to your pointer with the wrong IP being used. I was using a TP Link wifi repeater in order to convert my wifi signal to an ethernet port (bare metal ESXi doesn't accept wifi connections). This was messing the network (the extender had its own dhcp server, but even when using the right dhcp server from the internet box, it still didn't work). So I have to connect by ethernet directly to my home internet box and now it's working.

Thank you for your help, I honestly didn't know where to look at in the logs.

[elmiko]

thanks for sharing your update on the progress