Internal Load Balancer Implementation W/Passthrough LB Does Not Appear to Support More Than Single Node Control Plane Cluster

[bender100]

/kind bug

What steps did you take and what happened:
The implementation for Internal Load Balancer uses Passthrough Network LB which causes issues while spinning up a control plane with more than one node. When subsequent control plane nodes try to join they will fail during preflight checks due to the fact that it uses Direct Server Return and requests are routed to the same host without taking into account whether its a healthy backend as this is prior to static pods being deployed so nothing can respond to the requests. I'm not 100% certain if this is a bug due to the implementation choice or if there is a work around for this issue however it seems like using an Interrnal Proxy Network Load Balancer is a better fit for this use case as it will only route to healthy backends which addresses this issue.

Reference in docs:
If the client VM is a backend VM of the load balancer, connections sent to the IP address of the load balancer's forwarding rule are always answered by the client/backend VM. This happens regardless of whether the backend VM is healthy. It happens for all traffic sent to the load balancer's IP address, not just traffic on the protocol and ports specified in the load balancer's internal forwarding rule.

What did you expect to happen:
Expected a control plane with more than one node to be provisioned successfully

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:
N/A This is implementation detail of feature rather then specific to a release

• Cluster-api version:
• Minikube/KIND version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):