[EKS [request]: CNI custom networking to support more than one subnet.

[mehrajk]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Customers using VPC CNI custom networking can create ENIConfig with one subnet per AZ. This request is to allow more than one subnet per AZ for ENIConfig. Referring to the original requested from amazon-vpc-cni-k8s for additional inputs.

With this feature, customers can start with a smaller subnet/CIDR block first with ENIConfig and easily expand by adding more subnets in the future to have more IPs available for pods when needed.

Example scenario:
The expectation is that start with a single subnet for pod IPs (e.g., CUST_SNET1=100.64.0.0/22 which allows ~1000 pods for a subnet within a single AZ1). Run cluster for a while. Let's suppose, it runs out of pod IP capacity one day. Add another subnet (e.g., CUST_SNET2=100.65.0.0/22 for additional ~1000 pods to the same AZ1). Update ENIConfig for the AZ1 with two subnets now. Same ENIConfig, then pick up IPs (or delegated prefix) from these two subnets. Please note that second CIDR for the subnet CUST_SNET2=100.65.0.0/22 is not allocated to the ENIConfig on day-1, rather it will be added later if needed. It allows continuing expansion by adding more CIDR/subnets as needed without developing a complex logic of multiple ENIConfigs per subnets and automating individual node annotations as the new node join the EKS cluster.

This feature would also help when it is hard for customers to allocate a large contiguous CIDR block vs adding multiple smaller non-contiguous CIDR blocks to the ENIConfig.

cat <<EOF | kubectl apply -f -
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
 name: $AZ1
spec:
  securityGroups: 
    - sg-xxxxxxxxxxxx
  subnet:                            #<== Allow more than one subnets
    - $CUST_SNET1
    - $CUST_SNET2
EOF

Which service(s) is this request for?
EKS [CNI custom networking]

Are you currently working around this issue?
Currently there are a few complex workaround possible.
(1) Create another ENIConfig with a subnet with a large CIDR block. This is a disruptive process that requires draining nodes, terminating/restarting, etc. Also, it is hard for customers to allocate a large contiguous CIDR block vs adding multiple smaller non-contiguous CIDR blocks to the ENIConfig.
(2) Customers can also create several ENIConfigs per subnet using different subnets, however, they need to develop complex logic for managing multiple ENIConfigs per subnet and automating individual node annotations with the right ENIConfig as the new node join the EKS cluster.

[defyjoy]

This is very much needed as we have multiple subnets per AZ in VPC . We want to add those shorter ranges calculated instead of huge chunk of range and control the security groups more efficiently.

[dfroberg]

This is still super needed!
Any chance this will be picked up?
Expanding the number of subnets per AZ to grow out the available IP pools for clusters seems like a normal thing to do and should be supported in an un convoluted & disruptive way.

[vgunapati]

This issue is very important and needs to be prioritized. It's essential for provisioning smaller VPCs and expanding them over time. Additionally, creating large CIDR VPCs can lead to wasted IP addresses.