Overlay network_plugin_mode not idempotent

[benkoben]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.4.4

AzureRM Provider Version

3.61.0

Affected Resource(s)/Data Source(s)

azurerm_kubernetes_cluster

Terraform Configuration Files

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">= 3.61.0"
    }
  }
  required_version = ">= 1.4.4"
}


provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "example" {
  name     = "example-resources"
  location = "West Europe"
}

resource "azurerm_kubernetes_cluster" "example" {
  name                = "example-aks1"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  dns_prefix          = "exampleaks1"

  default_node_pool {
    name       = "default"
    node_count = 1
    vm_size    = "Standard_D2_v2"
  }

  network_profile {
    network_plugin      = "azure"
    network_policy      = "calico"
    docker_bridge_cidr  = "172.16.0.0/12"
    network_plugin_mode = "Overlay"
    outbound_type       = "managedNATGateway"
    service_cidr        = "192.168.100.0/23"
    dns_service_ip      = "192.168.101.254"
  }

  identity {
    type = "SystemAssigned"
  }
}

Debug Output/Panic Output

https://gist.github.com/benkoben/db223ca286aa9ef667a00ca921f88637

Expected Behaviour

network_plugin_mode, when set to Overlay, should not force replace on second apply.

Actual Behaviour

network_plugin_mode, when set to Overlay, forces replacement which re-creates whole AKS cluster. It seems a lowered case value is written to state while the config is set to a capitalized value. Using a lowered case value in config raises an error by the azurerm API.

The following output is shown on second apply:

~ network_profile {
    ~ network_plugin_mode = "overlay" -> "Overlay" # forces replacement
    ~ pod_cidr            = "10.244.0.0/16" -> (known after apply)
    ~ pod_cidrs           = [
        - "10.244.0.0/16",
      ] -> (known after apply)
    ~ service_cidrs       = [
        - "192.168.100.0/23",
      ] -> (known after apply)

Steps to Reproduce

1. terraform apply to first deploy the resources
2. run a second terraform apply without any changes to config

Important Factoids

No response

References

No response

[stephybun]

Thanks for raising this issue @benkoben.

v3.61.0 of the provider updates the AKS API version to 2023-04-02-preview which contains a breaking case change in this enum. We've submitted a fix that will allow users to update their config to the new casing overlay without replacing the cluster which will go into the next release v3.62.0.

Since this is related to #22151 and a fix has already been merged I'm going to close this issue.

[Shabahang]

Hello,
Bug is not solved in 3.62.0.
it keeps destroying and recreating cluster.

[stephybun]

@Shabahang can you please provide some more information - what do you have in your config and what is the output from terraform?

[Shabahang]

@Shabahang can you please provide some more information - what do you have in your config and what is the output from terraform?

Terraform v1.4.5
on windows_amd64
......

• provider registry.terraform.io/hashicorp/azuread v2.39.0
• provider registry.terraform.io/hashicorp/azurerm v3.62.0

I have aks with Azure CNI - Overlay .
I have no changes in terraform config file , except the azurerm provider version from 3.58.0 to 3.61.0 or 3.62.0.
when I submit "terraform apply" commnad:

Plan: 1 to add, 0 to change, 1 to destroy.

and this line shows the cause:

      ~ network_plugin_mode = "overlay" -> "Overlay" # forces replacement

Then , just for test, I confirmed the action and aks was recreated.
when I repeated the terrafrom apply again, same happend.

Note that same config with provider version 3.58.0 has no update .

[Shabahang]

I need to add that I have another problem too.
when with version 3.61.0 and 3.62.0 , the object azurerm_role_assignment will be planed to replace , while it is not the case on 3.58.0.

   # azurerm_role_assignment.<.....> must be replaced

-/+ resource "azurerm_role_assignment" "<.....>" {
...
...
~ principal_id = "......." -> "........." # forces replacement

Plan: 1 to add, 0 to change, 1 to destroy.

[stephybun]

There's a breaking case change in the AKS API explained in this comment - you need to update the casing of Overlay in your config to overlay that will fix the issue you're having.

Regarding this problem

I need to add that I have another problem too. when with version 3.61.0 and 3.62.0 , the object azurerm_role_assignment will be planed to replace , while it is not the case on 3.58.0.

   # azurerm_role_assignment.<.....> must be replaced

-/+ resource "azurerm_role_assignment" "<.....>" { ... ... ~ principal_id = "......." -> "........." # forces replacement

Plan: 1 to add, 0 to change, 1 to destroy.

Could you please raise a new issue and fill out the bug information so we have some more context - it's difficult to look into issues without it.

Thanks!

[Shabahang]

I have created another Issue:
#22283

about changing the case in config file, Overlay to overlay, is it a permanent solution or temporary ?
I mean should we expect our config, to work with "Overlay" in future provider versions?

[myc2h6o]

This line is converting networkPluginMode to the value defined in the swagger, which is being changed from Overlay to overlay. The document seems still using the old value, which will cause a diff with the new version. I think we need to update the possible value in document as well to overlay?

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.