Deploy Azure Keyvault with Private Endpoint using AzureRM Terraform in Gitlab Pipelines

[jakthar]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.5.7

AzureRM Provider Version

3.72.1

Affected Resource(s)/Data Source(s)

azurerm_key_vault

Terraform Configuration Files

#1 ### GITLAB_SETTINGS_1. YAML ###

variables:
  HTTP_PROXY: "http://myproxy:8080"
  HTTPS_PROXY: "http://myproxy:8080"
  NO_PROXY: "169.254.169.254"
  CHECKPOINT_DISABLE: "true"

 #2 ### GITLAB_SETTINGS_2. YAML ###

variables:
  HTTP_PROXY: "http://myproxy:8080"
  HTTPS_PROXY: "http://myproxy:8080"
  NO_PROXY: "vault.azure.net"
  CHECKPOINT_DISABLE: "true"



### KEYVAULT & KEY_CREATION.TF ####

resource "azurerm_resource_group" "kv_rg" {
  name     = "kv_name_01"
  location = local.location
  tags     = local.["tags"]
}
 
 
resource "azurerm_key_vault" "kv" {
 
  name                      = "kv_name_test_01"
  location                  = local.location
  resource_group_name       = azurerm_resource_group.kv_rg.name
  tenant_id                 = data.azurerm_client_config.current.tenant_id
  sku_name                  = "standard"
  enable_rbac_authorization = true
 
  enabled_for_deployment          = false
  enabled_for_disk_encryption     = true
  enabled_for_template_deployment = false
  soft_delete_retention_days      = 90
  purge_protection_enabled        = true
  public_network_access_enabled   = false
  tags                            = local.["tags"]
  network_acls {
    default_action = "Deny"
    bypass         = "AzureServices"
  }
}
 

resource "azurerm_private_endpoint" "kv_private_endpoint" {
  depends_on          = azurerm_key_vault.kv
  name                = lower("${azurerm_key_vault.kv.name}-ep")
  location            = local.location
  resource_group_name = azurerm_resource_group.kv_rg.name
  subnet_id           = data.azurerm_subnet.subnet.id
  tags                = local.["tags"]
 
  private_dns_zone_group {
    name                 = "privatelink.vaultcore.azure.net"
    private_dns_zone_ids = [local.["key_vault_dns_zone_id"]]
  }
 
  private_service_connection {
    name                           = lower("${azurerm_key_vault.kv.name}-psc")
    private_connection_resource_id = azurerm_key_vault.kv.id
    is_manual_connection           = false
    subresource_names              = ["Vault"]
  }
}
 
 
resource "azurerm_key_vault_key" "cmkkey" {
  name            = "Key_name_test_01"
  key_vault_id    = azurerm_key_vault.kv.id
  key_type        = "RSA"
  key_size        = 2048
  key_opts        = var.key_opts
 
  tags = local.["tags"]
 
}

Debug Output/Panic Output

Following Error Received with #1 ### GITLAB_SETTINGS_1. YAML ###

│ Error: checking for presence of existing Key "cmk-test-01" (Key Vault "https://kvtemp-test-99.vault.azure.net/"): 
keyvault.BaseClient#GetKey: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. 
Status=403 Code="Forbidden" Message="Public network access is disabled and request is not from a trusted service nor via an approved private link.
\r\nCaller: appid=00000_000000_000000;iss=https://sts.windows.net/8e823e99-cbcb-430f-a0f6-af1365c21e22/;
xms_mirid=/subscriptions/XXXXXXXXX/resourcegroups/kv_name_01/providers/Microsoft.Compute/virtualMachines/RUNNERPC;
xms_az_rid=/subscriptions/XXXXXXXX/resourcegroups/GitlabRunner-RG/providers/Microsoft.Compute/virtualMachines/RUNNERPC
\r\nVault: kvtemp-test-99;location=australiaeast" InnerError={"code":"ForbiddenByConnection"}





Following Error Received with #2 ### GITLAB_SETTINGS_2. YAML ###


│ Error: waiting for Key Vault (Subscription: "XXXXXXXXX"
 
│ Resource Group Name: "kv_name_01"
 
│ Key Vault Name: "kvtemp-test-99") to become available: connecting to "https://kvtemp-test-99.vault.azure.net/": 
Get "https://kvkvtemp-test-99.vault.azure.net/": dial tcp: lookup kv-dev-des-test-99.vault.azure.net on DNSIP:53: no such host

Expected Behaviour

Deploy Azure KeyVault with Public Network Access Disabled
Create Private Endpoint for Azure Key Vault
Create Private Services Connection for Key Vault with the created Private EndPoint
Create Customer Managed Keys in the KeyVault

All above need to complete in single Pipeline execution.

Actual Behaviour

Attached the flow of the Steps to reproduce the error.

Terraform Apply with #1. GITLAB_SETTINGS_1. YAML(WITH PROXY) fails with Error "Status=403 Code="Forbidden" Message="Public network access is disabled and request is not from a trusted service nor via an approved private link."

Terraform Apply with #2. GITLAB_SETTINGS_2. YAML(WITH PROXY BYPASS to KV URL) "failed with Error "Key Vault Name: "kvtemp-test-99") to become available: connecting to "https://kvtemp-test-99.vault.azure.net/": Get "https://kvkvtemp-test-99.vault.azure.net/": dial tcp: lookup kv-dev-des-test-99.vault.azure.net on DNSIP:53: no such host"

Terraform Apply with #1 and #2 Running #1. GITLAB_SETTINGS_1. YAML (WITH PROXY) and after first error Running #2. GITLAB_SETTINGS_2. YAML(WITH PROXY BYPASS to KV URL) sequentially completes the pipeline and configuration successful and deploys the resources.

Steps to Reproduce

Attached the flow of the Steps to reproduce the error.

Terraform Apply with #1 ### GITLAB_SETTINGS_1. YAML(WITH PROXY) ### failed with Error "Status=403 Code="Forbidden" Message="Public network access is disabled and request is not from a trusted service nor via an approved private link."

Terraform Apply with #2 ### GITLAB_SETTINGS_2. YAML(WITH PROXY BYPASS to KV URL) ### failed with Error "Key Vault Name: "kvtemp-test-99") to become available: connecting to "https://kvtemp-test-99.vault.azure.net/": Get "https://kvkvtemp-test-99.vault.azure.net/": dial tcp: lookup kv-dev-des-test-99.vault.azure.net on DNSIP:53: no such host"

Terraform Apply Get successful running #1. GITLAB_SETTINGS_1. YAML and #2. GITLAB_SETTINGS_2. YAML completes the configuration successful.

Important Factoids

No response

References

[harshavmb]

Hi @jakthar ,

This is by no means an azurerm provider issue. Since you restricted the usage of keyvault over private endpoints your GitLab environment/proxy must be able to access over private network which means the name resolution of the keyvault to private IP also plays the key role.

• I assume your GitLab environment is accessible over public internet infrastructure & it can only resolve the keyvault to a public IP & Azure firewall restricts the access as you disabled public_network_access_enabled. Your first error message tells the same. If you know the outbound NAT IP addresses of GitLab infrastructure you could whitelist them but it's still over internet.
• Using proxy to resolve the KV IP address into private IP is a good idea but I think in your case the proxy server can't resolve the IP address & that's why you have dial tcp: lookup kv-dev-des-test-99.vault.azure.net on DNSIP:53: no such host error.

If you could logon to proxy server & you can try to resolve the kv-dev-des-test-99.vault.azure.net address either with nslookup or dig. My gut feeling is that it will fail. Few are the reasons I've on my mind.

• the DNS resolver isn't configured properly
• the proxy server & the keyvault vnets are not peered
• firewall/network device blocking DNS traffic in between (your internal n/w only, not Azure)

All in all it shouldn't be an issue with keyvault service alone but with many managed services Azure offers over private endpoints.

[rcskosir]

Thank you for taking the time to raise this! We use Issues in this repository to track feature enhancements and bugs in the Azure Provider. So that we can maintain focus on that, we instead ask that broader questions are raised using one of the Community Resources, such as the Azure Provider forum. With that in mind, I'm going to close this issue.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.