How do I grant acrPull access to AKS using bicep?

[ecmcn]

I have an existing ACR instance and am creating a new AKS resource via Bicep that I want to assign the acrPull role to. The role assignment happens but AKS gets auth errors trying to pull images. If instead I grant ACR access via "az aks update --attach-acr" everything works fine.

I'm using two modules, one to create the AKS resource and one to assign the role:

module aks 'AksCluster.bicep' = {
  name: 'AksCluster'
  scope: rg
  params: {
    aks_name: aks_name
  }
}

module rbac 'AksRoleAssignments.bicep' = {
  name: 'AksRoleAssignments'
  scope: resourceGroup(acr_repo.subscription_id, acr_repo.resource_group)
  params: {
     acr_name: acr_repo.name
     aks_principal_id: aks.outputs.aks_principal_id
  }
}

// from AksCluster.bicep

resource aks 'Microsoft.ContainerService/managedClusters@2021-03-01' = {
  name: aks_name
  location: resourceGroup().location
  identity: {
     type: 'SystemAssigned'
  }
  sku: {
    name: 'Basic'
    tier: 'Free'
  }
  properties: {
    kubernetesVersion: kubernetes_version
    dnsPrefix: aks_name
    enableRBAC: true
    nodeResourceGroup: kube_aks_nodes
    addonProfiles: {
      // ingress controller
      httpApplicationRouting: {
        enabled: true
      }
    }
    agentPoolProfiles:[
      {
        name: 'agentpool'
        count: aks_nodeCount
        vnetSubnetID: aks_vnet_subnet_id
        vmSize: 'Standard_D3_v2'
        osType: 'Linux'
        osSKU: 'Ubuntu'
        osDiskSizeGB: 0
        osDiskType: 'Managed'
        maxPods: 110
        mode: 'System'
        type:'VirtualMachineScaleSets'
      }
    ]
    networkProfile: {
      networkPlugin: 'azure'
      networkPolicy:'azure'
      serviceCidr: '10.0.0.0/16'
      dnsServiceIP: '10.0.0.10'
      dockerBridgeCidr: '172.16.0.1/16'
      outboundType: 'loadBalancer'
      loadBalancerSku: 'standard'
      loadBalancerProfile: {
        managedOutboundIPs: {
          count: 1
        }
      }
    }
   apiServerAccessProfile: {
      enablePrivateCluster: false
    }
    servicePrincipalProfile: {
      clientId: 'msi'
    }
  }  
  dependsOn: [
    vnet1
  ]
}
output aks_principal_id string = aks.identity.principalId

// from AksRoleAssignments.bicep

resource acr 'Microsoft.ContainerRegistry/registries@2020-11-01-preview' existing = {
  name: acr_name
}
resource  AssignAcrPullToAks 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
  name: guid(resourceGroup().id, acr_name, aks_principal_id, 'AssignAcrPullToAks')       // want consistent GUID on each run
  scope: acr
  properties: {
    description: 'Assign AcrPull role to AKS'
    principalId: aks_principal_id
    principalType: 'ServicePrincipal'
    roleDefinitionId: roleAcrPull
  }
}

The end result is ACR has a role assignment to my AKS cluster, showing in the Portal as type "App", which I believe means Managed Identity, but AKS can't pull images.

If I set this all up via "az aks update --attach-acr" everything works fine, with the visible difference being that the role assignment it creates is for my AKS agentpool managed identity as opposed to AKS cluster's identity. The agentpool is part of an AKS nodes resource group that's created alongside my AKS resource group, and I don't understand how to reference that from bicep. Any idea what I'm doing wrong?

[brwilkinson]

This is the docs...

https://docs.microsoft.com/en-us/azure/aks/use-managed-identity#summary-of-managed-identities

It's the Kublet identity that needs the role assignment against ACR. . . you can also BYO User assigned identity here as well, that way you can perform the Role assignment prior to deploying the AKS cluster.

This should be helpful to get the principal identity that you are looking for

// using this as an exampple, you already have your template for this
resource myAKS 'Microsoft.ContainerService/managedClusters@2021-03-01' existing = {
  name: 'aksapp01'
}

// I don't have a cluster up right now to test if you need 'properties there' ... so I am not sure if that path is accurate, however it should be close.
output principalID2 string = myAKS.properties.identityProfile.kubeletidentity.objectId

// this is another way, that i decompiled from working JSON, however could be turned to more 'bicep' as above to simplify, i added this because i know the path is correct in this case.
output principalID1 string = reference('Microsoft.ContainerService/managedClusters/myaks', '2020-12-01').identityProfile.kubeletidentity.objectId

I believe you example was using the identity of the system assigned managed identity of the cluster itself? You can also BYO with that one as well, via user assigned identity, however that is besides the point here.

[ecmcn]

That works - thanks! The only hiccup is a warning it generates (and VSCode reflects the same error):

Warning BCP053: The type "schemas:53_provisioningInfo" does not contain property "objectId". Available properties include "error".

on the line:
output aks_principal_id string = aks.properties.identityProfile.kubeletidentity.objectId

VSCode says identityProfile is type Dictionary<string,Schemas53ProvisioningInfo>, and it doesn't list any available properties on the object.

[brwilkinson]

Okay great. Yes I also had the schema validation warning on the linting, glad that you were able to test to confirm its working.

There are some other items open on the schema validation, I'll take a note and add it over there.

[brwilkinson]

@ecmcn also if you are happy with the above solution, you can mark it as "answered".

[justinmchase]

What is the value of roleAcrPull on the RoleAssignment? How are you creating that?

[brwilkinson]

sorry to answer you specific q. you are actually getting the ID of the AKS kubelet identity, then you assign that to be able to access the Container registry....

[justinmchase]

Right right, I think the only question is how to get the guid for the roleDefinitionId... It seems like you're saying they're actually globally constant unless I define my own? So for these well known roles I can just hack the guid into the code?

[brwilkinson]

Get-azRoleDefinition | out-gridview

That is my favorite way to view, then you can filter in the search/pop up.

AZ Module in PowerShell.

It's documentated online as well.

[brwilkinson]

It's actually a gap that at the moment we cannot use the friendly names.

Most workarounds involve exporting out a json table of the names /id's, then using that to do a friendly lookup.

So yeah, the guids are common, for the roles, across subscriptions, except for custom roles.

[justinmchase]

It would be nice if the various resources exported their roles upon creation so I could do:

resource ACR 'Microsoft.ContainerRegistry/registries@2021-06-01-preview' = {
  name: '...'
  location: resourceGroup().location
  sku: {
    name: sku
  }
  properties: {
    adminUserEnabled: true
  }
}

resource AssignAcrPullToAks 'Microsoft.Authorization/roleAssignments@2020-10-01-preview' = {
  name: guid(resourceGroup().id, RecoveryContainerRegistry.name, principalId, 'AcrPull')
  scope: RecoveryContainerRegistry
  properties: {
    principalId: principalId
    principalType: 'ServicePrincipal'
    roleDefinitionId: ACR.accessControl.roles.AcrPull // <-- has all the non-custom roles that apply to this resource strongly typed
  }
}

[HariRajan2014]

I saw another example here , I think it is for assigning a rbac role at the subscription level (scope) using subscriptionresourceid function.
https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/bicep-functions-resource#subscriptionresourceid-example

[justinmchase]

This would have saved me a lot of time yesterday, thanks!