Transient failures when deploying a module that deploys a module

[vexingpos]

Hello all,

I have a policy file file that looks a bit like this:

resource ProductNameTagOnResource_InheritFromRG_IfMissing 'Microsoft.Authorization/policyDefinitions@2020-09-01' = {
  name: 'ProductNameTagOnResource_InheritFromRG_IfMissing'
  properties: {
    displayName: 'ProductNameTagOnResource_InheritFromRG_IfMissing'
    policyType: 'Custom'
    mode: 'Indexed'
    parameters: {}
    policyRule: {
      // full definition removed for ease of readability
    }
  }
}

I also have an initiative file that uses the policy from above, as a module, like this:

module ProductNameTagOnResource_InheritFromRG_IfMissing 'policy.bicep' = {
  name: 'ProductNameTagOnResource_InheritFromRG_IfMissing'
}

resource Tagging 'Microsoft.Authorization/policySetDefinitions@2020-09-01' = {
  name: 'Initiative-Tagging'
  properties: {
    policyType: 'Custom'
    displayName: 'Initiative-Tagging'
    description: 'Initiative-Tagging'
    metadata: {
      category: 'Tags'
    }
    parameters: {}
    policyDefinitions: [
      {
        policyDefinitionId: extensionResourceId(mgId, 'Microsoft.Authorization/policyDefinitions', ProductNameTagOnResource_InheritFromRG_IfMissing.name)
      }
    ]
  }
}

And to top it all off, I then have a policy assignment file (assignment.bicep) that uses the initiative as a module and deploys it and assigns it, like this:

// Deploy the tagging initiative
module taggingInitiative 'initiative' = {
  name: 'Initiative-Tagging'
}

// Assign the tagging initiative
resource taggingInitiativeAssignment 'Microsoft.Authorization/policyAssignments@2020-09-01' = {
  name: 'Policy-Tagging'
  properties: {
    displayName: 'Initiative-Tagging'
    description: 'Initiative-Tagging'
    enforcementMode: 'Default'
    policyDefinitionId: extensionResourceId(mgId, 'Microsoft.Authorization/policySetDefinitions', taggingInitiative.name)
  }
  location: 'eastus2'
  identity: {
    type: 'SystemAssigned'
  }
}

And to deploy this, I run

New-AzManagementGroupDeployment -Name "policy" -ManagementGroupId "XXX" -Location "eastus2" -TemplateFile "assignment.bicep"

When I run this, I get a transient deployment failure:

The deployment 'policy' failed with error(s). Showing 1 out of 1 error(s).
Status Message: The policy assignment create request is invalid. The policy set definition '/providers/Microsoft.Management/managementGroups/xxx/providers/Microsoft.Authorization/policySetDefinitions/Initiative-Tagging' could not be found.

And yet when I look in the Portal, I can see that the initiative called Initiative-Tagging clearly exists! If I hit redeploy, the second attempt works successfully. So what's up??

The annoying thing is, when I attempt all of this without modules (i.e. policy and initiative as "inline resources" all in one file), everything works perfectly the first time round.

It's almost as if the use of modules/nested deployments creates a race condition causing the initiative assignment to fail.

[brwilkinson]

Since your policy name matches your module name, then you reference it, that should add a dependson in your json template.

• Be careful with this since if the resource name doesn't match the module name (which is actually the deployment name), below will not work, in that case you need to add the dependson yourself explicitly.

// this should add the dependson: ProductNameTagOnResource_InheritFromRG_IfMissing.name
policyDefinitionId: extensionResourceId(mgId, 'Microsoft.Authorization/policyDefinitions', ProductNameTagOnResource_InheritFromRG_IfMissing.name)

// also for this: taggingInitiative.name
policyDefinitionId: extensionResourceId(mgId, 'Microsoft.Authorization/policySetDefinitions', taggingInitiative.name)

you can right click on the bicep file and build to the json template to confirm that, you should see a dependson.

Do you have this in a public repo? I would like to test deploy it myself to figure out what is going on?

You can test with an explicit dependson as well, as below... like I say I don't think this is needed in this specific case, however test it out.

resource Tagging 'Microsoft.Authorization/policySetDefinitions@2020-09-01' = {
  name: 'Initiative-Tagging'
  properties: {
    policyType: 'Custom'
    displayName: 'Initiative-Tagging'
    description: 'Initiative-Tagging'
    metadata: {
      category: 'Tags'
    }
    parameters: {}
    policyDefinitions: [
      {
        policyDefinitionId: extensionResourceId(mgId, 'Microsoft.Authorization/policyDefinitions', ProductNameTagOnResource_InheritFromRG_IfMissing.name)
      }
    ]
  }
  dependsOn: [
    ProductNameTagOnResource_InheritFromRG_IfMissing
  ]
}

[brwilkinson]

Sounds good, I'll give it a test run, see what I can come up with.

I have a feeling it's a platform issue with the management group scope + policy.

Will see if there are any workarounds or other reports of similar.

[brwilkinson]

Hi @vexingpos ...

Thanks for reporting this issue.

I was easily able to reproduce this error with your samples in both Management Group and Subscription Scope deployments.

• I haven't finished my search for other reports of this issue so far, however I will be doing some follow up to find the right place and path to report this and see if there is a potential solution/fix.

To describe the issue:

• When you do a deployment into an MG or Sub scope, the deployment completes before the resources have been completely provisioned.
  • The outcome of this via example, is when you create a policy definition within that deployment, it is not available to reference in a later module/deployment, for the policy assignment.

So now a few possible workarounds for you, until a solution might be available.

I reviewed a few other repos who have Policy deployments, such as @globalbao 'azure-policy-as-code' (also tagging @jesseloudon)
https://github.com/globalbao/azure-policy-as-code/blob/6f728f64095d7908a805f2e196348c35cc9d883e/Bicep/modules/mg_main.bicep#L59

What I noticed was a large amount of policies being deployed, plus an explicit depends on for the order:

1. policyDefinitions e.g. mg_definitions
2. policySetDefinitions e.g. mg_initiatives
3. policyAssignments e.g. mg_assignments

Now your sample also has those dependencies and order, however you only have a single policyDefinition. Each of those above must wait for the complete list of the policies in the previous step to deploy first, so I assume that gives it enough time to complete.

I found a few workarounds, I'll summarize, then document two of them below:

1. Put all 3 of the above stages in a single module and it works as expected (this does not seem like an attractive option for code re-use)
2. I found that I only needed to deploy 1 (or 2) extra policy Definitions to give it enough time to create/replicate the definition for the rest of the deployments to complete successfully

workaround 1, this mainly demonstrates what the underlying issue is here, which will be needed to report this issue

• assignment.bicep - this main file calls a single module with all 3 components being deployed

targetScope = 'managementGroup'

module initiativeTagging 'Initiative-Tagging.bicep' = {
  name: 'initiativeTagging'
    params: {
    initiativeName: 'Tagging'
    category: 'tags'
  }
}

output mgid string = managementGroup().id
output policyDefinitionId string = initiativeTagging.outputs.policydefinitionId
output policySetDefinitionId string = initiativeTagging.outputs.policySetDefinitionId
output policyAssignmentId string = initiativeTagging.outputs.policyAssignmentId

• assignment.bicep

param initiativeName string
param category string

targetScope = 'managementGroup'

resource productNameTagOnResource_InheritFromRG_IfMissing 'Microsoft.Authorization/policyDefinitions@2020-09-01' = {
  name: 'productNameTagOnResource_InheritFromRG_IfMissing'
  properties: {
    displayName: 'ProductNameTagOnResource_InheritFromRG_IfMissing'
    description: 'Resources will inherit the ProductName tag from the resource group if no value is provided'
    policyType: 'Custom'
    mode: 'Indexed'
    metadata: {
      category: 'Tags'
    }
    parameters: {}
    policyRule: {
      if: {
        allOf: [
          {
            anyOf: [
              {
                field: 'tags[\'ProductName\']'
                exists: false
              }
              {
                field: 'tags[\'ProductName\']'
                equals: '' 
              }
            ]
          }
          {
            allOf: [
              {
                value: '[resourceGroup().tags[\'ProductName\']]'
                exists: true
              }
              {
                value: '[resourceGroup().tags[\'ProductName\']]'
                notEquals: ''
              }
            ]
          }
        ]
      }
      then: {
        effect: 'modify'
        details: {
          roleDefinitionIds: [
            '/providers/microsoft.authorization/roleDefinitions/4a9ae827-6dc8-4573-8ac7-8239d42aa03f'
          ]
          operations: [
            {
              operation: 'addOrReplace'
              field: 'tags[\'ProductName\']'
              value: '[resourceGroup().tags[\'ProductName\']]'
            }
          ]
        }
      }
    }
  }
}

resource policySet 'Microsoft.Authorization/policySetDefinitions@2020-09-01' = {
  name: 'policySet-${initiativeName}'
  properties: {
    policyType: 'Custom'
    displayName: 'Initiative-${initiativeName}'
    description: 'Initiative-${initiativeName}'
    metadata: {
      category: category
    }
    parameters: {}
    policyDefinitions: [
      {
        policyDefinitionId: productNameTagOnResource_InheritFromRG_IfMissing.id
      }
    ]
  }
}

resource policyAssignment 'Microsoft.Authorization/policyAssignments@2020-09-01' = {
  name: 'policyAssignment-${initiativeName}'
  properties: {
    displayName: 'Initiative-${initiativeName}'
    description: 'Initiative-${initiativeName}'
    enforcementMode: 'Default'
    policyDefinitionId: policySet.id
  }
  location: deployment().location
  identity: {
    type: 'SystemAssigned'
  }
  dependsOn:[
    productNameTagOnResource_InheritFromRG_IfMissing
  ]
}

output policyAssignmentId string = policyAssignment.id
output policySetDefinitionId string = policySet.id
output policydefinitionId string = productNameTagOnResource_InheritFromRG_IfMissing.id

workaround 2, this is a little different from what you started with, however it demonstrates the technique

targetScope = 'managementGroup'

var policyDefinitions = [
  'productNameTagOnResource_InheritFromRG_IfMissing'
  'policy2'
  'policy3'
]

@batchSize(1)
module policyDefinition './nested_productNameTagOnResource_InheritFromRG_IfMissing.bicep' = [for item in policyDefinitions: {
  name: item
}]

module initiativeTagging 'Initiative-Tagging.bicep' = {
  name: 'initiativeTagging'
    params: {
    initiativeName: 'Tagging'
    definitionName: policyDefinition[0].name
    category: 'tags'
  }
  dependsOn: [
    policyDefinition
  ]
}

output mgid string = managementGroup().id
output policyDefinitionId string = policyDefinition[0].outputs.policydefinitionId
output policySetDefinitionId string = initiativeTagging.outputs.policySetDefinitionId
output policyAssignmentId string = initiativeTagging.outputs.policyAssignmentId

Notice above I shifted the module for the policyDefinition to the top level

• then I am now actually looping over deploying that same policydefinition 3 times.
  • Assuming in your case you will have 3 separate policy definitions here
• I opted to do that serially, so I added the batchsize(1) so that it does them 1 at a time.

I found that was enough time, while the second, then third policydefinition was being created for later steps to complete successfully.

• AGAIN, I just demo'd with the exact same policydefinition three times, however I imagine you will have other different policies to deploy.

I moved the policyDefinitionSet (initiative) and the policyAssignment into the same module as below, I am not sure if that would work for you, however if they were in separate modules/deployments the same workaround as above should work.

param initiativeName string
param definitionName string
param category string

targetScope = 'managementGroup'

resource PD 'Microsoft.Authorization/policyDefinitions@2020-09-01' existing = {
  name: definitionName
}


resource policySet 'Microsoft.Authorization/policySetDefinitions@2020-09-01' = {
  name: 'policySet-${initiativeName}'
  properties: {
    policyType: 'Custom'
    displayName: 'Initiative-${initiativeName}'
    description: 'Initiative-${initiativeName}'
    metadata: {
      category: category
    }
    parameters: {}
    policyDefinitions: [
      {
        policyDefinitionId: PD.id
      }
    ]
  }
}

resource policyAssignment 'Microsoft.Authorization/policyAssignments@2020-09-01' = {
  name: 'policyAssignment-${initiativeName}'
  properties: {
    displayName: 'Initiative-${initiativeName}'
    description: 'Initiative-${initiativeName}'
    enforcementMode: 'Default'
    policyDefinitionId: policySet.id
  }
  location: deployment().location
  identity: {
    type: 'SystemAssigned'
  }
  dependsOn:[
    PD
  ]
}

output policyAssignmentId string = policyAssignment.id
output policySetDefinitionId string = policySet.id

So let me know if you have any feedback and also if this workaround fits your short-term needs or let me know if you test it out?

I'll keep exploring options and also find the right avenue to escalate this issue.

[vexingpos]

Thanks for that very detailed write-up, it's appreciated. And likewise, thank you for confirming this is indeed a bug with how the ARM backend operates.

I'm trying to get my head around workaround 2. Is the gist of it that you basically deploy each policy multiple times?

• So if I have 5 custom policies, each custom policy needs to be deployed three times each? So 15 policy "deployments" in total before we deploy the initiative?

[brwilkinson]

If you have 1 policy, then deploy it 3 times.

Start time off deployment is 0.

10 Seconds to deploy policy1 the first time,.
5 seconds to deploy policy 1 second time.
5 seconds to deploy policy1 third time.

Total time before stage 2 starts is 20 seconds.

Now If you have a second policy, you still need to delay by the same 20 seconds.

10 seconds to deploy policy 1
10 seconds to deploy policy 2

Total time before stage 2 starts is 20 seconds.

So it's likely the issue goes away by itself after you add the second policy.

If not, you could still loop 1 extra time over the last policy, so that you have a min loop size of 3.

Ensure you use the batchsize(1) so they run serially (one at a time).

[brwilkinson]

Also just to add, after going through this exercise, I would say:

• if you just have 1 policy, use workaround 1 to group all items together in the single module/deployment
• if you have multiples, then this issue will likely go away on it's own after the 2nd or maybe 3rd policy is added, if you use batchsize(1)