This repository contains sample only code to demonstrate how secrets can be marshaled from Key Vault into a Container Instance running Nextflow for the purpose of dispatching Nextflow pipelines on Azure Batch.
Deploying the Azure resources supporting this sample is left to you.
It is assumed that this infrastructure would be deployed using GitHub workflows and maintained examples are provided. Stale examples for DevOps pipelines are provided but are not working since moving to rust and will not be maintained.
It is assumed that regardless of deployment method, GitHub is used for hosting the forked repository.
-
Fork this repository on GitHub.
-
Install Azure CLI version 2.45.0 or newer and login.
# Check version az --version # Login az login # Show active subscription az account show
-
Define these variables (change values as needed).
az_subId="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" az_location="australiaeast" az_rgName="GitHubFederatedIdentities" az_msiName="GitHubActions" gh_org="xxxxxx" # this repo's org or username gh_repo="azure-nextflow" # this repo's name
-
Create a Resource Group for the Managed Identity.
# Create Resource Group az group create --name $az_rgName --location $az_location --subscription $az_subId
-
Create a Managed Identity (user assigned) for GitHub to authenticate with Azure.
# Create MSI az identity create --name $az_msiName --resource-group $az_rgName --subscription $az_subId # Federate with GitHub az identity federated-credential create --name "${gh_org}--${gh_repo}" --identity-name $az_msiName --subject "repo:${gh_org}/${gh_repo}:ref:refs/heads/main" --issuer "https://token.actions.githubusercontent.com" --resource-group $az_rgName --subscription $az_subId # Show details (needed to create the AZURE_MSI GitHub secret) az identity show --name $az_msiName --resource-group $az_rgName
-
Assign ARM permissions to the Managed Identity
# Assign ARM permissions az role assignment create --assignee $az_msiName --role 'Owner' --scope /subscriptions/$az_subId
Note: The permissions above are an example, use the least privileged approach when assigning permissions.
The following uses the provided GitHub workflows to build and deploy the sample.
The deployment is targeted at the subscription level and will create the target resource group if it does not exist. This is why the deployment principal requires permission at the Subscription scope.
The deployment creates and assigns permissions to a number of Managed Identities. This is why the deployment principal requires the Owner permission.
Important: As a minimum the deployment principal will require
Contributorpermission on the subscription andOwnerpermission on the target resource group provided the target resource group is created ahead of time.
-
Use the JSON output from the pre-requisite steps to create a new
AZURE_MSIGitHub Secret, details in GitHub Docs.Name
AZURE_MSIValue
{ "clientId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx1111", "id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/nxfutil001/providers/Microsoft.ManagedIdentity/userAssignedIdentities/GitHubActions", "location": "australiaeast", "name": "GitHubActions", "principalId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx2222", "resourceGroup": "GitHubFederatedIdentities", "tags": {}, "tenantId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx3333", "type": "Microsoft.ManagedIdentity/userAssignedIdentities" } -
Run the workflow called
GitHub Workflows.
To validate set up was successful, trigger the Azure Function using its webhook.
az_funcAppName="myFuncAppName"
curl --get "https://$az_funcAppName.azurewebsites.net/api/nxfutil"
# returns
Hello, World!Used to get status of api and list available api paths
az_funcAppName="myFuncAppName"
curl --get "https://$az_funcAppName.azurewebsites.net/api/nxfutil"
# returns
# [GET,POST] api/nxfutil/dispatch
# [GET,POST] api/nxfutil/status
# [GET,POST] api/nxfutil/terminate Will show the json payload required for the corresponding POST method.
Will create a new nxfutil Container Instance. See Annex for nxfutil details.
The http trigger requires a json payload to provide the nextflow job with it's required config, pipeline and parameters files. Provide the URI for each file's location.
NOTE: providing empty values will trigger a default deployment which uses the nextflow files in this repository.
{
"config_uri": "",
"pipeline_uri": "",
"parameters_uri": "",
"auto_delete": true
}The http trigger currently accepts 1 (optional) argument.
- A boolean
whatif=trueargument can be provided to perform a mock deployment.
az_funcAppName="nxfutil"
curl -X POST "https://$az_funcAppName.azurewebsites.net/api/nxfutil?whatif=true" -H 'Content-Type: application/json' -d '{"config_uri":"", "pipeline_uri":"", "parameters_uri":""}'Will show the json payload required for the corresponding POST method.
Will return the progress of the nextflow job by dumping messages from the nextflow storage queue.
- Use
summary=trueto show only the latest event from a given nextflow job id. - Use
dequeue=trueto delete the messages as they are read.
Important: If no process is dequeueing the messages from the storage queue, only the oldest 32 messages can be summarized.
az_funcAppName="myFuncAppName"
curl -X POST "https://$az_funcAppName.azurewebsites.net/api/nxfutil_status" -H 'Content-Type: application/json' -d '{"summary": true "message_count": 1, "dequeue": false}'
# returns
# [
# {
# "event": "completed",
# "runId": "d58d0af4-1373-4dc2-b61d-e8ee06d18849",
# "runName": "maniac_easley",
# "utcTime": "2023-02-21T21:42:15Z",
# "metadata": {
# "parameters": {
# "dispatcher": "nextflow-20230220-8010e3cc-130f-4b86-838b-5572e69266d1"
# },
# "workflow": {
# "errorMessage": null
# }
# }
# }
# ]Will show the json payload required for the corresponding POST method.
Will delete the container instance specified in the request body.
The http trigger currently accepts 1 (optional) argument.
- A boolean
whatif=trueargument can be provided to perform a mock deployment.
az_funcAppName="myFuncAppName"
curl -X POST "https://$az_funcAppName.azurewebsites.net/api/nxfutil_status" -H 'Content-Type: application/json' -d '{"ci_name": "nextflow-20230220-8010e3cc-130f-4b86-838b-5572e69266d1"}'