This repository contains sample only code to demonstrate how secrets can be marshaled from Key Vault into a Container Instance running Nextflow for the purpose of dispatching Nextflow pipelines on Azure Batch.
Deploying the Azure resources supporting this sample is left to you.
It is assumed that this infrastructure would be deployed using GitHub workflows or DevOps pipelines; samples of each are provided.
It is assumed that regardless of deployment method, GitHub is used for hosting the forked repository.
-
Fork this repository on GitHub.
-
Create a Service Principal for connecting to Azure.
az_subId="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" az ad sp create-for-rbac --name "DeploymentConnection" --role owner --scopes /subscriptions/$az_subId--sdk-auth
-
(optional) Access can be granted at the resource group scope instead of the subscription scope if the resource group is created before running the deployment workflow/pipeline.
az_subId="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" az_rgName="myRgName" az ad sp create-for-rbac --name "GitHubConnection" --role owner --scopes /subscriptions/$az_subId/resourceGroups/$az_rgName --sdk-auth
Important: The principal will need Owner permission on the target resource group scope as a minimum.
The following uses the provided GitHub workflows or DevOps pipelines to build and deploy the sample. See Annex below for manual set up instructions.
-
Create a new
AZURE_CREDENTIALSGitHub Secret using the Service Principal created earlier; details in Azure Docs. -
Run the workflow called
GitHub Workflows.
-
Create a new
Azure Resource Manager - Service principal (manual)Service Connection in DevOps using the Service Principal created earlier; details in Azure Docs. -
Creating a new pipeline and link to the
./.devops/pipelines/cicd.ymlfile that's on GitHub by selectingGitHub (YAML)and thenExisting Azure Pipelines YAML file. -
Run the newly created pipeline.
To validate set up was successful, trigger the Azure Function using its webhook.
az_funcAppName="myFuncAppName"
curl --get "https://$az_funcAppName.azurewebsites.net/api/nxfutil"
# returns
Successfully created nxfutil container instance nextflow-xxxxxx.When the Function App is triggered it will create a new nxfutil Container Instance. See Annex for nxfutil details.
The http trigger currently accepts 3 (optional) arguments.
- A URI to a Nextflow
configfile. - A URI to a Nextflow
pipelinefile. - A URI to a Nextflow
parametersfile.
az_funcAppName="nxfutil-py"
nxf_configUri="https://raw.githubusercontent.com/axgonz/azure-nextflow/main/nextflow/pipelines/nextflow.config"
nxf_pipelineUri="https://raw.githubusercontent.com/axgonz/azure-nextflow/main/nextflow/pipelines/helloWorld/pipeline.nf"
nxf_parametersUri="https://raw.githubusercontent.com/axgonz/azure-nextflow/main/nextflow/pipelines/helloWorld/parameters.json"
curl --get "https://$az_funcAppName.azurewebsites.net/api/nxfutil?config=$nxf_configUri&pipeline=$nxf_pipelineUri¶meters=$nxf_parametersUri"