Add and configure a catalog from GitHub or Azure Repos

Article
05/21/2024

This article explains how to add and configure a catalog for your Azure Deployment Environments dev center or project.

Catalogs help you provide a set of curated infrastructure-as-code(IaC) templates, known as environment definitions for your development teams to create environments. You can attach your own source control repository from GitHub or Azure DevOps as a catalog and specify the folder with your environment definitions. Deployment Environments scans the folder for environment definitions and makes them available for dev teams to create environments.

To further secure your templates, the catalog is encrypted; Azure Deployment Environments supports encryption at rest with platform-managed encryption keys, which Microsoft for Azure Services manages.

To learn how to host a repository in GitHub, see Get started with GitHub.
To learn how to host a Git repository in an Azure Repos project, see Azure Repos.

Microsoft offers a quick start catalog that you can add to the dev center or project, and a sample catalog that you can use as your repository. You also can use your own private repository, or you can fork and customize the environment definitions in the sample catalog.

In this article, you learn how to:

Configure project-level catalogs
Configure a managed identity
Add a catalog from Azure Repos or GitHub
Update a catalog
Delete a catalog
Troubleshoot catalog sync errors

Configure project-level catalogs

Attaching catalogs at the project level enables platform engineers to provide curated environment definitions that are specific to the development teams. Additionally, it empowers dev team leads assigned as Project Admins to manage the environment definitions made available to their teams.

Platform engineers have full control over the use of catalogs at the project level. The use of project level catalogs must be enabled at the dev center level before a catalog can be added to a project. Platform engineers can also configure which types of catalogs items, such as environment definitions, can be consumed at the project level.

By default, use of catalogs at the project level is disabled and none of the catalog item types are enabled. Environment definitions from a project-level catalog are synced and usable under two conditions. First, you must enable project-based catalogs at the corresponding dev center level. Second, you must enable the use of environment definitions for the project.

Add a catalog to a project

You must enable project-level catalogs at the dev center level before you can add a catalog to a project. You should also enable the use of environment definitions at the project level.

To enable the use of project-level catalogs at the dev center level:

In the Azure portal, navigate to your dev center.
In the left menu, under Settings, select Configuration.
In the Project level catalogs pane, select Enable catalogs per project, and then select Apply.

To enable the use of environment definitions in the project:

In the Azure portal, navigate to your project.
In the left menu, under Settings, select Catalogs.
On the Catalogs page, select Catalog item permissions.
In the Catalog item settings pane, select Azure deployment environment definitions to enable the use of environment definitions at the project level.

Now, you can add a catalog to the project.

For catalogs that use a managed identity or Personal Access Token (PAT) for authentication, you must assign a managed identity for the project. For catalogs that use a PAT, you must store the PAT in a key vault and grant the managed identity access to the key vault secret.

Configure a managed identity

Before you can attach a catalog to a dev center or project, you must configure a managed identity, also called a Managed Service Identity (MSI). You can attach either a system-assigned managed identity (system-assigned MSI) or a user-assigned managed identity (user-assigned MSI). You then assign roles to the managed identity to allow the dev center or project to create environment types in your subscription and read the Azure Repos project that contains the catalog repo.

If your dev center or project doesn't have an MSI attached, follow the steps in Configure a managed identity to create one and to assign roles for the managed identity.

To learn more about managed identities, see What are managed identities for Azure resources?

Add a catalog

You can add a catalog from an Azure Repos repository or a GitHub repository. You can choose to authenticate by assigning permissions to an MSI or by using a PAT, which you store in a key vault.

Select the tab for the type of repository and authentication you want to use.

To add a catalog, complete the following tasks:

Assign permissions in Azure Repos for the managed identity.
Add your repository as a catalog.

Assign permissions in Azure Repos for the managed identity

You must give the managed identity permissions to the repository in Azure Repos.

Sign in to your Azure DevOps organization.

Note

Your Azure DevOps organization must be in the same directory as the Azure subscription that contains your dev center or project.
Select Organization settings.
On the Overview page, select Users.
On the Users page, select Add users.

Complete Add new users by entering or selecting the following information, and then select Add:

Name	Value
Users or Service Principals	Enter the name of your dev center or project. When you use a system-assigned MSI, specify the name of the dev center or project, not the object ID of the managed account. When you use a user-assigned MSI, use the name of the managed account.
Access level	Select Basic.
Add to projects	Select the project that contains your repository.
Azure DevOps Groups	Select Project Readers.
Send email invites (to Users only)	Clear the checkbox.

Add your repository as a catalog

Azure Deployment Environments supports attaching Azure Repos repositories and GitHub repositories. You can store a set of curated IaC templates in a repository. Attaching the repository to a dev center or project as a catalog gives your development teams access to the templates and enables them to quickly create consistent environments.

The following steps let you attach an Azure Repos repository.

In the Azure portal, navigate to your dev center or project.
In the left menu under Environment configuration, select Catalogs, and then select Add.

In Add catalog, enter the following information, and then select Add:

Field	Value
Name	Enter a name for the catalog.
Catalog location	Select Azure DevOps.
Authentication type	Select Managed Identity.
Organization	Select your Azure DevOps organization.
Project	From the list of projects, select the project that stores the repo.
Repo	From the list of repos, select the repo you want to add.
Branch	Select the branch.
Folder path	Dev Box retrieves a list of folders in your branch. Select the folder that stores your IaC templates.

In Catalogs for the dev center or project, verify that your catalog appears. When the connection is successful, the Status reads Sync successful. Connecting to a catalog can take a few minutes the first time.

To add a catalog, complete the following tasks:

Get the clone URL for your Azure Repos repository.
Create a personal access token (PAT).
Store the PAT as a key vault secret in Azure Key Vault.
Add your repository as a catalog.

Get the clone URL for your Azure Repos repository

Go to the home page of your team collection (for example, https://contoso-web-team.visualstudio.com), and then select your project.
Get the clone URL of your Azure Repos Git repo.
Copy and save the URL.

Create a personal access token in Azure Repos

Go to the home page of your team collection (for example, https://contoso-web-team.visualstudio.com) and select your project.
Create a PAT.
Copy and save the generated token to use later.

Create a Key Vault

You need an Azure Key Vault to store the PAT used to grant Azure access to your repository. Key vaults can control access with either access policies or role-based access control (RBAC). If you have an existing key vault, you can use it, but you should check whether it uses access policies or RBAC assignments to control access. For help with configuring an access policy for a key vault, see Assign a Key Vault access policy.

Use the following steps to create an RBAC key vault:

Sign in to the Azure portal.
In the Search box, enter Key Vault.
From the results list, select Key Vault.
On the Key Vault page, select Create.

On the Create key vault tab, provide the following information:

Name	Value
Name	Enter a name for the key vault.
Subscription	Select the subscription in which you want to create the key vault.
Resource group	Either use an existing resource group or select Create new and enter a name for the resource group.
Location	Select the location or region where you want to create the key vault.

Leave the other options at their defaults.

On the Access policy tab, select Azure role-based access control, and then select Review + create.
On the Review + create tab, select Create.

If your organization's policies require you to keep your Key Vault private from the internet, you can set your Key Vault to allow trusted Microsoft services to bypass your firewall rule.

To learn how to allow trusted Microsoft services to bypass the firewall, see Configure Azure Key Vault networking settings.

Store the personal access token in the key vault

In the Key Vault, on the left menu, select Secrets.
On the Secrets page, select Generate/Import.
On the Create a secret page:
- In the Name box, enter a descriptive name for your secret.
- In the Secret value box, paste the PAT that you copied earlier.
- Select Create.

Get the secret identifier

Get the path to the secret you created in the key vault.

In the Azure portal, navigate to your key vault.
On the key vault page, from the left menu, select Secrets.
On the Secrets page, select the secret you created earlier.
On the versions page, select the CURRENT VERSION.
On the current version page, for the Secret identifier, select copy.

Add your repository as a catalog

In the Azure portal, go to your dev center or project.
Ensure that the identity attached to the dev center or project has access to the key vault secret where your personal access token is stored.
In the left menu under Environment configuration, select Catalogs, and then select Add.

In Add catalog, enter the following information, and then select Add:

Field	Value
Name	Enter a name for the catalog.
Catalog location	Select Azure DevOps.
Authentication type	Select Personal Access Token.
Organization	Select the organization that hosts the catalog repo.
Project	Select the project that stores the catalog repo.
Rep	Select the repo that stores the catalog.
Folder path	Select the folder that holds your IaC templates.
Secret identifier	Enter the secret identifier that contains your PAT for the repository. When you copy a secret identifier, the connection string includes a version identifier at the end, like in this example: `https://contoso-kv.vault.azure.net/secrets/GitHub-repo-pat/9376b432b72441a1b9e795695708ea5a`. Removing the version identifier ensures that Deployment Environments fetches the latest version of the secret from the key vault. If your PAT expires, only the key vault needs to be updated. Example secret identifier: `https://contoso-kv.vault.azure.net/secrets/GitHub-repo-pat`

In Catalogs, verify that your catalog appears. If the connection is successful, the Status is Connected.

To add a catalog, complete the following tasks:

Install and configure the Microsoft Dev Center app
Assign permissions in GitHub for the repos.
Add your repository as a catalog.

Install Microsoft Dev Center app

Sign in to the Azure portal.
Navigate to your dev center or project.
In the left menu under Environment configuration, select Catalogs, and then select Add.
In the Add catalog pane, enter, or select the following:

Field Value

Name Enter a name for the catalog.

Catalog source Select GitHub.

Authentication type Select GitHub app.
To install the Microsoft Dev Center app, select configure your repositories.
If you're prompted to authenticate to GitHub, authenticate.
On the Microsoft DevCenter page, select Configure.
Select the GitHub organization that contains the repository you want to add as a catalog. You must be an owner of the organization to install this app.
On the Install Microsoft DevCenter page, select Only select repositories, select the repository you want to add as a catalog, and then select Install.

You can select multiple repositories to add as catalogs. You must add each repository as a separate catalog, as described in Add your repository as a catalog.
On the Microsoft DevCenter by Microsoft would like permission to: page, review the permissions required, and then select Authorize Microsoft Dev Center.

Add your repository as a catalog

Switch back to the Azure portal.

In Add catalog, enter the following information, and then select Add:

Field	Value
Repo	Select the repository that you want to add as a catalog.
Branch	Select the branch.
Folder path	Select the folder that contains subfolders that hold your environment definitions.

In Catalogs, verify that your catalog appears. When the connection is successful, the Status reads Sync successful.

To add a catalog, complete the following tasks:

Get the clone URL of your GitHub repository.
Create a personal access token (PAT) in GitHub.
Store the PAT as a key vault secret in Azure Key Vault.
Add your repository as a catalog.

Get the clone URL of your GitHub repository

Go to the home page of the GitHub repository that contains the template definitions.
Get the GitHub repository clone URL.
Copy and save the URL.

Create a personal access token in GitHub

Azure Deployment Environments supports authenticating to GitHub repositories by using either classic tokens or fine-grained tokens. In this example, you create a fine-grained token.

Go to the home page of the GitHub repository that contains the template definitions.
In the upper-right corner of GitHub, select the profile image, and then select Settings.
On the left sidebar, select Developer settings > Personal access tokens > Fine-grained tokens.
Select Generate new token.

On the New fine-grained personal access token page, provide the following information:

Name	Value
Token name	Enter a descriptive name for the token.
Expiration	Select the token expiration period in days.
Description	Enter a description for the token.
Resource owner	Select the owner of the repository.
Repository access	Select Only select repositories.
Select repositories	Select the repository that contains the environment definitions.
Repository permissions	Expand Repository permissions, and for Contents, from the Access list, select Code read.

Select Generate token.
Copy and save the generated token to use later.

Important

When working with a private repository stored within a GitHub organization, you must ensure that the GitHub PAT is configured to give access to the correct organization and the repositories within it.

Classic tokens within the organization must be SSO authorized to the specific organization after they are created.
Fine-grained tokens must have the owner of the token set as the organization itself to be authorized.

Incorrectly configured PATs can result in a Repository not found error.

Create a Key Vault

You need an Azure Key Vault to store the PAT that is used to grant Azure access to your repository. Key vaults can control access with either access policies or role-based access control (RBAC). If you have an existing key vault, you can use it, but you should check whether it uses access policies or RBAC assignments to control access. For help with configuring an access policy for a key vault, see Assign a key vault access policy.

Use the following steps to create an RBAC key vault:

Sign in to the Azure portal.
In the Search box, enter key vault.
From the results list, select Key Vault.
On the Key Vault page, select Create.

On the Create key vault tab, provide the following information:

Name	Value
Name	Enter a name for the key vault.
Subscription	Select the subscription in which you want to create the key vault.
Resource group	Either use an existing resource group or select Create new and enter a name for the resource group.
Location	Select the location or region where you want to create the key vault.

Leave the other options at their defaults.

On the Access policy tab, select Azure role-based access control, and then select Review + create.
On the Review + create tab, select Create.

If your organization's policies require you to keep your Key Vault private from the internet, you can set your Key Vault to allow trusted Microsoft services to bypass your firewall rule.

To learn how to allow trusted Microsoft services to bypass the firewall, see Configure Azure Key Vault networking settings.

Store the personal access token in the key vault

In the Key Vault, on the left menu, select Secrets.
On the Secrets page, select Generate/Import.
On the Create a secret page:
- In the Name box, enter a descriptive name for your secret.
- In the Secret value box, paste your PAT.
- Select Create.

Get the secret identifier

Get the path to the secret you created in the key vault.

In the Azure portal, navigate to your key vault.
On the key vault page, from the left menu, select Secrets.
On the Secrets page, select the secret you created earlier.
On the versions page, select the CURRENT VERSION.
On the current version page, for the Secret identifier, select copy.

Add your repository as a catalog

In the Azure portal, go to your dev center or project.
Ensure that the managed identity attached to the dev center or project has access to the key vault secret where your personal access token is stored.
In the left menu under Environment configuration, select Catalogs, and then select Add.

In Add catalog, enter the following information, and then select Add.

Field	Value
Name	Enter a name for the catalog.
Catalog location	Select GitHub.
Repo	Enter or paste the clone URL for either your GitHub repository or your Azure Repos repository. Sample catalog example: `https://github.com/Azure/deployment-environments.git`
Branch	Enter the repository branch to connect to. Sample catalog example: `main`
Folder path	Enter the folder path relative to the clone URI that contains subfolders that hold your environment definitions. The folder path is for the folder with subfolders containing environment definition environment files, not for the folder with the environment definition environment file itself. The following image shows the sample catalog folder structure. Sample catalog example: `/Environments` The folder path can begin with or without a forward slash (`/`).
Secret identifier	Enter the secret identifier that contains your PAT for the repository. When you copy a secret identifier, the connection string includes a version identifier at the end, like in this example: `https://contoso-kv.vault.azure.net/secrets/GitHub-repo-pat/9376b432b72441a1b9e795695708ea5a`. Removing the version identifier ensures that Deployment Environments fetch the latest version of the secret from the key vault. If your PAT expires, only the key vault needs to be updated. Example secret identifier: `https://contoso-kv.vault.azure.net/secrets/GitHub-repo-pat`

In Catalogs, verify that your catalog appears. When the connection is successful, the Status reads Sync successful.

View synced catalog items

Regardless of the type of repository you use, you can view the catalog items that are synced from the catalog.

On the left menu for your dev center or project, under Environment configuration, select Catalogs.
In the Catalogs pane, select the catalog name.
You see a list of successfully synced catalog items.

Update a catalog

If you update the definition or template contents in the attached repository, you can provide the latest set of environment definitions to your development teams by syncing the catalog. You can sync a catalog manually or automatically.

Manually sync a catalog

When you manually sync a catalog, Deployment Environments scans through the repository and makes the latest list of environment definitions available to all of the associated projects in the dev center.

On the left menu for your dev center, under Environment configuration, select Catalogs.
Select the specific catalog, and then from the command bar, select Sync.

Automatically sync a catalog

When you configure a catalog to sync automatically, Deployment Environments scans through the repository every 30 minutes and makes the latest list of environment definitions available to all of the associated projects in the dev center.

On the left menu for your dev center or project, under Environment configuration, select Catalogs.
Select the specific catalog, and then select edit.
In the Edit catalog pane, select Automatically sync this catalog, and then select Save.

Delete a catalog

You can delete a catalog to remove it from the Azure Deployment Environments dev center or project. Templates in a deleted catalog aren't available to development teams when they deploy new environments. Update the environment definition reference for any existing environments that were created by using the environment definitions in the deleted catalog. If the reference isn't updated and the environment is redeployed, the deployment fails.

To delete a catalog:

On the left menu for your dev center or project, under Environment configuration, select Catalogs.
Select the specific catalog, and then select Delete.
In the Delete catalog dialog, select Continue to delete the catalog.

Troubleshoot catalog sync errors

When you add or sync a catalog, you might encounter a sync error or warning. A sync error indicates that a catalog failed to sync successfully, A sync warning indicates that some or all of the catalog items have errors. You can view the sync status and errors in the Azure portal, or use the Azure CLI and REST API to troubleshoot and resolve the errors.

View catalog sync status

In the Azure portal, you can get more information about the catalog sync status and any warnings or errors by selecting the status link. The status link opens a pane that shows the sync status, the number of environment definitions that were added, and the number of environment definitions that were ignored or failed.

View catalog sync failures

On the left menu for your dev center or project, under Environment configuration, select Catalogs.
In the Status column, select the status link for the catalog that failed to sync.
You see a details pane that shows the changes in the last sync, the number of sync errors, and the type of errors.

View catalog sync warnings

On the left menu for your dev center or project, under Environment configuration, select Catalogs.
In the Status column, select the status link for the catalog that synced but reports a warning.
You see a details pane that shows the changes in the last sync, the number of item errors, and the type and source of each error.
You can view items that have synced successfully from a catalog that also reports sync errors. From the Catalogs pane, select the catalog name.
You see a list of successfully synced catalog items.

Troubleshoot catalog sync errors by using the Azure CLI

Use the Azure CLI or the REST API to GET the catalog. The GET response shows you the type of error:

Ignored environment definitions that were detected to be duplicates.
Invalid environment definitions that failed due to schema, reference, or validation errors.

Resolve ignored environment definition errors

An ignored environment definition error occurs if you add two or more environment definitions that have the same name. You can resolve this issue by renaming environment definitions so that each environment definition has a unique name within the catalog.

Resolve invalid environment definition errors

An invalid environment definition error might occur for various reasons:

Manifest schema errors. Ensure that your environment definition environment file matches the required schema.
Validation errors. Check the following items to resolve validation errors:
- Ensure that the environment file's engine type is correctly configured.
- Ensure that the environment definition name is between 3 and 63 characters.
- Ensure that the environment definition name includes only characters that are valid for a URL, which are alphanumeric characters and these symbols: ~ ! , . ' ; : = - _ + ( ) * & $ @
Reference errors. Ensure that the template path that the environment file references is a valid relative path to a file in the repository.

Share via

Add and configure a catalog from GitHub or Azure Repos