How to enable External Data Sources (EDS) Preview?

External Data Sources (EDS) is a capability in OSDU® that allows data from an OSDU® compliant external data source to be shared with an Azure Data Manager for Energy resource. EDS is designed to pull specified data (metadata) from OSDU-compliant data sources via scheduled jobs while leaving associated dataset files (LAS, SEG-Y, etc.) stored at the external source for retrieval on demand.

For more information about External Data Sources (EDS), see The OSDU Forum 2Q 2022 Newsletter - EDS.

Prerequisites

1. Create a new or use an existing key vault to store secrets managed by OSDU® secret service. To learn how to create a key vault with the Azure portal, see Quickstart: Create a key vault using the Azure portal.

   Important

   Your key vault must exist in the same tenant and subscription as your Azure Data Manager for Energy resource.

   Tip

   When you create the key vault, select Enable purge protection (enforce a mandatory retention period for deleted vaults and vault objects).

2. In the Access configuration tab, under Permission model select Vault access policy.

3. Give permission to Azure Data Manager for Energy Service Principal (SPN) on key vault (existing or new). How to give access to Azure Data Manager for Energy SPN - Assign an Azure Key Vault access policy (CLI) | Microsoft Learn.

   1. Under Access Policies, select Create to create an access policy:
      1. In the Permissions tab, select the permissions.
      2. Under the Principal selection pane, enter AzureEnergyRpaaSAppProd.
      3. Review + Create the access policy.

To enable EDS Preview, create an Azure support request

To enable External Data Sources Preview on your Azure Data Manager for Energy, create an Azure support ticket with the following information:

• Subscription ID
• Azure Data Manager for Energy developer tier resource name
• Data partition name (the data partition in which EDS needs to be enabled)
• Key Vault name (created in Prerequisites)

We notify you once EDS preview is enabled in your Azure Data Manager for Energy resource.

Known issues

• Below issues are specific to OSDU® M18 release:
  • EDS ingest DAG results in failures when the data supplier’s wrapper Search service is unavailable.
  • EDS Dataset service response provides an empty response when data supplier’s Dataset wrapper service is unavailable.
  • Secret service responds with 5xx HTTP response code instead of 4xx in some cases. For example,
    • When an application tries to recover a deleted secret, which isn't deleted.
    • When an application tries to get an invalid deleted secret.

Limitations

Some EDS capabilities like Naturalization, Reverse Naturalization, Reference data mapping are unavailable in the M18 OSDU® release (available in later releases), and hence unavailable in Azure Data Manager for Energy M18 release. These features are available once we upgrade to subsequent OSDU® milestone release.

FAQ

See External data sources FAQ.