Tutorial: Build and deploy a question and answer copilot with prompt flow in Azure AI Studio

In this Azure AI Studio tutorial, you use generative AI and prompt flow to build, configure, and deploy a copilot for your retail company called Contoso. Your retail company specializes in outdoor camping gear and clothing.

The copilot should answer questions about your products and services. It should also answer questions about your customers. For example, the copilot can answer questions such as "How much do the TrailWalker hiking shoes cost?" and "How many TrailWalker hiking shoes did Daniel Wilson buy?".

The steps in this tutorial are:

1. Add your data to the chat playground.
2. Create a prompt flow from the playground.
3. Customize prompt flow with multiple data sources.
4. Evaluate the flow using a question and answer evaluation dataset.
5. Deploy the flow for consumption.

Prerequisites

• An Azure subscription - Create one for free.

• Access granted to Azure OpenAI in the desired Azure subscription.

  Currently, access to this service is granted only by application. You can apply for access to Azure OpenAI by completing the form at https://aka.ms/oai/access. Open an issue on this repo to contact us if you have an issue.

• An AI Studio hub, project, and deployed Azure OpenAI chat model. Complete the AI Studio playground quickstart to create these resources if you haven't already.

• An Azure AI Search service connection to index the sample product and customer data.

• You need a local copy of product and customer data. The Azure-Samples/aistudio-python-quickstart-sample repository on GitHub contains sample retail customer and product information that's relevant for this tutorial scenario. Clone the repository or copy the files from 1-customer-info and 3-product-info.

Add your data and try the chat model again

In the AI Studio playground quickstart (that's a prerequisite for this tutorial), you can observe how your model responds without your data. Now you add your data to the model to help it answer questions about your products.

To complete this section, you need a local copy of product data. The Azure-Samples/aistudio-python-quickstart-sample repository on GitHub contains sample retail customer and product information that's relevant for this tutorial scenario. Clone the repository or copy the files from 3-product-info.

Follow these steps to add your data in the chat playground to help the assistant answer questions about your products. You're not changing the deployed model itself. Your data is stored separately and securely in your Azure subscription.

1. Go to your project in Azure AI Studio.

2. Select Playgrounds > Chat from the left pane.

3. Select your deployed chat model from the Deployment dropdown.

   

4. On the left side of the chat playground, select Add your data > + Add a new data source.

   

5. In the Data source dropdown, select Upload files.

   

6. Select Upload > Upload files to browse your local files.

7. Select the files you want to upload. Select the product information files (3-product-info) that you downloaded or created earlier. Add all of the files now. You won't be able to add more files later in the same playground session.

8. Select Upload to upload the file to your Azure Blob storage account. Then select Next.

   

9. Select an Azure AI Search service. In this example we select Connect other Azure AI Search resource from the Select Azure AI Search service dropdown. If you don't have a search resource, you can create one by selecting Create a new Azure AI Search resource. Then return to this step to connect and select it.

   

10. Browse for your Azure AI Search service, and select Add connection.

    

11. For the Index name, enter product-info and select Next.

12. On the Search settings page under Vector settings, deselect the Add vector search to this search resource checkbox. This setting helps determine how the model responds to requests. Then select Next.

    Note

    If you add vector search, more options would be available here for an additional cost.

13. Review your settings and select Create.

14. In the playground, you can see that your data ingestion is in progress. This process might take several minutes. Before proceeding, wait until you see the data source and index name in place of the status.

    

15. Enter a name for the playground configuration and select Save > Save configuration. All configuration items are saved by default. Items include deployment, system message, safety message, parameters, added data, examples, and variables. Saving a configuration with the same name will save over the previous version.

    

16. You can now chat with the model asking the same question as before ("How much are the TrailWalker hiking shoes"), and this time it uses information from your data to construct the response. You can expand the references button to see the data that was used.

Create a prompt flow from the playground

Now you might ask "How can I further customize this copilot?" You might want to add multiple data sources, compare different prompts or the performance of multiple models. A prompt flow serves as an executable workflow that streamlines the development of your LLM-based AI application. It provides a comprehensive framework for managing data flow and processing within your application. You use prompt flow to optimize the messages that are sent to the copilot's chat model.

In this section, you learn how to transition to prompt flow from the playground. You export the playground chat environment including connections to the data that you added. Later in this tutorial, you evaluate the flow and then deploy the flow for consumption.

You can create a prompt flow from the playground by following these steps:

1. Go to your project in AI Studio.

2. Select Playgrounds > Chat from the left pane.

3. Since we're using our own data, you need to select Add your data. You should already have an index named product-info that you created previously in the chat playground. Select it from the Select available project index dropdown. Otherwise, first create an index with your product data and then return to this step.

4. Select Prompt flow from the menu above the chat session pane.

5. Enter a folder name for your prompt flow. Then select Open. AI Studio exports the playground chat environment to prompt flow. The export includes the connections to the data that you added.

   

Within a flow, nodes take center stage, representing specific tools with unique capabilities. These nodes handle data processing, task execution, and algorithmic operations, with inputs and outputs. By connecting nodes, you establish a seamless chain of operations that guides the flow of data through your application. For more information, see prompt flow tools.

To facilitate node configuration and fine-tuning, a visual representation of the workflow structure is provided through a DAG (Directed Acyclic Graph) graph. This graph showcases the connectivity and dependencies between nodes, providing a clear overview of the entire workflow. The nodes in the graph shown here are representative of the playground chat experience that you exported to prompt flow.

In prompt flow, you should also see:

• Save button: You can save your prompt flow at any time by selecting Save from the top menu. Be sure to save your prompt flow periodically as you make changes in this tutorial.
• Start compute session button: You need to start a compute session to run your prompt flow. You can start the session later in the tutorial. You incur costs for compute instances while they are running. For more information, see how to create a compute session.

You can return to the prompt flow anytime by selecting Prompt flow from Tools in the left menu. Then select the prompt flow folder that you created previously.

Customize prompt flow with multiple data sources

previously in the AI Studio chat playground, you added your data to create one search index that contained product data for the Contoso copilot. So far, users can only inquire about products with questions such as "How much do the TrailWalker hiking shoes cost?". But they can't get answers to questions such as "How many TrailWalker hiking shoes did Daniel Wilson buy?" To enable this scenario, we add another index with customer information to the flow.

Create the customer info index

To proceed, you need a local copy of example customer information. For more information and links to example data, see the prerequisites.

Follow these instructions on how to create a new index. You'll return to your prompt flow later in this tutorial to add the customer info to the flow. You can open a new tab in your browser to follow these instructions and then return to your prompt flow.

1. Go to your project in AI Studio.

2. Select Index from the left menu. Notice that you already have an index named product-info that you created previously in the chat playground.

   

3. Select + New index. You're taken to the Create an index wizard.

4. On the Source data page, select Upload files from the Data source dropdown. Then select Upload > Upload files to browse your local files.

5. Select the customer info files that you downloaded or created previously. See the prerequisites. Then select Next.

   

6. Select the same Azure AI Search service connection (contosooutdooraisearch) that you used for your product info index. Then select Next.

7. Enter customer-info for the index name.

   

8. Select a virtual machine to run indexing jobs. The default option is Auto select. Then select Next.

9. On the Search settings page under Vector settings, deselect the Add vector search to this search resource checkbox. This setting helps determine how the model responds to requests. Then select Next.

   Note

   If you add vector search, more options would be available here for an additional cost.

10. Review the details you entered, and select Create.

    

    Note

    You use the customer-info index and the contosooutdooraisearch connection to your Azure AI Search service in prompt flow later in this tutorial. If the names you enter differ from what's specified here, make sure to use the names you entered in the rest of the tutorial.

11. You're taken to the index details page where you can see the status of your index creation.

    

For more information on how to create an index, see Create an index.

Create a compute session that's needed for prompt flow

After you're done creating your index, return to your prompt flow and start the compute session. Prompt flow requires a compute session to run.

1. Go to your project.
2. Select Prompt flow from Tools in the left menu. Then select the prompt flow folder that you created previously.
3. Select Start compute session from the top menu.

To create a compute instance and a compute session, you can also follow the steps in how to create a compute session.

To complete the rest of the tutorial, make sure that your compute session is running.

Add customer information to the flow

After you're done creating your index, return to your prompt flow and follow these steps to add the customer info to the flow:

1. Make sure you have a compute session running. If you don't have one, see create a compute session in the previous section.

2. Select + More tools from the top menu and then select Index Lookup from the list of tools.

   

3. Name the new node queryCustomerIndex and select Add.

4. Select the mlindex_content textbox in the queryCustomerIndex node.

   

   The Generate dialog opens. You use this dialog to configure the queryCustomerIndex node to connect to your customer-info index.

5. For the index_type value, select Azure AI Search.

6. Select or enter the following values:

   Name	Value
   acs_index_connection	The name of your Azure AI Search service connection (such as contosooutdooraisearch)
   acs_index_name	customer-info
   acs_content_field	content
   acs_metadata_field	meta_json_string
   semantic_configuration	azuremldefault
   embedding_type	None

7. Select Save to save your settings.

8. Select or enter the following values for the queryCustomerIndex node:

   Name	Value
   queries	${extractSearchIntent.output}
   query_type	Keyword
   topK	5

   You can see the queryCustomerIndex node is connected to the extractSearchIntent node in the graph.

   

9. Select Save from the top menu to save your changes. Remember to save your prompt flow periodically as you make changes.

Connect the customer info to the flow

In the next section, you aggregate the product and customer info to output it in a format that the large language model can use. But first, you need to connect the customer info to the flow.

1. Select the ellipses icon next to + More tools and then select Raw file mode to switch to raw file mode. This mode allows you to copy and paste nodes in the graph.

   

2. Replace all instances of querySearchResource with queryProductIndex in the graph. We're renaming the node to better reflect that it retrieves product info and contrasts with the queryCustomerIndex node that you added to the flow.

3. Rename and replace all instances of chunkDocuments with chunkProductDocuments in the graph.

4. Rename and replace all instances of selectChunks with selectProductChunks in the graph.

5. Copy and paste the chunkProductDocuments and selectProductChunks nodes to create similar nodes for the customer info. Rename the new nodes chunkCustomerDocuments and selectCustomerChunks respectively.

6. Within the chunkCustomerDocuments node, replace the ${queryProductIndex.output} input with ${queryCustomerIndex.output}.

7. Within the selectCustomerChunks node, replace the ${chunkProductDocuments.output} input with ${chunkCustomerDocuments.output}.

8. Select Save from the top menu to save your changes.

   

   By now, the flow.dag.yaml file should include nodes (among others) that look similar to the following example:

   - name: chunkProductDocuments
     type: python
     source:
       type: code
       path: chunkProductDocuments.py
     inputs:
       data_source: Azure AI Search
       max_tokens: 1050
       queries: ${extractSearchIntent.output}
       query_type: Keyword
       results: ${queryProductIndex.output}
       top_k: 5
     use_variants: false
   - name: selectProductChunks
     type: python
     source:
       type: code
       path: filterChunks.py
     inputs:
       min_score: 0.3
       results: ${chunkProductDocuments.output}
       top_k: 5
     use_variants: false
   - name: chunkCustomerDocuments
     type: python
     source:
       type: code
       path: chunkCustomerDocuments.py
     inputs:
       data_source: Azure AI Search
       max_tokens: 1050
       queries: ${extractSearchIntent.output}
       query_type: Keyword
       results: ${queryCustomerIndex.output}
       top_k: 5
     use_variants: false
   - name: selectCustomerChunks
     type: python
     source:
       type: code
       path: filterChunks.py
     inputs:
       min_score: 0.3
       results: ${chunkCustomerDocuments.output}
       top_k: 5
     use_variants: false
   

Aggregate product and customer info

At this point, the prompt flow only uses the product information.

• extractSearchIntent extracts the search intent from the user's question.
• queryProductIndex retrieves the product info from the product-info index.
• The LLM tool (for large language models) receives a formatted reply via the chunkProductDocuments > selectProductChunks > formatGeneratedReplyInputs nodes.

You need to connect and aggregate the product and customer info to output it in a format that the LLM tool can use. Follow these steps to aggregate the product and customer info:

1. Select Python from the list of tools.

2. Name the tool aggregateChunks and select Add.

3. Copy and paste the following Python code to replace all contents in the aggregateChunks code block.

   from promptflow import tool
   from typing import List
   
   @tool
   def aggregate_chunks(input1: List, input2: List) -> str:
       interleaved_list = []
       for i in range(max(len(input1), len(input2))):
           if i < len(input1):
               interleaved_list.append(input1[i])
           if i < len(input2):
               interleaved_list.append(input2[i])
       return interleaved_list
   

4. Select the Validate and parse input button to validate the inputs for the aggregateChunks node. If the inputs are valid, prompt flow parses the inputs and creates the necessary variables for you to use in your code.

   

5. Edit the aggregateChunks node to connect the product and customer info. Set the inputs to the following values:

   Name	Type	Value
   input1	list	${selectProductChunks.output}
   input2	list	${selectCustomerChunks.output}

   

6. Select the shouldGenerateReply node from the graph. Select or enter ${aggregateChunks.output} for the chunks input.

7. Select the formatGenerateReplyInputs node from the graph. Select or enter ${aggregateChunks.output} for the chunks input.

8. Select the outputs node from the graph. Select or enter ${aggregateChunks.output} for the chunks input.

9. Select Save from the top menu to save your changes. Remember to save your prompt flow periodically as you make changes.

Now you can see the aggregateChunks node in the graph. The node connects the product and customer info to output it in a format that the LLM tool can use.

Chat in prompt flow with product and customer info

By now you have both the product and customer info in prompt flow. You can chat with the model in prompt flow and get answers to questions such as "How many TrailWalker hiking shoes did Daniel Wilson buy?" Before proceeding to a more formal evaluation, you can optionally chat with the model to see how it responds to your questions.

1. Continue from the previous section with the outputs node selected. Make sure that the reply output has the Chat output radio button selected. Otherwise, the full set of documents are returned in response to the question in chat.

2. Select Chat from the top menu in prompt flow to try chat.

3. Enter "How many TrailWalker hiking shoes did Daniel Wilson buy?" and then select the right arrow icon to send.

   Note

   It might take a few seconds for the model to respond. You can expect the response time to be faster when you use a deployed flow.

4. The response is what you expect. The model uses the customer info to answer the question.

   

Evaluate the flow using a question and answer evaluation dataset

In AI Studio, you want to evaluate the flow before you deploy the flow for consumption.

In this section, you use the built-in evaluation to evaluate your flow with a question and answer evaluation dataset. The built-in evaluation uses AI-assisted metrics to evaluate your flow: groundedness, relevance, and retrieval score. For more information, see built-in evaluation metrics.

Create an evaluation

You need a question and answer evaluation dataset that contains questions and answers that are relevant to your scenario. Create a new file locally named qa-evaluation.jsonl. Copy and paste the following questions and answers ("truth") into the file.

{"question": "What color is the CozyNights Sleeping Bag?", "truth": "Red", "chat_history": [], }
{"question": "When did Daniel Wilson order the BaseCamp Folding Table?", "truth": "May 7th, 2023", "chat_history": [] }
{"question": "How much does TrailWalker Hiking Shoes cost? ", "truth": "$110", "chat_history": [] }
{"question": "What kind of tent did Sarah Lee buy?", "truth": "SkyView 2 person tent", "chat_history": [] }
{"question": "What is Melissa Davis's phone number?", "truth": "555-333-4444", "chat_history": [] }
{"question": "What is the proper care for trailwalker hiking shoes?", "truth": "After each use, remove any dirt or debris by brushing or wiping the shoes with a damp cloth.", "chat_history": [] }
{"question": "Does TrailMaster Tent come with a warranty?", "truth": "2 years", "chat_history": [] }
{"question": "How much did David Kim spend on the TrailLite Daypack?", "truth": "$240", "chat_history": [] }
{"question": "What items did Amanda Perez purchase?", "truth": "TrailMaster X4 Tent, TrekReady Hiking Boots (quantity 3), CozyNights Sleeping Bag, TrailBlaze Hiking Pants, RainGuard Hiking Jacket, and CompactCook Camping Stove", "chat_history": [] }
{"question": "What is the Brand for TrekReady Hiking Boots", "truth": "TrekReady", "chat_history": [] }
{"question": "How many items did Karen Williams buy?", "truth": "three items of the Summit Breeze Jacket", "chat_history": [] }
{"question": "France is in Europe", "truth": "Sorry, I can only truth questions related to outdoor/camping gear and equipment", "chat_history": [] }

Now that you have your evaluation dataset, you can evaluate your flow by following these steps:

1. Select Evaluate > Built-in evaluation from the top menu in prompt flow.

   

   You're taken to the Create a new evaluation wizard.

2. Enter a name for your evaluation and select a compute session.

3. Select Question and answer without context from the scenario options.

4. Select the flow to evaluate. In this example, select Contoso outdoor flow or whatever you named your flow. Then select Next.

   

5. Select Add your dataset on the Configure test data page.

   

6. Select Upload file, browse files, and select the qa-evaluation.jsonl file that you created previously.

7. After the file is uploaded, you need to configure your data columns to match the required inputs for prompt flow to execute a batch run that generate output for evaluation. Enter or select the following values for each data set mapping for prompt flow.

   

   Name	Description	Type	Data source
   chat_history	The chat history	list	${data.chat_history}
   query	The query	string	${data.question}

8. Select Next.

9. Select the metrics you want to use to evaluate your flow. In this example, select Coherence, Fluency, GPT similarity, and F1 score.

10. Select a connection and model to use for evaluation. In this example, select gpt-35-turbo-16k. Then select Next.

    

    Note

    Evaluation with AI-assisted metrics needs to call another GPT model to do the calculation. For best performance, use a model that supports at least 16k tokens such as gpt-4-32k or gpt-35-turbo-16k model. If you didn't previously deploy such a model, you can deploy another model by following the steps in the AI Studio chat playground quickstart. Then return to this step and select the model you deployed.

11. You need to configure your data columns to match the required inputs to generate evaluation metrics. Enter the following values to map the dataset to the evaluation properties:

    Name	Description	Type	Data source
    question	A query seeking specific information.	string	${data.question}
    answer	The response to question generated by the model as answer.	string	${run.outputs.reply}
    documents	String with context from retrieved documents.	string	${run.outputs.documents}

12. Select Next.

13. Review the evaluation details and then select Submit. You're taken to the Metric evaluations page.

View the evaluation status and results

Now you can view the evaluation status and results by following these steps:

1. After you create an evaluation, if you aren't there already go to the Evaluation. On the Metric evaluations page, you can see the evaluation status and the metrics that you selected. You might need to select Refresh after a couple of minutes to see the Completed status.

   

2. Stop your compute session in prompt flow. Go to your prompt flow and select Compute session running > Stop compute session from the top menu.

   

   Tip

   Once the evaluation is in Completed status, you don't need a compute session to complete the rest of this tutorial. You can stop your compute instance to avoid incurring unnecessary Azure costs. For more information, see how to start and stop compute.

3. Select the name of the evaluation (such as evaluation_evaluate_from_flow_variant_0) to see the evaluation metrics.

   

For more information, see view evaluation results.

Deploy the flow

Now that you built a flow and completed a metrics-based evaluation, it's time to create your online endpoint for real-time inference. That means you can use the deployed flow to answer questions in real time.

Follow these steps to deploy a prompt flow as an online endpoint from AI Studio.

1. Have a prompt flow ready for deployment. If you don't have one, see the previous sections or how to build a prompt flow.

2. Optional: Select Chat to test if the flow is working correctly. Testing your flow before deployment is recommended best practice.

3. Select Deploy on the flow editor.

   

4. Provide the requested information on the Basic Settings page in the deployment wizard. Select Next to proceed to the advanced settings pages.

   

5. On the Advanced settings - Endpoint page, leave the default settings and select Next.

6. On the Advanced settings - Deployment page, leave the default settings and select Next.

7. On the Advanced settings - Outputs & connections page, make sure all outputs are selected under Included in endpoint response.

   

8. Select Review + Create to review the settings and create the deployment.

9. Select Create to deploy the prompt flow.

   

For more information, see how to deploy a flow.

Use the deployed flow

Your copilot application can use the deployed prompt flow to answer questions in real time. You can use the REST endpoint or the SDK to use the deployed flow.

1. To view the status of your deployment in AI Studio, select Deployments from the left navigation.

   

   Once the deployment is created successfully, you can select the deployment to view the details.

   Note

   If you see a message that says "Currently this endpoint has no deployments" or the State is still Updating, you might need to select Refresh after a couple of minutes to see the deployment.

2. Optionally, the details page is where you can change the authentication type or enable monitoring.

   

3. Select the Consume tab. You can see code samples and the REST endpoint for your copilot application to use the deployed flow.

   

Clean up resources

To avoid incurring unnecessary Azure costs, you should delete the resources you created in this tutorial if they're no longer needed. To manage resources, you can use the Azure portal.

You can also stop or delete your compute instance in AI Studio as needed.

Next steps

• Learn more about prompt flow.
• Deploy an enterprise chat web app.