Quickstart: Use images in your AI chats

In this chat session, you're instructing the assistant to aid in understanding images that you input.

1. To start, select your GPT-4 Turbo with Vision deployment from the dropdown.

2. In the Assistant setup pane, provide a System Message to guide the assistant. The default System Message is: "You are an AI assistant that helps people find information." You can tailor the System Message to the image or scenario that you're uploading.

   Note

   It is recommended to update the System Message to be specific to the task in order to avoid unhelpful responses from the model.

3. Save your changes, and when prompted to confirm updating the system message, select Continue.

4. In the Chat session pane, enter a text prompt like "Describe this image," and upload an image with the attachment button. You can use a different text prompt for your use case. Then select Send.

5. Observe the output provided. Consider asking follow-up questions related to the analysis of your image to learn more.

GPT-4 Turbo with Vision provides exclusive access to Azure AI Services tailored enhancements. When combined with Azure AI Vision, it enhances your chat experience by providing the chat model with more detailed information about visible text in the image and the locations of objects.

The Optical Character Recognition (OCR) integration allows the model to produce higher quality responses for dense text, transformed images, and number-heavy financial documents. It also covers a wider range of languages.

The object grounding integration brings a new layer to data analysis and user interaction, as the feature can visually distinguish and highlight important elements in the images it processes.

In this chat session, you try out the capabilities of the enhanced Vision model.

1. To start, select your GPT-4 Turbo with Vision deployment from the dropdown.

2. In the Configuration tab on the right side of the chat experience, turn on the option for Vision under the Enhancements section.

3. You're required to select a Computer Vision resource to try the enhanced Vision API. Select your resource, and Save.

4. In the Assistant setup pane, provide a System Message to guide the assistant. The default System Message is: "You are an AI assistant that helps people find information." You can tailor the System Message to the image or scenario that you're uploading.

   Note

   It is recommended to update the System Message to be specific to the task in order to avoid unhelpful responses from the model.

5. Save your changes, and when prompted to confirm updating the system message, select Continue.

6. In the Chat session pane, enter a text prompt like "Describe this image," and upload an image with the attachment button. You can use a different text prompt for your use case. Then select Send.

7. You should receive a response with more detailed information about visible text in the image and the locations of objects. Consider asking follow-up questions related to the analysis of your image to learn more.

GPT-4 Turbo with Vision provides exclusive access to Azure AI Services tailored enhancements. The video prompt integration uses Azure AI Vision video retrieval to sample a set of frames from a video and create a transcript of the speech in the video. It enables the AI model to give summaries and answers about video content.

1. To start, select your GPT-4 Turbo with Vision deployment from the dropdown.

2. In the Configuration tab on the right side of the chat experience, turn on the option for Vision under the Enhancements section.

3. You're required to select a Computer Vision resource to try the enhanced Vision API. Select your resource, and Save.

4. In the Assistant setup pane, provide a System Message to guide the assistant. The default System Message is: "You are an AI assistant that helps people find information." You can tailor the System Message to the video or scenario that you're uploading.

   Note

   It is recommended to update the System Message to be specific to the task in order to avoid unhelpful responses from the model.

5. Save your changes, and when prompted to confirm updating the system message, select Continue.

6. In the chat session pane, enter a question about the video like: "Describe this video in detail. Focus on brands, technology and people." You can use a different text prompt for your use case. Upload a video using the attachment button and then select Send.

   Note

   Currently the chat playground only supports videos that are less than 3 minutes long.

7. You should receive a response describing the video. Consider asking follow-up questions related to the analysis of your video to learn more.

1. Replace the contents of quickstart.py with the following code.

   # Packages required:
   import requests 
   import json 
   
   api_base = '<your_azure_openai_endpoint>' 
   deployment_name = '<your_deployment_name>'
   API_KEY = '<your_azure_openai_key>'
   
   base_url = f"{api_base}openai/deployments/{deployment_name}" 
   headers = {   
       "Content-Type": "application/json",   
       "api-key": API_KEY 
   } 
   
   # Prepare endpoint, headers, and request body 
   endpoint = f"{base_url}/chat/completions?api-version=2023-12-01-preview" 
   data = { 
       "messages": [ 
           { "role": "system", "content": "You are a helpful assistant." }, 
           { "role": "user", "content": [  
               { 
                   "type": "text", 
                   "text": "Describe this picture:" 
               },
               { 
                   "type": "image_url",
                   "image_url": {
                       "url": "<image URL>"
                   }
               }
           ] } 
       ], 
       "max_tokens": 2000 
   }   
   
   # Make the API call   
   response = requests.post(endpoint, headers=headers, data=json.dumps(data))   
   
   print(f"Status Code: {response.status_code}")   
   print(response.text)
   

2. Make the following changes:

   1. Enter your endpoint URL and key in the appropriate fields.

   2. Enter your GPT-4 Turbo with Vision deployment name in the appropriate field.

   3. Change the value of the "image" field to the URL of your image.

      Tip

      You can also use a base 64 encoded image data instead of a URL. For more information, see the GPT-4 Turbo with Vision how-to guide.

3. Run the application with the python command:

   python quickstart.py
   

GPT-4 Turbo with Vision provides exclusive access to Azure AI Services tailored enhancements. When combined with Azure AI Vision, it enhances your chat experience by providing the chat model with more detailed information about visible text in the image and the locations of objects.

The Optical Character Recognition (OCR) integration allows the model to produce higher quality responses for dense text, transformed images, and number-heavy financial documents. It also covers a wider range of languages.

The object grounding integration brings a new layer to data analysis and user interaction, as the feature can visually distinguish and highlight important elements in the images it processes.

1. Replace the contents of quickstart.py with the following code.

   # Packages required:
   import requests 
   import json 
   
   api_base = '<your_azure_openai_endpoint>' 
   deployment_name = '<your_deployment_name>'
   API_KEY = '<your_azure_openai_key>'
   
   base_url = f"{api_base}openai/deployments/{deployment_name}" 
   headers = {   
       "Content-Type": "application/json",   
       "api-key": API_KEY 
   } 
   
   # Prepare endpoint, headers, and request body 
   endpoint = f"{base_url}/extensions/chat/completions?api-version=2023-12-01-preview" 
   data = {
       "model": "gpt-4-vision-preview",
       "enhancements": {
           "ocr": {
             "enabled": True
           },
           "grounding": {
             "enabled": True
           }
       },
       "dataSources": [
       {
           "type": "AzureComputerVision",
           "parameters": {
               "endpoint": "<your_computer_vision_endpoint>",
               "key": "<your_computer_vision_key>"
           }
       }],
       "messages": [ 
           { "role": "system", "content": "You are a helpful assistant." }, 
           { "role": "user", 
           "content": [  
               { 
                   "type": "text", 
                   "text": "Describe this picture:" 
               },
               { 
                   "type": "image_url", 
                   "image_url": {
                       "url" : "<image URL>"
                   }
               }
           ]} 
       ], 
       "max_tokens": 2000 
   }   
   
   # Make the API call   
   response = requests.post(endpoint, headers=headers, data=json.dumps(data))   
   
   print(f"Status Code: {response.status_code}")   
   print(response.text)
   

2. Make the following changes:

   1. Enter your GPT-4 Turbo with Vision deployment name in the appropriate field.
   2. Enter your Computer Vision endpoint URL and key in the appropriate fields.
   3. Change the value of the "image" field to the URL of your image.

      Tip

      You can also use a base 64 encoded image data instead of a URL. For more information, see the GPT-4 Turbo with Vision how-to guide.

3. Run the application with the python command:

   python quickstart.py
   

Video prompt integration is outside the scope of this quickstart. See the GPT-4 Turbo with Vision how-to guide for detailed instructions on setting up video prompts in chat completions programmatically.

setx AZURE_OPENAI_API_KEY "REPLACE_WITH_YOUR_KEY_VALUE_HERE" 
setx AZURE_OPENAI_ENDPOINT "REPLACE_WITH_YOUR_ENDPOINT_HERE" 

[System.Environment]::SetEnvironmentVariable('AZURE_OPENAI_API_KEY', 'REPLACE_WITH_YOUR_KEY_VALUE_HERE', 'User')
[System.Environment]::SetEnvironmentVariable('AZURE_OPENAI_ENDPOINT', 'REPLACE_WITH_YOUR_ENDPOINT_HERE', 'User')

export AZURE_OPENAI_API_KEY="REPLACE_WITH_YOUR_KEY_VALUE_HERE"
export AZURE_OPENAI_ENDPOINT="REPLACE_WITH_YOUR_ENDPOINT_HERE"

1. Replace the contents of quickstart.py with the following code.

   from openai import AzureOpenAI
   
   api_base = os.getenv("AZURE_OPENAI_ENDPOINT")
   api_key= os.getenv("AZURE_OPENAI_API_KEY")
   deployment_name = '<your_deployment_name>'
   api_version = '2023-12-01-preview' # this might change in the future
   
   client = AzureOpenAI(
       api_key=api_key,  
       api_version=api_version,
       base_url=f"{api_base}/openai/deployments/{deployment_name}"
   )
   
   response = client.chat.completions.create(
       model=deployment_name,
       messages=[
           { "role": "system", "content": "You are a helpful assistant." },
           { "role": "user", "content": [  
               { 
                   "type": "text", 
                   "text": "Describe this picture:" 
               },
               { 
                   "type": "image_url",
                   "image_url": {
                       "url": "<image URL>"
                   }
               }
           ] } 
       ],
       max_tokens=2000 
   )
   
   print(response)
   

2. Make the following changes:

   1. Enter the name of your GPT-4 Turbo with Vision deployment in the appropriate field.
   2. Change the value of the "url" field to the URL of your image.

      Tip

      You can also use a base 64 encoded image data instead of a URL. For more information, see the GPT-4 Turbo with Vision how-to guide.

3. Run the application with the python command:

   python quickstart.py
   

GPT-4 Turbo with Vision provides exclusive access to Azure AI Services tailored enhancements. When combined with Azure AI Vision, it enhances your chat experience by providing the chat model with more detailed information about visible text in the image and the locations of objects.

The Optical Character Recognition (OCR) integration allows the model to produce higher quality responses for dense text, transformed images, and number-heavy financial documents. It also covers a wider range of languages.

The object grounding integration brings a new layer to data analysis and user interaction, as the feature can visually distinguish and highlight important elements in the images it processes.

1. Replace the contents of quickstart.py with the following code.

   from openai import AzureOpenAI
   
   api_base = os.getenv("AZURE_OPENAI_ENDPOINT")
   api_key= os.getenv("AZURE_OPENAI_API_KEY")
   deployment_name = '<your_deployment_name>'
   api_version = '2023-12-01-preview' # this might change in the future
   
   client = AzureOpenAI(
       api_key=api_key,  
       api_version=api_version,
       base_url=f"{api_base}/openai/deployments/{deployment_name}/extensions",
   )
   
   response = client.chat.completions.create(
       model=deployment_name,
       messages=[
           { "role": "system", "content": "You are a helpful assistant." },
           { "role": "user", "content": [  
               { 
                   "type": "text", 
                   "text": "Describe this picture:" 
               },
               { 
                   "type": "image_url",
                   "image_url": {
                       "url": "<image URL>"
                   }
               }
           ] } 
       ],
       extra_body={
           "dataSources": [
               {
                   "type": "AzureComputerVision",
                   "parameters": {
                       "endpoint": "<your_computer_vision_endpoint>",
                       "key": "<your_computer_vision_key>"
                   }
               }],
           "enhancements": {
               "ocr": {
                   "enabled": True
               },
               "grounding": {
                   "enabled": True
               }
           }
       },
       max_tokens=2000
   )
   
   print(response)
   

2. Make the following changes:

   1. Enter your GPT-4 Turbo with Vision deployment name in the appropriate field.
   2. Enter your Computer Vision endpoint URL and key in the appropriate fields.
   3. Change the value of the "url" field to the URL of your image.

      Tip

      You can also use a base 64 encoded image data instead of a URL. For more information, see the GPT-4 Turbo with Vision how-to guide.

3. Run the application with the python command:

   python quickstart.py
   

Video prompt integration is outside the scope of this quickstart. See the GPT-4 Turbo with Vision how-to guide for detailed instructions on setting up video prompts in chat completions programmatically.