Edit

Share via

Content filtering

  • Article

Important

The content filtering system isn't applied to prompts and completions processed by the Whisper model in Azure OpenAI Service. Learn more about the Whisper model in Azure OpenAI.

Azure OpenAI Service includes a content filtering system that works alongside core models, including DALL-E image generation models. This system works by running both the prompt and completion through an ensemble of classification models aimed at detecting and preventing the output of harmful content. The content filtering system detects and takes action on specific categories of potentially harmful content in both input prompts and output completions. Variations in API configurations and application design might affect completions and thus filtering behavior.

The content filtering models for the hate, sexual, violence, and self-harm categories have been specifically trained and tested on the following languages: English, German, Japanese, Spanish, French, Italian, Portuguese, and Chinese. However, the service can work in many other languages, but the quality might vary. In all cases, you should do your own testing to ensure that it works for your application.

In addition to the content filtering system, the Azure OpenAI Service performs monitoring to detect content and/or behaviors that suggest use of the service in a manner that might violate applicable product terms. For more information about understanding and mitigating risks associated with your application, see the Transparency Note for Azure OpenAI. For more information about how data is processed for content filtering and abuse monitoring, see Data, privacy, and security for Azure OpenAI Service.

The following sections provide information about the content filtering categories, the filtering severity levels and their configurability, and API scenarios to be considered in application design and implementation.

Content filtering categories

The content filtering system integrated in the Azure OpenAI Service contains:

  • Neural multi-class classification models aimed at detecting and filtering harmful content; the models cover four categories (hate, sexual, violence, and self-harm) across four severity levels (safe, low, medium, and high). Content detected at the 'safe' severity level is labeled in annotations but isn't subject to filtering and isn't configurable.
  • Other optional classification models aimed at detecting jailbreak risk and known content for text and code; these models are binary classifiers that flag whether user or model behavior qualifies as a jailbreak attack or match to known text or source code. The use of these models is optional, but use of protected material code model may be required for Customer Copyright Commitment coverage.

Risk categories

Category Description
Hate and fairness Hate and fairness-related harms refer to any content that attacks or uses pejorative or discriminatory language with reference to a person or Identity groups on the basis of certain differentiating attributes of these groups including but not limited to race, ethnicity, nationality, gender identity groups and expression, sexual orientation, religion, immigration status, ability status, personal appearance, and body size. 

 Fairness is concerned with ensuring that AI systems treat all groups of people equitably without contributing to existing societal inequities. Similar to hate speech, fairness-related harms hinge upon disparate treatment of Identity groups.  
Sexual Sexual describes language related to anatomical organs and genitals, romantic relationships, acts portrayed in erotic or affectionate terms, pregnancy, physical sexual acts, including those portrayed as an assault or a forced sexual violent act against one’s will, prostitution, pornography, and abuse.  
Violence Violence describes language related to physical actions intended to hurt, injure, damage, or kill someone or something; describes weapons, guns and related entities, such as manufactures, associations, legislation, etc.  
Self-Harm Self-harm describes language related to physical actions intended to purposely hurt, injure, damage one’s body or kill oneself.
Protected Material for Text* Protected material text describes known text content (for example, song lyrics, articles, recipes, and selected web content) that can be outputted by large language models.
Protected Material for Code Protected material code describes source code that matches a set of source code from public repositories, which can be outputted by large language models without proper citation of source repositories.

* If you are an owner of text material and want to submit text content for protection, please file a request.

Prompt Shields

Type Description
Prompt Shield for Jailbreak Attacks Jailbreak Attacks are User Prompts designed to provoke the Generative AI model into exhibiting behaviors it was trained to avoid or to break the rules set in the System Message. Such attacks can vary from intricate roleplay to subtle subversion of the safety objective.
Prompt Shield for Indirect Attacks Indirect Attacks, also referred to as Indirect Prompt Attacks or Cross-Domain Prompt Injection Attacks, are a potential vulnerability where third parties place malicious instructions inside of documents that the Generative AI system can access and process. Requires document embedding and formatting.

Text content

Warning

The Severity definitions tab in this document contains examples of harmful content that may be disturbing to some readers.

Image content

Warning

The Severity definitions tab in this document contains examples of harmful content that may be disturbing to some readers.

Configurability (preview)

The default content filtering configuration for the GPT model series is set to filter at the medium severity threshold for all four content harm categories (hate, violence, sexual, and self-harm) and applies to both prompts (text, multi-modal text/image) and completions (text). This means that content that is detected at severity level medium or high is filtered, while content detected at severity level low isn't filtered by the content filters. For DALL-E, the default severity threshold is set to low for both prompts (text) and completions (images), so content detected at severity levels low, medium, or high is filtered. The configurability feature is available in preview and allows customers to adjust the settings, separately for prompts and completions, to filter content for each content category at different severity levels as described in the table below:

Severity filtered Configurable for prompts Configurable for completions Descriptions
Low, medium, high Yes Yes Strictest filtering configuration. Content detected at severity levels low, medium and high is filtered.
Medium, high Yes Yes Content detected at severity level low isn't filtered, content at medium and high is filtered.
High Yes Yes Content detected at severity levels low and medium isn't filtered. Only content at severity level high is filtered. Requires approval1.
No filters If approved1 If approved1 No content is filtered regardless of severity level detected. Requires approval1.

1 For Azure OpenAI models, only customers who have been approved for modified content filtering have full content filtering control and can turn content filters off. Apply for modified content filters via this form: Azure OpenAI Limited Access Review: Modified Content Filters For Azure Government customers, please apply for modified content filters via this form: Azure Government - Request Modified Content Filtering for Azure OpenAI Service.

This preview feature is available for the following Azure OpenAI models:

  • GPT model series (text)
  • GPT-4 Turbo Vision 2024-04-09 (multi-modal text/image)
  • DALL-E 2 and 3 (image)

Content filtering configurations are created within a Resource in Azure AI Studio, and can be associated with Deployments. Learn more about configurability here.

Customers are responsible for ensuring that applications integrating Azure OpenAI comply with the Code of Conduct.

Scenario details

When the content filtering system detects harmful content, you receive either an error on the API call if the prompt was deemed inappropriate, or the finish_reason on the response will be content_filter to signify that some of the completion was filtered. When building your application or system, you'll want to account for these scenarios where the content returned by the Completions API is filtered, which might result in content that is incomplete. How you act on this information will be application specific. The behavior can be summarized in the following points:

  • Prompts that are classified at a filtered category and severity level will return an HTTP 400 error.
  • Non-streaming completions calls won't return any content when the content is filtered. The finish_reason value will be set to content_filter. In rare cases with longer responses, a partial result can be returned. In these cases, the finish_reason will be updated.
  • For streaming completions calls, segments will be returned back to the user as they're completed. The service will continue streaming until either reaching a stop token, length, or when content that is classified at a filtered category and severity level is detected.

Scenario: You send a non-streaming completions call asking for multiple outputs; no content is classified at a filtered category and severity level

The table below outlines the various ways content filtering can appear:

HTTP response code Response behavior
200 In the cases when all generation passes the filters as configured, no content moderation details are added to the response. The finish_reason for each generation will be either stop or length.

Example request payload:

{
    "prompt":"Text example", 
    "n": 3,
    "stream": false
}

Example response JSON:

{
    "id": "example-id",
    "object": "text_completion",
    "created": 1653666286,
    "model": "davinci",
    "choices": [
        {
            "text": "Response generated text",
            "index": 0,
            "finish_reason": "stop",
            "logprobs": null
        }
    ]
}

Scenario: Your API call asks for multiple responses (N>1) and at least one of the responses is filtered

HTTP Response Code Response behavior
200 The generations that were filtered will have a finish_reason value of content_filter.

Example request payload:

{
    "prompt":"Text example",
    "n": 3,
    "stream": false
}

Example response JSON:

{
    "id": "example",
    "object": "text_completion",
    "created": 1653666831,
    "model": "ada",
    "choices": [
        {
            "text": "returned text 1",
            "index": 0,
            "finish_reason": "length",
            "logprobs": null
        },
        {
            "text": "returned text 2",
            "index": 1,
            "finish_reason": "content_filter",
            "logprobs": null
        }
    ]
}

Scenario: An inappropriate input prompt is sent to the completions API (either for streaming or non-streaming)

HTTP Response Code Response behavior
400 The API call fails when the prompt triggers a content filter as configured. Modify the prompt and try again.

Example request payload:

{
    "prompt":"Content that triggered the filtering model"
}

Example response JSON:

"error": {
    "message": "The response was filtered",
    "type": null,
    "param": "prompt",
    "code": "content_filter",
    "status": 400
}

Scenario: You make a streaming completions call; no output content is classified at a filtered category and severity level

HTTP Response Code Response behavior
200 In this case, the call will stream back with the full generation and finish_reason will be either 'length' or 'stop' for each generated response.

Example request payload:

{
    "prompt":"Text example",
    "n": 3,
    "stream": true
}

Example response JSON:

{
    "id": "cmpl-example",
    "object": "text_completion",
    "created": 1653670914,
    "model": "ada",
    "choices": [
        {
            "text": "last part of generation",
            "index": 2,
            "finish_reason": "stop",
            "logprobs": null
        }
    ]
}

Scenario: You make a streaming completions call asking for multiple completions and at least a portion of the output content is filtered

HTTP Response Code Response behavior
200 For a given generation index, the last chunk of the generation includes a non-null finish_reason value. The value is content_filter when the generation was filtered.

Example request payload:

{
    "prompt":"Text example",
    "n": 3,
    "stream": true
}

Example response JSON:

 {
    "id": "cmpl-example",
    "object": "text_completion",
    "created": 1653670515,
    "model": "ada",
    "choices": [
        {
            "text": "Last part of generated text streamed back",
            "index": 2,
            "finish_reason": "content_filter",
            "logprobs": null
        }
    ]
}

Scenario: Content filtering system doesn't run on the completion

HTTP Response Code Response behavior
200 If the content filtering system is down or otherwise unable to complete the operation in time, your request will still complete without content filtering. You can determine that the filtering wasn't applied by looking for an error message in the content_filter_result object.

Example request payload:

{
    "prompt":"Text example",
    "n": 1,
    "stream": false
}

Example response JSON:

{
    "id": "cmpl-example",
    "object": "text_completion",
    "created": 1652294703,
    "model": "ada",
    "choices": [
        {
            "text": "generated text",
            "index": 0,
            "finish_reason": "length",
            "logprobs": null,
            "content_filter_result": {
                "error": {
                    "code": "content_filter_error",
                    "message": "The contents are not filtered"
                }
            }
        }
    ]
}

Annotations

Content filters

When annotations are enabled as shown in the code snippet below, the following information is returned via the API for the categories hate and fairness, sexual, violence, and self-harm:

  • content filtering category (hate, sexual, violence, self_harm)
  • the severity level (safe, low, medium or high) within each content category
  • filtering status (true or false).

Optional models

Optional models can be enabled in annotate (returns information when content was flagged, but not filtered) or filter mode (returns information when content was flagged and filtered).

When annotations are enabled as shown in the code snippets below, the following information is returned by the API for optional models:

Model Output
jailbreak detected (true or false),
filtered (true or false)
indirect attacks detected (true or false),
filtered (true or false)
protected material text detected (true or false),
filtered (true or false)
protected material code detected (true or false),
filtered (true or false),
Example citation of public GitHub repository where code snippet was found,
The license of the repository

When displaying code in your application, we strongly recommend that the application also displays the example citation from the annotations. Compliance with the cited license may also be required for Customer Copyright Commitment coverage.

See the following table for the annotation availability in each API version:

Category 2024-02-01 GA 2024-04-01-preview 2023-10-01-preview 2023-06-01-preview
Hate ✅ ✅ ✅ ✅
Violence ✅ ✅ ✅ ✅
Sexual ✅ ✅ ✅ ✅
Self-harm ✅ ✅ ✅ ✅
Prompt Shield for jailbreak attacks ✅ ✅ ✅ ✅
Prompt Shield for indirect attacks ✅
Protected material text ✅ ✅ ✅ ✅
Protected material code ✅ ✅ ✅ ✅
Profanity blocklist ✅ ✅ ✅ ✅
Custom blocklist ✅ ✅ ✅ 
# os.getenv() for the endpoint and key assumes that you are using environment variables.

import os
from openai import AzureOpenAI
client = AzureOpenAI(
    api_key=os.getenv("AZURE_OPENAI_API_KEY"),  
    api_version="2024-03-01-preview",
    azure_endpoint = os.getenv("AZURE_OPENAI_ENDPOINT") 
    )

response = client.completions.create(
    model="gpt-35-turbo-instruct", # model = "deployment_name".
    prompt="{Example prompt where a severity level of low is detected}" 
    # Content that is detected at severity level medium or high is filtered, 
    # while content detected at severity level low isn't filtered by the content filters.
)

print(response.model_dump_json(indent=2))

Output

{ 
  "choices": [ 
    { 
      "content_filter_results": { 
        "hate": { 
          "filtered": false, 
          "severity": "safe" 
        }, 
        "protected_material_code": { 
          "citation": { 
            "URL": " https://github.com/username/repository-name/path/to/file-example.txt", 
            "license": "EXAMPLE-LICENSE" 
          }, 
          "detected": true,
          "filtered": false 
        }, 
        "protected_material_text": { 
          "detected": false, 
          "filtered": false 
        }, 
        "self_harm": { 
          "filtered": false, 
          "severity": "safe" 
        }, 
        "sexual": { 
          "filtered": false, 
          "severity": "safe" 
        }, 
        "violence": { 
          "filtered": false, 
          "severity": "safe" 
        } 
      }, 
      "finish_reason": "stop", 
      "index": 0, 
      "message": { 
        "content": "Example model response will be returned ", 
        "role": "assistant" 
      } 
    } 
  ], 
  "created": 1699386280, 
  "id": "chatcmpl-8IMI4HzcmcK6I77vpOJCPt0Vcf8zJ", 
  "model": "gpt-35-turbo-instruct", 
  "object": "text.completion",
  "usage": { 
    "completion_tokens": 40, 
    "prompt_tokens": 11, 
    "total_tokens": 417 
  },  
  "prompt_filter_results": [ 
    { 
      "content_filter_results": { 
        "hate": { 
          "filtered": false, 
          "severity": "safe" 
        }, 
        "jailbreak": { 
          "detected": false, 
          "filtered": false 
        }, 
        "profanity": { 
          "detected": false, 
          "filtered": false 
        }, 
        "self_harm": { 
          "filtered": false, 
          "severity": "safe" 
        }, 
        "sexual": { 
          "filtered": false, 
          "severity": "safe" 
        }, 
        "violence": { 
          "filtered": false, 
          "severity": "safe" 
        } 
      }, 
      "prompt_index": 0 
    } 
  ]
}

For details on the inference REST API endpoints for Azure OpenAI and how to create Chat and Completions please follow Azure OpenAI Service REST API reference guidance. Annotations are returned for all scenarios when using any preview API version starting from 2023-06-01-preview, as well as the GA API version 2024-02-01.

Example scenario: An input prompt containing content that is classified at a filtered category and severity level is sent to the completions API

{
    "error": {
        "message": "The response was filtered due to the prompt triggering Azure Content management policy. 
                   Please modify your prompt and retry. To learn more about our content filtering policies
                   please read our documentation: https://go.microsoft.com/fwlink/?linkid=21298766",
        "type": null,
        "param": "prompt",
        "code": "content_filter",
        "status": 400,
        "innererror": {
            "code": "ResponsibleAIPolicyViolation",
            "content_filter_result": {
                "hate": {
                    "filtered": true,
                    "severity": "high"
                },
                "self-harm": {
                    "filtered": true,
                    "severity": "high"
                },
                "sexual": {
                    "filtered": false,
                    "severity": "safe"
                },
                "violence": {
                    "filtered":true,
                    "severity": "medium"
                }
            }
        }
    }
}

Document embedding in prompts

A key aspect of Azure OpenAI's Responsible AI measures is the content safety system. This system runs alongside the core GPT model to monitor any irregularities in the model input and output. Its performance is improved when it can differentiate between various elements of your prompt like system input, user input, and AI assistant's output.

For enhanced detection capabilities, prompts should be formatted according to the following recommended methods.

Chat Completions API

The Chat Completion API is structured by definition. It consists of a list of messages, each with an assigned role.

The safety system will parse this structured format and apply the following behavior:

  • On the latest “user” content, the following categories of RAI Risks will be detected:
    • Hate
    • Sexual
    • Violence
    • Self-Harm
    • Jailbreak (optional)

This is an example message array:

{"role": "system", "content": "Provide some context and/or instructions to the model."}, 
{"role": "user", "content": "Example question goes here."}, 
{"role": "assistant", "content": "Example answer goes here."}, 
{"role": "user", "content": "First question/message for the model to actually respond to."}

Embedding documents in your prompt

In addition to detection on last user content, Azure OpenAI also supports the detection of specific risks inside context documents via Prompt Shields – Indirect Prompt Attack Detection. You should identify parts of the input that are a document (e.g. retrieved website, email, etc.) with the following document delimiter.

<documents> 
*insert your document content here* 
</documents>

When you do so, the following options are available for detection on tagged documents:

  • On each tagged “document” content, detect the following categories:
    • Indirect attacks (optional)

Here is an example chat completion messages array:

{"role": "system", "content": "Provide some context and/or instructions to the model, including document context. \"\"\" <documents>\n*insert your document content here*\n<\\documents> \"\"\""}, 

{"role": "user", "content": "First question/message for the model to actually respond to."}

JSON escaping

When you tag unvetted documents for detection, the document content should be JSON-escaped to ensure successful parsing by the Azure OpenAI safety system.

For example, see the following email body:

Hello Josè, 

I hope this email finds you well today.

With JSON escaping, it would read:

Hello Jos\u00E9,\nI hope this email finds you well today.

The escaped text in a chat completion context would read:

{"role": "system", "content": "Provide some context and/or instructions to the model, including document context. \"\"\" <documents>\n Hello Jos\\u00E9,\\nI hope this email finds you well today. \n<\\documents> \"\"\""}, 

{"role": "user", "content": "First question/message for the model to actually respond to."}

Content streaming

This section describes the Azure OpenAI content streaming experience and options. Customers have the option to receive content from the API as it's generated, instead of waiting for chunks of content that have been verified to pass your content filters.

Default

The content filtering system is integrated and enabled by default for all customers. In the default streaming scenario, completion content is buffered, the content filtering system runs on the buffered content, and – depending on the content filtering configuration – content is either returned to the user if it doesn't violate the content filtering policy (Microsoft's default or a custom user configuration), or it’s immediately blocked and returns a content filtering error, without returning the harmful completion content. This process is repeated until the end of the stream. Content is fully vetted according to the content filtering policy before it's returned to the user. Content isn't returned token-by-token in this case, but in “content chunks” of the respective buffer size.

Asynchronous Filter

Customers can choose the Asynchronous Filter as an additional option, providing a new streaming experience. In this case, content filters are run asynchronously, and completion content is returned immediately with a smooth token-by-token streaming experience. No content is buffered, which allows for a fast streaming experience with zero latency associated with content safety.

Customers must be aware that while the feature improves latency, it's a trade-off against the safety and real-time vetting of smaller sections of model output. Because content filters are run asynchronously, content moderation messages and policy violation signals are delayed, which means some sections of harmful content that would otherwise have been filtered immediately could be displayed to the user.

Annotations: Annotations and content moderation messages are continuously returned during the stream. We strongly recommend you consume annotations in your app and implement additional AI content safety mechanisms, such as redacting content or returning additional safety information to the user.

Content filtering signal: The content filtering error signal is delayed. In case of a policy violation, it’s returned as soon as it’s available, and the stream is stopped. The content filtering signal is guaranteed within a ~1,000-character window of the policy-violating content.

Customer Copyright Commitment: Content that is retroactively flagged as protected material may not be eligible for Customer Copyright Commitment coverage.

To enable Asynchronous Filter in Azure OpenAI Studio, follow the Content filter how-to guide to create a new content filtering configuration, and select Asynchronous Filter in the Streaming section.

Comparison of content filtering modes

Compare Streaming - Default Streaming - Asynchronous Filter
Status GA Public Preview
Eligibility All customers Customers approved for modified content filtering
How to enable Enabled by default, no action needed Customers approved for modified content filtering can configure it directly in Azure OpenAI Studio (as part of a content filtering configuration, applied at the deployment level)
Modality and availability Text; all GPT models Text; all GPT models except gpt-4-vision
Streaming experience Content is buffered and returned in chunks Zero latency (no buffering, filters run asynchronously)
Content filtering signal Immediate filtering signal Delayed filtering signal (in up to ~1,000-character increments)
Content filtering configurations Supports default and any customer-defined filter setting (including optional models) Supports default and any customer-defined filter setting (including optional models)

Annotations and sample responses

Prompt annotation message

This is the same as default annotations.

data: { 
    "id": "", 
    "object": "", 
    "created": 0, 
    "model": "", 
    "prompt_filter_results": [ 
        { 
            "prompt_index": 0, 
            "content_filter_results": { ... } 
        } 
    ], 
    "choices": [], 
    "usage": null 
}

Completion token message

Completion messages are forwarded immediately. No moderation is performed first, and no annotations are provided initially.

data: { 
    "id": "chatcmpl-7rAJvsS1QQCDuZYDDdQuMJVMV3x3N", 
    "object": "chat.completion.chunk", 
    "created": 1692905411, 
    "model": "gpt-35-turbo", 
    "choices": [ 
        { 
            "index": 0, 
            "finish_reason": null, 
            "delta": { 
                "content": "Color" 
            } 
        } 
    ], 
    "usage": null 
}

Annotation message

The text field will always be an empty string, indicating no new tokens. Annotations will only be relevant to already-sent tokens. There may be multiple annotation messages referring to the same tokens.

"start_offset" and "end_offset" are low-granularity offsets in text (with 0 at beginning of prompt) to mark which text the annotation is relevant to.

"check_offset" represents how much text has been fully moderated. It's an exclusive lower bound on the "end_offset" values of future annotations. It's non-decreasing.

data: { 
    "id": "", 
    "object": "", 
    "created": 0, 
    "model": "", 
    "choices": [ 
        { 
            "index": 0, 
            "finish_reason": null, 
            "content_filter_results": { ... }, 
            "content_filter_raw": [ ... ], 
            "content_filter_offsets": { 
                "check_offset": 44, 
                "start_offset": 44, 
                "end_offset": 198 
            } 
        } 
    ], 
    "usage": null 
}

Sample response stream (passes filters)

Below is a real chat completion response using Asynchronous Filter. Note how the prompt annotations aren't changed, completion tokens are sent without annotations, and new annotation messages are sent without tokens—they are instead associated with certain content filter offsets.

{"temperature": 0, "frequency_penalty": 0, "presence_penalty": 1.0, "top_p": 1.0, "max_tokens": 800, "messages": [{"role": "user", "content": "What is color?"}], "stream": true}

data: {"id":"","object":"","created":0,"model":"","prompt_annotations":[{"prompt_index":0,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}],"choices":[],"usage":null} 

data: {"id":"chatcmpl-7rCNsVeZy0PGnX3H6jK8STps5nZUY","object":"chat.completion.chunk","created":1692913344,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"role":"assistant"}}],"usage":null} 

data: {"id":"chatcmpl-7rCNsVeZy0PGnX3H6jK8STps5nZUY","object":"chat.completion.chunk","created":1692913344,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":"Color"}}],"usage":null} 

data: {"id":"chatcmpl-7rCNsVeZy0PGnX3H6jK8STps5nZUY","object":"chat.completion.chunk","created":1692913344,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":" is"}}],"usage":null} 

data: {"id":"chatcmpl-7rCNsVeZy0PGnX3H6jK8STps5nZUY","object":"chat.completion.chunk","created":1692913344,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":" a"}}],"usage":null} 

... 

data: {"id":"","object":"","created":0,"model":"","choices":[{"index":0,"finish_reason":null,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}},"content_filter_offsets":{"check_offset":44,"start_offset":44,"end_offset":198}}],"usage":null} 

... 

data: {"id":"chatcmpl-7rCNsVeZy0PGnX3H6jK8STps5nZUY","object":"chat.completion.chunk","created":1692913344,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":"stop","delta":{}}],"usage":null} 

data: {"id":"","object":"","created":0,"model":"","choices":[{"index":0,"finish_reason":null,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}},"content_filter_offsets":{"check_offset":506,"start_offset":44,"end_offset":571}}],"usage":null} 

data: [DONE]

Sample response stream (blocked by filters)

{"temperature": 0, "frequency_penalty": 0, "presence_penalty": 1.0, "top_p": 1.0, "max_tokens": 800, "messages": [{"role": "user", "content": "Tell me the lyrics to \"Hey Jude\"."}], "stream": true}

data: {"id":"","object":"","created":0,"model":"","prompt_filter_results":[{"prompt_index":0,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}],"choices":[],"usage":null} 

data: {"id":"chatcmpl-8JCbt5d4luUIhYCI7YH4dQK7hnHx2","object":"chat.completion.chunk","created":1699587397,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"role":"assistant"}}],"usage":null} 

data: {"id":"chatcmpl-8JCbt5d4luUIhYCI7YH4dQK7hnHx2","object":"chat.completion.chunk","created":1699587397,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":"Hey"}}],"usage":null} 

data: {"id":"chatcmpl-8JCbt5d4luUIhYCI7YH4dQK7hnHx2","object":"chat.completion.chunk","created":1699587397,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":" Jude"}}],"usage":null} 

data: {"id":"chatcmpl-8JCbt5d4luUIhYCI7YH4dQK7hnHx2","object":"chat.completion.chunk","created":1699587397,"model":"gpt-35-turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":","}}],"usage":null} 

... 

data: {"id":"chatcmpl-8JCbt5d4luUIhYCI7YH4dQK7hnHx2","object":"chat.completion.chunk","created":1699587397,"model":"gpt-35- 

turbo","choices":[{"index":0,"finish_reason":null,"delta":{"content":" better"}}],"usage":null} 

data: {"id":"","object":"","created":0,"model":"","choices":[{"index":0,"finish_reason":null,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}},"content_filter_offsets":{"check_offset":65,"start_offset":65,"end_offset":1056}}],"usage":null} 

data: {"id":"","object":"","created":0,"model":"","choices":[{"index":0,"finish_reason":"content_filter","content_filter_results":{"protected_material_text":{"detected":true,"filtered":true}},"content_filter_offsets":{"check_offset":65,"start_offset":65,"end_offset":1056}}],"usage":null} 

data: [DONE]

Important

When content filtering is triggered for a prompt and a "status": 400 is received as part of the response there will be a charge for this request as the prompt was evaluated by the service. Due to the asynchronous nature of the content filtering system, a charge for both the prompt and completion tokens will occur. Charges will also occur when a "status":200 is received with "finish_reason": "content_filter". In this case the prompt did not have any issues, but the completion generated by the model was detected to violate the content filtering rules which results in the completion being filtered.

Best practices

As part of your application design, consider the following best practices to deliver a positive experience with your application while minimizing potential harms:

  • Decide how you want to handle scenarios where your users send prompts containing content that is classified at a filtered category and severity level or otherwise misuse your application.
  • Check the finish_reason to see if a completion is filtered.
  • Check that there's no error object in the content_filter_result (indicating that content filters didn't run).
  • If you're using the protected material code model in annotate mode, display the citation URL when you're displaying the code in your application.

Next steps