How to troubleshoot with resource logs

This how-to guide provides an overview of Azure Web PubSub resource logs and some tips for using the logs to troubleshoot certain problems. Logs can be used for issue identification, connection tracking, message tracing, HTTP request tracing, and analysis.

What are resource logs?

There are three types of resource logs: Connectivity, Messaging, and HTTP requests.

• Connectivity logs provide detailed information for Azure Web PubSub hub connections. For example, basic information (user ID, connection ID, and so on) and event information (connect, disconnect, and so on).
• Messaging logs provide tracing information for the Azure Web PubSub hub messages received and sent via Azure Web PubSub service. For example, tracing ID and message type of the message.
• HTTP requests logs provide tracing information for HTTP requests to the Azure Web PubSub service. For example, HTTP method and status code. Typically the HTTP request is recorded when it arrives at or leave from service.

Capture resource logs by using the live trace tool

The Azure Web PubSub service live trace tool has ability to collect resource logs in real time, which is helpful for troubleshooting problems in your development environment. The live trace tool can capture connectivity logs, messaging logs, and HTTP request logs.

Launch the live trace tool

Steps for access key enabled

1. Go to the Azure portal and your SignalR Service page.

2. From the menu on the left, under Monitoring select Live trace settings.

3. Select Enable Live Trace.

4. Select Save button. It will take a moment for the changes to take effect.

5. When updating is complete, select Open Live Trace Tool.

   

Steps for access key disabled

Assign live trace tool API permission to yourself

1. Go to the Azure portal and your SignalR Service page.
2. Select Access control (IAM).
3. In the new page, Click +Add, then click Role assignment.
4. In the new page, focus on Job function roles tab, Select SignalR Service Owner role, and then click Next.
5. In Members page, click +Select members.
6. In the new panel, search and select members, and then click Select.
7. Click Review + assign, and wait for the completion notification.

Visit live trace tool

1. Go to the Azure portal and your SignalR Service page.

2. From the menu on the left, under Monitoring select Live trace settings.

3. Select Enable Live Trace.

4. Select Save button. It will take a moment for the changes to take effect.

5. When updating is complete, select Open Live Trace Tool.

   

Sign in with your Microsoft account

1. The live trace tool will pop up a Microsoft sign in window. If no window is pop up, check and allow pop up windows in your browser.
2. Wait for Ready showing in the status bar.

Capture the resource logs

The live trace tool provides functionality to help you capture the resource logs for troubleshooting.

• Capture: Begin to capture the real-time resource logs from Azure Web PubSub.
• Clear: Clear the captured real-time resource logs.
• Log filter: The live trace tool lets you filter the captured real-time resource logs with one specific key word. The common separators (for example, space, comma, semicolon, and so on) will be treated as part of the key word.
• Status: The status shows whether the live trace tool is connected or disconnected with the specific instance.

The real-time resource logs captured by live trace tool contain detailed information for troubleshooting.

Capture resource logs with Azure Monitor

How to enable resource logs

Currently Azure Web PubSub supports integration with Azure Storage.

1. Go to Azure portal.

2. On Diagnostic settings page of your Azure Web PubSub service instance, select + Add diagnostic setting.

3. In Diagnostic setting name, input the setting name.

4. In Category details, select any log category you need.

5. In Destination details, check Archive to a storage account.

   

6. Select Save to save the diagnostic setting.

   Note

   The storage account should be in the same region as Azure Web PubSub service.

Archive to an Azure Storage Account

Logs are stored in the storage account that's configured in the Diagnostics setting pane. A container named insights-logs-<CATEGORY_NAME> is created automatically to store resource logs. Inside the container, logs are stored in the file resourceId=/SUBSCRIPTIONS/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/RESOURCEGROUPS/XXXX/PROVIDERS/MICROSOFT.SIGNALRSERVICE/SIGNALR/XXX/y=YYYY/m=MM/d=DD/h=HH/m=00/PT1H.json. The path is combined by resource ID and Date Time. The log files are split by hour. The minute value is always m=00.

All logs are stored in JavaScript Object Notation (JSON) format. Each entry has string fields that use the format described in the following sections.

Archive log JSON strings include elements listed in the following tables:

Format

Properties Table

The following code is an example of an archive log JSON string:

{
  "properties": {
    "message": "Connection started",
    "collection": "Connection",
    "connectionId": "LW61bMG2VQLIMYIVBMmyXgb3c418200",
    "userId": null
  },
  "operationName": "ConnectionStarted",
  "category": "ConnectivityLogs",
  "level": "Informational",
  "callerIpAddress": "167.220.255.79",
  "resourceId": "/SUBSCRIPTIONS/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/RESOURCEGROUPS/MYGROUP/PROVIDERS/MICROSOFT.SIGNALRSERVICE/WEBPUBSUB/MYWEBPUBSUB",
  "time": "2021-09-17T05:25:05Z",
  "location": "westus"
}

Archive to Azure Log Analytics

To send logs to a Log Analytics workspace:

1. On the Diagnostic setting page, under Destination details, select **Send to Log Analytics workspace.
2. Select the Subscription you want to use.
3. Select the Log Analytics workspace to use as the destination for the logs.

To view the resource logs, follow these steps:

1. Select Logs in your target Log Analytics.

   

2. Enter WebPubSubConnectivity, WebPubSubMessaging or WebPubSubHttpRequest, and then select the time range to query the log. For advanced queries, see Get started with Log Analytics in Azure Monitor.

   

To use a sample query for SignalR service, follow the steps below.

1. Select Logs in your target Log Analytics.
2. Select Queries to open query explorer.
3. Select Resource type to group sample queries in resource type.
4. Select Run to run the script.

Archive log columns include elements listed in the following table.

Troubleshoot with the resource logs

If you find unexpected changes in the number of connections, either increasing or decreasing, you can take advantage of resource logs to troubleshoot the problem. Typical issues are often about connections' unexpected quantity changes, connections reach connection limits, and authorization failure.

Unexpected changes in number of connections

Unexpected connection dropping

If a connection disconnects, the resource logs will record the disconnection event with ConnectionAborted or ConnectionEnded in operationName.

The difference between ConnectionAborted and ConnectionEnded is that ConnectionEnded is an expected disconnection that is triggered by the client or server side. While the ConnectionAborted is usually an unexpected connection dropping event, and the reason for disconnection will be provided in message.

The abort reasons are listed in the following table:

Unexpected increase in connections

When the number of client connections unexpectedly increases, the first thing you need to do is to filter out the superfluous connections. Add a unique test user ID to your test client connection. Then check the resource logs; if you see more than one client connection has the same test user ID or IP, then it's likely the client is creating more connections than expected. Check your client code to find the source of the extra connections.

Authorization failure

If you get 401 Unauthorized returned for client requests, check your resource logs. If you find Failed to validate audience. Expected Audiences: <valid audience>. Actual Audiences: <actual audience>, it means all audiences in your access token are invalid. Try to use the valid audiences suggested in the log.

Throttling

If you find that you can't establish client connections to Azure Web PubSub service, check your resource logs. If you see Connection count reaches limit in the resource log, you established too many connections to Azure Web PubSub service and reached the connection count limit. Consider scaling up your Azure Web PubSub service instance. If you see Message count reaches limit in the resource log and you're using the Free tier, it means you used up the quota of messages. If you want to send more messages, consider changing your Azure Web PubSub service instance to Standard tier. For more information, see Azure Web PubSub service Pricing.