Standard load balancer diagnostics with metrics, alerts, and resource health

The metric for data path availability describes the health within the region of the data path to the compute host where your VMs are located. The metric is a reflection of the health of the Azure infrastructure. You can use the metric to:

• Monitor the external availability of your service.

• Investigate the platform where your service is deployed and determine if it's healthy. Determine if your guest OS or application instance is healthy.

• Isolate whether an event is related to your service or the underlying data plane. Don’t confuse this metric with the health probe status ("Backend instance availability").

To get the data path availability for your standard load balancer resources:

1. Make sure the correct load balancer resource is selected.

2. In the Metric drop-down list, select Data Path Availability.

3. In the Aggregation drop-down list, select Avg.

4. Additionally, add a filter on the frontend IP address or frontend port as the dimension with the required frontend IP address or frontend port. Then group them by the selected dimension.

Figure: Load balancer frontend probing details

The metric is generated by an active, in-band measurement. A probing service within the region originates traffic for the measurement. The service is activated as soon as you create a deployment with a public front end, and it continues until you remove the front end.

A packet matching your deployment's front end and rule is generated periodically. It traverses the region from the source to the host where a VM in the backend pool is located. The load balancer infrastructure performs the same load balancing and translation operations as it does for all other traffic. This probe is in-band on your load-balanced endpoint. After the probe arrives on the compute host, where a healthy VM in the backend pool is located, the compute host generates a response to the probing service. Your VM doesn’t see this traffic.

Data path availability fails for the following reasons:

• Your deployment has no healthy VMs remaining in the backend pool.

• An infrastructure outage has occurred.

For diagnostic purposes, you can use the Metric for data path availability together with the health probe status.

Use Average as the aggregation for most scenarios.

The health probe status metric describes the health of your application deployment as configured by you when you configure the health probe of your load balancer. The load balancer uses the status of the health probe to determine where to send new flows. Health probes originate from an Azure infrastructure address and are visible within the guest OS of the VM.

To get the health probe status for your standard load balancer resources:

1. Select the Health Probe Status metric with Avg aggregation type.

2. Apply a filter on the required frontend IP address or port (or both).

Health probes fail for the following reasons:

• You configure a health probe to a port that isn’t listening or not responding or is using the wrong protocol. If your service is using direct server return or floating IP rules, verify the service is listening on the IP address of the NIC's IP configuration and the loopback that's configured with the frontend IP address.

• Your Network Security Group, the VM's guest OS firewall, or the application layer filters don't allow the health probe traffic.

Use Average as the aggregation for most scenarios.

The SNAT connections metric describes the volume of successful and failed connections for outbound flows.

A failed connections volume of greater than zero indicates SNAT port exhaustion. You must investigate further to determine what may be causing these failures. SNAT port exhaustion manifests as a failure to establish an outbound flow. Review the article about outbound connections to understand the scenarios and mechanisms at work, and to learn how to mitigate and design to avoid SNAT port exhaustion.

To get SNAT connection statistics:

1. Select SNAT Connections metric type and Sum as aggregation.

2. Group by Connection State for successful and failed SNAT connection counts to be represented by different lines.

Figure: Load balancer SNAT connection count

The used SNAT ports metric tracks how many SNAT ports are being consumed to maintain outbound flows. This metric indicates how many unique flows are established between an internet source and a backend VM or virtual machine scale set that is behind a load balancer and doesn’t have a public IP address. By comparing the number of SNAT ports you’re using with the Allocated SNAT Ports metric, you can determine if your service is experiencing or at risk of SNAT exhaustion and resulting outbound flow failure.

If your metrics indicate risk of outbound flow failure, reference the article and take steps to mitigate this to ensure service health.

To view SNAT port usage and allocation:

1. Set the time aggregation of the graph to 1 minute to ensure desired data is displayed.

2. Select Used SNAT Ports and/or Allocated SNAT Ports as the metric type and Average as the aggregation.

   • By default, these metrics are the average number of SNAT ports allocated to or used by each backend VM or virtual machine scale set. They correspond to all frontend public IPs mapped to the load balancer, aggregated over TCP and UDP.

   • To view total SNAT ports used by or allocated for the load balancer use metric aggregation Sum.

3. Filter to a specific Protocol Type, a set of Backend IPs, and/or Frontend IPs.

4. To monitor health per backend or frontend instance, apply splitting.

   • Note splitting only allows for a single metric to be displayed at a time.

5. For example, to monitor SNAT usage for TCP flows per machine, aggregate by Average, split by Backend IPs and filter by Protocol Type.

Figure: Average TCP SNAT port allocation and usage for a set of backend VMs

Figure: TCP SNAT port usage per backend instance

For more information on outbound connections, see Source Network Address Translation (SNAT) for outbound connections

Use Sum as the aggregation for most scenarios.

Figure: Load balancer SYN count

The bytes and packet counters metric describes the volume of bytes and packets that are sent or received by your service on a per-frontend basis.

Use Sum as the aggregation for most scenarios.

To get byte or packet count statistics:

1. Select the Bytes Count and/or Packet Count metric type, with Sum as the aggregation.

2. Do either of the following:

   • Apply a filter on a specific frontend IP, frontend port, backend IP, or backend port.

   • Get overall statistics for your load balancer resource without any filtering.

Figure: Load balancer byte count

By using a combination of the data path availability and health probe status metrics on a single chart, you can identify where to look for the problem and resolve the problem. You can gain assurance that Azure is working correctly and use this knowledge to conclusively determine that the configuration or application is the root cause.

You can use health probe metrics to understand how Azure views the health of your deployment as per the configuration you’ve provided. Looking at health probes is always a great first step in monitoring or determining a cause.

You can take it a step further and use data path availability metric to gain insight into how Azure views the health of the underlying data plane that's responsible for your specific deployment. When you combine both metrics, you can isolate where the fault might be, as illustrated in this example:

Figure: Combining data path availability and health probe status metrics

The chart displays the following information:

• The infrastructure hosting your VMs was unavailable and at 0 percent at the beginning of the chart. Later, the infrastructure was healthy and the VMs were reachable, and more than one VM was placed in the back end. This information is indicated by the blue trace for data path availability, which was later at 100 percent.

• The health probe status, indicated by the purple trace, is at 0 percent at the beginning of the chart. The circled area in green highlights where the health probe status became healthy, and at which point the customer's deployment was able to accept new flows.

The chart allows customers to troubleshoot the deployment on their own without having to guess or ask support whether other issues are occurring. The service was unavailable because health probes were failing due to either a misconfiguration or a failed application.