Debug high CPU usage in .NET Core

This article applies to: ✔️ .NET Core 3.1 SDK and later versions

In this tutorial, you'll learn how to debug an excessive CPU usage scenario. Using the provided example ASP.NET Core web app source code repository, you can cause a deadlock intentionally. The endpoint will stop responding and experience thread accumulation. You'll learn how you can use various tools to diagnose this scenario with several key pieces of diagnostics data.

In this tutorial, you will:

Prerequisites

The tutorial uses:

• .NET Core 3.1 SDK or a later version.
• Sample debug target to trigger the scenario.
• dotnet-trace to list processes and generate a profile.
• dotnet-counters to monitor cpu usage.

CPU counters

Before attempting to collect diagnostics data, you need to observe a high CPU condition. Run the sample application using the following command from the project root directory.

dotnet run

To find the process ID, use the following command:

dotnet-trace ps

Take note of the process ID from your command output. Our process ID was 22884, but yours will be different. To check the current CPU usage, use the dotnet-counters tool command:

dotnet-counters monitor --refresh-interval 1 -p 22884

The refresh-interval is the number of seconds between the counter polling CPU values. The output should be similar to the following:

Press p to pause, r to resume, q to quit.
    Status: Running

[System.Runtime]
    % Time in GC since last GC (%)                         0
    Allocation Rate / 1 sec (B)                            0
    CPU Usage (%)                                          0
    Exception Count / 1 sec                                0
    GC Heap Size (MB)                                      4
    Gen 0 GC Count / 60 sec                                0
    Gen 0 Size (B)                                         0
    Gen 1 GC Count / 60 sec                                0
    Gen 1 Size (B)                                         0
    Gen 2 GC Count / 60 sec                                0
    Gen 2 Size (B)                                         0
    LOH Size (B)                                           0
    Monitor Lock Contention Count / 1 sec                  0
    Number of Active Timers                                1
    Number of Assemblies Loaded                          140
    ThreadPool Completed Work Item Count / 1 sec           3
    ThreadPool Queue Length                                0
    ThreadPool Thread Count                                7
    Working Set (MB)                                      63

With the web app running, immediately after startup, the CPU isn't being consumed at all and is reported at 0%. Navigate to the api/diagscenario/highcpu route with 60000 as the route parameter:

https://localhost:5001/api/diagscenario/highcpu/60000

Now, rerun the dotnet-counters command. If interested in monitoring just the cpu-usage counter, add '--counters System.Runtime[cpu-usage]` to the previous command. We are unsure if the CPU is being consumed, so we will monitor the same list of counters as above to verify counter values are within expected range for our application.

dotnet-counters monitor -p 22884 --refresh-interval 1

You should see an increase in CPU usage as shown below (depending on the host machine, expect varying CPU usage):

Press p to pause, r to resume, q to quit.
    Status: Running

[System.Runtime]
    % Time in GC since last GC (%)                         0
    Allocation Rate / 1 sec (B)                            0
    CPU Usage (%)                                         25
    Exception Count / 1 sec                                0
    GC Heap Size (MB)                                      4
    Gen 0 GC Count / 60 sec                                0
    Gen 0 Size (B)                                         0
    Gen 1 GC Count / 60 sec                                0
    Gen 1 Size (B)                                         0
    Gen 2 GC Count / 60 sec                                0
    Gen 2 Size (B)                                         0
    LOH Size (B)                                           0
    Monitor Lock Contention Count / 1 sec                  0
    Number of Active Timers                                1
    Number of Assemblies Loaded                          140
    ThreadPool Completed Work Item Count / 1 sec           3
    ThreadPool Queue Length                                0
    ThreadPool Thread Count                                7
    Working Set (MB)                                      63

Throughout the duration of the request, the CPU usage will hover around the increased percentage.

At this point, you can safely say the CPU is running higher than you expect. Identifying the effects of a problem is key to finding the cause. We will use the effect of high CPU consumption in addition to diagnostic tools to find the cause of the problem.

Analyze High CPU with Profiler

When analyzing an app with high CPU usage, you need a diagnostics tool that can provide insights into what the code is doing. The usual choice is a profiler, and there are different profiler options to choose from. dotnet-trace can be used on all operating systems, however, its limitations of safe-point bias and managed-only callstacks result in more general information compared to a kernel-aware profiler like 'perf' for Linux or ETW for Windows. If your performance investigation involves only managed code, generally dotnet-trace will be sufficient.

export DOTNET_PerfMapEnabled=1
dotnet run

sudo perf record -p 2266 -g

sudo perf report -f

git clone --depth=1 https://github.com/BrendanGregg/FlameGraph
sudo perf script | FlameGraph/stackcollapse-perf.pl | FlameGraph/flamegraph.pl > flamegraph.svg

dotnet-trace collect -p 22884 --providers Microsoft-DotNETCore-SampleProfiler

Analyzing High CPU Data with Visual Studio

All *.nettrace files can be analyzed in Visual Studio. To analyze a Linux *.nettrace file in Visual Studio, transfer the *.nettrace file, in addition to the other necessary documents, to a Windows machine, and then open the *.nettrace file in Visual Studio. For more information, see Analyze CPU Usage Data.

See also

• dotnet-trace to list processes
• dotnet-counters to check managed memory usage
• dotnet-dump to collect and analyze a dump file
• dotnet/diagnostics

Next steps