Orleans observability

Metrics

Metrics are numerical measurements reported over time. They're most often used to monitor the health of an application and generate alerts. For more information, see Metrics in .NET. Orleans uses the System.Diagnostics.Metrics APIs to collect metrics. The metrics are exposed to the OpenTelemetry project, which exports the metrics to various monitoring systems.

To monitor your app without making any code changes at all, you can use the dotnet counters .NET diagnostic tool. To monitor Orleans ActivitySource counters, given your desired <ProcessName> to monitor, use the dotnet counters monitor command as shown:

dotnet counters monitor -n <ProcessName> --counters Microsoft.Orleans

Imagine that you're running the Orleans GPS Tracker sample app, and in a separate terminal, you're monitoring it with the dotnet counters monitor command. The following output is typical:

Press p to pause, r to resume, q to quit.
    Status: Running
[Microsoft.Orleans]
    orleans-app-requests-latency-bucket (Count / 1 sec)                    0
        duration=10000ms                                                   0
        duration=1000ms                                                    0
        duration=100ms                                                     0
        duration=10ms                                                      0
        duration=15000ms                                                   0
        duration=1500ms                                                    0
        duration=1ms                                                   2,530
        duration=2000ms                                                    0
        duration=200ms                                                     0
        duration=2ms                                                       0
        duration=400ms                                                     0
        duration=4ms                                                       0
        duration=5000ms                                                    0
        duration=50ms                                                      0
        duration=6ms                                                       0
        duration=800ms                                                     0
        duration=8ms                                                       0
        duration=9223372036854775807ms                                     0
    orleans-app-requests-latency-count (Count / 1 sec)                 2,530
    orleans-app-requests-latency-sum (Count / 1 sec)                       0
    orleans-catalog-activation-working-set                                36
    orleans-catalog-activations                                           38
    orleans-consistent-ring-range-percentage-average                     100
    orleans-consistent-ring-range-percentage-local                       100
    orleans-consistent-ring-size                                           1
    orleans-directory-cache-size                                          27
    orleans-directory-partition-size                                      26
    orleans-directory-ring-local-portion-average-percentage              100
    orleans-directory-ring-local-portion-distance                          0
    orleans-directory-ring-local-portion-percentage                        0
    orleans-directory-ring-size                                        1,295
    orleans-gateway-received (Count / 1 sec)                           1,291
    orleans-gateway-sent (Count / 1 sec)                               2,582
    orleans-messaging-processing-activation-data                           0
    orleans-messaging-processing-dispatcher-forwarded (Count / 1           0
    orleans-messaging-processing-dispatcher-processed (Count / 1       2,543
        Direction=Request,Status=Ok                                    2,582
    orleans-messaging-processing-dispatcher-received (Count / 1        1,271
        Context=Grain,Direction=Request                                1,291
        Context=None,Direction=Request                                 1,291
    orleans-messaging-processing-ima-enqueued (Count / 1 sec)          5,113

For more information, see Investigate performance counters (dotnet-counters).

Orleans meters

Orleans uses the System.Diagnostics.Metrics APIs to collect metrics. Orleans categorizes each meter into domain-centric concerns, such as networking, messaging, gateway, and so on. The following subsections describe the meters that Orleans uses.

Networking

The following table represents a collection of networking meters that are used to monitor the Orleans networking layer.

Messaging

The following table represents a collection of messaging meters that are used to monitor the Orleans messaging layer.

Gateway

The following table represents a collection of gateway meters that are used to monitor the Orleans gateway layer.

Runtime

The following table represents a collection of runtime meters that are used to monitor the Orleans runtime layer.

Catalog

The following table represents a collection of catalog meters that are used to monitor the Orleans catalog layer.

Directory

The following table represents a collection of directory meters that are used to monitor the Orleans directory layer.

Consistent ring

The following table represents a collection of consistent ring meters that are used to monitor the Orleans consistent ring layer.

Watchdog

The following table represents a collection of watchdog meters that are used to monitor the Orleans watchdog layer.

Client

The following table represents a collection of client meters that are used to monitor the Orleans client layer.

Miscellaneous

The following table represents a collection of miscellaneous meters that are used to monitor various layers.

App requests

The following table represents a collection of app request meters that are used to monitor the Orleans app request layer.

Reminders

The following table represents a collection of reminder meters that are used to monitor the Orleans reminder layer.

Storage

The following table represents a collection of storage meters that are used to monitor the Orleans storage layer.

Streams

The following table represents a collection of stream meters that are used to monitor the Orleans stream layer.

Transactions

The following table represents a collection of transaction meters that are used to monitor the Orleans transaction layer.

Prometheus

There are various third-party metrics providers that you can use with Orleans. One popular example is Prometheus, which can be used to collect metrics from your app with OpenTelemetry.

To use OpenTelemetry and Prometheus with Orleans, call the following IServiceCollection extension method:

builder.Services.AddOpenTelemetry()
    .WithMetrics(metrics =>
    {
        metrics
            .AddPrometheusExporter()
            .AddMeter("Microsoft.Orleans");
    });

The AddPrometheusExporter method ensures that the PrometheusExporter is added to the builder. Orleans makes use of a Meter named "Microsoft.Orleans" to create Counter<T> instances for many Orleans-specific metrics. The AddMeter method is used to specify the name of the meter to subscribe to, in this case "Microsoft.Orleans".

After the exporter has been configured, and your app has been built, you must call MapPrometheusScrapingEndpoint on the IEndpointRouteBuilder (the app instance) to expose the metrics to Prometheus. For example:

WebApplication app = builder.Build();

app.MapPrometheusScrapingEndpoint();
app.Run();

Distributed tracing

Distributed tracing is a set of tools and practices to monitor and troubleshoot distributed applications. Distributed tracing is a key component of observability, and it's a critical tool for developers to understand the behavior of their apps. Orleans also supports distributed tracing with OpenTelemetry.

Regardless of the distributed tracing exporter you choose, you call:

• AddActivityPropagation(ISiloBuilder): which enables distributed tracing for the silo.
• AddActivityPropagation(IClientBuilder): which enables distributed tracing for the client.

Referring back to the Orleans GPS Tracker sample app, you can use the Zipkin distributed tracing system to monitor the app by updating the Program.cs. To use OpenTelemetry and Zipkin with Orleans, call the following IServiceCollection extension method:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        // Set a service name
        tracing.SetResourceBuilder(
            ResourceBuilder.CreateDefault()
                .AddService(serviceName: "GPSTracker", serviceVersion: "1.0"));

        tracing.AddSource("Microsoft.Orleans.Runtime");
        tracing.AddSource("Microsoft.Orleans.Application");

        tracing.AddZipkinExporter(zipkin =>
        {
            zipkin.Endpoint = new Uri("http://localhost:9411/api/v2/spans");
        });
    });

The Zipkin trace is shown in the Jaeger UI (which is an alternative to Zipkin but uses the same data format):

For more information, see Distributed tracing.

Orleans outputs its runtime statistics and metrics through the ITelemetryConsumer interface. The application can register one or more telemetry consumers for their silos and clients, to receive statistics and metrics that the Orleans runtime periodically publishes. These can be consumers for popular telemetry analytics solutions or custom ones for any other destination and purpose. Three telemetry consumers are currently included in the Orleans codebase.

They're released as separate NuGet packages:

• Microsoft.Orleans.OrleansTelemetryConsumers.AI for publishing to Azure Application Insights.

• Microsoft.Orleans.OrleansTelemetryConsumers.Counters for publishing to Windows performance counters. The Orleans runtime continually updates many them. The CounterControl.exe tool, included in the Microsoft.Orleans.CounterControl NuGet package, helps register necessary performance counter categories. It has to run with elevated privileges. The performance counters can be monitored using any of the standard monitoring tools.

• Microsoft.Orleans.OrleansTelemetryConsumers.NewRelic, for publishing to New Relic.

To configure your silo and client to use telemetry consumers, silo configuration code looks like this:

var siloHostBuilder = new HostBuilder()
    .UseOrleans(c =>
    {
        c.AddApplicationInsightsTelemetryConsumer("INSTRUMENTATION_KEY");
    });

Client configuration code look like this:

var clientBuilder = new ClientBuilder();
clientBuilder.AddApplicationInsightsTelemetryConsumer("INSTRUMENTATION_KEY");

To use a custom defined TelemetryConfiguration (which may have TelemetryProcessors, TelemetrySinks, and so on), silo configuration code looks like this:

var telemetryConfiguration = TelemetryConfiguration.CreateDefault();
var siloHostBuilder = new HostBuilder()
    .UseOrleans(c =>
    {
        c.AddApplicationInsightsTelemetryConsumer(telemetryConfiguration);
    });

Client configuration code look like this:

var clientBuilder = new ClientBuilder();
var telemetryConfiguration = TelemetryConfiguration.CreateDefault();
clientBuilder.AddApplicationInsightsTelemetryConsumer(telemetryConfiguration);