configuring-opentelemetry-dotnet

SKILL.md

Configuring OpenTelemetry in .NET

When to Use

Adding distributed tracing to an ASP.NET Core application

Setting up OpenTelemetry exporters (OTLP is the primary protocol; Jaeger accepts OTLP natively; Prometheus OTLP ingestion requires explicit opt-in)

Creating custom metrics or trace spans for business operations

Troubleshooting distributed trace context propagation across services

When Not to Use

The user wants application-level logging only (use ILogger, Serilog)

The user is using Application Insights SDK directly (different API)

The user needs APM with a commercial vendor's proprietary SDK

Inputs

Input
Required
Description

ASP.NET Core project
Yes
The application to instrument

Observability backend
No
Where to export: OTLP collector, Aspire dashboard, Jaeger (accepts OTLP natively)

Workflow

Step 1: Install the correct packages

There are many OpenTelemetry NuGet packages. Install exactly these:

# Core SDK + ASP.NET Core instrumentation + logging integration
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http

# Exporter
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol  # OTLP exporter for traces, metrics, AND logs

# Optional — dev/local debugging only (do NOT include in production deployments)
# dotnet add package OpenTelemetry.Exporter.Console

Do NOT install OpenTelemetry alone — you need OpenTelemetry.Extensions.Hosting for proper DI integration.

Optional: additional auto-instrumentation packages

Install only the packages that match the libraries your application uses:

dotnet add package OpenTelemetry.Instrumentation.SqlClient           # SQL Server queries
dotnet add package OpenTelemetry.Instrumentation.EntityFrameworkCore  # EF Core
dotnet add package OpenTelemetry.Instrumentation.GrpcNetClient       # gRPC calls
dotnet add package OpenTelemetry.Instrumentation.Runtime             # GC, thread pool metrics

Step 2: Configure all signals in Program.cs

using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using OpenTelemetry.Metrics;
using OpenTelemetry.Logs;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(serviceName: builder.Environment.ApplicationName))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation(options =>
        {
            // Filter out health check endpoints from traces
            options.Filter = httpContext =>
                !httpContext.Request.Path.StartsWithSegments("/healthz");
        })
        .AddHttpClientInstrumentation(options =>
        {
            options.RecordException = true;
        })
        // Optional: add SQL instrumentation if using SqlClient directly
        // .AddSqlClientInstrumentation(options =>
        // {
        //     options.SetDbStatementForText = true;
        //     options.RecordException = true;
        // })
        // Custom activity sources (must match ActivitySource names in your code)
        .AddSource("MyApp.Orders")
        .AddSource("MyApp.Payments")
        .AddSource("MyApp.Messaging"))
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        // Optional: .AddRuntimeInstrumentation() for GC and thread pool metrics
        //   (requires OpenTelemetry.Instrumentation.Runtime package)
        // Custom meters (must match Meter names in your code)
        .AddMeter("MyApp.Metrics"))
    .WithLogging(logging =>
    {
        logging.IncludeScopes = true;
        // logging.IncludeFormattedMessage = true;  // Enable if you need the formatted message string in log exports
    })
    // Single OTLP exporter for all signals — reads OTEL_EXPORTER_OTLP_ENDPOINT
    // env var (defaults to http://localhost:4317). Override via environment variable
    // or appsettings.json configuration.
    .UseOtlpExporter();

Step 3: Understanding log–trace correlation

The .WithLogging() call in Step 2 integrates ILogger with OpenTelemetry:

Each log entry automatically includes TraceId and SpanId for correlation with traces

The service resource from .ConfigureResource() propagates to logs automatically

UseOtlpExporter() applies to logs alongside traces and metrics

No additional packages or separate SetResourceBuilder call needed

Step 4: Create custom spans (Activities) for business operations

using System.Diagnostics;
using Microsoft.Extensions.Logging;

public class OrderService
{
    // Create an ActivitySource matching what you registered in Step 2
    private static readonly ActivitySource ActivitySource = new("MyApp.Orders");
    private readonly ILogger<OrderService> _logger;

    public OrderService(ILogger<OrderService> logger) => _logger = logger;

    public async Task<Order> ProcessOrderAsync(CreateOrderRequest request)
    {
        // Start a new span
        using var activity = ActivitySource.StartActivity("ProcessOrder");

        // Add attributes (tags) to the span
        activity?.SetTag("order.customer_id", request.CustomerId);
        activity?.SetTag("order.item_count", request.Items.Count);

        try
        {
            // Child span for validation
            using (var validationActivity = ActivitySource.StartActivity("ValidateOrder"))
            {
                await ValidateOrderAsync(request);
                validationActivity?.SetTag("validation.result", "passed");
            }

            // Child span for payment
            using (var paymentActivity = ActivitySource.StartActivity("ProcessPayment",
                ActivityKind.Client))  // Client = outgoing call
            {
                paymentActivity?.SetTag("payment.method", request.PaymentMethod);
                await ProcessPaymentAsync(request);
            }

            var order = new Order { Id = Guid.NewGuid(), CustomerId = request.CustomerId, Status = "Completed" };

            activity?.SetTag("order.status", "completed");
            activity?.SetStatus(ActivityStatusCode.Ok);

            return order;
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            // Log via ILogger — OpenTelemetry captures this with trace correlation.
            // Prefer logging over activity.RecordException() as OTel is deprecating
            // span events for exception recording in favor of log-based exceptions.
            _logger.LogError(ex, "Order processing failed for customer {CustomerId}", request.CustomerId);
            throw;
        }
    }
}

Critical: ActivitySource name must match AddSource("...") in configuration. Unmatched sources are silently ignored — this is the #1 debugging issue.

Step 5: Create custom metrics

Use IMeterFactory (injected via DI) to create meters — this ensures proper lifetime management and testability.

using System.Diagnostics;
using System.Diagnostics.Metrics;

public class OrderMetrics
{
    private readonly Counter<long> _ordersProcessed;
    private readonly Histogram<double> _orderProcessingDuration;
    private readonly UpDownCounter<int> _activeOrders;

    public OrderMetrics(IMeterFactory meterFactory)
    {
        // Meter name must match AddMeter("...") in configuration
        var meter = meterFactory.Create("MyApp.Metrics");

        // Counter — use for things that only go up
        _ordersProcessed = meter.CreateCounter<long>(
            "orders.processed", "orders", "Total orders successfully processed");

        // Histogram — use for measuring distributions (latency, sizes)
        _orderProcessingDuration = meter.CreateHistogram<double>(
            "orders.processing_duration", "ms", "Time to process an order");

        // UpDownCounter — use for things that go up AND down
        _activeOrders = meter.CreateUpDownCounter<int>(
            "orders.active", "orders", "Currently processing orders");
    }

    public void RecordOrderProcessed(string region, double durationMs)
    {
        // Tags enable dimensional filtering (by region, status, etc.)
        var tags = new TagList
        {
            { "region", region },
            { "order.type", "standard" }
        };

        _ordersProcessed.Add(1, tags);
        _orderProcessingDuration.Record(durationMs, tags);
    }
}

Register OrderMetrics in DI:

builder.Services.AddSingleton<OrderMetrics>();

Step 6: Configure context propagation for distributed scenarios

Trace context propagation is automatic for HTTP calls when using AddHttpClientInstrumentation(). For non-HTTP scenarios:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using OpenTelemetry.Context.Propagation;

// ActivitySource should be static — register via .AddSource("MyApp.Messaging") in Step 2
private static readonly ActivitySource MessageSource = new("MyApp.Messaging");

// Manual context propagation (e.g., across message queues)
// On the SENDING side:
var propagator = Propagators.DefaultTextMapPropagator;
var activityContext = Activity.Current?.Context ?? default;
var context = new PropagationContext(activityContext, Baggage.Current);
var carrier = new Dictionary<string, string>();

propagator.Inject(context, carrier, (dict, key, value) => dict[key] = value);
// Send carrier dictionary as message headers

// On the RECEIVING side:
var parentContext = propagator.Extract(default, carrier,
    (dict, key) => dict.TryGetValue(key, out var value) ? new[] { value } : Array.Empty<string>());

Baggage.Current = parentContext.Baggage;
using var activity = MessageSource.StartActivity("ProcessMessage",
    ActivityKind.Consumer,
    parentContext.ActivityContext);  // Links to parent trace!

Validation

 Traces appear in the observability backend (Jaeger, Aspire dashboard, etc.)

 HTTP requests automatically create spans with correct verb, URL, status code

 Custom ActivitySource names match AddSource() registrations

 Custom Meter names match AddMeter() registrations

 Logs include TraceId and SpanId for correlation

 Health check endpoints are filtered from traces

 Exception details appear on error spans

Common Pitfalls

Pitfall
Solution

ActivitySource.StartActivity returns null
Source name doesn't match any AddSource() — names must match exactly

Traces not appearing in exporter
Check OTLP endpoint: gRPC uses port 4317, HTTP uses 4318

Missing HTTP client spans
Ensure AddHttpClientInstrumentation() is registered; it works for both IHttpClientFactory/DI and new HttpClient() (use IHttpClientFactory for lifetime management)

High cardinality tags
Don't use user IDs, request IDs, or UUIDs as metric tags — explodes storage

OTLP gRPC vs HTTP mismatch
Default is gRPC (port 4317); if collector only accepts HTTP, set OtlpExportProtocol.HttpProtobuf

Meter / ActivitySource lifecycle
ActivitySource should be static; create Meter via IMeterFactory from DI (not new Meter()) for proper lifetime management and testability