OpenTelemetry in ConnectSoft Microservice Template¶

Purpose & Overview¶

OpenTelemetry is an open-source observability framework that provides a unified approach to collecting, processing, and exporting telemetry data (traces, metrics, and logs) from cloud-native applications. In the ConnectSoft Microservice Template, OpenTelemetry serves as the foundation for observability, enabling teams to gain comprehensive insights into application behavior, performance, and health across distributed systems.

OpenTelemetry provides:

Unified Observability: Single framework for traces, metrics, and logs
Vendor Neutrality: Open standard that works with any observability backend
Automatic Instrumentation: Out-of-the-box instrumentation for common frameworks
Custom Instrumentation: Flexible API for application-specific telemetry
Trace Context Propagation: Automatic correlation across service boundaries
Resource Attributes: Service identity and environment metadata
Multiple Exporters: Support for OTLP, Seq, Console, and other backends
Industry Standard: CNCF project with broad industry adoption

OpenTelemetry Philosophy

OpenTelemetry treats observability as a first-class concern, providing a standardized, vendor-neutral approach to collecting telemetry data. By instrumenting applications once with OpenTelemetry, teams can export to any observability backend, ensuring flexibility and avoiding vendor lock-in while maintaining consistency across the entire system.

Architecture Overview¶

Three Pillars of Observability¶

OpenTelemetry collects three types of telemetry data:

Application Code
    ↓
OpenTelemetry SDK
    ├── Traces (Distributed Tracing)
    │   ├── Request flow across services
    │   ├── Span hierarchy and timing
    │   └── Trace context propagation
    ├── Metrics (Quantitative Measurements)
    │   ├── Performance counters
    │   ├── Business metrics
    │   └── Resource utilization
    └── Logs (Structured Events)
        ├── Application events
        ├── Error logs
        └── Trace-linked logs
    ↓
Resource Attributes
    ├── Service identity
    ├── Environment information
    └── Deployment metadata
    ↓
Exporters
    ├── OTLP Exporter (OpenTelemetry Collector)
    ├── Seq Exporter (OTLP endpoint)
    └── Console Exporter (development)
    ↓
Observability Backends
    ├── Grafana LGTM Stack (Loki, Tempo, Mimir) - Comprehensive observability
    ├── Jaeger, Zipkin (traces)
    ├── Prometheus, Grafana (metrics)
    ├── Seq, Application Insights (unified)
    └── Other OTLP-compatible systems

OpenTelemetry Components¶

Component	Purpose	Location
OpenTelemetry SDK	Core instrumentation framework	`OpenTelemetry.Extensions.Hosting`
TracerProvider	Manages trace collection	`OpenTelemetry.Trace`
MeterProvider	Manages metric collection	`OpenTelemetry.Metrics`
Resource	Service identity and metadata	`OpenTelemetry.Resources`
ActivitySource	Creates custom spans	`System.Diagnostics`
Meter	Creates custom metrics	`System.Diagnostics.Metrics`
Exporters	Sends telemetry to backends	`OpenTelemetry.Exporter.*`

Service Registration¶

OpenTelemetry Configuration¶

Registration:

// MicroserviceRegistrationExtensions.cs
#if OpenTelemetry
    services.AddMicroserviceOpenTelemetry(webHostEnvironment);
#endif

Implementation:

// OpenTelemetryExtensions.cs
internal static IServiceCollection AddMicroserviceOpenTelemetry(
    this IServiceCollection services, 
    IWebHostEnvironment webHostEnvironment)
{
    ArgumentNullException.ThrowIfNull(services);
    ArgumentNullException.ThrowIfNull(webHostEnvironment);

    // Configure OpenTelemetry with traces, metrics, and resource
    services.AddOpenTelemetry()
        .ConfigureResource(resourceBuilder =>
        {
            resourceBuilder.ConfigureMicroserviceOpenTelemetryResource(webHostEnvironment);
        })
        .WithMetrics(metricsBuilder =>
        {
            metricsBuilder.ConfigureMicroserviceOpenTelemetryMetrics(OptionsExtensions.OpenTelemetryOptions);
        })
        .WithTracing(tracingBuilder =>
        {
            tracingBuilder.ConfigureMicroserviceOpenTelemetryTracing(webHostEnvironment, OptionsExtensions.OpenTelemetryOptions);
        });

    // Enable ActivitySource support for Azure SDKs
    AppContext.SetSwitch("Azure.Experimental.EnableActivitySource", isEnabled: true);

    return services;
}

Logging Integration¶

OpenTelemetry logging is registered separately:

// Program.cs
#if OpenTelemetry
    logging.AddOpenTelemetry(options =>
    {
        options.IncludeScopes = true;
        options.ParseStateValues = true;
    });
#endif

Resource Configuration¶

Service Identity¶

Resource attributes identify the service and environment:

private static void ConfigureMicroserviceOpenTelemetryResource(
    this ResourceBuilder resourceBuilder, 
    IWebHostEnvironment webHostEnvironment)
{
    // Add service attributes
    resourceBuilder.AddService(
        serviceName: webHostEnvironment.ApplicationName,
        serviceVersion: "1.0.0",
        serviceNamespace: "ConnectSoft.MicroserviceTemplate",
        serviceInstanceId: Environment.MachineName);

    // Add SDK version
    resourceBuilder.AddTelemetrySdk();

    // Add custom attributes
    resourceBuilder.AddAttributes(
        new KeyValuePair<string, object>[]
        {
            new (OpenTelemetryAttributeName.Deployment.Environment, webHostEnvironment.EnvironmentName),
            new (OpenTelemetryAttributeName.Host.Name, Environment.MachineName),
            new (OpenTelemetryAttributeName.OperatingSystem.Description, Environment.OSVersion.VersionString),
        });

    // Detect environment variables (e.g., K8s pod info)
    resourceBuilder.AddEnvironmentVariableDetector();
}

Resource Attributes¶

All telemetry data includes resource attributes:

Attribute	Description	Example
`service.name`	Service identifier	`"ConnectSoft.MicroserviceTemplate"`
`service.version`	Service version	`"1.0.0"`
`service.namespace`	Service namespace	`"ConnectSoft.MicroserviceTemplate"`
`service.instance.id`	Instance identifier	`"kube-node-1"`
`deployment.environment`	Environment name	`"Production"`, `"Development"`
`host.name`	Host machine name	`"kube-node-1"`
`os.description`	Operating system	`"Linux 5.10.0"`

Distributed Tracing¶

Overview¶

OpenTelemetry provides distributed tracing capabilities, enabling end-to-end visibility into requests as they flow through services. Each trace represents a complete request journey, composed of spans (individual operations).

Key Concepts:

Trace: Complete request journey across services
Span: Individual operation within a trace
Activity: .NET representation of a span
ActivitySource: Factory for creating activities
Trace Context: Propagation metadata (trace ID, span ID)

Automatic Instrumentation¶

The template automatically instruments:

ASP.NET Core: HTTP requests and responses
HttpClient: Outgoing HTTP calls
gRPC: Client and server calls
SQL Client: Database operations (NHibernate)
Redis: Cache operations
MassTransit: Message publishing and consumption
NServiceBus: Message handling
Orleans: Actor invocations
SignalR: Real-time connections
Hangfire: Background jobs
MongoDB: Document operations
Semantic Kernel: AI operations
OpenAI: API calls
MCP Tools: Model Context Protocol tool invocations

Custom Activities¶

Create custom spans using ActivitySource:

using System.Diagnostics;

public static class Tracing
{
    private static readonly ActivitySource ActivitySource = 
        new("ConnectSoft.MicroserviceTemplate");

    public static ActivitySource Source => ActivitySource;
}

// Usage
using var activity = Tracing.Source.StartActivity("ProcessOrder");
activity?.SetTag("order.id", orderId);
activity?.SetTag("order.total", totalAmount);

try
{
    await ProcessOrderAsync(order);
    activity?.SetStatus(ActivityStatusCode.Ok);
}
catch (Exception ex)
{
    activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
    activity?.AddException(ex);
    throw;
}

Register ActivitySource:

// OpenTelemetryExtensions.cs
tracingBuilder
    .AddSource("ConnectSoft.MicroserviceTemplate") // Subscribes to all ActivitySources starting with this name
#if UseMCP
    .AddSource("ConnectSoft.MicroserviceTemplate.MCP") // MCP tool invocation tracing
#endif
    ;

MCP Tool Tracing¶

The template provides McpToolActivitySource for tracing MCP tool invocations:

using ConnectSoft.MicroserviceTemplate.ModelContextProtocol;
using System.Diagnostics;

[McpServerTool]
public EchoResponse Echo(string text, bool uppercase = false)
{
    using var activity = McpToolActivitySource.StartToolInvocationActivity("Echo", "DemoStringTools");

    try
    {
        // Tool logic...
        activity?.SetStatus(ActivityStatusCode.Ok);
        return result;
    }
    catch (Exception ex)
    {
        activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
        activity?.AddException(ex);
        throw;
    }
}

MCP Activity Tags:

mcp.tool.name: Tool name (automatically set)
mcp.tool.type: Tool class/type (automatically set)
Custom tags: Additional context-specific tags

For comprehensive MCP observability patterns, see Model Context Protocol.

Trace Context Propagation¶

OpenTelemetry automatically propagates trace context via W3C Trace Context headers:

traceparent: W3C Trace Context header
tracestate: Additional trace state

Automatic Propagation:

// Outgoing HTTP request automatically includes trace context
using var httpClient = new HttpClient();
var response = await httpClient.GetAsync("https://api.example.com/users");
// Trace context is automatically included in headers

For comprehensive distributed tracing documentation, see Distributed Tracing.

Metrics¶

Overview¶

OpenTelemetry Metrics API provides quantitative measurements of system behavior, performance, and business operations. Metrics are aggregated over time and exported to observability backends.

Key Concepts:

Meter: Factory for creating instruments
Counter: Monotonically increasing metric
Histogram: Distribution of values
Gauge: Current value snapshot
UpDownCounter: Value that can increase or decrease

Automatic Instrumentation¶

The template automatically collects metrics from:

Runtime Instrumentation: .NET runtime metrics (GC, threads, etc.)
ASP.NET Core: HTTP request metrics
Process: System resource metrics (CPU, memory)
HttpClient: Outgoing HTTP metrics
MassTransit: Messaging metrics
NServiceBus: Messaging metrics
Orleans: Actor metrics
Semantic Kernel: AI operation metrics

Custom Metrics¶

Create custom metrics using Meter:

using System.Diagnostics.Metrics;

public class MicroserviceTemplateMetrics
{
    private static readonly Meter Meter = new("ConnectSoft.MicroserviceTemplate.Metrics");

    private static readonly Counter<long> OrdersCreatedCounter = 
        Meter.CreateCounter<long>("orders.created", "orders", "Number of orders created");

    private static readonly Histogram<double> OrderProcessingDuration = 
        Meter.CreateHistogram<double>("orders.processing.duration", "ms", "Order processing duration");

    public void RecordOrderCreated()
    {
        OrdersCreatedCounter.Add(1);
    }

    public void RecordOrderProcessingDuration(double durationMs)
    {
        OrderProcessingDuration.Record(durationMs);
    }
}

Register Meter:

// OpenTelemetryExtensions.cs
metricsBuilder
    .AddMeter("ConnectSoft.MicroserviceTemplate.Metrics");

For comprehensive metrics documentation, see Metrics.

Logging¶

Overview¶

OpenTelemetry logging integrates with structured logging to add trace context to log records, enabling correlation between logs, traces, and metrics.

Log Integration¶

OpenTelemetry automatically enriches logs with trace context:

// Program.cs
#if OpenTelemetry
    logging.AddOpenTelemetry(options =>
    {
        options.IncludeScopes = true;
        options.ParseStateValues = true;
    });
#endif

Automatic Enrichment¶

When OpenTelemetry is enabled, logs automatically include:

traceId: Distributed trace identifier
spanId: Current span identifier
parentId: Parent span identifier (when applicable)
traceFlags: Trace flags

Example Trace-Linked Log:

{
  "timestamp": "2025-01-15T17:22:58Z",
  "level": "Information",
  "message": "Processing order {OrderId}",
  "orderId": "ORD-7788",
  "traceId": "00-ab12cd34ef567890abcdef1234567890",
  "spanId": "1a2b3c4d5e6f7890",
  "parentId": "00-xyz1234567890abcdef1234567890",
  "traceFlags": "01",
  "service": "OrderService",
  "environment": "Production"
}

Benefits¶

End-to-End Visibility: Every log linked to a trace span
Root Cause Isolation: See logs before/after failures in trace context
Cross-Service Diagnostics: Correlate logs across service boundaries
Telemetry-to-Log Drilldown: Navigate from traces to specific log records

For comprehensive logging documentation, see Logging.

Automatic Instrumentation¶

ASP.NET Core Instrumentation¶

HTTP Request Tracing:

tracingBuilder
    .AddAspNetCoreInstrumentation(options =>
    {
        options.EnrichWithHttpRequest = EnrichWithHttpRequest;
        options.EnrichWithHttpResponse = EnrichWithHttpResponse;
        options.RecordException = true;
    });

Automatic Capture:

HTTP method and path
Request/response headers
Status codes
Request/response sizes
Duration
Client IP address
User identity (if authenticated)

Request Enrichment:

private static void EnrichWithHttpRequest(Activity activity, HttpRequest request)
{
    var context = request.HttpContext;
    activity.AddTag(OpenTelemetryAttributeName.Http.ClientIP, context.Connection.RemoteIpAddress);
    activity.AddTag(OpenTelemetryAttributeName.Http.RequestContentLength, request.ContentLength);
    activity.AddTag(OpenTelemetryAttributeName.Http.RequestContentType, request.ContentType);

    var user = context.User;
    if (user.Identity?.Name is not null)
    {
        activity.AddTag(OpenTelemetryAttributeName.EndUser.Id, user.Identity.Name);
        activity.AddTag(OpenTelemetryAttributeName.EndUser.Scope, string.Join(',', user.Claims.Select(x => x.Value)));
    }
}

HttpClient Instrumentation¶

Outgoing HTTP Calls:

tracingBuilder
    .AddHttpClientInstrumentation(options =>
    {
        options.DefineHttpClientInstrumentationOptions();
    });

Filter Configuration:

private static void DefineHttpClientInstrumentationOptions(
    this HttpClientTraceInstrumentationOptions options)
{
    options.FilterHttpRequestMessage = message =>
        message is not null &&
        message.RequestUri is not null &&
        !message.RequestUri.Host.Contains("visualstudio", StringComparison.Ordinal) &&
        !message.RequestUri.Host.Contains("applicationinsights", StringComparison.Ordinal);
}

SQL Client Instrumentation¶

Database Operations (NHibernate):

#if UseNHibernate
tracingBuilder
    .AddSqlClientInstrumentation(options =>
    {
        options.SetDbStatementForText = true;
        options.RecordException = true;
        options.Enrich = (activity, eventName, rawObject) =>
        {
            if (eventName.Equals("OnCustom", StringComparison.Ordinal) && rawObject is SqlCommand msCmd)
            {
                activity.SetTag("db.commandTimeout", msCmd.CommandTimeout);
            }
        };
    });
#endif

Messaging Instrumentation¶

MassTransit:

#if UseMassTransit
tracingBuilder
    .AddSource(MassTransit.Logging.DiagnosticHeaders.DefaultListenerName);

metricsBuilder
    .AddMeter(MassTransit.Monitoring.InstrumentationOptions.MeterName);
#endif

NServiceBus:

#if UseNServiceBus
tracingBuilder
    .AddSource("NServiceBus.Core");

metricsBuilder
    .AddMeter("NServiceBus.Core");
#endif

Orleans Instrumentation¶

Actor Invocations:

#if UseOrleans
tracingBuilder
    .AddSource("Microsoft.Orleans.Runtime")
    .AddSource("Microsoft.Orleans.Application");

metricsBuilder
    .AddMeter("Microsoft.Orleans");
#endif

MongoDB Instrumentation¶

Document Operations:

#if UseMongoDb
tracingBuilder
    .AddSource("MongoDB.Driver.Core.Extensions.DiagnosticSources");
#endif

Hangfire Instrumentation¶

Background Jobs:

#if UseHangFire
tracingBuilder
    .AddHangfireInstrumentation();
#endif

SignalR Instrumentation¶

Real-Time Connections:

#if UseSignalR
tracingBuilder
    .AddSource("Microsoft.AspNetCore.SignalR.Server");
#endif

AI Instrumentation¶

Semantic Kernel:

#if UseSemanticKernel
tracingBuilder
    .AddSource("Microsoft.SemanticKernel*");

metricsBuilder
    .AddMeter("Microsoft.SemanticKernel*");
#endif

OpenAI:

#if (UseMicrosoftExtensionsAIOpenAIProvider || UseSemanticKernelOpenAIConnector)
tracingBuilder
    .AddSource("OpenAI.*");

metricsBuilder
    .AddMeter("OpenAI.*");
#endif

Microsoft.Extensions.AI:

#if UseMicrosoftExtensionsAI
tracingBuilder
    .AddSource("Experimental.Microsoft.Extensions.AI*");

metricsBuilder
    .AddMeter("Experimental.Microsoft.Extensions.AI*");
#endif

For comprehensive documentation on AI observability, including detailed information on traces, metrics, and logging for AI operations, see AI Observability.

Exporters¶

OTLP Exporter¶

OpenTelemetry Protocol (OTLP) exporter sends telemetry to OpenTelemetry Collector or OTLP-compatible backends:

tracingBuilder.AddOtlpExporter(options =>
{
    options.Protocol = (OtlpExportProtocol)otelOptions.OtlpExporter.OtlpExportProtocol;
    options.Endpoint = new Uri(otelOptions.OtlpExporter.Endpoint);
});

metricsBuilder.AddOtlpExporter(options =>
{
    options.Protocol = (OtlpExportProtocol)otelOptions.OtlpExporter.OtlpExportProtocol;
    options.Endpoint = new Uri(otelOptions.OtlpExporter.Endpoint);
});

Supported Protocols: - Grpc: gRPC protocol (default) - HttpProtobuf: HTTP with Protocol Buffers

Seq Exporter¶

Seq exporter sends telemetry to Seq's OTLP endpoint:

tracingBuilder.AddOtlpExporter(options =>
{
    options.Protocol = (OtlpExportProtocol)otelOptions.OtlpSeqExporter.OtlpExportProtocol;
    options.Endpoint = new Uri($"{otelOptions.OtlpSeqExporter.Endpoint}/ingest/otlp/v1/traces");
});

metricsBuilder.AddOtlpExporter(options =>
{
    options.Protocol = (OtlpExportProtocol)otelOptions.OtlpSeqExporter.OtlpExportProtocol;
    options.Endpoint = new Uri($"{otelOptions.OtlpSeqExporter.Endpoint}/ingest/otlp/v1/logs");
});

Console Exporter¶

Development Debugging:

if (otelOptions.EnableConsoleExporter)
{
    tracingBuilder.AddConsoleExporter(options =>
    {
        options.Targets = ConsoleExporterOutputTargets.Console | ConsoleExporterOutputTargets.Debug;
    });

    metricsBuilder.AddConsoleExporter();
}

Prometheus Exporter (Optional)¶

Prometheus Metrics Endpoint:

// Uncomment to enable Prometheus exporter
// metricsBuilder.AddPrometheusExporter();
// app.MapPrometheusScrapingEndpoint();

Azure Monitor Exporter (Optional)¶

Application Insights:

// Uncomment to enable Azure Monitor exporter
// services.AddOpenTelemetry()
//     .UseAzureMonitor();

Grafana LGTM Stack Exporter¶

Grafana LGTM Stack (Loki, Tempo, Mimir) via OpenTelemetry Collector:

The Grafana LGTM stack is configured through the OpenTelemetry Collector, which routes telemetry to:

Loki: Logs via Loki exporter
Tempo: Traces via OTLP exporter
Mimir: Metrics via Prometheus Remote Write exporter

See Grafana LGTM Stack Example for complete setup instructions.

Configuration¶

OpenTelemetry Options¶

Configuration Class:

public sealed class OpenTelemetryOptions
{
    public const string OpenTelemetryOptionsSectionName = "OpenTelemetry";

    [Required]
    required public bool EnableConsoleExporter { get; set; }

    [Required]
    [ValidateObjectMembers]
    required public OtlpExporterOptions OtlpExporter { get; set; }

    [Required]
    [ValidateObjectMembers]
    required public OtlpSeqExporterOptions OtlpSeqExporter { get; set; }
}

appsettings.json:

{
  "OpenTelemetry": {
    "EnableConsoleExporter": false,
    "OtlpExporter": {
      "OtlpExportProtocol": "Grpc",
      "Endpoint": "http://localhost:4317"
    },
    "OtlpSeqExporter": {
      "OtlpExportProtocol": "HttpProtobuf",
      "Endpoint": "http://localhost:5341"
    }
  }
}

OTLP Exporter Options¶

public sealed class OtlpExporterOptions
{
    [Required]
    [Url]
    required public string Endpoint { get; set; }

    [Required]
    [EnumDataType(typeof(OtlpExportProtocolType))]
    required public OtlpExportProtocolType OtlpExportProtocol { get; set; }
}

Supported Protocols: - Grpc: gRPC protocol (default, port 4317) - HttpProtobuf: HTTP with Protocol Buffers (port 4318)

Sampling¶

Development Sampling¶

Always On (Development):

if (webHostEnvironment.IsDevelopment())
{
    // Sample all traces in development
    tracingBuilder.SetSampler(new AlwaysOnSampler());
}

Production Sampling¶

Configurable Sampling:

// Example: Sample 10% of traces
tracingBuilder.SetSampler(new TraceIdRatioBasedSampler(0.1));

// Example: Sample based on parent decision
tracingBuilder.SetSampler(new ParentBasedSampler(new TraceIdRatioBasedSampler(0.1)));

Sampling Strategies:

AlwaysOnSampler: Sample all traces
AlwaysOffSampler: Sample no traces
TraceIdRatioBasedSampler: Sample based on ratio (0.0 to 1.0)
ParentBasedSampler: Respect parent sampling decision

Best Practices¶

Do's¶

Use Semantic Conventions

// ✅ GOOD - Use standard attribute names
activity?.SetTag("http.method", "GET");
activity?.SetTag("http.status_code", 200);
activity?.SetTag("db.system", "postgresql");

Create Meaningful Activity Names

// ✅ GOOD - Descriptive activity names
using var activity = Tracing.Source.StartActivity("Order.Validate");

// ❌ BAD - Generic names
using var activity = Tracing.Source.StartActivity("Process");

Set Activity Status

// ✅ GOOD - Always set status
try
{
    await ProcessAsync();
    activity?.SetStatus(ActivityStatusCode.Ok);
}
catch (Exception ex)
{
    activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
    activity?.AddException(ex);
    throw;
}

Add Relevant Tags

// ✅ GOOD - Add business context
activity?.SetTag("order.id", orderId);
activity?.SetTag("order.total", totalAmount);

Use ActivitySource for Custom Activities

// ✅ GOOD - Use ActivitySource
private static readonly ActivitySource ActivitySource = new("ConnectSoft.MicroserviceTemplate");

// ❌ BAD - Don't use Activity directly
var activity = new Activity("MyActivity");

Enable Resource Attributes

// ✅ GOOD - Identify service and environment
resourceBuilder.AddService(
    serviceName: "MyService",
    serviceVersion: "1.0.0",
    serviceInstanceId: Environment.MachineName);

Link Logs to Traces

// ✅ GOOD - Enable OpenTelemetry logging
logging.AddOpenTelemetry(options =>
{
    options.IncludeScopes = true;
    options.ParseStateValues = true;
});

Don'ts¶

Don't Over-Instrument

// ❌ BAD - Too many activities
using var activity1 = Tracing.Source.StartActivity("Step1");
using var activity2 = Tracing.Source.StartActivity("Step2");
using var activity3 = Tracing.Source.StartActivity("Step3");
// ... many more

// ✅ GOOD - Instrument meaningful operations
using var activity = Tracing.Source.StartActivity("ProcessOrder");

Don't Store Sensitive Data in Tags

// ❌ BAD - Sensitive data in tags
activity?.SetTag("password", password);
activity?.SetTag("apiKey", apiKey);

// ✅ GOOD - Use safe identifiers
activity?.SetTag("user.id", userId);
activity?.SetTag("order.id", orderId);

Don't Create Activities in Hot Paths

// ❌ BAD - Activity in tight loop
for (int i = 0; i < 10000; i++)
{
    using var activity = Tracing.Source.StartActivity("ProcessItem");
    // ...
}

// ✅ GOOD - Instrument at higher level
using var activity = Tracing.Source.StartActivity("ProcessBatch");
for (int i = 0; i < 10000; i++)
{
    // ...
}

Don't Ignore Sampling

// ❌ BAD - Always sample in production
tracingBuilder.SetSampler(new AlwaysOnSampler());

// ✅ GOOD - Use appropriate sampling
tracingBuilder.SetSampler(new TraceIdRatioBasedSampler(0.1));

Troubleshooting¶

Issue: No Telemetry Data Appearing¶

Symptoms: Traces, metrics, or logs not appearing in observability backend.

Solutions:

Verify Exporter Configuration: Check endpoint URLs and protocols
Check Network Connectivity: Ensure exporter endpoints are reachable
Verify Resource Attributes: Check service name and version are set
Enable Console Exporter: Use for development debugging
Check Sampling: Verify sampling isn't filtering all traces
Review Logs: Check for exporter errors in application logs

Issue: High Overhead¶

Symptoms: Performance degradation with OpenTelemetry enabled.

Solutions:

Enable Sampling: Reduce trace volume in production
Review Instrumentation: Remove unnecessary instrumentation
Check Exporters: Use efficient exporters (gRPC vs HTTP)
Monitor Resource Usage: Check CPU and memory consumption
Batch Exports: Use batching exporters when available

Issue: Missing Trace Context¶

Symptoms: Traces not correlated across services.

Solutions:

Verify W3C Trace Context: Ensure headers are propagated
Check HttpClient Instrumentation: Verify automatic instrumentation
Review Custom Code: Ensure trace context is preserved
Check Middleware Order: Ensure instrumentation runs before custom middleware

Issue: Duplicate Telemetry¶

Symptoms: Same telemetry appearing multiple times.

Solutions:

Check Multiple Exporters: Ensure exporters aren't duplicated
Review Instrumentation: Verify instrumentation isn't registered multiple times
Check OpenTelemetry Collector: Verify collector isn't duplicating data

Distributed Tracing: Comprehensive guide to distributed tracing with OpenTelemetry
Metrics: Detailed metrics collection and instrumentation
Logging: Structured logging with trace correlation
AI Observability: Comprehensive guide to AI observability with OpenTelemetry, logging, and metrics
Application Insights: Azure Application Insights integration

Summary¶

OpenTelemetry in the ConnectSoft Microservice Template provides:

✅ Unified Observability: Single framework for traces, metrics, and logs
✅ Vendor Neutrality: Open standard compatible with any observability backend
✅ Automatic Instrumentation: Out-of-the-box instrumentation for common frameworks
✅ Custom Instrumentation: Flexible API for application-specific telemetry
✅ Trace Context Propagation: Automatic correlation across service boundaries
✅ Resource Attributes: Service identity and environment metadata
✅ Multiple Exporters: Support for OTLP, Seq, Console, and other backends
✅ Log Correlation: Trace-linked logging for end-to-end visibility
✅ Sampling: Configurable sampling for production efficiency

By following these patterns, teams can:

Instrument Once: Use OpenTelemetry to instrument applications
Export Anywhere: Send telemetry to any observability backend
Correlate Everything: Link traces, metrics, and logs together
Debug Efficiently: Trace requests through complex distributed systems
Monitor Performance: Track system behavior and performance
Maintain Flexibility: Avoid vendor lock-in while maintaining consistency

OpenTelemetry is the foundation of observability in the ConnectSoft Microservice Template, providing the standardized, vendor-neutral approach needed to understand, debug, and optimize cloud-native applications.

OpenTelemetry in ConnectSoft Microservice Template¶

Purpose & Overview¶

Architecture Overview¶

Three Pillars of Observability¶

OpenTelemetry Components¶

Service Registration¶

OpenTelemetry Configuration¶

Logging Integration¶

Resource Configuration¶

Service Identity¶

Resource Attributes¶

Distributed Tracing¶

Overview¶

Automatic Instrumentation¶

Custom Activities¶

MCP Tool Tracing¶

Trace Context Propagation¶

Metrics¶

Overview¶

Automatic Instrumentation¶

Custom Metrics¶

Logging¶

Overview¶

Log Integration¶

Automatic Enrichment¶

Benefits¶

Automatic Instrumentation¶

ASP.NET Core Instrumentation¶

HttpClient Instrumentation¶

SQL Client Instrumentation¶

Messaging Instrumentation¶

Orleans Instrumentation¶

MongoDB Instrumentation¶

Hangfire Instrumentation¶

SignalR Instrumentation¶

AI Instrumentation¶

Exporters¶

OTLP Exporter¶

Seq Exporter¶

Console Exporter¶

Prometheus Exporter (Optional)¶

Azure Monitor Exporter (Optional)¶

Grafana LGTM Stack Exporter¶

Configuration¶

OpenTelemetry Options¶

OTLP Exporter Options¶

Sampling¶

Development Sampling¶

Production Sampling¶

Best Practices¶

Do's¶

Don'ts¶

Troubleshooting¶

Issue: No Telemetry Data Appearing¶

Issue: High Overhead¶

Issue: Missing Trace Context¶

Issue: Duplicate Telemetry¶

Related Documentation¶

Summary¶