Metrics and Observability

Complete guide to implementing metrics and distributed tracing in libraries created with the ConnectSoft Library Template.

Overview

The template provides built-in support for observability through:

• Metrics: OpenTelemetry-compatible metrics for performance monitoring
• Tracing: ActivitySource-based distributed tracing
• Logging: Structured logging with correlation (see Features)

These features work together to provide comprehensive observability for your library.

Metrics Implementation

Overview

Metrics are implemented using Microsoft.Extensions.Diagnostics and follow OpenTelemetry semantic conventions.

Generated Metrics Class

When UseMetrics=true, the template generates a sample metrics class:

namespace YourLibraryName.Metrics
{
    using System;
    using System.Diagnostics.Metrics;

    /// <summary>
    /// Provides custom application metrics for request counting and processing durations.
    /// </summary>
    public class LibraryTemplateMetrics
    {
        private const string MetricsPrefix = "connectsoft.librarytemplate";

        private readonly Counter<long> requestCounter;
        private readonly Histogram<double> requestDuration;

        public LibraryTemplateMetrics(IMeterFactory meterFactory)
        {
            ArgumentNullException.ThrowIfNull(meterFactory);

            var meter = meterFactory.Create("ConnectSoft.LibraryTemplate");

            requestCounter = meter.CreateCounter<long>(
                name: $"{MetricsPrefix}.request.count",
                unit: "requests",
                description: "Counts the number of handled custom requests.");

            requestDuration = meter.CreateHistogram<double>(
                name: $"{MetricsPrefix}.request.duration",
                unit: "milliseconds",
                description: "Measures the processing time of custom handled requests in milliseconds.");
        }

        public void IncrementRequestCounter()
        {
            requestCounter.Add(1);
        }

        public void RecordRequestProcessingTime(double durationMilliseconds)
        {
            requestDuration.Record(durationMilliseconds);
        }
    }
}

Metric Types

The template demonstrates two common metric types:

Counter

Counts events (monotonically increasing):

private readonly Counter<long> requestCounter;

requestCounter = meter.CreateCounter<long>(
    name: "mylibrary.requests.count",
    unit: "requests",
    description: "Total number of requests processed.");

// Usage
requestCounter.Add(1);
requestCounter.Add(1, new KeyValuePair<string, object?>("status", "success"));

Use Cases:

• Request counts
• Error counts
• Operation counts

Histogram

Measures distributions (durations, sizes):

private readonly Histogram<double> requestDuration;

requestDuration = meter.CreateHistogram<double>(
    name: "mylibrary.requests.duration",
    unit: "milliseconds",
    description: "Request processing duration.");

// Usage
requestDuration.Record(123.45);
requestDuration.Record(67.89, new KeyValuePair<string, object?>("operation", "process"));

Use Cases:

• Request durations
• Response sizes
• Processing times

Creating Custom Metrics

Step 1: Define Metrics Class

public class MyLibraryMetrics
{
    private const string MetricsPrefix = "mylibrary";
    private readonly Counter<long> errorCounter;
    private readonly Histogram<double> cacheHitDuration;

    public MyLibraryMetrics(IMeterFactory meterFactory)
    {
        var meter = meterFactory.Create("MyLibrary");

        errorCounter = meter.CreateCounter<long>(
            name: $"{MetricsPrefix}.errors.count",
            unit: "errors",
            description: "Total number of errors.");

        cacheHitDuration = meter.CreateHistogram<double>(
            name: $"{MetricsPrefix}.cache.hit.duration",
            unit: "milliseconds",
            description: "Cache hit lookup duration.");
    }

    public void RecordError(string errorType)
    {
        errorCounter.Add(1, new KeyValuePair<string, object?>("error.type", errorType));
    }

    public void RecordCacheHit(double durationMs)
    {
        cacheHitDuration.Record(durationMs);
    }
}

Step 2: Register Metrics Service

services.AddSingleton<MyLibraryMetrics>();

Step 3: Use Metrics in Code

public class MyService
{
    private readonly MyLibraryMetrics _metrics;

    public MyService(MyLibraryMetrics metrics)
    {
        _metrics = metrics;
    }

    public void ProcessRequest()
    {
        var sw = Stopwatch.StartNew();
        try
        {
            // Your logic here
            _metrics.RecordCacheHit(sw.ElapsedMilliseconds);
        }
        catch (Exception ex)
        {
            _metrics.RecordError(ex.GetType().Name);
            throw;
        }
    }
}

Metric Naming Conventions

Follow OpenTelemetry semantic conventions:

Format: {library}.{domain}.{metric}.{unit}

Examples:

• mylibrary.requests.count (counter)
• mylibrary.requests.duration (histogram, milliseconds)
• mylibrary.errors.count (counter)
• mylibrary.cache.hits (counter)

Best Practices:

• Use lowercase with dots as separators
• Be consistent with naming
• Include units in description
• Use descriptive names

Metric Tags

Add tags for dimensional data:

requestCounter.Add(1, 
    new KeyValuePair<string, object?>("status", "success"),
    new KeyValuePair<string, object?>("operation", "process"));

Common Tags:

• status: success, error, timeout
• operation: operation name
• method: HTTP method (for APIs)
• endpoint: endpoint name

Distributed Tracing

Overview

Distributed tracing is implemented using ActivitySource and follows OpenTelemetry conventions.

Generated Diagnostics Class

When UseActivitySource=true, the template generates a diagnostics class:

namespace YourLibraryName.Diagnostics
{
    using System;
    using System.Collections.Generic;
    using System.Diagnostics;
    using Microsoft.Extensions.Logging;

    /// <summary>
    /// First-class tracing entry point for the library.
    /// </summary>
    public static class LibraryTemplateDiagnostics
    {
        public const string ActivitySourceName = "ConnectSoft.LibraryTemplate";
        public static readonly ActivitySource ActivitySource = new(ActivitySourceName);

        public static Activity? StartActivity(
            string name,
            IEnumerable<KeyValuePair<string, object?>>? tags = null)
        {
            var activity = ActivitySource.StartActivity(name, ActivityKind.Internal);
            if (activity is null)
            {
                return null;
            }

            if (tags is not null)
            {
                foreach (var tag in tags)
                {
                    activity.SetTag(tag.Key, tag.Value);
                }
            }

            return activity;
        }

        public static IDisposable? StartActivityScope(
            ILogger logger,
            string name,
            IEnumerable<KeyValuePair<string, object?>>? tags = null)
        {
            var activity = StartActivity(name, tags);
            if (activity is null)
            {
                return null;
            }

            var scope = logger.BeginScope(new Dictionary<string, object?>
            {
                ["traceId"] = activity.TraceId.ToString(),
                ["spanId"] = activity.SpanId.ToString(),
            });

            return new CompositeDisposable(activity, scope!);
        }
    }
}

Basic Tracing

Simple Activity

using var activity = LibraryTemplateDiagnostics.StartActivity("MyOperation");
// Your code here
// Activity automatically ends when disposed

Activity with Tags

var tags = new List<KeyValuePair<string, object?>>
{
    new("operation.type", "process"),
    new("item.id", itemId),
    new("item.type", "document")
};

using var activity = LibraryTemplateDiagnostics.StartActivity("ProcessItem", tags);
// Your code here

Activity with Events

using var activity = LibraryTemplateDiagnostics.StartActivity("ProcessItem");
activity?.AddEvent(new ActivityEvent("ItemReceived"));
// Process item
activity?.AddEvent(new ActivityEvent("ItemProcessed"));

Log Correlation

Correlate logs with traces:

using var scope = LibraryTemplateDiagnostics.StartActivityScope(
    logger, 
    "ProcessItem",
    new[] { new KeyValuePair<string, object?>("item.id", itemId) });

logger.LogInformation("Processing item {ItemId}", itemId);
// Your code here
// Both activity and log scope end when disposed

Benefits:

• Logs include trace ID and span ID
• Easy correlation in observability platforms
• Request flow visualization

Nested Activities

Create child activities:

using var parentActivity = LibraryTemplateDiagnostics.StartActivity("ParentOperation");

// Child activity automatically becomes child of parent
using var childActivity = LibraryTemplateDiagnostics.StartActivity("ChildOperation");
// Child code here

Activity Status

Set activity status for errors:

using var activity = LibraryTemplateDiagnostics.StartActivity("ProcessItem");
try
{
    // Your code
    activity?.SetStatus(ActivityStatusCode.Ok);
}
catch (Exception ex)
{
    activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
    activity?.SetTag("error.type", ex.GetType().Name);
    throw;
}

Observability Best Practices

1. Use Structured Logging

// Good: Structured logging
logger.LogInformation("Processing item {ItemId} with status {Status}", itemId, status);

// Avoid: String interpolation
logger.LogInformation($"Processing item {itemId} with status {status}");

Benefits:

• Better searchability
• Efficient storage
• Query capabilities

2. Add Context to Metrics

// Good: Metrics with tags
metrics.RecordRequest(durationMs, 
    new KeyValuePair<string, object?>("status", "success"),
    new KeyValuePair<string, object?>("operation", "process"));

// Avoid: Metrics without context
metrics.RecordRequest(durationMs);

Benefits:

• Dimensional analysis
• Better insights
• Filtering capabilities

3. Correlate Logs and Traces

// Good: Correlated logs and traces
using var scope = LibraryTemplateDiagnostics.StartActivityScope(logger, "Operation");
logger.LogInformation("Operation started");

// Avoid: Separate logs and traces
using var activity = LibraryTemplateDiagnostics.StartActivity("Operation");
logger.LogInformation("Operation started"); // No correlation

Benefits:

• Unified view
• Easy debugging
• Request flow tracking

4. Use Appropriate Metric Types

// Counter: For counts (monotonically increasing)
counter.Add(1);

// Histogram: For distributions (durations, sizes)
histogram.Record(value);

// Gauge: For current values (can go up or down)
// (Not shown in template, but available)

5. Set Activity Status

// Always set status
activity?.SetStatus(ActivityStatusCode.Ok);
// or
activity?.SetStatus(ActivityStatusCode.Error, errorMessage);

Benefits:

• Clear success/failure indication
• Better error tracking
• Consistent status reporting

6. Use Meaningful Names

// Good: Descriptive names
"mylibrary.requests.process.duration"
"mylibrary.cache.hits.count"

// Avoid: Vague names
"metric1"
"duration"

7. Include Units

// Good: Units in description
meter.CreateHistogram<double>(
    name: "mylibrary.requests.duration",
    unit: "milliseconds",
    description: "Request processing duration in milliseconds.");

// Avoid: No units
meter.CreateHistogram<double>(
    name: "mylibrary.requests.duration",
    description: "Request duration.");

Integration with Observability Platforms

Application Insights

Metrics and traces automatically flow to Application Insights when configured:

// In your application (not library)
services.AddApplicationInsightsTelemetry();

Prometheus

Export metrics to Prometheus:

// In your application
services.AddOpenTelemetry()
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddPrometheusExporter());

Jaeger

Export traces to Jaeger:

// In your application
services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddJaegerExporter());

Example: Complete Observability

public class MyService
{
    private readonly ILogger<MyService> _logger;
    private readonly MyLibraryMetrics _metrics;

    public MyService(ILogger<MyService> logger, MyLibraryMetrics metrics)
    {
        _logger = logger;
        _metrics = metrics;
    }

    public async Task ProcessItemAsync(string itemId)
    {
        var sw = Stopwatch.StartNew();

        using var scope = LibraryTemplateDiagnostics.StartActivityScope(
            _logger,
            "ProcessItem",
            new[] { new KeyValuePair<string, object?>("item.id", itemId) });

        try
        {
            _logger.LogInformation("Processing item {ItemId}", itemId);

            // Your processing logic
            await DoWorkAsync(itemId);

            sw.Stop();
            _metrics.RecordRequestProcessingTime(sw.ElapsedMilliseconds);
            _metrics.IncrementRequestCounter();

            _logger.LogInformation("Item processed successfully in {Duration}ms", sw.ElapsedMilliseconds);
        }
        catch (Exception ex)
        {
            sw.Stop();
            _metrics.RecordError(ex.GetType().Name);

            _logger.LogError(ex, "Error processing item {ItemId}", itemId);
            throw;
        }
    }
}

Testing Metrics and Tracing

Testing Metrics

[TestClass]
public class MyLibraryMetricsTests
{
    [TestMethod]
    public void IncrementRequestCounter_ShouldIncrementCounter()
    {
        // Arrange
        var meterFactory = new TestMeterFactory();
        var metrics = new MyLibraryMetrics(meterFactory);

        // Act
        metrics.IncrementRequestCounter();

        // Assert
        // Verify counter increment using TestMeterFactory
    }
}

Testing Tracing

[TestClass]
public class TracingTests
{
    [TestMethod]
    public void StartActivity_ShouldCreateActivity()
    {
        // Arrange
        using var listener = new ActivityListener
        {
            ShouldListenTo = source => source.Name == "MyLibrary",
            Sample = (ref ActivityCreationOptions<ActivityContext> _) => ActivitySamplingResult.AllData
        };
        ActivitySource.AddActivityListener(listener);

        // Act
        using var activity = LibraryTemplateDiagnostics.StartActivity("TestOperation");

        // Assert
        Assert.IsNotNull(activity);
        Assert.AreEqual("TestOperation", activity.OperationName);
    }
}

Related Documentation

• Features - Feature overview
• Architecture - Design patterns
• Configuration - Configuration patterns
• Use Cases - Real-world examples

References

• OpenTelemetry Metrics
• OpenTelemetry Tracing
• .NET Metrics
• .NET Tracing
• ActivitySource Documentation