Metrics in ConnectSoft Microservice Template

Purpose & Overview

Metrics in the ConnectSoft Microservice Template provide quantitative measurements of system behavior, performance, and business operations. Built on OpenTelemetry Metrics API, the template enables teams to collect, aggregate, and export metrics for monitoring, alerting, and performance analysis.

Metrics provide:

• Performance Monitoring: Track latency, throughput, and resource utilization
• Business Metrics: Measure business operations and outcomes
• Error Tracking: Count failures and exceptions
• Resource Monitoring: Track CPU, memory, and other system resources
• SLO Tracking: Monitor service-level objectives and compliance
• Custom Metrics: Application-specific measurements for business intelligence
• Automatic Instrumentation: Built-in metrics for common operations
• Multiple Exporters: Support for Prometheus, OTLP, Application Insights, and more

Metrics Philosophy

Metrics are the quantitative foundation of observability, providing aggregated measurements over time that enable trend analysis, alerting, and capacity planning. The template uses OpenTelemetry Metrics API for standardized, vendor-neutral metrics collection, ensuring compatibility with modern observability platforms while enabling custom business metrics for domain-specific insights.

Architecture Overview

Metrics Stack

Application Code
    ↓
OpenTelemetry Metrics API
    ├── Automatic Instrumentation
    │   ├── Runtime Instrumentation (.NET runtime)
    │   ├── ASP.NET Core (HTTP requests)
    │   ├── HttpClient (outgoing HTTP)
    │   ├── Process (system resources)
    │   ├── MassTransit (messaging)
    │   ├── NServiceBus (messaging)
    │   ├── Orleans (actor metrics)
    │   └── Semantic Kernel (AI operations)
    ├── Custom Metrics
    │   ├── MicroserviceTemplateMetrics
    │   ├── FeatureAMetrics
    │   ├── BankAccountActorMetrics
    │   └── McpToolMetrics
    └── MeterProvider
        ├── Meter Creation
        ├── Metric Collection
        └── Aggregation
    ↓
Exporters
    ├── OTLP Exporter (OpenTelemetry Collector)
    ├── Console Exporter (development)
    ├── Prometheus Exporter (optional)
    └── Application Insights (via OTLP)
    ↓
Observability Backends
    ├── Prometheus
    ├── Grafana
    ├── Application Insights
    └── OpenTelemetry Collector

Metrics Components

Component	Purpose	Location
MeterProvider	Creates and manages meters	OpenTelemetry SDK
Meter	Factory for creating instruments	OpenTelemetry Metrics API
IMeterFactory	Dependency injection for meter creation	ASP.NET Core
Counter	Monotonically increasing metric	Custom metrics classes
Histogram	Distribution of values	Custom metrics classes
Gauge	Current value snapshot	Custom metrics classes
UpDownCounter	Value that can increase or decrease	Custom metrics classes
MicroserviceTemplateMetrics	Business metrics for aggregate roots	Metrics project
FeatureAMetrics	Feature-specific metrics	Metrics project
McpToolMetrics	MCP tool invocation metrics	Metrics project

Service Registration

Metrics Registration

AddMicroserviceMetrics:

// MicroserviceMetricsExtensions.cs
internal static IServiceCollection AddMicroserviceMetrics(this IServiceCollection services)
{
    ArgumentNullException.ThrowIfNull(services);

#if (UseNHibernate || UseMongoDb)
    services.AddSingleton<MicroserviceTemplateMetrics>();
#endif
    services.AddSingleton<FeatureAMetrics>();
#if UseOrleans
    services.AddSingleton<BankAccountActorMetrics>();
#endif
#if UseMCP
    services.AddSingleton<McpToolMetrics>();
#endif

#if (UseNHibernate || UseMongoDb)
    services.ActivateSingleton<MicroserviceTemplateMetrics>();
#endif
    services.ActivateSingleton<FeatureAMetrics>();
#if UseOrleans
    services.ActivateSingleton<BankAccountActorMetrics>();
#endif
#if UseMCP
    services.ActivateSingleton<McpToolMetrics>();
#endif

    return services;
}

Registration in Startup:

// MicroserviceRegistrationExtensions.cs
services.AddMicroserviceMetrics();

OpenTelemetry Metrics Configuration

Metrics Configuration:

// OpenTelemetryExtensions.cs
services.AddOpenTelemetry()
    .WithMetrics(metricsBuilder =>
    {
        metricsBuilder.ConfigureMicroserviceOpenTelemetryMetrics(
            OptionsExtensions.OpenTelemetryOptions);
    });

Metrics Builder Setup:

private static void ConfigureMicroserviceOpenTelemetryMetrics(
    this MeterProviderBuilder metricsBuilder, 
    OpenTelemetryOptions otelOptions)
{
    metricsBuilder
        // Automatic instrumentation
        .AddRuntimeInstrumentation()      // .NET runtime metrics
        .AddAspNetCoreInstrumentation()   // HTTP request metrics
        .AddProcessInstrumentation()      // Process metrics (CPU, memory)
        .AddHttpClientInstrumentation()   // HTTP client metrics

        // Framework meters
#if UseNServiceBus
        .AddMeter("NServiceBus.Core")
#endif
#if UseMassTransit
        .AddMeter(MassTransit.Monitoring.InstrumentationOptions.MeterName)
#endif
#if UseOrleans
        .AddMeter("Microsoft.Orleans")
#endif
#if UseSemanticKernel
        .AddMeter("Microsoft.SemanticKernel*")
#endif
#if (UseMicrosoftExtensionsAIOpenAIProvider || UseSemanticKernelOpenAIConnector)
        .AddMeter("OpenAI.*")
#endif
#if UseMicrosoftExtensionsAI
        .AddMeter("Experimental.Microsoft.Extensions.AI*")
#endif

        // Custom application meters
        .AddMeter(MicroserviceTemplateMetrics.MicroserviceTemplateMetricsMeterName)
        .AddMeter(FeatureAMetrics.FeatureAMetricsMeterName)
#if UseMCP
        .AddMeter(McpToolMetrics.McpToolMetricsMeterName)
#endif

        // Exporters
        .AddOtlpExporter(options =>
        {
            options.Protocol = (OtlpExportProtocol)otelOptions.OtlpExporter.OtlpExportProtocol;
            options.Endpoint = new Uri(otelOptions.OtlpExporter.Endpoint);
        })
        .AddOtlpExporter(options =>
        {
            // Seq OTLP exporter
            options.Protocol = (OtlpExportProtocol)otelOptions.OtlpSeqExporter.OtlpExportProtocol;
            options.Endpoint = new Uri($"{otelOptions.OtlpSeqExporter.Endpoint}/ingest/otlp/v1/logs");
        });

    if (otelOptions.EnableConsoleExporter)
    {
        metricsBuilder.AddConsoleExporter();
    }

    // Prometheus exporter (optional)
    // metricsBuilder.AddPrometheusExporter();
}

Metric Types

Counter

Purpose: Monotonically increasing metric for counting events.

Characteristics: - Only increases (never decreases) - Reset on application restart - Used for counting operations, requests, errors

Example:

public class MicroserviceTemplateMetrics
{
    private readonly Counter<long> added;

    public MicroserviceTemplateMetrics(IMeterFactory meterFactory)
    {
        var meter = meterFactory.Create(MicroserviceTemplateMetricsMeterName);

        this.added = meter.CreateCounter<long>(
            name: "connectsoft.microservicetemplate.aggregateroot.added",
            unit: "1",
            description: "Number of aggregate roots added");
    }

    public void AddMicroserviceAggregateRoot(string? tenantId, string? aggregate)
    {
        var tags = BuildTags(tenantId, aggregate);
        this.added.Add(1, tags);
    }
}

Usage:

// Increment counter
metrics.AddMicroserviceAggregateRoot(tenantId: "tenant-1", aggregate: "Order");

Histogram

Purpose: Track distribution of values (e.g., latency, sizes).

Characteristics: - Records individual measurements - Aggregates into buckets for percentile calculation - Used for latency, durations, sizes

Example:

public class MicroserviceTemplateMetrics
{
    private readonly Histogram<double> addDurationSeconds;

    public MicroserviceTemplateMetrics(IMeterFactory meterFactory)
    {
        var meter = meterFactory.Create(MicroserviceTemplateMetricsMeterName);

        this.addDurationSeconds = meter.CreateHistogram<double>(
            name: "connectsoft.microservicetemplate.aggregateroot.add.duration",
            unit: "s",
            description: "Add operation duration");
    }

    public void AddMicroserviceAggregateRoot(
        TimeSpan duration, 
        string? tenantId = null, 
        string? aggregate = null)
    {
        var tags = BuildTags(tenantId, aggregate);
        this.added.Add(1, tags);
        this.addDurationSeconds.Record(duration.TotalSeconds, tags);
    }
}

Usage:

var stopwatch = Stopwatch.StartNew();
// ... operation ...
stopwatch.Stop();

metrics.AddMicroserviceAggregateRoot(
    stopwatch.Elapsed, 
    tenantId: "tenant-1", 
    aggregate: "Order");

Benefits: - Calculate percentiles (p50, p95, p99, p999) - Understand value distribution - Track SLO compliance - Identify outliers

UpDownCounter

Purpose: Track value that can increase or decrease (e.g., active connections, queue length).

Characteristics: - Can increase or decrease - Represents current state - Used for counts that vary (e.g., active instances, queue size)

Example:

public class MicroserviceTemplateMetrics
{
    private readonly UpDownCounter<long> total;

    public MicroserviceTemplateMetrics(IMeterFactory meterFactory)
    {
        var meter = meterFactory.Create(MicroserviceTemplateMetricsMeterName);

        this.total = meter.CreateUpDownCounter<long>(
            name: "connectsoft.microservicetemplate.aggregateroot.total",
            unit: "1",
            description: "Live number of aggregate roots");
    }

    public void IncreaseTotalMicroserviceAggregateRoots(string? tenantId, string? aggregate)
    {
        var tags = BuildTags(tenantId, aggregate);
        this.total.Add(1, tags);
    }

    public void DecreaseTotalMicroserviceAggregateRoots(string? tenantId, string? aggregate)
    {
        var tags = BuildTags(tenantId, aggregate);
        this.total.Add(-1, tags);
    }
}

Usage:

// Create aggregate
metrics.IncreaseTotalMicroserviceAggregateRoots(tenantId: "tenant-1", aggregate: "Order");

// Delete aggregate
metrics.DecreaseTotalMicroserviceAggregateRoots(tenantId: "tenant-1", aggregate: "Order");

ObservableGauge

Purpose: Snapshot of current value (e.g., current memory usage, active connections).

Characteristics: - Provides current value on demand - Not cumulative - Used for current state measurements

Example:

public class ResourceMetrics
{
    private readonly ObservableGauge<double> currentMemory;

    public ResourceMetrics(IMeterFactory meterFactory)
    {
        var meter = meterFactory.Create("ConnectSoft.MicroserviceTemplate.Resources");

        this.currentMemory = meter.CreateObservableGauge<double>(
            name: "process.memory.usage",
            unit: "bytes",
            description: "Current memory usage",
            observeValues: ObserveCurrentMemory);
    }

    private Measurement<double> ObserveCurrentMemory()
    {
        var process = Process.GetCurrentProcess();
        return new Measurement<double>(
            process.WorkingSet64,
            new TagList { { "process.name", process.ProcessName } });
    }
}

Metrics Project

The Metrics Project (ConnectSoft.MicroserviceTemplate.Metrics) is a dedicated project that contains all custom business metrics classes for the microservice. It provides a clean, organized structure for defining domain-specific metrics following consistent patterns and conventions.

Project Structure

ConnectSoft.MicroserviceTemplate.Metrics/
├── ConnectSoft.MicroserviceTemplate.Metrics.csproj
├── MicroserviceTemplateMetrics.cs        # Aggregate root metrics
├── FeatureAMetrics.cs                    # Feature-specific metrics
├── BankAccountActorMetrics.cs            # Actor-specific metrics (Orleans)
├── DurationScope.cs                      # Timing scope helper
└── GlobalSuppressions.cs                 # Code analysis suppressions

Project Purpose

The Metrics project serves as:

• Centralized Metrics Definition: All custom business metrics in one location
• Domain Metrics: Metrics specific to business operations and aggregates
• Consistent Patterns: Standardized approach to creating metrics classes
• Reusability: Metrics classes can be injected and used across the application
• Testability: Metrics classes are easily mockable for unit testing
• Clean Architecture: Separates metrics definition from application logic

Metrics Class Architecture

Each metrics class follows a consistent structure:

1. Constants: Meter name and metric name constants
2. Constructor: Creates meter and instruments via IMeterFactory
3. Public Methods: Record metrics with optional tags and duration
4. Timing Scopes: Return IDisposable scopes for automatic timing
5. Tag Building: Private helper methods for constructing tags

Creating a New Metrics Class

Step 1: Define Constants

public class OrderMetrics
{
    // Meter name (used for registration)
    public const string OrderMetricsMeterName = "connectsoft.microservicetemplate.order";

    // Metric name constants
    public const string OrdersCreatedCounterName = 
        "connectsoft.microservicetemplate.order.created";

    public const string OrderCreationDurationName = 
        "connectsoft.microservicetemplate.order.creation.duration";

    // Module and component attributes for tags
    private const string ModuleAttribute = "ConnectSoft.MicroserviceTemplate";
    private const string ComponentAttribute = "Order";
}

Step 2: Create Instruments in Constructor

private readonly Counter<long> ordersCreated;
private readonly Histogram<double> orderCreationDurationSeconds;

public OrderMetrics(IMeterFactory meterFactory)
{
    var meter = meterFactory.Create(OrderMetricsMeterName);

    this.ordersCreated = meter.CreateCounter<long>(
        name: OrdersCreatedCounterName,
        unit: "1",
        description: "Number of orders created");

    this.orderCreationDurationSeconds = meter.CreateHistogram<double>(
        name: OrderCreationDurationName,
        unit: "s",
        description: "Order creation duration");
}

Step 3: Add Public Methods

/// <summary>
/// Records an order creation with optional duration and tags.
/// </summary>
public void RecordOrderCreated(
    TimeSpan? duration = null,
    string? tenantId = null,
    string? orderType = null)
{
    var tags = BuildTags(tenantId, orderType);
    this.ordersCreated.Add(1, tags);

    if (duration.HasValue)
    {
        this.orderCreationDurationSeconds.Record(duration.Value.TotalSeconds, tags);
    }
}

/// <summary>
/// Creates a timing scope for order creation.
/// </summary>
public IDisposable TimeOrderCreation(string? tenantId = null, string? orderType = null)
{
    return DurationScope.Seconds(
        this.orderCreationDurationSeconds, 
        BuildTags(tenantId, orderType));
}

Step 4: Add Tag Building Helper

private static KeyValuePair<string, object?>[] BuildTags(
    string? tenantId = null,
    string? orderType = null)
{
    var tags = new List<KeyValuePair<string, object?>>(4)
    {
        new ("module", ModuleAttribute),
        new ("component", ComponentAttribute),
    };

    if (!string.IsNullOrWhiteSpace(tenantId))
    {
        tags.Add(new ("tenant_id", tenantId));
    }

    if (!string.IsNullOrWhiteSpace(orderType))
    {
        tags.Add(new ("order_type", orderType));
    }

    return tags.ToArray();
}

Complete Example:

namespace ConnectSoft.MicroserviceTemplate.Metrics
{
    using System;
    using System.Collections.Generic;
    using System.Diagnostics.Metrics;

    /// <summary>
    /// Order-related business metrics.
    /// </summary>
    public class OrderMetrics
    {
        public const string OrderMetricsMeterName = "connectsoft.microservicetemplate.order";
        public const string OrdersCreatedCounterName = "connectsoft.microservicetemplate.order.created";
        public const string OrderCreationDurationName = "connectsoft.microservicetemplate.order.creation.duration";

        private const string ModuleAttribute = "ConnectSoft.MicroserviceTemplate";
        private const string ComponentAttribute = "Order";

        private readonly Counter<long> ordersCreated;
        private readonly Histogram<double> orderCreationDurationSeconds;

        public OrderMetrics(IMeterFactory meterFactory)
        {
            var meter = meterFactory.Create(OrderMetricsMeterName);

            this.ordersCreated = meter.CreateCounter<long>(
                name: OrdersCreatedCounterName,
                unit: "1",
                description: "Number of orders created");

            this.orderCreationDurationSeconds = meter.CreateHistogram<double>(
                name: OrderCreationDurationName,
                unit: "s",
                description: "Order creation duration");
        }

        public void RecordOrderCreated(
            TimeSpan? duration = null,
            string? tenantId = null,
            string? orderType = null)
        {
            var tags = BuildTags(tenantId, orderType);
            this.ordersCreated.Add(1, tags);

            if (duration.HasValue)
            {
                this.orderCreationDurationSeconds.Record(duration.Value.TotalSeconds, tags);
            }
        }

        public IDisposable TimeOrderCreation(string? tenantId = null, string? orderType = null)
        {
            return DurationScope.Seconds(
                this.orderCreationDurationSeconds,
                BuildTags(tenantId, orderType));
        }

        private static KeyValuePair<string, object?>[] BuildTags(
            string? tenantId = null,
            string? orderType = null)
        {
            var tags = new List<KeyValuePair<string, object?>>(4)
            {
                new ("module", ModuleAttribute),
                new ("component", ComponentAttribute),
            };

            if (!string.IsNullOrWhiteSpace(tenantId))
            {
                tags.Add(new ("tenant_id", tenantId));
            }

            if (!string.IsNullOrWhiteSpace(orderType))
            {
                tags.Add(new ("order_type", orderType));
            }

            return tags.ToArray();
        }
    }
}

Registering Metrics Classes

Step 1: Add to ApplicationModel

// MicroserviceMetricsExtensions.cs
internal static IServiceCollection AddMicroserviceMetrics(this IServiceCollection services)
{
    services.AddSingleton<OrderMetrics>();
    services.ActivateSingleton<OrderMetrics>();
    return services;
}

Step 2: Register Meter in OpenTelemetry

// OpenTelemetryExtensions.cs
metricsBuilder.AddMeter(OrderMetrics.OrderMetricsMeterName);

DurationScope Helper

The DurationScope class provides automatic timing for operations:

Purpose: Automatically records elapsed time when a scope is disposed.

Usage:

// Automatic timing on dispose
using (metrics.TimeOrderCreation(tenantId: "tenant-1", orderType: "standard"))
{
    // Operation automatically timed
    await CreateOrder(order);
}

Implementation:

public sealed class DurationScope : IDisposable
{
    private readonly Histogram<double> histogram;
    private readonly KeyValuePair<string, object?>[] tags;
    private readonly Stopwatch stopwatch;

    private DurationScope(Histogram<double> histogram, KeyValuePair<string, object?>[] tags)
    {
        this.histogram = histogram;
        this.tags = tags;
        this.stopwatch = Stopwatch.StartNew();
    }

    public static DurationScope Seconds(
        Histogram<double> histogram,
        params KeyValuePair<string, object?>[] tags)
    {
        return new DurationScope(histogram, tags);
    }

    public void Dispose()
    {
        this.stopwatch.Stop();
        this.histogram.Record(this.stopwatch.Elapsed.TotalSeconds, this.tags);
    }
}

Benefits: - Automatic timing on scope exit - Exception-safe (always records duration) - Consistent timing across codebase - Cleaner code (no manual stopwatch management)

Metrics Class Patterns

Pattern 1: CRUD Operations (MicroserviceTemplateMetrics)

Structure: Separate counters for Create, Read, Update, Delete operations with duration histograms and failure counters.

public class MicroserviceTemplateMetrics
{
    // Counters
    private readonly Counter<long> added;
    private readonly Counter<long> deleted;
    private readonly Counter<long> updated;
    private readonly Counter<long> read;

    // Duration histograms
    private readonly Histogram<double> addDurationSeconds;
    private readonly Histogram<double> deleteDurationSeconds;

    // Failure counters
    private readonly Counter<long> addFailed;

    // Methods
    public void AddMicroserviceAggregateRoot(TimeSpan duration, ...);
    public void AddMicroserviceAggregateRootFailed(...);
    public IDisposable TimeAdd(...);
}

Use Case: Aggregate root operations with comprehensive CRUD tracking.

Pattern 2: Feature Operations (FeatureAMetrics)

Structure: Success/failure counters with duration histogram and instance tracking.

public class FeatureAMetrics
{
    // Success/failure counters
    private readonly Counter<long> successes;
    private readonly Counter<long> failures;

    // Duration histogram
    private readonly Histogram<double> durationSeconds;

    // Instance tracking
    private readonly UpDownCounter<long> instances;

    // Methods
    public void RecordFeatureAUseCaseASuccess(TimeSpan? duration, ...);
    public void RecordFeatureAUseCaseAFailed(TimeSpan? duration, ...);
    public IDisposable TimeUseCaseASuccess(...);
    public IDisposable TimeUseCaseAFailure(...);
}

Use Case: Feature-specific use cases with success/failure tracking.

Pattern 3: Actor Operations (BankAccountActorMetrics)

Structure: Actor-specific metrics with business context (amounts, currency).

public class BankAccountActorMetrics
{
    // Operation counters
    private readonly Counter<long> withdrawSuccesses;
    private readonly Counter<long> withdrawFailures;

    // Duration and amount histograms
    private readonly Histogram<double> withdrawDurationSeconds;
    private readonly Histogram<double> withdrawAmount;

    // Instance tracking
    private readonly UpDownCounter<long> instances;

    // Methods
    public void RecordWithdrawSuccess(double amount, TimeSpan? duration, ...);
    public void RecordWithdrawFailure(double? amount, TimeSpan? duration, ...);
}

Use Case: Actor operations with business metrics (amounts, currencies).

Tag Building Patterns

Standard Tags

All metrics classes include standard tags:

private static KeyValuePair<string, object?>[] BuildTags(...)
{
    var tags = new List<KeyValuePair<string, object?>>(4)
    {
        new ("module", ModuleAttribute),      // Always included
        new ("component", ComponentAttribute), // Always included
    };

    // Optional tags
    if (!string.IsNullOrWhiteSpace(tenantId))
    {
        tags.Add(new ("tenant_id", tenantId));
    }

    // ... other optional tags

    return tags.ToArray();
}

Tag Best Practices

1. Always Include Module and Component

   new ("module", "ConnectSoft.MicroserviceTemplate"),
   new ("component", "Order"),
   

2. Use Low-Cardinality Values

   // ✅ GOOD - Limited values
   new ("result", "success"),  // success/failure only
   new ("order_type", "standard"),  // Limited order types
   
   // ❌ BAD - High cardinality
   new ("order_id", orderId),  // Unique per order
   new ("user_id", userId),    // Unique per user
   

3. Conditional Tags

   if (!string.IsNullOrWhiteSpace(tenantId))
   {
       tags.Add(new ("tenant_id", tenantId));
   }
   

4. Consistent Tag Names

   // ✅ GOOD - snake_case
   new ("tenant_id", tenantId),
   new ("order_type", orderType),
   
   // ❌ BAD - Mixed casing
   new ("TenantId", tenantId),
   new ("orderType", orderType),
   

Meter Naming Conventions

Meter Names: - Format: connectsoft.microservicetemplate.{component} - Example: connectsoft.microservicetemplate.order - Used for: Grouping related metrics

Metric Names: - Format: connectsoft.microservicetemplate.{component}.{operation}.{type} - Examples: - connectsoft.microservicetemplate.order.created - connectsoft.microservicetemplate.order.creation.duration - connectsoft.microservicetemplate.order.created.failed

Constants: - Meter name: {Class}MeterName - Metric names: {Metric}{Type}Name - Example: OrderMetricsMeterName, OrdersCreatedCounterName

Using Metrics in Application Code

Injection:

public class OrderProcessor
{
    private readonly OrderMetrics metrics;

    public OrderProcessor(OrderMetrics metrics)
    {
        this.metrics = metrics;
    }
}

Manual Timing:

public async Task<Order> CreateOrder(CreateOrderInput input)
{
    var stopwatch = Stopwatch.StartNew();
    try
    {
        var order = await SaveOrder(input);

        this.metrics.RecordOrderCreated(
            duration: stopwatch.Elapsed,
            tenantId: input.TenantId,
            orderType: input.OrderType);

        return order;
    }
    catch (Exception ex)
    {
        // Record failure
        this.metrics.RecordOrderCreatedFailed(
            tenantId: input.TenantId,
            orderType: input.OrderType);
        throw;
    }
}

Automatic Timing:

public async Task<Order> CreateOrder(CreateOrderInput input)
{
    using (this.metrics.TimeOrderCreation(
        tenantId: input.TenantId,
        orderType: input.OrderType))
    {
        var order = await SaveOrder(input);
        this.metrics.RecordOrderCreated(
            tenantId: input.TenantId,
            orderType: input.OrderType);
        return order;
    }
}

Testing Metrics Classes

Unit Testing:

[TestMethod]
public void OrderMetrics_ShouldRecordOrderCreated()
{
    // Arrange
    var meterFactory = new TestMeterFactory();
    var metrics = new OrderMetrics(meterFactory);

    // Act
    metrics.RecordOrderCreated(
        duration: TimeSpan.FromSeconds(1),
        tenantId: "tenant-1",
        orderType: "standard");

    // Assert
    var meter = meterFactory.GetMeter(OrderMetrics.OrderMetricsMeterName);
    // Verify metric was recorded
}

Mocking for Application Tests:

[TestMethod]
public void OrderProcessor_ShouldRecordMetrics_OnCreate()
{
    // Arrange
    var mockMetrics = new Mock<OrderMetrics>();
    var processor = new OrderProcessor(mockMetrics.Object);

    // Act
    await processor.CreateOrder(input);

    // Assert
    mockMetrics.Verify(m => m.RecordOrderCreated(
        It.IsAny<TimeSpan>(),
        It.IsAny<string>(),
        It.IsAny<string>()),
        Times.Once);
}

Metrics Project Best Practices

Do's

1. Follow Consistent Structure

   // ✅ GOOD - Consistent pattern
   public class OrderMetrics
   {
       public const string OrderMetricsMeterName = "...";
       private readonly Counter<long> ordersCreated;
   
       public OrderMetrics(IMeterFactory meterFactory) { ... }
       public void RecordOrderCreated(...) { ... }
       private static KeyValuePair<string, object?>[] BuildTags(...) { ... }
   }
   

2. Use DurationScope for Timing

   // ✅ GOOD - Automatic timing
   using (metrics.TimeOrderCreation(...))
   {
       await CreateOrder(...);
   }
   

3. Include Standard Tags

   // ✅ GOOD - Always include module and component
   new ("module", ModuleAttribute),
   new ("component", ComponentAttribute),
   

4. Document Metric Names

   /// <summary>
   /// Counter name for tracking the number of orders created.
   /// </summary>
   public const string OrdersCreatedCounterName = "...";
   

Don'ts

1. Don't Create Metrics in Hot Paths

   // ❌ BAD - Creating meter on every request
   public void Process()
   {
       var meter = new Meter("MyMeter");
       var counter = meter.CreateCounter<long>("requests");
   }
   
   // ✅ GOOD - Inject via DI
   public MyService(OrderMetrics metrics) { ... }
   

2. Don't Use High-Cardinality Tags

   // ❌ BAD - High cardinality
   tags.Add(new ("order_id", orderId));
   tags.Add(new ("user_id", userId));
   
   // ✅ GOOD - Low cardinality
   tags.Add(new ("order_type", orderType));
   tags.Add(new ("tenant_id", tenantId));
   

3. Don't Mix Business and Technical Metrics

   // ❌ BAD - Mixed concerns
   metrics.RecordMetric("order_created", 1, new TagList { { "http_status", "200" } });
   
   // ✅ GOOD - Separate metrics
   orderMetrics.RecordOrderCreated(...);
   httpMetrics.RecordHttpRequest(...);
   

Custom Metrics Implementation

MicroserviceTemplateMetrics

Purpose: Business metrics for aggregate root operations.

Metrics Provided:

Metric Type	Name	Description
Counter	aggregateroot.added	Count of aggregate roots created
Counter	aggregateroot.deleted	Count of aggregate roots deleted
Counter	aggregateroot.updated	Count of aggregate roots updated
Counter	aggregateroot.read	Count of aggregate roots read
UpDownCounter	aggregateroot.total	Current number of aggregate roots
Histogram	aggregateroot.add.duration	Duration of add operations
Histogram	aggregateroot.delete.duration	Duration of delete operations
Histogram	aggregateroot.update.duration	Duration of update operations
Histogram	aggregateroot.read.duration	Duration of read operations
Counter	aggregateroot.add.failed	Count of failed add operations
Counter	aggregateroot.delete.failed	Count of failed delete operations
Counter	aggregateroot.update.failed	Count of failed update operations
Counter	aggregateroot.read.failed	Count of failed read operations

Usage in Processor:

public class DefaultMicroserviceAggregateRootsProcessor
{
    private readonly MicroserviceTemplateMetrics meters;

    public async Task<IMicroserviceAggregateRoot> CreateMicroserviceAggregateRoot(
        CreateMicroserviceAggregateRootInput input,
        CancellationToken token = default)
    {
        var stopwatch = Stopwatch.StartNew();
        try
        {
            // Business logic
            var entity = await SaveNewEntity(input, token);

            // Record metrics
            this.meters.AddMicroserviceAggregateRoot(
                stopwatch.Elapsed, 
                tenantId: tenantId, 
                aggregate: aggregateName);
            this.meters.IncreaseTotalMicroserviceAggregateRoots(
                tenantId: tenantId, 
                aggregate: aggregateName);

            return entity;
        }
        catch (Exception ex)
        {
            // Record failure
            this.meters.AddMicroserviceAggregateRootFailed(
                tenantId: tenantId, 
                aggregate: aggregateName);
            throw;
        }
    }
}

FeatureAMetrics

Purpose: Feature-specific metrics for UseCaseA operations.

Metrics Provided:

Metric Type	Name	Description
Counter	featurea.usecasea.successes	Count of successful executions
Counter	featurea.usecasea.failures	Count of failed executions
Histogram	featurea.usecasea.duration	Duration of executions
UpDownCounter	featurea.instances	Current number of FeatureA instances

Usage:

public class FeatureAService
{
    private readonly FeatureAMetrics metrics;

    public async Task ExecuteUseCaseA(UseCaseAInput input)
    {
        var stopwatch = Stopwatch.StartNew();
        try
        {
            // Business logic
            await ProcessUseCaseA(input);

            // Record success with duration
            this.metrics.RecordFeatureAUseCaseASuccess(
                duration: stopwatch.Elapsed,
                tenantId: input.TenantId,
                reason: "completed");
        }
        catch (Exception ex)
        {
            // Record failure with duration
            this.metrics.RecordFeatureAUseCaseAFailed(
                duration: stopwatch.Elapsed,
                tenantId: input.TenantId,
                reason: "exception");
            throw;
        }
    }
}

Timing Scopes

DurationScope Pattern:

For automatic duration recording, use timing scopes:

public class MicroserviceTemplateMetrics
{
    public IDisposable TimeAdd(string? tenantId = null, string? aggregate = null)
    {
        return DurationScope.Seconds(
            this.addDurationSeconds, 
            BuildTags(tenantId, aggregate));
    }
}

Usage:

// Automatic timing on dispose
using (metrics.TimeAdd(tenantId: "tenant-1", aggregate: "Order"))
{
    // Operation automatically timed
    await CreateAggregate(input);
}

Automatic Instrumentation

Runtime Instrumentation

.NET Runtime Metrics:

metricsBuilder.AddRuntimeInstrumentation();

Collected Metrics: - GC collections (gen0, gen1, gen2) - GC pause time - Thread pool queue length - Thread pool threads - Exception count - Contention count - Timer resolution

ASP.NET Core Instrumentation

HTTP Request Metrics:

metricsBuilder.AddAspNetCoreInstrumentation();

Collected Metrics: - Request duration - Request count - Active requests - Response status codes

Process Instrumentation

Process Metrics:

metricsBuilder.AddProcessInstrumentation();

Collected Metrics: - CPU usage - Memory usage (working set, private bytes) - Virtual memory - Thread count - Handle count

HttpClient Instrumentation

Outgoing HTTP Metrics:

metricsBuilder.AddHttpClientInstrumentation();

Collected Metrics: - HTTP request duration - HTTP request count - HTTP response status codes - HTTP connection pool metrics

Tags and Dimensions

Tag Structure

Tags provide dimensions for metrics aggregation:

private static KeyValuePair<string, object?>[] BuildTags(
    string? tenantId, 
    string? aggregate)
{
    var tags = new List<KeyValuePair<string, object?>>(4)
    {
        new ("module", "ConnectSoft.MicroserviceTemplate"),
        new ("component", "AggregateRoot"),
    };

    if (!string.IsNullOrWhiteSpace(tenantId))
    {
        tags.Add(new ("tenant_id", tenantId));
    }

    if (!string.IsNullOrWhiteSpace(aggregate))
    {
        tags.Add(new ("aggregate", aggregate));
    }

    return tags.ToArray();
}

Tag Best Practices

1. Use Low-Cardinality Tags

   // ✅ GOOD - Low cardinality
   tags.Add(new ("result", "success"));  // success/failure only
   tags.Add(new ("reason", "timeout"));  // Limited set of reasons
   
   // ❌ BAD - High cardinality
   tags.Add(new ("user_id", userId));    // Unique per user
   tags.Add(new ("request_id", requestId)); // Unique per request
   

2. Standard Tag Names

   // ✅ GOOD - Standard names
   tags.Add(new ("tenant_id", tenantId));
   tags.Add(new ("aggregate", aggregate));
   tags.Add(new ("module", module));
   
   // ❌ BAD - Inconsistent naming
   tags.Add(new ("TenantId", tenantId));
   tags.Add(new ("AggregateName", aggregate));
   

3. Include Context Tags

   // ✅ GOOD - Contextual information
   var tags = new TagList
   {
       { "module", "ConnectSoft.MicroserviceTemplate" },
       { "component", "AggregateRoot" },
       { "tenant_id", tenantId },
       { "aggregate", aggregate }
   };
   

Metric Naming Conventions

OpenTelemetry Semantic Conventions

Metric Name Format:

<namespace>.<metric_name>

Examples:

connectsoft.microservicetemplate.aggregateroot.added
connectsoft.microservicetemplate.aggregateroot.add.duration
connectsoft.microservicetemplate.featurea.usecasea.successes

Naming Best Practices

1. Use Dot Notation

   // ✅ GOOD
   "connectsoft.microservicetemplate.aggregateroot.added"
   
   // ❌ BAD
   "connectsoft_microservicetemplate_aggregateroot_added"
   

2. Use Lowercase

   // ✅ GOOD
   "aggregateroot.added"
   
   // ❌ BAD
   "AggregateRoot.Added"
   

3. Include Units in Name

   // ✅ GOOD
   "aggregateroot.add.duration"  // Duration in seconds
   "process.memory.usage.bytes"  // Memory in bytes
   
   // ❌ BAD
   "aggregateroot.add"  // Unclear unit
   

Metric Exporters

OTLP Exporter

OpenTelemetry Protocol Exporter:

metricsBuilder.AddOtlpExporter(options =>
{
    options.Protocol = OtlpExportProtocol.Grpc;
    options.Endpoint = new Uri("https://otel-collector:4317");
});

Supported Backends: - OpenTelemetry Collector - Jaeger - Zipkin - Application Insights (via collector) - Prometheus (via collector)

Console Exporter

Development Exporter:

if (otelOptions.EnableConsoleExporter)
{
    metricsBuilder.AddConsoleExporter();
}

Output: Metrics printed to console (useful for debugging).

Prometheus Exporter

Prometheus Scraping Endpoint:

// Uncomment to enable Prometheus exporter
// metricsBuilder.AddPrometheusExporter();

// In middleware
// application.MapPrometheusScrapingEndpoint();

Access: GET /metrics endpoint for Prometheus scraping.

Querying Metrics

Prometheus Queries

Rate Calculation:

# Requests per second
rate(connectsoft_microservicetemplate_aggregateroot_added_total[5m])

# Error rate
rate(connectsoft_microservicetemplate_aggregateroot_add_failed_total[5m])

Percentiles:

# p95 latency
histogram_quantile(0.95, 
    rate(connectsoft_microservicetemplate_aggregateroot_add_duration_bucket[5m]))

# p99 latency
histogram_quantile(0.99, 
    rate(connectsoft_microservicetemplate_aggregateroot_add_duration_bucket[5m]))

Filtering by Tags:

# By tenant
connectsoft_microservicetemplate_aggregateroot_added_total{tenant_id="tenant-1"}

# By aggregate
connectsoft_microservicetemplate_aggregateroot_added_total{aggregate="Order"}

Application Insights Queries

Kusto Queries:

// Custom metrics
customMetrics
| where name == "connectsoft.microservicetemplate.aggregateroot.added"
| summarize count() by bin(timestamp, 1m)

// Percentiles
customMetrics
| where name == "connectsoft.microservicetemplate.aggregateroot.add.duration"
| summarize percentiles(value, 50, 95, 99) by bin(timestamp, 1m)

Testing Metrics

Unit Testing

Mock Metrics:

[TestMethod]
public void Processor_ShouldRecordMetrics_OnCreate()
{
    // Arrange
    var mockMetrics = new Mock<MicroserviceTemplateMetrics>();
    var processor = new DefaultMicroserviceAggregateRootsProcessor(
        mockMetrics.Object,
        /* other dependencies */);

    // Act
    await processor.CreateMicroserviceAggregateRoot(input);

    // Assert
    mockMetrics.Verify(m => m.AddMicroserviceAggregateRoot(
        It.IsAny<TimeSpan>(),
        It.IsAny<string>(),
        It.IsAny<string>()), 
        Times.Once);
}

Integration Testing

Verify Metrics Collection:

[TestMethod]
public async Task Metrics_ShouldBeExported_ToPrometheus()
{
    // Arrange
    var client = factory.CreateClient();

    // Act
    await client.PostAsync("/api/aggregates", content);

    var metricsResponse = await client.GetAsync("/metrics");
    var metricsText = await metricsResponse.Content.ReadAsStringAsync();

    // Assert
    Assert.IsTrue(metricsText.Contains("aggregateroot_added_total"));
}

Best Practices

Do's

1. Use Appropriate Metric Types

   // ✅ GOOD - Counter for counting events
   private readonly Counter<long> requests;
   
   // ✅ GOOD - Histogram for durations
   private readonly Histogram<double> duration;
   
   // ✅ GOOD - UpDownCounter for current state
   private readonly UpDownCounter<long> activeConnections;
   

2. Include Context in Tags

   // ✅ GOOD - Contextual tags
   this.counter.Add(1, new TagList
   {
       { "tenant_id", tenantId },
       { "aggregate", aggregate },
       { "result", "success" }
   });
   

3. Use Timing Scopes for Duration

   // ✅ GOOD - Automatic timing
   using (metrics.TimeAdd(tenantId, aggregate))
   {
       await CreateAggregate(input);
   }
   

4. Record Metrics for Both Success and Failure

   // ✅ GOOD - Track both outcomes
   try
   {
       await Process();
       metrics.RecordSuccess(duration);
   }
   catch (Exception ex)
   {
       metrics.RecordFailure(duration);
       throw;
   }
   

5. Use Low-Cardinality Tags

   // ✅ GOOD - Limited tag values
   tags.Add(new ("result", "success"));  // success/failure only
   tags.Add(new ("reason", "timeout"));  // Limited reasons
   

Don'ts

1. Don't Use High-Cardinality Tags

   // ❌ BAD - High cardinality
   tags.Add(new ("user_id", userId));       // Unique per user
   tags.Add(new ("request_id", requestId)); // Unique per request
   
   // ✅ GOOD - Low cardinality
   tags.Add(new ("tenant_id", tenantId));   // Limited tenants
   tags.Add(new ("aggregate", aggregate));  // Limited aggregates
   

2. Don't Create Metrics in Hot Paths

   // ❌ BAD - Creating meter on every request
   public void Process()
   {
       var meter = new Meter("MyMeter");
       var counter = meter.CreateCounter<long>("requests");
       counter.Add(1);
   }
   
   // ✅ GOOD - Create once, inject via DI
   public MyService(MicroserviceTemplateMetrics metrics)
   {
       this.metrics = metrics;
   }
   

3. Don't Mix Business and Technical Metrics

   // ❌ BAD - Mixed concerns
   metrics.RecordMetric("user_login", 1, new TagList { { "user_id", userId } });
   
   // ✅ GOOD - Separate metrics
   businessMetrics.RecordUserLogin(userId);
   technicalMetrics.RecordHttpRequest(method, path, statusCode);
   

4. Don't Record Metrics Synchronously in Critical Paths

   // ❌ BAD - Synchronous in hot path
   metrics.RecordDuration(sw.Elapsed); // Blocks
   
   // ✅ GOOD - Metrics are async by design in OpenTelemetry
   // Or use fire-and-forget if needed
   _ = Task.Run(() => metrics.RecordDuration(sw.Elapsed));
   

Troubleshooting

Issue: Metrics Not Appearing

Symptoms: Metrics not showing in observability backend.

Solutions: 1. Verify meter is registered: AddMeter("MeterName") 2. Check exporter configuration 3. Verify metrics class is registered in DI: AddSingleton<MetricsClass>() 4. Check network connectivity to exporter endpoint 5. Enable console exporter for debugging

Issue: High Cardinality

Symptoms: Too many unique metric series.

Solutions: 1. Review tag values for high cardinality 2. Remove or aggregate high-cardinality tags 3. Use sampling for high-volume metrics 4. Consider using logs for high-cardinality data

Issue: Missing Metrics

Symptoms: Some metrics not collected.

Solutions: 1. Verify meter name matches in AddMeter() 2. Check metrics class is activated: ActivateSingleton<MetricsClass>() 3. Verify metric names match exactly 4. Check metric is being called in code path

Summary

Metrics in the ConnectSoft Microservice Template provide:

• ✅ Multiple Metric Types: Counter, Histogram, UpDownCounter, ObservableGauge
• ✅ Automatic Instrumentation: Runtime, ASP.NET Core, HttpClient, Process
• ✅ Custom Business Metrics: Domain-specific metrics for business operations
• ✅ OpenTelemetry Integration: Vendor-neutral metrics collection
• ✅ Multiple Exporters: OTLP, Prometheus, Console, Application Insights
• ✅ Tag Support: Contextual dimensions for metric aggregation
• ✅ Timing Scopes: Automatic duration recording
• ✅ Testing Support: Mockable metrics for unit testing
• ✅ Best Practices: Low-cardinality tags, appropriate metric types

By following these patterns, teams can:

• Monitor Performance: Track latency, throughput, and resource usage
• Measure Business Operations: Track domain-specific metrics
• Alert on Issues: Set up alerts based on metric thresholds
• Plan Capacity: Use metrics for capacity planning and optimization
• Debug Issues: Use metrics to identify performance problems
• Track SLOs: Monitor service-level objectives with metrics

Metrics are a critical component of observability, providing quantitative measurements that enable teams to understand system behavior, track business outcomes, and ensure reliable service delivery.