Resource Monitoring in ConnectSoft Microservice Template¶

Purpose & Overview¶

Resource Monitoring in the ConnectSoft Microservice Template provides continuous measurement and tracking of resource utilization (CPU, memory, disk) to ensure optimal performance, detect resource constraints, and prevent service degradation. The template uses Microsoft.Extensions.Diagnostics.ResourceMonitoring for real-time resource monitoring and integrates it with health checks, metrics, and observability systems.

Resource monitoring provides:

Real-Time Resource Tracking: Continuous measurement of CPU and memory utilization
Health Check Integration: Resource utilization health checks with configurable thresholds
OpenTelemetry Integration: Automatic metrics export to observability backends
Threshold-Based Alerts: Degraded and unhealthy states based on resource usage
System-Level Monitoring: Memory, disk, and garbage collection tracking
Cross-Platform Support: Windows and Linux resource monitoring (with platform-specific considerations)
Capacity Planning: Historical resource usage data for capacity planning

Resource Monitoring Philosophy

Resource monitoring is essential for maintaining service health and preventing resource exhaustion. By continuously tracking CPU, memory, and disk usage with configurable thresholds, the template enables proactive alerting and automatic scaling decisions. Resource monitoring integrates seamlessly with health checks and observability systems, ensuring that resource constraints are detected early and addressed before they impact service availability.

Architecture Overview¶

Resource Monitoring Stack¶

Application Process
    ↓
Microsoft.Extensions.Diagnostics.ResourceMonitoring
    ├── IResourceMonitor Interface
    │   ├── GetUtilization() → ResourceUtilization
    │   ├── CPU Usage Tracking
    │   └── Memory Usage Tracking
    ├── Automatic Instrumentation
    │   ├── CPU Metrics
    │   ├── Memory Metrics
    │   └── OpenTelemetry Export
    └── Health Check Integration
        ├── ResourceUtilizationHealthCheck
        ├── CPU Thresholds
        └── Memory Thresholds
    ↓
Observability Systems
    ├── OpenTelemetry (Metrics)
    ├── Health Checks (/health endpoint)
    ├── Application Insights
    └── Prometheus (optional)

Resource Monitoring Components¶

Component	Purpose	Location
IResourceMonitor	Interface for retrieving resource utilization data	`Microsoft.Extensions.Diagnostics.ResourceMonitoring`
ResourceUtilizationHealthCheck	Health check that monitors CPU and memory thresholds	`HealthChecks.Extensions.ResourceMonitoring`
ResourceMonitoringExtensions	Service registration extensions	`ApplicationModel` project
HealthChecksExtensions	Health check registration with resource monitoring	`ApplicationModel` project

Service Registration¶

AddMicroserviceResourceMonitoring Extension¶

Resource monitoring is registered via AddMicroserviceResourceMonitoring():

// ResourceMonitoringExtensions.cs
internal static IServiceCollection AddMicroserviceResourceMonitoring(this IServiceCollection services)
{
    ArgumentNullException.ThrowIfNull(services);

    // Resource monitoring involves the continuous measurement of resource utilization
    // Currently Windows-only due to Linux bug: https://github.com/dotnet/extensions/issues/4885
    if (OperatingSystem.IsWindows())
    {
        services.AddResourceMonitoring();
    }

    return services;
}

Registration in MicroserviceRegistrationExtensions:

// MicroserviceRegistrationExtensions.cs
services.AddMicroserviceResourceMonitoring();

Platform Support¶

Windows: Fully supported - CPU monitoring via performance counters - Memory monitoring via process working set - Full OpenTelemetry integration

Linux: Limited support - Known issue: https://github.com/dotnet/extensions/issues/4885 - Resource monitoring is conditionally disabled on Linux - Alternative: Use system-level health checks (disk, memory via GC)

Resource Utilization Health Check¶

Configuration¶

Resource utilization health checks are configured with CPU and memory thresholds:

// HealthChecksExtensions.cs
#if ResourceMonitoring
private static IHealthChecksBuilder AddAndConfigureResourceUtilizationHealthCheck(
    this IHealthChecksBuilder healthChecksBuilder)
{
    if (OperatingSystem.IsWindows())
    {
        return healthChecksBuilder
            .AddResourceUtilizationHealthCheck(
                options =>
                {
                    options.CpuThresholds = new ResourceUsageThresholds
                    {
                        DegradedUtilizationPercentage = 80,
                        UnhealthyUtilizationPercentage = 90,
                    };
                    options.MemoryThresholds = new ResourceUsageThresholds
                    {
                        DegradedUtilizationPercentage = 80,
                        UnhealthyUtilizationPercentage = 90,
                    };
                },
                tags: "resource utilization");
    }

    return healthChecksBuilder;
}
#endif

Threshold Configuration¶

ResourceUsageThresholds:

Property	Type	Description	Default
`DegradedUtilizationPercentage`	`int`	Percentage at which resource is considered degraded	`80`
`UnhealthyUtilizationPercentage`	`int`	Percentage at which resource is considered unhealthy	`90`

Health Status Mapping:

CPU/Memory Usage	Health Status	Description
< Degraded Threshold	`Healthy`	Resource usage is normal
>= Degraded, < Unhealthy	`Degraded`	Resource usage is elevated but acceptable
>= Unhealthy Threshold	`Unhealthy`	Resource usage is critical

Health Check Response¶

Example Response:

{
  "status": "Degraded",
  "totalDuration": "00:00:00.0123456",
  "entries": {
    "resource-utilization": {
      "status": "Degraded",
      "description": "CPU utilization is at 85%, exceeding degraded threshold of 80%",
      "duration": "00:00:00.0123456",
      "data": {
        "cpu_usage_percentage": "85",
        "memory_usage_percentage": "72",
        "cpu_threshold_degraded": "80",
        "cpu_threshold_unhealthy": "90",
        "memory_threshold_degraded": "80",
        "memory_threshold_unhealthy": "90"
      }
    }
  }
}

Using IResourceMonitor¶

Retrieving Resource Utilization¶

Inject IResourceMonitor to retrieve real-time resource utilization data:

public class ResourceMonitoringService
{
    private readonly IResourceMonitor resourceMonitor;
    private readonly ILogger<ResourceMonitoringService> logger;

    public ResourceMonitoringService(
        IResourceMonitor resourceMonitor,
        ILogger<ResourceMonitoringService> logger)
    {
        this.resourceMonitor = resourceMonitor;
        this.logger = logger;
    }

    public void LogResourceUsage()
    {
        var utilization = this.resourceMonitor.GetUtilization();

        this.logger.LogInformation(
            "Resource utilization - CPU: {CpuUsage}%, Memory: {MemoryUsage}%",
            utilization.CpuUsedPercentage,
            utilization.MemoryUsedPercentage);
    }
}

ResourceUtilization Properties¶

ResourceUtilization provides:

Property	Type	Description
`CpuUsedPercentage`	`double`	CPU utilization percentage (0-100)
`MemoryUsedPercentage`	`double`	Memory utilization percentage (0-100)
`CpuUsedUnits`	`double`	CPU usage in abstract units
`MemoryUsedUnits`	`double`	Memory usage in abstract units
`CpuTotalUnits`	`double`	Total CPU capacity in abstract units
`MemoryTotalUnits`	`double`	Total memory capacity in abstract units

Example Usage:

var utilization = resourceMonitor.GetUtilization();

if (utilization.CpuUsedPercentage > 80)
{
    logger.LogWarning(
        "High CPU usage detected: {CpuUsage}%",
        utilization.CpuUsedPercentage);
}

if (utilization.MemoryUsedPercentage > 80)
{
    logger.LogWarning(
        "High memory usage detected: {MemoryUsage}%",
        utilization.MemoryUsedPercentage);
}

System-Level Health Checks¶

Memory Health Checks¶

The template includes multiple memory health checks:

// HealthChecksExtensions.cs
private static void AddSystemHealthChecks(this IHealthChecksBuilder builder)
{
    const int oneHoundredMb = 104857600; // 100 MB

    // Process allocated memory check (cross-platform)
    builder.AddProcessAllocatedMemoryHealthCheck(
        maximumMegabytesAllocated: 1024, 
        tags: new string[] { "allocatedmemory" });

    // Windows-specific memory checks
    if (OperatingSystem.IsWindows())
    {
        builder
            .AddPrivateMemoryHealthCheck(
                Process.GetCurrentProcess().PrivateMemorySize64 + (oneHoundredMb * 10), 
                tags: new string[] { "privatememory" })
            .AddVirtualMemorySizeHealthCheck(
                Process.GetCurrentProcess().VirtualMemorySize64 + (oneHoundredMb * 10), 
                tags: new string[] { "virtualmemory" });
    }
    else
    {
        // Cross-platform memory check using GC
        builder.AddCrossPlatformMemoryHealthCheck();
    }
}

Cross-Platform Memory Check¶

Implementation:

private static void AddCrossPlatformMemoryCheck(this IHealthChecksBuilder builder)
{
    const long threshold = 1024L * 1024L * 1024L; // 1 GB

    builder.AddCheck(
        "Memory",
        () =>
        {
            var totalMemory = GC.GetTotalMemory(forceFullCollection: false);

            return totalMemory < threshold
                ? HealthCheckResult.Healthy($"Total memory is {totalMemory / 1_000_000} MB.")
                : HealthCheckResult.Unhealthy($"Total memory is {totalMemory / 1_000_000} MB, exceeds threshold of {threshold / 1_000_000} MB.");
        },
        tags: new string[] { "memory" });
}

Disk Storage Health Check¶

Disk storage monitoring:

builder.AddDiskStorageHealthCheck(storage =>
{
    if (OperatingSystem.IsWindows())
    {
        string rootDrive = Path.GetPathRoot(Environment.SystemDirectory);
        storage.AddDrive(rootDrive, 1024 * 5); // 5 GB free minimum
    }
    else if (OperatingSystem.IsLinux())
    {
        storage.AddDrive("/", 1024 * 5); // 5 GB free minimum
    }
    else
    {
        storage.AddDrive("//", 1024 * 5); // 5 GB free minimum
    }
}, tags: new string[] { "diskstorage" });

OpenTelemetry Integration¶

Automatic Metrics Export¶

Resource monitoring automatically publishes metrics to OpenTelemetry:

Metrics Exported:

Metric Name	Type	Description
`process.cpu.usage`	Gauge	CPU utilization percentage
`process.memory.usage`	Gauge	Memory utilization percentage
`process.cpu.time`	Counter	CPU time in abstract units
`process.memory.usage_bytes`	Gauge	Memory usage in bytes

Automatic Export: - No manual configuration required - Metrics automatically exported to configured OpenTelemetry exporters - Available in Prometheus, Application Insights, OTLP collectors

Custom Resource Metrics¶

Create custom resource metrics using OpenTelemetry:

public class ResourceMetrics
{
    private readonly ObservableGauge<double> cpuUsage;
    private readonly ObservableGauge<double> memoryUsage;
    private readonly IResourceMonitor resourceMonitor;

    public ResourceMetrics(IMeterFactory meterFactory, IResourceMonitor resourceMonitor)
    {
        var meter = meterFactory.Create("ConnectSoft.MicroserviceTemplate.Resources");
        this.resourceMonitor = resourceMonitor;

        this.cpuUsage = meter.CreateObservableGauge<double>(
            name: "process.cpu.usage",
            unit: "%",
            description: "CPU utilization percentage",
            observeValues: ObserveCpuUsage);

        this.memoryUsage = meter.CreateObservableGauge<double>(
            name: "process.memory.usage",
            unit: "%",
            description: "Memory utilization percentage",
            observeValues: ObserveMemoryUsage);
    }

    private Measurement<double> ObserveCpuUsage()
    {
        var utilization = this.resourceMonitor.GetUtilization();
        return new Measurement<double>(utilization.CpuUsedPercentage);
    }

    private Measurement<double> ObserveMemoryUsage()
    {
        var utilization = this.resourceMonitor.GetUtilization();
        return new Measurement<double>(utilization.MemoryUsedPercentage);
    }
}

Configuration¶

Resource Monitoring Options¶

Resource monitoring can be configured via appsettings.json:

{
  "ResourceMonitoring": {
    "Enabled": true,
    "HealthCheck": {
      "Enabled": true,
      "CpuThresholds": {
        "DegradedUtilizationPercentage": 80,
        "UnhealthyUtilizationPercentage": 90
      },
      "MemoryThresholds": {
        "DegradedUtilizationPercentage": 80,
        "UnhealthyUtilizationPercentage": 90
      }
    }
  }
}

Environment-Specific Configuration¶

Development:

{
  "ResourceMonitoring": {
    "HealthCheck": {
      "CpuThresholds": {
        "DegradedUtilizationPercentage": 70,
        "UnhealthyUtilizationPercentage": 85
      },
      "MemoryThresholds": {
        "DegradedUtilizationPercentage": 70,
        "UnhealthyUtilizationPercentage": 85
      }
    }
  }
}

Production:

{
  "ResourceMonitoring": {
    "HealthCheck": {
      "CpuThresholds": {
        "DegradedUtilizationPercentage": 80,
        "UnhealthyUtilizationPercentage": 90
      },
      "MemoryThresholds": {
        "DegradedUtilizationPercentage": 80,
        "UnhealthyUtilizationPercentage": 90
      }
    }
  }
}

Best Practices¶

Do's¶

Configure Appropriate Thresholds

// ✅ GOOD - Realistic thresholds based on workload
options.CpuThresholds = new ResourceUsageThresholds
{
    DegradedUtilizationPercentage = 80,
    UnhealthyUtilizationPercentage = 90,
};

Monitor Both CPU and Memory

// ✅ GOOD - Monitor both resources
options.CpuThresholds = new ResourceUsageThresholds { ... };
options.MemoryThresholds = new ResourceUsageThresholds { ... };

Use Health Checks for Orchestration

// ✅ GOOD - Health checks for Kubernetes/load balancers
builder.AddResourceUtilizationHealthCheck(options => { ... }, tags: "ready");

Export Metrics to Observability Systems

// ✅ GOOD - Automatic OpenTelemetry export
services.AddResourceMonitoring(); // Automatically exports to OpenTelemetry

Handle Platform-Specific Limitations

// ✅ GOOD - Conditional registration for Windows
if (OperatingSystem.IsWindows())
{
    services.AddResourceMonitoring();
}

Log Resource Utilization

// ✅ GOOD - Log resource usage for debugging
var utilization = resourceMonitor.GetUtilization();
logger.LogInformation(
    "CPU: {CpuUsage}%, Memory: {MemoryUsage}%",
    utilization.CpuUsedPercentage,
    utilization.MemoryUsedPercentage);

Don'ts¶

Don't Set Thresholds Too Low

// ❌ BAD - Too aggressive thresholds
options.CpuThresholds = new ResourceUsageThresholds
{
    DegradedUtilizationPercentage = 50,  // Too low
    UnhealthyUtilizationPercentage = 60,
};

// ✅ GOOD - Realistic thresholds
options.CpuThresholds = new ResourceUsageThresholds
{
    DegradedUtilizationPercentage = 80,
    UnhealthyUtilizationPercentage = 90,
};

Don't Ignore Platform Limitations

// ❌ BAD - Assumes Linux support
services.AddResourceMonitoring(); // May fail on Linux

// ✅ GOOD - Conditional registration
if (OperatingSystem.IsWindows())
{
    services.AddResourceMonitoring();
}

Don't Poll Too Frequently

// ❌ BAD - Too frequent polling
while (true)
{
    var utilization = resourceMonitor.GetUtilization();
    Thread.Sleep(100); // Too frequent
}

// ✅ GOOD - Reasonable polling interval
while (true)
{
    var utilization = resourceMonitor.GetUtilization();
    await Task.Delay(TimeSpan.FromSeconds(5));
}

Don't Skip Health Check Integration

// ❌ BAD - No health check integration
services.AddResourceMonitoring();
// No health check registration

// ✅ GOOD - Integrated with health checks
services.AddResourceMonitoring();
builder.AddResourceUtilizationHealthCheck(options => { ... });

Common Scenarios¶

Scenario 1: Monitoring High-Resource Operations¶

Use Case: Monitor resource usage during batch processing operations.

public class BatchProcessor
{
    private readonly IResourceMonitor resourceMonitor;
    private readonly ILogger<BatchProcessor> logger;

    public async Task ProcessBatchAsync(IEnumerable<Item> items)
    {
        var initialUtilization = this.resourceMonitor.GetUtilization();

        this.logger.LogInformation(
            "Starting batch processing - CPU: {Cpu}%, Memory: {Memory}%",
            initialUtilization.CpuUsedPercentage,
            initialUtilization.MemoryUsedPercentage);

        foreach (var item in items)
        {
            // Process item
            await ProcessItemAsync(item);

            // Check resource usage periodically
            var currentUtilization = this.resourceMonitor.GetUtilization();
            if (currentUtilization.CpuUsedPercentage > 90)
            {
                this.logger.LogWarning(
                    "High CPU usage detected: {Cpu}%",
                    currentUtilization.CpuUsedPercentage);

                // Add delay to reduce load
                await Task.Delay(TimeSpan.FromSeconds(1));
            }
        }
    }
}

Scenario 2: Health Check-Based Scaling¶

Use Case: Kubernetes horizontal pod autoscaling based on resource utilization.

# Kubernetes HPA configuration
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: microservice-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: microservice
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

Scenario 3: Resource-Based Circuit Breaking¶

Use Case: Prevent operations when resources are critically low.

public class ResourceAwareService
{
    private readonly IResourceMonitor resourceMonitor;
    private readonly ILogger<ResourceAwareService> logger;

    public async Task<bool> CanProcessRequestAsync()
    {
        var utilization = this.resourceMonitor.GetUtilization();

        if (utilization.CpuUsedPercentage > 90 || 
            utilization.MemoryUsedPercentage > 90)
        {
            this.logger.LogWarning(
                "Resource utilization too high - CPU: {Cpu}%, Memory: {Memory}%",
                utilization.CpuUsedPercentage,
                utilization.MemoryUsedPercentage);

            return false; // Circuit breaker: reject requests
        }

        return true; // Proceed with request
    }
}

Troubleshooting¶

Issue: Resource Monitoring Not Working on Linux¶

Symptoms: Resource monitoring not collecting data on Linux.

Solutions: 1. Known Limitation: Linux support has a known bug (https://github.com/dotnet/extensions/issues/4885) 2. Use Alternative Checks: Use system-level health checks (GC memory, disk) 3. Use External Monitoring: Use Kubernetes metrics or Prometheus node exporter

Issue: Health Check Always Unhealthy¶

Symptoms: Resource utilization health check always reports unhealthy.

Solutions: 1. Check Thresholds: Verify thresholds are appropriate for your workload 2. Review Actual Usage: Check actual CPU/memory usage vs. thresholds 3. Adjust Thresholds: Increase thresholds if they're too aggressive 4. Check Platform: Ensure running on Windows (Linux has limitations)

Issue: Metrics Not Appearing in OpenTelemetry¶

Symptoms: Resource metrics not exported to observability backend.

Solutions: 1. Verify OpenTelemetry Registration: Ensure OpenTelemetry is configured 2. Check Meter Registration: Verify meter is registered in MeterProvider 3. Review Exporter Configuration: Ensure exporters are properly configured 4. Check Logs: Review OpenTelemetry logs for export errors

Issue: High CPU Usage from Monitoring¶

Symptoms: Resource monitoring itself consumes significant CPU.

Solutions: 1. Reduce Polling Frequency: Poll less frequently 2. Use Health Checks: Rely on health checks instead of frequent polling 3. Use Async Operations: Ensure monitoring operations are async 4. Review Thresholds: Adjust thresholds to reduce unnecessary checks

Health Checks: Comprehensive health check documentation including resource monitoring integration
Metrics: Metrics collection and OpenTelemetry integration
Application Insights: Application Insights performance counters and resource monitoring
OpenTelemetry: OpenTelemetry metrics and resource monitoring export

Summary¶

Resource monitoring in the ConnectSoft Microservice Template provides:

✅ Real-Time Tracking: Continuous CPU and memory utilization monitoring
✅ Health Check Integration: Resource utilization health checks with configurable thresholds
✅ OpenTelemetry Export: Automatic metrics export to observability backends
✅ System-Level Monitoring: Memory, disk, and GC health checks
✅ Platform Support: Windows support with Linux alternatives
✅ Threshold-Based Alerts: Degraded and unhealthy states for proactive alerting
✅ Capacity Planning: Resource usage data for capacity planning and scaling decisions

By following these patterns, teams can:

Monitor Resource Health: Continuous tracking of CPU, memory, and disk usage
Prevent Resource Exhaustion: Early detection of resource constraints
Enable Auto-Scaling: Health check-based scaling decisions
Optimize Performance: Identify and address resource bottlenecks
Plan Capacity: Historical data for capacity planning

Resource monitoring is essential for maintaining service health, preventing resource exhaustion, and enabling data-driven scaling and capacity planning decisions.

Resource Monitoring in ConnectSoft Microservice Template¶

Purpose & Overview¶

Architecture Overview¶

Resource Monitoring Stack¶

Resource Monitoring Components¶

Service Registration¶

AddMicroserviceResourceMonitoring Extension¶

Platform Support¶

Resource Utilization Health Check¶

Configuration¶

Threshold Configuration¶

Health Check Response¶

Using IResourceMonitor¶

Retrieving Resource Utilization¶

ResourceUtilization Properties¶

System-Level Health Checks¶

Memory Health Checks¶

Cross-Platform Memory Check¶

Disk Storage Health Check¶

OpenTelemetry Integration¶

Automatic Metrics Export¶

Custom Resource Metrics¶

Configuration¶

Resource Monitoring Options¶

Environment-Specific Configuration¶

Best Practices¶

Do's¶

Don'ts¶

Common Scenarios¶

Scenario 1: Monitoring High-Resource Operations¶

Scenario 2: Health Check-Based Scaling¶

Scenario 3: Resource-Based Circuit Breaking¶

Troubleshooting¶

Issue: Resource Monitoring Not Working on Linux¶

Issue: Health Check Always Unhealthy¶

Issue: Metrics Not Appearing in OpenTelemetry¶

Issue: High CPU Usage from Monitoring¶

Related Documentation¶

Summary¶