Startup and Warmup¶

Purpose & Overview¶

The Startup Warmup mechanism in the ConnectSoft Microservice Template provides a configurable grace period after application startup and manual gating to control when the instance is considered ready to receive traffic. This is particularly important for:

Kubernetes Startup Probes: Ensures the application is fully initialized before accepting traffic
Dependency Initialization: Allows time for database connections, caches, and other dependencies to initialize
False Positive Prevention: Prevents premature "healthy" status during startup
Graceful Startup: Enables controlled startup sequencing for complex initialization tasks

Startup Warmup Philosophy

Applications need time to initialize dependencies, warm caches, and establish connections before they're ready to serve traffic. The StartupWarmupGate provides both automatic grace periods and manual control over readiness, ensuring reliable startup behavior in orchestrated environments like Kubernetes.

StartupWarmupGate¶

Overview¶

The StartupWarmupGate is a hosted service that manages application readiness during startup. It provides:

Configurable Grace Period: Automatic readiness after a specified time period
Manual Control: Explicit marking of readiness when initialization completes
Hold Mechanism: Ability to block readiness until specific tasks complete
Kubernetes Integration: Works seamlessly with Kubernetes startup probes

Implementation¶

namespace ConnectSoft.MicroserviceTemplate.ApplicationModel
{
    using System;
    using System.Threading;
    using System.Threading.Tasks;
    using Microsoft.Extensions.Hosting;

    /// <summary>
    /// Represents a configurable "grace" period after application startup and
    /// provides manual gating to control when the instance is considered ready.
    /// </summary>
    public sealed class StartupWarmupGate : IHostedService
    {
        private const int StartupWarmupGateInSeconds = 20;

        private readonly TimeSpan grace;
        private DateTimeOffset readyAtUtc;
        private int holds;                // > 0 means still warming up
        private int manuallyReadyFlag;    // 1 == MarkReady() called

        public StartupWarmupGate(TimeSpan? grace = null)
        {
            this.grace = grace ?? TimeSpan.FromSeconds(StartupWarmupGateInSeconds);
            this.readyAtUtc = DateTimeOffset.UtcNow + this.grace;
        }

        /// <summary>
        /// Gets a value indicating whether the startup grace period has elapsed.
        /// </summary>
        public bool IsReady
        {
            get
            {
                return (DateTimeOffset.UtcNow >= this.readyAtUtc || 
                       Volatile.Read(ref this.manuallyReadyFlag) == 1) && 
                       Volatile.Read(ref this.holds) == 0;
            }
        }

        /// <summary>
        /// Manually mark the instance as ready (ignores remaining grace).
        /// Outstanding holds still block readiness.
        /// </summary>
        public void MarkReady()
        {
            Interlocked.Exchange(ref this.manuallyReadyFlag, 1);
        }

        /// <summary>
        /// Prevent readiness until <see cref="ReleaseHold"/> is called the same number of times.
        /// Use from subsystems that must complete before serving (e.g., migrations, bus connect).
        /// </summary>
        public void AcquireHold()
        {
            Interlocked.Increment(ref this.holds);
        }

        /// <summary>
        /// Releases a previously acquired hold.
        /// </summary>
        public void ReleaseHold()
        {
            if (Interlocked.Decrement(ref this.holds) < 0)
            {
                Interlocked.Exchange(ref this.holds, 0); // safety
            }
        }

        /// <summary>
        /// Extends the grace window (useful if you detect slow warmup).
        /// </summary>
        public void ExtendGrace(TimeSpan extra)
        {
            if (extra <= TimeSpan.Zero)
            {
                return;
            }

            this.readyAtUtc = this.readyAtUtc + extra;
        }

        public Task StartAsync(CancellationToken cancellationToken)
        {
            // reset on every start
            this.readyAtUtc = DateTimeOffset.UtcNow + this.grace;
            Volatile.Write(ref this.holds, Volatile.Read(ref this.holds)); // no-op: keeps external holds if set early
            return Task.CompletedTask;
        }

        public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;
    }
}

Registration¶

The StartupWarmupGate is registered as a hosted service in the application model:

internal static IServiceCollection AddMicroserviceHostedServices(this IServiceCollection services)
{
    ArgumentNullException.ThrowIfNull(services);

    // --- StartupWarmupGate (used by dependency-heavy health checks like SignalR) ---
    var warmupSeconds = OptionsExtensions.MicroserviceOptions.StartupWarmupSeconds;
    services.AddSingleton(new StartupWarmupGate(TimeSpan.FromSeconds(warmupSeconds)));
    services.AddHostedService(sp => sp.GetRequiredService<StartupWarmupGate>());

    return services;
}

Configuration¶

The warmup period is configurable via options:

{
  "Microservice": {
    "StartupWarmupSeconds": 30
  }
}

Default: 20 seconds if not specified

Usage Patterns¶

Pattern 1: Automatic Grace Period¶

The simplest pattern relies on the automatic grace period:

// Application starts
// StartupWarmupGate automatically blocks readiness for 20-30 seconds
// After grace period, IsReady returns true
// Health checks can now report healthy

Use Case: Simple applications with predictable startup time

Pattern 2: Manual Marking¶

For applications that know when initialization is complete:

public class StartupInitializationService : IHostedService
{
    private readonly StartupWarmupGate warmupGate;

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        // Perform initialization tasks
        await RunMigrationsAsync();
        await WarmCacheAsync();
        await ConnectToMessageBusAsync();

        // Mark ready when done
        warmupGate.MarkReady();
    }
}

Use Case: Applications with explicit initialization tasks

Pattern 3: Hold Mechanism¶

For subsystems that need to complete before readiness:

public class DatabaseMigrationService : IHostedService
{
    private readonly StartupWarmupGate warmupGate;

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        warmupGate.AcquireHold(); // Block readiness

        try
        {
            await RunMigrationsAsync();
        }
        finally
        {
            warmupGate.ReleaseHold(); // Allow readiness
        }
    }
}

Use Case: Critical initialization tasks that must complete

Pattern 4: Combined Approach¶

Combining automatic grace period with manual control:

public class StartupOrchestrator
{
    private readonly StartupWarmupGate warmupGate;

    public async Task InitializeAsync()
    {
        warmupGate.AcquireHold(); // Block until migrations complete

        // Run critical initialization
        await RunMigrationsAsync();
        await WarmCacheAsync();

        warmupGate.ReleaseHold(); // Allow readiness if grace elapsed

        // Continue with non-critical initialization
        await InitializeBackgroundServicesAsync();

        // Explicitly mark ready (if not already ready)
        warmupGate.MarkReady();
    }
}

Use Case: Complex applications with multiple initialization phases

Health Check Integration¶

The StartupWarmupGate integrates with health checks to prevent premature readiness:

public class StartupWarmupHealthCheck : IHealthCheck
{
    private readonly StartupWarmupGate warmupGate;

    public Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context, 
        CancellationToken cancellationToken = default)
    {
        if (warmupGate.IsReady)
        {
            return Task.FromResult(
                HealthCheckResult.Healthy("Warmup complete"));
        }

        return Task.FromResult(
            HealthCheckResult.Unhealthy("Warming up (startup grace period not elapsed)"));
    }
}

Health Check Registration¶

services.AddHealthChecks()
    .AddCheck<StartupWarmupHealthCheck>("startup", tags: new[] { "startup" });

Health Check Endpoint¶

The startup endpoint checks if the service has completed startup warmup:

app.MapHealthChecks("/startup", new HealthCheckOptions
{
    Predicate = check => check.Tags.Contains("startup")
});

Behavior:

During warmup: Returns Unhealthy with message "Warming up (startup grace period not elapsed)"
After warmup: Returns Healthy with message "Warmup complete"

Kubernetes Integration¶

Startup Probe Configuration¶

The startup warmup mechanism is designed for Kubernetes startup probes:

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: microservice
    startupProbe:
      httpGet:
        path: /startup
        port: 8080
      initialDelaySeconds: 0
      periodSeconds: 5
      timeoutSeconds: 3
      failureThreshold: 30  # Allow up to 150 seconds for startup
    livenessProbe:
      httpGet:
        path: /health
        port: 8080
      initialDelaySeconds: 30
      periodSeconds: 10
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      initialDelaySeconds: 30
      periodSeconds: 5

Benefits:

Prevents Premature Traffic: Kubernetes won't route traffic until startup probe succeeds
Graceful Startup: Application has time to initialize dependencies
Failure Recovery: Startup probe allows longer initialization time than readiness probe

Initialization Tasks¶

Common initialization tasks that benefit from startup warmup:

Database Migrations¶

public class MigrationInitializationService : IHostedService
{
    private readonly StartupWarmupGate warmupGate;
    private readonly IMigrationRunner migrationRunner;

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        warmupGate.AcquireHold();

        try
        {
            await migrationRunner.MigrateUpAsync(cancellationToken);
        }
        finally
        {
            warmupGate.ReleaseHold();
        }
    }
}

Cache Warming¶

public class CacheWarmingService : IHostedService
{
    private readonly StartupWarmupGate warmupGate;
    private readonly ICache cache;

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        warmupGate.AcquireHold();

        try
        {
            // Warm frequently accessed data
            await cache.WarmPolicyCacheAsync();
            await cache.WarmTenantMetadataAsync();
        }
        finally
        {
            warmupGate.ReleaseHold();
        }
    }
}

Message Bus Connection¶

public class MessageBusInitializationService : IHostedService
{
    private readonly StartupWarmupGate warmupGate;
    private readonly IBusControl busControl;

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        warmupGate.AcquireHold();

        try
        {
            await busControl.StartAsync(cancellationToken);
        }
        finally
        {
            warmupGate.ReleaseHold();
        }
    }
}

External Service Connectivity¶

public class ExternalServiceVerificationService : IHostedService
{
    private readonly StartupWarmupGate warmupGate;
    private readonly IHttpClientFactory httpClientFactory;

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        warmupGate.AcquireHold();

        try
        {
            // Verify critical external services are reachable
            await VerifyDatabaseConnectivityAsync();
            await VerifyMessageBrokerConnectivityAsync();
            await VerifyConfigurationServiceConnectivityAsync();
        }
        finally
        {
            warmupGate.ReleaseHold();
        }
    }
}

Best Practices¶

Do's¶

Use Appropriate Warmup Periods

// ✅ GOOD - Based on actual initialization time
var warmupSeconds = configuration.GetValue<int>("Microservice:StartupWarmupSeconds", 30);

Acquire Holds for Critical Tasks

// ✅ GOOD - Block readiness until migrations complete
warmupGate.AcquireHold();
try { await RunMigrationsAsync(); }
finally { warmupGate.ReleaseHold(); }

Mark Ready When Initialization Completes

// ✅ GOOD - Explicit readiness marking
await InitializeAsync();
warmupGate.MarkReady();

Integrate with Health Checks

// ✅ GOOD - Health checks respect warmup gate
services.AddHealthChecks()
    .AddCheck<StartupWarmupHealthCheck>("startup");

Don'ts¶

Don't Skip Warmup for Production

// ❌ BAD - No warmup in production
if (environment.IsDevelopment())
{
    warmupGate.MarkReady(); // Never skip in production!
}

Don't Forget to Release Holds

// ❌ BAD - Hold never released, application never becomes ready
warmupGate.AcquireHold();
await RunMigrationsAsync();
// Missing: warmupGate.ReleaseHold();

Don't Use Too Short Warmup Periods

// ❌ BAD - Too short, may not complete initialization
var warmupGate = new StartupWarmupGate(TimeSpan.FromSeconds(1));

// ✅ GOOD - Appropriate warmup period
var warmupGate = new StartupWarmupGate(TimeSpan.FromSeconds(30));

Don't Block on Non-Critical Initialization

// ❌ BAD - Non-critical task blocks readiness
warmupGate.AcquireHold();
await InitializeNonCriticalFeatureAsync(); // Shouldn't block
warmupGate.ReleaseHold();

// ✅ GOOD - Non-critical initialization doesn't block
_ = Task.Run(async () => await InitializeNonCriticalFeatureAsync());

Troubleshooting¶

Issue: Application Never Becomes Ready¶

Symptom: Health checks always report unhealthy, startup probe fails.

Solutions:

Check if holds are being released
Verify MarkReady() is being called if using manual control
Ensure warmup period is sufficient
Review initialization tasks for blocking operations

Issue: Application Becomes Ready Too Early¶

Symptom: Health checks report healthy before initialization completes.

Solutions:

Increase warmup period
Use hold mechanism for critical initialization tasks
Verify initialization tasks complete before marking ready

Issue: Startup Probe Times Out¶

Symptom: Kubernetes startup probe fails after maximum attempts.

Solutions:

Increase failureThreshold in startup probe configuration
Increase warmup period in application configuration
Optimize initialization tasks to complete faster
Use hold mechanism to ensure critical tasks complete

Health Checks: Health check configuration and integration
Application Model: Application model and service registration
Kubernetes: Kubernetes deployment patterns
Solution Structure: Solution organization and project structure
Configuration: Configuration options and StartupWarmupSeconds setting

Summary¶

The Startup Warmup mechanism in the ConnectSoft Microservice Template provides:

✅ Configurable Grace Period: Automatic readiness after specified time
✅ Manual Control: Explicit marking of readiness
✅ Hold Mechanism: Block readiness until tasks complete
✅ Health Check Integration: Seamless integration with health checks
✅ Kubernetes Support: Works with Kubernetes startup probes
✅ Flexible Patterns: Multiple usage patterns for different scenarios

By following these patterns, teams can:

Ensure Reliable Startup: Applications initialize completely before serving traffic
Prevent Premature Traffic: Kubernetes won't route traffic until ready
Handle Complex Initialization: Support multiple initialization phases
Integrate with Orchestration: Works seamlessly with Kubernetes and other orchestrators

The Startup Warmup mechanism ensures that applications are fully initialized and ready to serve traffic before accepting requests, preventing startup-related failures and ensuring reliable operation in production environments.

Startup and Warmup¶

Purpose & Overview¶

StartupWarmupGate¶

Overview¶

Implementation¶

Registration¶

Configuration¶

Usage Patterns¶

Pattern 1: Automatic Grace Period¶

Pattern 2: Manual Marking¶

Pattern 3: Hold Mechanism¶

Pattern 4: Combined Approach¶

Health Check Integration¶

Health Check Registration¶

Health Check Endpoint¶

Kubernetes Integration¶

Startup Probe Configuration¶

Initialization Tasks¶

Database Migrations¶

Cache Warming¶

Message Bus Connection¶

External Service Connectivity¶

Best Practices¶

Do's¶

Don'ts¶

Troubleshooting¶

Issue: Application Never Becomes Ready¶

Issue: Application Becomes Ready Too Early¶

Issue: Startup Probe Times Out¶

Related Documentation¶

Summary¶