Hangfire in ConnectSoft Microservice Template

Purpose & Overview

Hangfire is a powerful background job processing library for .NET that provides reliable, persistent, and easy-to-use job scheduling capabilities. In the ConnectSoft Microservice Template, Hangfire is integrated as part of the Scheduler Model, enabling background job processing, recurring tasks, and scheduled operations with a web-based dashboard for monitoring and management.

Why Hangfire?

Hangfire offers several key benefits for microservices:

• Persistent Storage: Jobs are stored in a database (SQL Server), ensuring they survive application restarts
• Reliability: Automatic retry mechanisms for failed jobs
• Dashboard: Built-in web UI for monitoring and managing jobs
• Scalability: Multiple server instances can process jobs concurrently
• Flexibility: Support for fire-and-forget, delayed, and recurring jobs
• Integration: Seamless integration with ASP.NET Core dependency injection
• Observability: OpenTelemetry instrumentation for distributed tracing

Hangfire Philosophy

Hangfire enables fire-and-forget, delayed, and recurring background jobs in .NET applications without requiring separate Windows services or task schedulers. It provides a simple, reliable way to offload work from HTTP request threads and process it asynchronously.

Architecture Overview

Hangfire in Clean Architecture

Hangfire sits in the Application Layer as part of the Scheduler Model:

Application Layer (SchedulerModel)
├── IMicroserviceTemplateScheduler (Interface)
└── HangFireMicroserviceTemplateScheduler (Implementation)
    ↓
Hangfire Infrastructure
├── Storage (SQL Server)
├── Background Job Server (IHostedService)
├── Dashboard (Web UI)
└── OpenTelemetry Instrumentation
    ↓
Domain Layer
└── Business Logic Execution

Components

Component	Purpose	Location
HangFireExtensions	Configuration and registration	ApplicationModel
HangFireOptions	Configuration options	Options
IMicroserviceTemplateScheduler	Scheduler interface	SchedulerModel
HangFireMicroserviceTemplateScheduler	Hangfire implementation	SchedulerModel.Hangfire
Dashboard	Web UI for monitoring	/hangfire endpoint

Configuration

Options Configuration

Hangfire configuration is managed through strongly-typed options:

{
  "HangFire": {
    "HangFireSqlServerConnectionStringKey": "ConnectSoft.MicroserviceTemplateHangFireConnectionString",
    "AutomaticRetries": {
      "MaximumAttempts": 0
    },
    "BackgroundJobServerOptions": {
      "ShutdownTimeout": "00:00:15",
      "StopTimeout": "00:00:00.500",
      "HeartbeatInterval": "00:00:30",
      "ServerCheckInterval": "00:05:00",
      "ServerTimeout": "00:05:00",
      "CancellationCheckInterval": "00:00:15",
      "SchedulePollingInterval": "00:00:15"
    },
    "DashboardOptions": {
      "DashboardPath": "/hangfire",
      "DashboardTitle": "ConnectSoft.MicroserviceTemplate HangFire Dashboard",
      "ApplicationPath": "/",
      "StatsPollingInterval": 2000
    }
  }
}

Connection String

Connection String Configuration:

{
  "ConnectionStrings": {
    "ConnectSoft.MicroserviceTemplateHangFireConnectionString": "Server=localhost,1433;Database=HANGFIRE_DATABASE;User Id=sa;Password=Password@123;MultipleActiveResultSets=true;Encrypt=false;TrustServerCertificate=true;"
  }
}

Service Registration

AddMicroserviceHangFire:

// HangFireExtensions.cs
internal static IServiceCollection AddMicroserviceHangFire(
    this IServiceCollection services, 
    IConfiguration configuration)
{
    ArgumentNullException.ThrowIfNull(services);
    ArgumentNullException.ThrowIfNull(configuration);

    string connectionString = configuration.GetConnectionString(
        OptionsExtensions.HangFireOptions.HangFireSqlServerConnectionStringKey);
    ArgumentNullException.ThrowIfNull(connectionString);

    // Create database if it doesn't exist
    new SqlServerDatabaseHelper().CreateIfNotExists(connectionString);

    // Add HangFire services
    services.AddHangfire(configuration =>
    {
        configuration
            .UseConsole()  // Hangfire.Console for real-time logging
            .SetDataCompatibilityLevel(CompatibilityLevel.Version_180)
            .UseSimpleAssemblyNameTypeSerializer()
            .UseRecommendedSerializerSettings()
            .UseSqlServerStorage(connectionString);
    });

    // Configure global retry policy
    GlobalJobFilters.Filters.Add(new AutomaticRetryAttribute 
    { 
        Attempts = OptionsExtensions.HangFireOptions.AutomaticRetries.MaximumAttempts 
    });

    // Register scheduler implementation
    services.AddScoped<IMicroserviceTemplateScheduler, HangFireMicroserviceTemplateScheduler>();

    // Add Hangfire Console extensions
    services.AddHangfireConsoleExtensions();

    // Add the processing server as IHostedService
    services.AddHangfireServer(configureOptions =>
    {
        configureOptions.HeartbeatInterval = OptionsExtensions.HangFireOptions.BackgroundJobServerOptions.HeartbeatInterval;
        configureOptions.CancellationCheckInterval = OptionsExtensions.HangFireOptions.BackgroundJobServerOptions.CancellationCheckInterval;
        configureOptions.ServerCheckInterval = OptionsExtensions.HangFireOptions.BackgroundJobServerOptions.ServerCheckInterval;
        configureOptions.SchedulePollingInterval = OptionsExtensions.HangFireOptions.BackgroundJobServerOptions.SchedulePollingInterval;
        configureOptions.StopTimeout = OptionsExtensions.HangFireOptions.BackgroundJobServerOptions.StopTimeout;
        configureOptions.ShutdownTimeout = OptionsExtensions.HangFireOptions.BackgroundJobServerOptions.ShutdownTimeout;
        configureOptions.ServerTimeout = OptionsExtensions.HangFireOptions.BackgroundJobServerOptions.ServerTimeout;
        configureOptions.WorkerCount = Math.Min(Environment.ProcessorCount * 5, 20);
        configureOptions.Queues = new string[1] { "default" };
    });

    return services;
}

Middleware Configuration

UseMicroserviceHangFire:

// HangFireExtensions.cs
internal static IApplicationBuilder UseMicroserviceHangFire(
    this IApplicationBuilder application)
{
    ArgumentNullException.ThrowIfNull(application);

    // Add HangFire dashboard
    if (OptionsExtensions.HangFireOptions.DashboardOptions != null)
    {
        DashboardOptions dashboardOptions = new()
        {
            AppPath = OptionsExtensions.HangFireOptions.DashboardOptions.ApplicationPath,
            DashboardTitle = OptionsExtensions.HangFireOptions.DashboardOptions.DashboardTitle,
            StatsPollingInterval = OptionsExtensions.HangFireOptions.DashboardOptions.StatsPollingInterval,
            DarkModeEnabled = true,
        };

        application.UseHangfireDashboard(
            pathMatch: OptionsExtensions.HangFireOptions.DashboardOptions.DashboardPath,
            options: dashboardOptions);
    }

    // Register recurring jobs
    RecurringJob.AddOrUpdate<IMicroserviceTemplateScheduler>(
        nameof(HangFireMicroserviceTemplateScheduler),
        x => x.RunJob(),
        Cron.Never); // Change to desired cron expression

    return application;
}

Integration in Program.cs

Service Registration:

// Program.cs
#if UseHangFire
    services.AddMicroserviceHangFire(configuration);
#endif

Middleware Configuration:

// Program.cs
#if UseHangFire
    application.UseMicroserviceHangFire();
#endif

Background Job Server Options

Configuration Options

HangFireBackgroundJobServerOptions:

Option	Default	Description
StopTimeout	00:00:00.500	Time to wait for jobs to finish before stopping
ShutdownTimeout	00:00:15	Time to wait for jobs to finish before shutdown
HeartbeatInterval	00:00:30	Interval between heartbeats sent to storage
ServerCheckInterval	00:05:00	Interval between recurring checks that server is running
ServerTimeout	00:05:00	Time to wait for server to start
CancellationCheckInterval	00:00:15	Interval between checks for cancellation requests
SchedulePollingInterval	00:00:15	Interval between recurring checks for scheduled jobs

Worker Count: - Automatically calculated: Math.Min(Environment.ProcessorCount * 5, 20) - Can be overridden if needed

Queues: - Default queue: "default" - Multiple queues can be configured for job prioritization

Background Jobs

Fire-and-Forget Jobs

Enqueue a job immediately:

public class OrderProcessingService
{
    private readonly IBackgroundJobClient backgroundJobClient;

    public OrderProcessingService(IBackgroundJobClient backgroundJobClient)
    {
        this.backgroundJobClient = backgroundJobClient;
    }

    public void ProcessOrder(Order order)
    {
        // Enqueue job to be processed immediately
        string jobId = this.backgroundJobClient.Enqueue<OrderProcessor>(
            processor => processor.ProcessOrderAsync(order.Id));

        // JobId can be used for tracking
        this.logger.LogInformation("Order processing job enqueued: {JobId}", jobId);
    }
}

Job Implementation:

public class OrderProcessor
{
    private readonly ILogger<OrderProcessor> logger;
    private readonly IOrderRepository orderRepository;

    public OrderProcessor(
        ILogger<OrderProcessor> logger,
        IOrderRepository orderRepository)
    {
        this.logger = logger;
        this.orderRepository = orderRepository;
    }

    [AutomaticRetry(Attempts = 3, DelaysInSeconds = new[] { 10, 30, 60 })]
    public async Task ProcessOrderAsync(Guid orderId)
    {
        using (this.logger.BeginScope(
            new Dictionary<string, object>
            {
                ["OrderId"] = orderId,
                ["JobType"] = "OrderProcessing"
            }))
        {
            this.logger.LogInformation("Processing order {OrderId}", orderId);

            try
            {
                var order = await this.orderRepository.GetByIdAsync(orderId);

                // Process order logic
                await this.ProcessOrderLogicAsync(order);

                this.logger.LogInformation("Order {OrderId} processed successfully", orderId);
            }
            catch (Exception ex)
            {
                this.logger.LogError(ex, "Failed to process order {OrderId}", orderId);
                throw; // Hangfire will retry based on [AutomaticRetry] attribute
            }
        }
    }

    private async Task ProcessOrderLogicAsync(Order order)
    {
        // Business logic here
    }
}

Delayed Jobs

Schedule a job for later execution:

public class NotificationService
{
    private readonly IBackgroundJobClient backgroundJobClient;

    public void SendDelayedNotification(User user, TimeSpan delay)
    {
        // Schedule job to run after delay
        string jobId = this.backgroundJobClient.Schedule<NotificationSender>(
            sender => sender.SendNotificationAsync(user.Id, "Welcome!"),
            delay);
    }

    public void SendScheduledNotification(User user, DateTimeOffset scheduledTime)
    {
        // Schedule job for specific time
        TimeSpan delay = scheduledTime - DateTimeOffset.UtcNow;
        string jobId = this.backgroundJobClient.Schedule<NotificationSender>(
            sender => sender.SendNotificationAsync(user.Id, "Reminder"),
            delay);
    }
}

Continuations

Chain jobs together:

public class WorkflowService
{
    private readonly IBackgroundJobClient backgroundJobClient;

    public void StartWorkflow(WorkflowRequest request)
    {
        // Start first job
        string firstJobId = this.backgroundJobClient.Enqueue<Step1Processor>(
            processor => processor.ProcessStep1Async(request.Id));

        // Chain second job after first completes
        this.backgroundJobClient.ContinueJobWith<Step2Processor>(
            firstJobId,
            processor => processor.ProcessStep2Async(request.Id));

        // Chain third job after second completes
        this.backgroundJobClient.ContinueJobWith<Step3Processor>(
            firstJobId,
            processor => processor.ProcessStep3Async(request.Id));
    }
}

Recurring Jobs

Basic Recurring Job

Register a recurring job:

// In UseMicroserviceHangFire or Startup
RecurringJob.AddOrUpdate<IDataCleanupService>(
    "data-cleanup",
    service => service.CleanupOldDataAsync(),
    Cron.Daily); // Run daily at midnight

Cron Expressions

Common Cron Patterns:

// Every minute
Cron.Minutely

// Every hour
Cron.Hourly

// Daily at midnight
Cron.Daily

// Weekly on Monday at midnight
Cron.Weekly

// Monthly on first day at midnight
Cron.Monthly

// Yearly on January 1st at midnight
Cron.Yearly

// Custom cron expression
Cron.Cron("0 */6 * * *")  // Every 6 hours
Cron.Cron("0 9 * * 1-5")  // Every weekday at 9 AM
Cron.Cron("0 0 1 * *")     // First day of every month at midnight

Cron Expression Format:

┌───────────── minute (0 - 59)
│ ┌───────────── hour (0 - 23)
│ │ ┌───────────── day of month (1 - 31)
│ │ │ ┌───────────── month (1 - 12)
│ │ │ │ ┌───────────── day of week (0 - 6) (Sunday to Saturday)
│ │ │ │ │
* * * * *

Scheduler Implementation

IMicroserviceTemplateScheduler:

// IMicroserviceTemplateScheduler.cs
public interface IMicroserviceTemplateScheduler
{
    /// <summary>
    /// Run the job.
    /// </summary>
    /// <returns>A <see cref="Task"/> representing the asynchronous operation.</returns>
    Task RunJob();
}

HangFireMicroserviceTemplateScheduler:

// HangFireMicroserviceTemplateScheduler.cs
public class HangFireMicroserviceTemplateScheduler(
    ILogger<HangFireMicroserviceTemplateScheduler> logger,
    TimeProvider dateTimeProvider)
    : IMicroserviceTemplateScheduler
{
    private readonly ILogger<HangFireMicroserviceTemplateScheduler> logger = logger;
    private readonly TimeProvider dateTimeProvider = dateTimeProvider;

    /// <inheritdoc/>
    public Task RunJob()
    {
        using (this.logger.BeginScope(
            new Dictionary<string, object>(StringComparer.Ordinal)
            {
                ["ApplicationFlowName"] = nameof(HangFireMicroserviceTemplateScheduler) + "/" + nameof(this.RunJob),
            }))
        {
            try
            {
                DateTimeOffset utcNow = this.dateTimeProvider.GetUtcNow();
                this.logger.Here(log => log.LogInformation(
                    message: "Run ConnectSoft.MicroserviceTemplate's scheduler at {UtcNow}...",
                    args: utcNow));

                // Job logic here
                // await this.ProcessScheduledTasksAsync();

                this.logger.Here(log => log.LogInformation(
                    message: "Run ConnectSoft.MicroserviceTemplate's scheduler at {UtcNow} successfully completed...",
                    args: utcNow));
            }
            catch (Exception ex)
            {
                this.logger.Here(log => log.LogError(
                    exception: ex,
                    message: "Failed to run ConnectSoft.MicroserviceTemplate's scheduler job"));
                throw;
            }
        }

        return Task.CompletedTask;
    }
}

Register Recurring Job:

// In UseMicroserviceHangFire
RecurringJob.AddOrUpdate<IMicroserviceTemplateScheduler>(
    nameof(HangFireMicroserviceTemplateScheduler),
    x => x.RunJob(),
    Cron.Never); // Change to desired cron expression, e.g., Cron.Daily

Automatic Retries

Global Retry Configuration

Configure Global Retry Policy:

// In AddMicroserviceHangFire
GlobalJobFilters.Filters.Add(new AutomaticRetryAttribute 
{ 
    Attempts = OptionsExtensions.HangFireOptions.AutomaticRetries.MaximumAttempts 
});

Configuration:

{
  "HangFire": {
    "AutomaticRetries": {
      "MaximumAttempts": 3  // Set to 0 to disable automatic retries
    }
  }
}

Per-Job Retry Configuration

Attribute-based Retry:

[AutomaticRetry(Attempts = 3, DelaysInSeconds = new[] { 10, 30, 60 })]
public class OrderProcessor
{
    public async Task ProcessOrderAsync(Guid orderId)
    {
        // Job logic
    }
}

Retry Options:

[AutomaticRetry(
    Attempts = 3,                                    // Maximum retry attempts
    DelaysInSeconds = new[] { 10, 30, 60 },         // Delay between retries
    OnAttemptsExceeded = AttemptsExceededAction.Delete,  // Action when max attempts exceeded
    LogEvents = true,                                // Log retry events
    OnRetryStateChanged = typeof(CustomRetryFilter))]  // Custom retry filter
public class MyJob
{
    // Job implementation
}

Custom Retry Logic

Exception-based Retry:

public class ResilientJobService
{
    [AutomaticRetry(Attempts = 3)]
    public async Task ProcessWithRetryAsync(Guid itemId)
    {
        try
        {
            await this.ProcessItemAsync(itemId);
        }
        catch (TransientException ex)
        {
            // Transient exceptions will be retried
            this.logger.LogWarning(ex, "Transient error, will retry");
            throw;
        }
        catch (DomainModelException ex)
        {
            // Domain exceptions should not be retried
            this.logger.LogError(ex, "Domain validation failed, not retrying");
            throw;
        }
    }
}

Dashboard

Dashboard Configuration

Dashboard Options:

{
  "HangFire": {
    "DashboardOptions": {
      "DashboardPath": "/hangfire",
      "DashboardTitle": "ConnectSoft.MicroserviceTemplate HangFire Dashboard",
      "ApplicationPath": "/",
      "StatsPollingInterval": 2000
    }
  }
}

Access Dashboard: - URL: https://your-service.com/hangfire - Real-time job monitoring - Job history and statistics - Manual job triggering - Job state management

Dashboard Features

Available Features: - Jobs: View all jobs (enqueued, processing, succeeded, failed) - Recurring Jobs: View and manage recurring jobs - Retries: View retry attempts and failures - Servers: Monitor Hangfire server instances - Queues: View queue status and job distribution - Statistics: Overview of job execution metrics

Dashboard Security

Authorization (if needed):

application.UseHangfireDashboard(
    pathMatch: "/hangfire",
    options: new DashboardOptions
    {
        Authorization = new[] { new HangfireAuthorizationFilter() }
    });

Custom Authorization Filter:

public class HangfireAuthorizationFilter : IDashboardAuthorizationFilter
{
    public bool Authorize(DashboardContext context)
    {
        var httpContext = context.GetHttpContext();
        // Implement your authorization logic
        return httpContext.User.IsInRole("Admin");
    }
}

Integration with Template Features

OpenTelemetry Integration

Automatic Instrumentation:

// OpenTelemetryExtensions.cs
#if UseHangFire
    // Hangfire instrumentation for distributed tracing
    .AddHangfireInstrumentation()
#endif

Trace Context Propagation: - Hangfire jobs automatically include trace context - Distributed tracing across job execution - Correlation IDs for job tracking

Logging Integration

Structured Logging:

public class ScheduledJobService
{
    private readonly ILogger<ScheduledJobService> logger;

    public async Task ProcessScheduledTask()
    {
        using (this.logger.BeginScope(
            new Dictionary<string, object>
            {
                ["JobType"] = "ScheduledTask",
                ["JobId"] = BackgroundJob.Id,
                ["RetryCount"] = BackgroundJob.GetJobParameter<int>("RetryCount")
            }))
        {
            this.logger.LogInformation("Processing scheduled task");

            try
            {
                // Job logic
                await this.ProcessTaskAsync();

                this.logger.LogInformation("Scheduled task completed successfully");
            }
            catch (Exception ex)
            {
                this.logger.LogError(ex, "Scheduled task failed");
                throw;
            }
        }
    }
}

Exception Handling

Exception Handling in Jobs:

public class ScheduledJobService
{
    private readonly ILogger<ScheduledJobService> logger;
    private readonly IMicroserviceAggregateRootsProcessor processor;

    [AutomaticRetry(Attempts = 3, DelaysInSeconds = new[] { 10, 30, 60 })]
    public async Task ProcessScheduledTask()
    {
        try
        {
            // Job logic
            await processor.ProcessBatchAsync();

            this.logger.LogInformation("Scheduled job completed successfully");
        }
        catch (DomainModelException ex)
        {
            // Domain exceptions: log and fail (don't retry)
            this.logger.LogWarning(
                ex,
                "Domain validation failed in scheduled job");

            throw; // Hangfire will mark job as failed
        }
        catch (Exception ex)
        {
            // Infrastructure exceptions: log and retry
            this.logger.LogError(
                ex,
                "Error in scheduled job");

            throw; // Hangfire will retry according to [AutomaticRetry] attribute
        }
    }
}

Health Checks

Hangfire Health Check (if implemented):

services.AddHealthChecks()
    .AddHangfire(options =>
    {
        options.MinimumAvailableServers = 1;
        options.MaximumJobsFailed = 5;
    });

Best Practices

Do's

1. Use Dependency Injection

   // ✅ GOOD - Use DI for job classes
   public class OrderProcessor
   {
       private readonly IOrderRepository repository;
   
       public OrderProcessor(IOrderRepository repository)
       {
           this.repository = repository;
       }
   }
   

2. Use Scoped Services Correctly

   // ✅ GOOD - Hangfire creates scopes for each job
   // Services registered as Scoped are available per job execution
   services.AddScoped<IMicroserviceTemplateScheduler, HangFireMicroserviceTemplateScheduler>();
   

3. Implement Proper Logging

   // ✅ GOOD - Structured logging with context
   using (this.logger.BeginScope(new Dictionary<string, object> { ["JobId"] = jobId }))
   {
       this.logger.LogInformation("Processing job {JobId}", jobId);
   }
   

4. Configure Retry Policies

   // ✅ GOOD - Explicit retry configuration
   [AutomaticRetry(Attempts = 3, DelaysInSeconds = new[] { 10, 30, 60 })]
   public async Task ProcessAsync()
   {
       // Job logic
   }
   

5. Use Idempotent Jobs

   // ✅ GOOD - Job can be safely retried
   public async Task ProcessOrderAsync(Guid orderId)
   {
       // Check if already processed
       if (await this.IsProcessedAsync(orderId))
       {
           return; // Idempotent: already processed
       }
   
       // Process order
       await this.ProcessOrderLogicAsync(orderId);
   }
   

6. Handle Cancellation

   // ✅ GOOD - Respect cancellation tokens
   public async Task LongRunningJobAsync(CancellationToken cancellationToken)
   {
       while (!cancellationToken.IsCancellationRequested)
       {
           await this.ProcessBatchAsync(cancellationToken);
           await Task.Delay(TimeSpan.FromSeconds(1), cancellationToken);
       }
   }
   

Don'ts

1. Don't Use Static State

   // ❌ BAD - Static state shared across jobs
   private static int processedCount = 0;
   
   // ✅ GOOD - Use database or external state
   private readonly IStateRepository stateRepository;
   

2. Don't Block Threads

   // ❌ BAD - Blocking call
   Task.Run(() => Process()).Wait();
   
   // ✅ GOOD - Async/await
   await ProcessAsync();
   

3. Don't Ignore Exceptions

   // ❌ BAD - Swallowing exceptions
   try
   {
       await ProcessAsync();
   }
   catch { }
   
   // ✅ GOOD - Log and rethrow
   catch (Exception ex)
   {
       this.logger.LogError(ex, "Job failed");
       throw;
   }
   

4. Don't Use Long-Running Operations in HTTP Context

   // ❌ BAD - Long operation in HTTP request
   [HttpPost]
   public IActionResult ProcessOrder(Order order)
   {
       this.ProcessOrder(order); // Takes 5 minutes
       return Ok();
   }
   
   // ✅ GOOD - Enqueue background job
   [HttpPost]
   public IActionResult ProcessOrder(Order order)
   {
       this.backgroundJobClient.Enqueue<OrderProcessor>(
           processor => processor.ProcessOrderAsync(order.Id));
       return Accepted();
   }
   

5. Don't Store Sensitive Data in Job Arguments

   // ❌ BAD - Sensitive data in job arguments
   this.backgroundJobClient.Enqueue<UserService>(
       service => service.ProcessUserAsync(user.Password));
   
   // ✅ GOOD - Store IDs only
   this.backgroundJobClient.Enqueue<UserService>(
       service => service.ProcessUserAsync(user.Id));
   

Testing

Unit Testing

Test Job Logic:

[TestMethod]
public async Task ProcessOrderAsync_ValidOrder_ProcessesSuccessfully()
{
    // Arrange
    var logger = new Mock<ILogger<OrderProcessor>>();
    var repository = new Mock<IOrderRepository>();
    var processor = new OrderProcessor(logger.Object, repository.Object);

    var order = new Order { Id = Guid.NewGuid() };
    repository.Setup(r => r.GetByIdAsync(order.Id))
        .ReturnsAsync(order);

    // Act
    await processor.ProcessOrderAsync(order.Id);

    // Assert
    repository.Verify(r => r.SaveAsync(order), Times.Once);
}

Integration Testing

Test Hangfire Integration:

[TestMethod]
public async Task EnqueueJob_JobExecutesSuccessfully()
{
    // Arrange
    var host = new WebApplicationFactory<Program>()
        .WithWebHostBuilder(builder =>
        {
            builder.ConfigureServices(services =>
            {
                // Configure test services
            });
        });

    var client = host.Services.GetRequiredService<IBackgroundJobClient>();

    // Act
    string jobId = client.Enqueue<TestJob>(job => job.ExecuteAsync());

    // Wait for job to complete
    await Task.Delay(TimeSpan.FromSeconds(5));

    // Assert
    var storage = host.Services.GetRequiredService<JobStorage>();
    using var connection = storage.GetConnection();
    var jobData = connection.GetJobData(jobId);
    Assert.AreEqual("Succeeded", jobData.State);
}

BDD Testing

Reqnroll Steps (from template):

[When(@"I enqueue the scheduler job now")]
public void WhenIEnqueueTheSchedulerJobNow()
{
    var client = BeforeAfterTestRunHooks.HostInstance!
        .Services.GetRequiredService<IBackgroundJobClient>();

    this.jobId = client.Enqueue<IMicroserviceTemplateScheduler>(
        svc => svc.RunJob());

    Assert.IsFalse(string.IsNullOrWhiteSpace(this.jobId));
}

[Then(@"the job succeeds within (.*) seconds")]
public void ThenTheJobSucceedsWithinSeconds(int seconds)
{
    var storage = BeforeAfterTestRunHooks.HostInstance!
        .Services.GetRequiredService<JobStorage>();

    var sw = Stopwatch.StartNew();
    var deadline = TimeSpan.FromSeconds(Math.Max(1, seconds));

    while (sw.Elapsed < deadline)
    {
        using var conn = storage.GetConnection();
        var jobData = conn.GetJobData(this.jobId);
        var state = jobData?.State;

        if (string.Equals(state, "Succeeded", StringComparison.OrdinalIgnoreCase))
        {
            return; // success
        }

        if (string.Equals(state, "Failed", StringComparison.OrdinalIgnoreCase))
        {
            Assert.Fail("Hangfire job moved to Failed state.");
        }

        Thread.Sleep(100);
    }

    Assert.Fail($"Job did not succeed within {seconds} seconds.");
}

Troubleshooting

Issue: Jobs Not Processing

Symptoms: Jobs remain in "Enqueued" state and never execute.

Solutions: 1. Verify Hangfire server is running: Check dashboard "Servers" tab 2. Check connection string is valid 3. Verify AddHangfireServer() is called 4. Check worker count is greater than 0 5. Review server logs for errors

Issue: Jobs Failing Immediately

Symptoms: Jobs fail without retry attempts.

Solutions: 1. Check job method is public and accessible 2. Verify dependency injection is configured correctly 3. Review exception details in dashboard 4. Check MaximumAttempts is greater than 0 5. Verify job method signature matches enqueue call

Issue: Database Connection Errors

Symptoms: Errors connecting to Hangfire database.

Solutions: 1. Verify connection string is correct 2. Check database exists (created automatically, but verify) 3. Ensure SQL Server is accessible 4. Check firewall rules 5. Verify credentials have sufficient permissions

Issue: Dashboard Not Accessible

Symptoms: 404 or access denied when accessing /hangfire.

Solutions: 1. Verify UseHangfireDashboard() is called in middleware pipeline 2. Check dashboard path matches configuration 3. Verify middleware order (before UseRouting() if needed) 4. Check authorization filters if configured 5. Review application logs for routing errors

Issue: Recurring Jobs Not Running

Symptoms: Recurring jobs registered but never execute.

Solutions: 1. Verify cron expression is correct (not Cron.Never) 2. Check SchedulePollingInterval is reasonable (not too high) 3. Verify recurring job is registered: Check dashboard "Recurring Jobs" tab 4. Check if job is enabled (not disabled) 5. Review server logs for scheduling errors

Advanced Patterns

Multiple Queues

Configure Multiple Queues:

services.AddHangfireServer(options =>
{
    options.Queues = new[] { "critical", "default", "low-priority" };
    options.WorkerCount = 10;
});

Enqueue to Specific Queue:

this.backgroundJobClient.Enqueue<CriticalJob>(
    job => job.ProcessAsync(),
    "critical");  // Queue name

Job State Management

Access Job Context:

public class JobWithContext
{
    public async Task ProcessAsync()
    {
        // Get current job ID
        string jobId = BackgroundJob.Id;

        // Get job parameters
        int retryCount = BackgroundJob.GetJobParameter<int>("RetryCount");

        // Set job state
        BackgroundJob.SetJobParameter("CustomData", "value");

        // Cancel job
        BackgroundJob.Delete(jobId);
    }
}

Batch Jobs

Create Job Batches:

var batchId = BatchJob.StartNew(x =>
{
    x.Enqueue<Job1>(j => j.Execute());
    x.Enqueue<Job2>(j => j.Execute());
    x.Enqueue<Job3>(j => j.Execute());
});

// Continuation after all jobs complete
BatchJob.ContinueBatchWith(batchId, x =>
{
    x.Enqueue<FinalJob>(j => j.Execute());
});

Summary

Hangfire in the ConnectSoft Microservice Template provides:

• ✅ Persistent Storage: Jobs survive application restarts
• ✅ Reliability: Automatic retry mechanisms
• ✅ Dashboard: Web UI for monitoring and management
• ✅ Scalability: Multiple server instances
• ✅ Flexibility: Fire-and-forget, delayed, and recurring jobs
• ✅ Integration: Seamless ASP.NET Core DI integration
• ✅ Observability: OpenTelemetry instrumentation
• ✅ Testing: Support for unit and integration tests

By following these patterns, teams can:

• Offload Work: Process long-running operations asynchronously
• Schedule Tasks: Recurring jobs for maintenance and cleanup
• Ensure Reliability: Automatic retries for transient failures
• Monitor Jobs: Dashboard for real-time job monitoring
• Scale Processing: Multiple servers for high throughput
• Maintain Quality: Structured logging and exception handling
• Test Effectively: Unit and integration testing support

Hangfire transforms background job processing from a complex, error-prone task into a simple, reliable, and observable operation that integrates seamlessly with the microservice architecture.