HTTP Logging in ConnectSoft Microservice Template¶

Purpose & Overview¶

HTTP Logging in the ConnectSoft Microservice Template provides comprehensive visibility into HTTP request and response details for debugging, monitoring, and troubleshooting. The template supports two complementary HTTP logging mechanisms:

ASP.NET Core HTTP Logging: Low-level HTTP request/response logging with detailed protocol information
Serilog Request Logging: High-level structured request completion logging with enriched context

Both mechanisms work together to provide complete HTTP request observability, from raw protocol details to business-level context.

HTTP Logging Philosophy

HTTP logging in ConnectSoft provides visibility into all HTTP traffic while maintaining security and performance. Raw HTTP logging captures protocol-level details for debugging, while structured request logging provides business-level context for operational monitoring. Both mechanisms respect security boundaries and can be configured per environment.

Architecture Overview¶

Dual HTTP Logging System¶

Incoming HTTP Request
    ↓
ASP.NET Core HTTP Logging Middleware
    ├── Logs Request Headers
    ├── Logs Request Body (if enabled)
    ├── Logs Response Headers
    └── Logs Response Body (if enabled)
    ↓
Request Processing
    ↓
Serilog Request Logging Middleware
    ├── Logs Request Completion
    ├── Enriches with Context (Flow, CorrelationId, etc.)
    ├── Logs Status Code and Duration
    └── Uses Structured Logging
    ↓
Response Sent

Component Comparison¶

Feature	ASP.NET Core HTTP Logging	Serilog Request Logging
Purpose	Raw protocol-level logging	Structured business-level logging
Level	Low-level HTTP details	High-level request completion
Format	Raw HTTP format	Structured JSON with context
Body Logging	Supported with limits	Not included (use ASP.NET Core logging)
Performance	Higher overhead	Lower overhead
Use Case	Debugging, protocol analysis	Monitoring, operations, analytics
Enrichment	Minimal	Rich context (Flow, CorrelationId, etc.)
Security	Requires careful configuration	Context-aware, less sensitive

ASP.NET Core HTTP Logging¶

Overview¶

ASP.NET Core HTTP Logging provides low-level HTTP protocol logging, capturing raw request and response details including headers, bodies, and protocol information. This is useful for debugging protocol-level issues, analyzing request/response payloads, and understanding HTTP communication patterns.

Configuration¶

Service Registration:

// HttpLoggingExtensions.cs
internal static IServiceCollection AddMicroserviceHttpLogging(
    this IServiceCollection services, 
    IConfiguration configuration)
{
    ArgumentNullException.ThrowIfNull(services);
    ArgumentNullException.ThrowIfNull(configuration);

    bool isHttpLoggingEnabled = configuration.GetValue<bool>("HttpLogging:Enabled");
    if (isHttpLoggingEnabled)
    {
        services.AddHttpLogging(logging =>
        {
            logging.LoggingFields = HttpLoggingFields.All;
            logging.RequestBodyLogLimit = 4096;
            logging.ResponseBodyLogLimit = 4096;
        });
    }

    return services;
}

Middleware Registration:

// MicroserviceRegistrationExtensions.cs
internal static IApplicationBuilder UseMicroserviceHttpLogging(
    this IApplicationBuilder application, 
    IConfiguration configuration)
{
    ArgumentNullException.ThrowIfNull(application);
    ArgumentNullException.ThrowIfNull(configuration);

    bool isHttpLoggingEnabled = configuration.GetValue<bool>("HttpLogging:Enabled");
    if (isHttpLoggingEnabled)
    {
        application.UseHttpLogging();
    }

    return application;
}

Configuration File:

{
  "HttpLogging": {
    "Enabled": true
  }
}

Middleware Placement¶

HTTP logging middleware must be placed early in the pipeline, before routing but after exception handling:

// Middleware order (important!)
application.UseForwardedHeaders();           // 1. Process forwarded headers
application.UseExceptionHandler();            // 2. Handle exceptions
application.UseConversationId();             // 3. Set correlation ID
application.UseMicroserviceHttpLogging();    // 4. Log HTTP requests (before routing)
application.UseRouting();                    // 5. Route requests
application.UseMicroserviceSerilogRequestLogging(); // 6. Log request completion (after routing)

Why Before Routing? - HTTP logging captures raw request details before routing modifies the request - Allows logging of requests that don't match any route - Captures protocol-level information early in the pipeline

Logging Fields¶

HttpLoggingFields Options:

services.AddHttpLogging(logging =>
{
    // Log all fields
    logging.LoggingFields = HttpLoggingFields.All;

    // Or selectively enable fields
    logging.LoggingFields = HttpLoggingFields.RequestProperties | 
                           HttpLoggingFields.ResponseProperties |
                           HttpLoggingFields.RequestHeaders |
                           HttpLoggingFields.ResponseHeaders |
                           HttpLoggingFields.RequestBody |
                           HttpLoggingFields.ResponseBody |
                           HttpLoggingFields.RequestTrailers |
                           HttpLoggingFields.ResponseTrailers;
});

Available Fields:

Field	Description	Example
`RequestProperties`	Method, path, protocol, scheme	`GET /api/orders HTTP/1.1`
`RequestHeaders`	All request headers	`Authorization: Bearer ...`
`RequestBody`	Request body content	`{"orderId": "123"}`
`RequestTrailers`	Request trailers	`Trailer: ...`
`ResponseProperties`	Status code, headers	`200 OK`
`ResponseHeaders`	All response headers	`Content-Type: application/json`
`ResponseBody`	Response body content	`{"id": "123", "status": "created"}`
`ResponseTrailers`	Response trailers	`Trailer: ...`

Body Logging¶

Request Body Logging:

services.AddHttpLogging(logging =>
{
    logging.LoggingFields = HttpLoggingFields.RequestBody;
    logging.RequestBodyLogLimit = 4096; // Log first 4KB
});

Response Body Logging:

services.AddHttpLogging(logging =>
{
    logging.LoggingFields = HttpLoggingFields.ResponseBody;
    logging.ResponseBodyLogLimit = 4096; // Log first 4KB
});

Body Logging Limits: - RequestBodyLogLimit: Maximum bytes of request body to log (default: 32KB) - ResponseBodyLogLimit: Maximum bytes of response body to log (default: 32KB) - Bodies larger than the limit are truncated - Set to 0 to disable body logging

Log Output Format¶

Example Log Entry:

info: Microsoft.AspNetCore.HttpLogging.HttpLoggingMiddleware[1]
      Request:
      Protocol: HTTP/1.1
      Method: POST
      Scheme: https
      Path: /api/orders
      PathBase: 
      QueryString: ?status=pending
      Headers:
      Content-Type: application/json
      Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...
      Body:
      {"orderId":"123","customerId":"456"}
      info: Microsoft.AspNetCore.HttpLogging.HttpLoggingMiddleware[2]
      Response:
      StatusCode: 201
      Headers:
      Content-Type: application/json
      Location: /api/orders/123
      Body:
      {"id":"123","status":"created"}

Selective Header Logging¶

Include Specific Headers:

services.AddHttpLogging(logging =>
{
    logging.LoggingFields = HttpLoggingFields.RequestHeaders | 
                           HttpLoggingFields.ResponseHeaders;

    // Log only specific headers
    logging.RequestHeaders.Add("X-Custom-Header");
    logging.RequestHeaders.Add("X-Correlation-Id");
    logging.ResponseHeaders.Add("X-Response-Time");
});

Exclude Sensitive Headers:

services.AddHttpLogging(logging =>
{
    logging.LoggingFields = HttpLoggingFields.RequestHeaders;

    // Remove sensitive headers from logging
    logging.RequestHeaders.Remove("Authorization");
    logging.RequestHeaders.Remove("Cookie");
    logging.RequestHeaders.Remove("X-Api-Key");
});

Default Excluded Headers: - Authorization - Cookie - Set-Cookie

Serilog Request Logging¶

Overview¶

Serilog Request Logging provides high-level structured logging of HTTP request completion events. It captures request metadata, status codes, duration, and enriched context (correlation IDs, flow names, etc.) in a structured format suitable for operational monitoring and analytics.

See Logging for detailed Serilog request logging documentation.

Integration with HTTP Logging¶

Complementary Usage:

// Both can be used together
application.UseMicroserviceHttpLogging(configuration);      // Raw HTTP details
application.UseRouting();
application.UseMicroserviceSerilogRequestLogging();         // Structured completion events

When to Use Each:

ASP.NET Core HTTP Logging: Use for debugging protocol issues, analyzing request/response payloads, development
Serilog Request Logging: Use for production monitoring, operational dashboards, business analytics

Security Considerations¶

Sensitive Data Protection¶

⚠️ Critical: HTTP logging can expose sensitive data including credentials, tokens, personal information, and business data.

Best Practices:

Disable Body Logging in Production:

services.AddHttpLogging(logging =>
{
    logging.LoggingFields = HttpLoggingFields.RequestProperties | 
                           HttpLoggingFields.ResponseProperties |
                           HttpLoggingFields.RequestHeaders |
                           HttpLoggingFields.ResponseHeaders;
    // Disable body logging
    logging.RequestBodyLogLimit = 0;
    logging.ResponseBodyLogLimit = 0;
});

Exclude Sensitive Headers:

services.AddHttpLogging(logging =>
{
    logging.LoggingFields = HttpLoggingFields.RequestHeaders;

    // Remove sensitive headers
    logging.RequestHeaders.Remove("Authorization");
    logging.RequestHeaders.Remove("Cookie");
    logging.RequestHeaders.Remove("X-Api-Key");
    logging.RequestHeaders.Remove("X-Secret");
});

Exclude Sensitive Endpoints:

services.AddHttpLogging(logging =>
{
    logging.LoggingFields = HttpLoggingFields.All;

    // Configure endpoint-specific logging
    // (Requires custom middleware or filtering)
});

Use Environment-Specific Configuration:

// appsettings.Development.json
{
  "HttpLogging": {
    "Enabled": true,
    "LogRequestBody": true,
    "LogResponseBody": true
  }
}

// appsettings.Production.json
{
  "HttpLogging": {
    "Enabled": true,
    "LogRequestBody": false,
    "LogResponseBody": false
  }
}

Considerations: - Request/response bodies may contain personal data - Headers may contain user identifiers - Logs may be subject to data retention policies - Consider log encryption and access controls

Recommendations: - Disable body logging in production - Exclude user-identifying headers - Implement log retention policies - Encrypt log storage - Restrict log access to authorized personnel

Performance Considerations¶

Impact on Performance¶

HTTP logging adds overhead to every request:

Memory Usage: Request/response bodies are buffered in memory
CPU Usage: Logging operations consume CPU cycles
I/O Operations: Log writes to sinks (file, network, etc.)
Network Bandwidth: Logs transmitted to remote sinks

Performance Optimization¶

1. Limit Body Logging:

services.AddHttpLogging(logging =>
{
    // Limit body size to reduce memory usage
    logging.RequestBodyLogLimit = 1024;  // 1KB instead of 4KB
    logging.ResponseBodyLogLimit = 1024;
});

2. Selective Field Logging:

services.AddHttpLogging(logging =>
{
    // Log only essential fields
    logging.LoggingFields = HttpLoggingFields.RequestProperties | 
                           HttpLoggingFields.ResponseProperties;
    // Exclude headers and bodies for performance
});

3. Disable in Production:

// appsettings.Production.json
{
  "HttpLogging": {
    "Enabled": false
  }
}

4. Use Async Logging: - Serilog request logging is asynchronous by default - ASP.NET Core HTTP logging writes to ILogger (which may be async depending on provider)

5. Filter Low-Value Requests:

// Custom middleware to filter health checks, static files, etc.
public class FilteredHttpLoggingMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILogger _logger;

    public async Task InvokeAsync(HttpContext context)
    {
        // Skip logging for health checks
        if (context.Request.Path.StartsWithSegments("/health"))
        {
            await _next(context);
            return;
        }

        // Use HTTP logging for other requests
        await _next(context);
    }
}

Configuration Options¶

Full Configuration¶

appsettings.json:

{
  "HttpLogging": {
    "Enabled": true,
    "LoggingFields": "All",
    "RequestBodyLogLimit": 4096,
    "ResponseBodyLogLimit": 4096,
    "RequestHeaders": {
      "Include": ["X-Correlation-Id", "X-Request-Id"],
      "Exclude": ["Authorization", "Cookie"]
    },
    "ResponseHeaders": {
      "Include": ["X-Response-Time", "X-Request-Id"],
      "Exclude": ["Set-Cookie"]
    }
  }
}

Programmatic Configuration:

services.AddHttpLogging(logging =>
{
    // Enable/disable
    // Controlled by HttpLogging:Enabled in config

    // Logging fields
    logging.LoggingFields = HttpLoggingFields.RequestProperties | 
                           HttpLoggingFields.ResponseProperties |
                           HttpLoggingFields.RequestHeaders |
                           HttpLoggingFields.ResponseHeaders;

    // Body limits
    logging.RequestBodyLogLimit = 4096;
    logging.ResponseBodyLogLimit = 4096;

    // Header filtering
    logging.RequestHeaders.Add("X-Custom-Header");
    logging.RequestHeaders.Remove("Authorization");
    logging.ResponseHeaders.Add("X-Response-Time");
    logging.ResponseHeaders.Remove("Set-Cookie");
});

Environment-Specific Configuration¶

Development (appsettings.Development.json):

{
  "HttpLogging": {
    "Enabled": true,
    "RequestBodyLogLimit": 8192,
    "ResponseBodyLogLimit": 8192
  }
}

Production (appsettings.Production.json):

{
  "HttpLogging": {
    "Enabled": true,
    "RequestBodyLogLimit": 0,
    "ResponseBodyLogLimit": 0
  }
}

Advanced Scenarios¶

Custom HTTP Logging Middleware¶

Create Custom Middleware:

public class CustomHttpLoggingMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILogger<CustomHttpLoggingMiddleware> _logger;

    public CustomHttpLoggingMiddleware(
        RequestDelegate next,
        ILogger<CustomHttpLoggingMiddleware> logger)
    {
        _next = next;
        _logger = logger;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        // Log request
        _logger.LogInformation(
            "Request: {Method} {Path} from {RemoteIp}",
            context.Request.Method,
            context.Request.Path,
            context.Connection.RemoteIpAddress);

        var stopwatch = Stopwatch.StartNew();

        await _next(context);

        stopwatch.Stop();

        // Log response
        _logger.LogInformation(
            "Response: {StatusCode} in {ElapsedMs}ms",
            context.Response.StatusCode,
            stopwatch.ElapsedMilliseconds);
    }
}

// Register
application.UseMiddleware<CustomHttpLoggingMiddleware>();

Endpoint-Specific Logging¶

Filter by Endpoint:

public class SelectiveHttpLoggingMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILogger _logger;

    public async Task InvokeAsync(HttpContext context)
    {
        var endpoint = context.GetEndpoint();

        // Skip logging for health checks
        if (endpoint?.DisplayName?.Contains("Health") == true)
        {
            await _next(context);
            return;
        }

        // Skip logging for static files
        if (context.Request.Path.StartsWithSegments("/static"))
        {
            await _next(context);
            return;
        }

        // Use standard HTTP logging for other endpoints
        await _next(context);
    }
}

Conditional Body Logging¶

Log Bodies Only for Specific Endpoints:

services.AddHttpLogging(logging =>
{
    // Disable body logging by default
    logging.RequestBodyLogLimit = 0;
    logging.ResponseBodyLogLimit = 0;

    // Enable body logging for specific endpoints via custom middleware
});

Integration with Observability¶

OpenTelemetry Integration¶

HTTP logging integrates with OpenTelemetry for distributed tracing:

// OpenTelemetry automatically instruments HTTP requests
services.AddOpenTelemetry()
    .WithTracing(builder => builder
        .AddHttpClientInstrumentation()  // Outgoing requests
        .AddAspNetCoreInstrumentation()  // Incoming requests (includes HTTP logging)
    );

Benefits: - HTTP logs automatically linked to trace spans - End-to-end request visibility - Correlation between logs and traces

Application Insights Integration¶

HTTP logging data is automatically captured by Application Insights:

services.AddApplicationInsightsTelemetry();

Captured Data: - Request/response telemetry - Duration and status codes - Custom properties from logs

Testing¶

Unit Testing¶

Test HTTP Logging Configuration:

[TestMethod]
public void HttpLogging_ShouldBeConfigured()
{
    // Arrange
    var services = new ServiceCollection();
    var configuration = new ConfigurationBuilder()
        .AddInMemoryCollection(new Dictionary<string, string>
        {
            ["HttpLogging:Enabled"] = "true"
        })
        .Build();

    // Act
    services.AddMicroserviceHttpLogging(configuration);
    var provider = services.BuildServiceProvider();

    // Assert
    var httpLoggingOptions = provider.GetService<IOptions<HttpLoggingOptions>>();
    Assert.IsNotNull(httpLoggingOptions);
}

Integration Testing¶

Test HTTP Logging Output:

[TestMethod]
public async Task HttpLogging_ShouldLogRequest()
{
    // Arrange
    var factory = new WebApplicationFactory<Program>();
    var client = factory.CreateClient();

    // Capture logs
    var logSink = new TestLogSink();
    // Configure test logger to use sink

    // Act
    var response = await client.GetAsync("/api/test");

    // Assert
    Assert.IsTrue(logSink.Logs.Any(log => 
        log.Contains("Request:") && 
        log.Contains("GET /api/test")));
}

Troubleshooting¶

Issue: HTTP Logging Not Working¶

Symptoms: No HTTP logs appearing in output.

Solutions: 1. Verify HttpLogging:Enabled is set to true in configuration 2. Check middleware order (must be before routing) 3. Verify logger is configured correctly 4. Check log level settings

Issue: Too Much Logging¶

Symptoms: Excessive log output, performance degradation.

Solutions: 1. Reduce RequestBodyLogLimit and ResponseBodyLogLimit 2. Limit LoggingFields to essential fields only 3. Disable body logging in production 4. Filter out health checks and static files

Issue: Sensitive Data in Logs¶

Symptoms: Credentials or PII appearing in logs.

Solutions: 1. Disable body logging: RequestBodyLogLimit = 0, ResponseBodyLogLimit = 0 2. Remove sensitive headers: RequestHeaders.Remove("Authorization") 3. Review log output for sensitive data 4. Implement log redaction if needed

Issue: Performance Impact¶

Symptoms: Slow request processing, high memory usage.

Solutions: 1. Reduce body logging limits 2. Disable body logging entirely 3. Use selective field logging 4. Consider disabling in production 5. Use async logging providers

Best Practices¶

Do's¶

Enable HTTP Logging in Development

// appsettings.Development.json
{
  "HttpLogging": {
    "Enabled": true
  }
}

Disable Body Logging in Production

// appsettings.Production.json
{
  "HttpLogging": {
    "Enabled": true,
    "RequestBodyLogLimit": 0,
    "ResponseBodyLogLimit": 0
  }
}

Exclude Sensitive Headers

services.AddHttpLogging(logging =>
{
    logging.RequestHeaders.Remove("Authorization");
    logging.RequestHeaders.Remove("Cookie");
});

Use Appropriate Middleware Order

application.UseExceptionHandler();      // Before HTTP logging
application.UseMicroserviceHttpLogging(); // Early in pipeline
application.UseRouting();               // After HTTP logging

Monitor Log Volume
Track log file sizes
Monitor log sink performance
Set up alerts for excessive logging

Don'ts¶

Don't Log Bodies in Production

// ❌ BAD - Exposes sensitive data
logging.RequestBodyLogLimit = 32768;
logging.ResponseBodyLogLimit = 32768;

// ✅ GOOD - Disable body logging
logging.RequestBodyLogLimit = 0;
logging.ResponseBodyLogLimit = 0;

Don't Log Sensitive Headers

// ❌ BAD - Exposes credentials
// Authorization header logged by default

// ✅ GOOD - Remove sensitive headers
logging.RequestHeaders.Remove("Authorization");

Don't Place Middleware After Routing

// ❌ BAD - Logs after routing (may miss some requests)
application.UseRouting();
application.UseMicroserviceHttpLogging();

// ✅ GOOD - Logs before routing
application.UseMicroserviceHttpLogging();
application.UseRouting();

Don't Use Unlimited Body Limits

// ❌ BAD - Memory risk
logging.RequestBodyLogLimit = int.MaxValue;

// ✅ GOOD - Reasonable limit
logging.RequestBodyLogLimit = 4096;

Summary¶

HTTP Logging in the ConnectSoft Microservice Template provides:

✅ Dual Logging System: ASP.NET Core HTTP logging for protocol details, Serilog for structured events
✅ Configurable Fields: Selective logging of headers, bodies, properties
✅ Security Conscious: Header filtering, body limits, sensitive data protection
✅ Performance Optimized: Configurable limits, environment-specific settings
✅ Observability Ready: Integration with OpenTelemetry and Application Insights
✅ Production Ready: Environment-specific configurations, best practices

By following these patterns, teams can:

Debug Effectively: Protocol-level visibility for troubleshooting
Monitor Operations: Structured request logging for dashboards and analytics
Maintain Security: Protection of sensitive data in logs
Optimize Performance: Configurable logging to balance visibility and performance
Comply with Regulations: GDPR-compliant logging practices

The HTTP logging infrastructure ensures that ConnectSoft microservices provide comprehensive HTTP request visibility while maintaining security, performance, and compliance requirements.

HTTP Logging in ConnectSoft Microservice Template¶

Purpose & Overview¶

Architecture Overview¶

Dual HTTP Logging System¶

Component Comparison¶

ASP.NET Core HTTP Logging¶

Overview¶

Configuration¶

Middleware Placement¶

Logging Fields¶

Body Logging¶

Log Output Format¶

Selective Header Logging¶

Serilog Request Logging¶

Overview¶

Integration with HTTP Logging¶

Security Considerations¶

Sensitive Data Protection¶

PII and GDPR Compliance¶

Performance Considerations¶

Impact on Performance¶

Performance Optimization¶

Configuration Options¶

Full Configuration¶

Environment-Specific Configuration¶

Advanced Scenarios¶

Custom HTTP Logging Middleware¶

Endpoint-Specific Logging¶

Conditional Body Logging¶

Integration with Observability¶

OpenTelemetry Integration¶

Application Insights Integration¶

Testing¶

Unit Testing¶

Integration Testing¶

Troubleshooting¶

Issue: HTTP Logging Not Working¶

Issue: Too Much Logging¶

Issue: Sensitive Data in Logs¶

Issue: Performance Impact¶

Best Practices¶

Do's¶

Don'ts¶

Summary¶