Data Redaction in ConnectSoft Microservice Template

Purpose & Overview

Data Redaction is the process of automatically masking or removing sensitive information from logs, telemetry, and responses to protect Personally Identifiable Information (PII), credentials, and other sensitive data. In the ConnectSoft Microservice Template, data redaction is implemented using Microsoft.Extensions.Compliance.Redaction and ConnectSoft.Extensions.Compliance, providing automatic, policy-driven redaction that ensures sensitive data never appears in logs or telemetry.

Data redaction provides:

• Automatic Protection: Sensitive data is automatically redacted in structured logs
• Policy-Driven: Classification-based redaction rules applied consistently
• Compliance Ready: Supports GDPR, CCPA, and other privacy regulations
• Zero Trust Logging: Ensures credentials and secrets never appear in logs
• Developer-Friendly: Simple attribute-based classification
• Performance Optimized: Low-overhead redaction for high-throughput logging
• Idempotent: Safe to apply multiple times without distortion

Redaction Philosophy

Data redaction is a security-first approach to logging and telemetry. By classifying data at the source and applying automatic redaction, we ensure that sensitive information never leaks into logs, telemetry, or error messages. The template uses a taxonomy-based classification system that maps data types to appropriate redaction strategies, ensuring consistent protection across all logging and telemetry outputs.

Architecture Overview

Redaction Flow

Application Code
    ├── DTOs with Classification Attributes
    │   ├── [EmailData] string Email
    │   ├── [PhoneData] string Phone
    │   └── [SecretData] string ApiKey
    ├── Logger with Classified Parameters
    │   └── LogInformation("User {Email}", [EmailData] email)
    └── Manual Redaction
        └── IRedactorProvider.GetRedactor(classification)
    ↓
Microsoft.Extensions.Compliance.Redaction
    ├── Classification → Redactor Mapping
    ├── Redactor Selection
    └── Redaction Execution
    ↓
ConnectSoft.Extensions.Compliance
    ├── ConnectSoftTaxonomy (Classifications)
    ├── Custom Redactors (Email, Phone, JWT, etc.)
    └── Profile Configuration
    ↓
Structured Logs
    └── Redacted Values (Email: j**e@example.com)

Redaction Components

Component	Purpose	Location
ConnectSoftTaxonomy	Data classification definitions	ConnectSoft.Extensions.Compliance
IRedactorProvider	Redactor factory and selection	Microsoft.Extensions.Compliance.Redaction
Redactor	Base class for redaction logic	Microsoft.Extensions.Compliance.Redaction
EmailRedactor	Email address masking	ConnectSoft.Extensions.Compliance
PhoneLast4Redactor	Phone number masking	ConnectSoft.Extensions.Compliance
JwtRedactor	JWT token masking	ConnectSoft.Extensions.Compliance
ErasingRedactor	Complete value erasure	Microsoft.Extensions.Compliance.Redaction

Service Registration

AddMicroserviceRedaction Extension

Redaction is registered via AddMicroserviceRedaction():

// MicroserviceRegistrationExtensions.cs
services.AddMicroserviceRedaction(configuration, environment);

Implementation:

// MicroserviceRedactionExtensions.cs
internal static IServiceCollection AddMicroserviceRedaction(
    this IServiceCollection services, 
    IConfiguration configuration, 
    IHostEnvironment environment)
{
    ArgumentNullException.ThrowIfNull(services);
    ArgumentNullException.ThrowIfNull(configuration);
    ArgumentNullException.ThrowIfNull(environment);

    services.AddConnectSoftCompliance(configuration, environment);

    return services;
}

Logging Integration

Redaction is automatically enabled in the logging pipeline:

// SerilogLoggingExtensions.cs
services.AddLogging(loggingBuilder =>
{
    // Enable redaction of classified data in logs
    loggingBuilder.EnableRedaction();

    // ... rest of logging configuration
});

Data Classification Taxonomy

ConnectSoftTaxonomy

The template uses a standardized taxonomy for data classification:

// ConnectSoftTaxonomy.cs
public static class ConnectSoftTaxonomy
{
    public static string Name => typeof(ConnectSoftTaxonomy).FullName!;

    // PII (Personally Identifiable Information)
    public static DataClassification Email => new(Name, nameof(Email));
    public static DataClassification Phone => new(Name, nameof(Phone));
    public static DataClassification PersonName => new(Name, nameof(PersonName));
    public static DataClassification PostalAddress => new(Name, nameof(PostalAddress));
    public static DataClassification IpAddress => new(Name, nameof(IpAddress));
    public static DataClassification GeoLocation => new(Name, nameof(GeoLocation));
    public static DataClassification GovernmentId => new(Name, nameof(GovernmentId));

    // Financial Information
    public static DataClassification PaymentCardPan => new(Name, nameof(PaymentCardPan));
    public static DataClassification BankAccount => new(Name, nameof(BankAccount));
    public static DataClassification FinancialId => new(Name, nameof(FinancialId));

    // Credentials and Secrets
    public static DataClassification Secret => new(Name, nameof(Secret));
    public static DataClassification OAuthToken => new(Name, nameof(OAuthToken));
    public static DataClassification Jwt => new(Name, nameof(Jwt));
    public static DataClassification SessionId => new(Name, nameof(SessionId));

    // Device and Tracking
    public static DataClassification DeviceId => new(Name, nameof(DeviceId));
    public static DataClassification UserAgent => new(Name, nameof(UserAgent));

    // Health Information
    public static DataClassification HealthInfo => new(Name, nameof(HealthInfo));
}

Classification Attributes

Use attributes to classify data in DTOs:

using ConnectSoft.Extensions.Compliance;

public sealed class UserDto
{
    public string UserName { get; set; }  // Public (no classification)

    [EmailData]
    public string Email { get; set; }

    [PhoneData]
    public string PhoneNumber { get; set; }

    [SecretData]
    public string ApiKey { get; set; }

    [JwtData]
    public string AccessToken { get; set; }

    [IpAddressData]
    public string? ClientIpAddress { get; set; }

    [PaymentCardPan]
    public string? CreditCardNumber { get; set; }
}

Available Attributes: - [EmailData]: Email addresses - [PhoneData]: Phone numbers - [PaymentCardPan]: Payment card numbers - [JwtData]: JSON Web Tokens - [IpAddressData]: IP addresses - [GuidData]: Device identifiers (GUIDs) - [SecretData]: Generic secrets (passwords, API keys) - [HealthData]: Health information (PHI)

Redactors

Built-in Redactors

The template includes custom redactors for common data types:

EmailRedactor

Purpose: Masks email addresses while preserving domain

Example:

Input:  "john.doe@example.com"
Output: "j**e@example.com"

Input:  "ab@example.com"
Output: "**@example.com"  // Short local part fully masked

Characteristics: - Preserves domain name - Masks local part (characters before @) - Handles short email addresses gracefully - Idempotent (safe to apply multiple times)

PhoneLast4Redactor

Purpose: Shows only last 4 digits of phone numbers

Example:

Input:  "+1-555-123-4567"
Output: "****4567"

Characteristics: - Preserves last 4 digits - Masks all other digits - Culture-aware formatting

PanLast4Redactor

Purpose: Shows only last 4 digits of payment card numbers

Example:

Input:  "4532-1234-5678-9010"
Output: "**** **** **** 9010"

Characteristics: - Preserves last 4 digits - Formats as masked card number - Handles various card number formats

JwtRedactor

Purpose: Masks JWT tokens while preserving structure

Example:

Input:  "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c"
Output: "<jwt>.<redacted>.<redacted>"

Characteristics: - Preserves JWT structure (3 parts) - Masks header and payload - Masks signature - Or erased in strict mode

IpAddressRedactor

Purpose: Masks IP addresses

Example:

Input (IPv4):  "192.168.1.100"
Output:        "192.168.1.x"

Input (IPv6):  "2001:0db8:85a3:0000:0000:8a2e:0370:7334"
Output:        "2001:0db8:85a3:0000::/64"

Characteristics: - IPv4: Masks last octet - IPv6: Masks last 64 bits - Preserves network portion

ErasingRedactor

Purpose: Completely erases sensitive values

Example:

Input:  "my-secret-api-key"
Output: ""  // Empty string

Characteristics: - Used for secrets, tokens, session IDs - Complete erasure (no trace) - Applied to highest-risk data

Redactor Mapping

Redactors are mapped to classifications during service registration:

// ConnectSoftComplianceServiceCollectionExtensions.cs
services.AddRedaction(cfg =>
{
    // Erase high-risk credential tokens
    cfg.SetRedactor<ErasingRedactor>(new DataClassificationSet(
        ConnectSoftTaxonomy.Secret,
        ConnectSoftTaxonomy.OAuthToken,
        ConnectSoftTaxonomy.SessionId));

    // Email / JWT: strict mode in development
    var devStrict = environment?.IsDevelopment() == true && options.StrictInDevelopment;

    if (devStrict)
    {
        cfg.SetRedactor<ErasingRedactor>(new DataClassificationSet(
            ConnectSoftTaxonomy.Email,
            ConnectSoftTaxonomy.Jwt));
    }
    else
    {
        cfg.SetRedactor<EmailRedactor>(new DataClassificationSet(ConnectSoftTaxonomy.Email));
        cfg.SetRedactor<JwtRedactor>(new DataClassificationSet(ConnectSoftTaxonomy.Jwt));
    }

    // Map core custom redactors
    cfg.SetRedactor<PhoneLast4Redactor>(new DataClassificationSet(ConnectSoftTaxonomy.Phone));
    cfg.SetRedactor<PanLast4Redactor>(new DataClassificationSet(ConnectSoftTaxonomy.PaymentCardPan));
    cfg.SetRedactor<IpAddressRedactor>(new DataClassificationSet(ConnectSoftTaxonomy.IpAddress));
    cfg.SetRedactor<GuidRedactor>(new DataClassificationSet(ConnectSoftTaxonomy.DeviceId));
});

Configuration

ConnectSoftComplianceOptions

Configuration Class:

// ConnectSoftComplianceOptions.cs
public class ConnectSoftComplianceOptions
{
    public const string ComplianceSectionName = "Compliance";

    [Required]
    required public string Profile { get; set; } = "default";

    [Required]
    required public bool EnableLoggingRedaction { get; set; } = true;

    [Required]
    required public bool StrictInDevelopment { get; set; } = true;
}

appsettings.json Configuration

Example Configuration:

{
  "Compliance": {
    "EnableLoggingRedaction": true,
    "StrictInDevelopment": true,
    "Profile": "default"
  }
}

Configuration Parameters:

Parameter	Type	Description	Default
EnableLoggingRedaction	bool	Enable automatic redaction in logs	true
StrictInDevelopment	bool	Use stricter redaction in Development	true
Profile	string	Redaction profile (default/audit/strict)	"default"

Environment-Specific Configuration

Development (Strict Mode):

{
  "Compliance": {
    "EnableLoggingRedaction": true,
    "StrictInDevelopment": true,
    "Profile": "default"
  }
}

Production:

{
  "Compliance": {
    "EnableLoggingRedaction": true,
    "StrictInDevelopment": false,
    "Profile": "default"
  }
}

Usage Patterns

Pattern 1: Source-Generated Logger with Classified Parameters

Recommended for automatic redaction:

public static partial class UserLogs
{
    [LoggerMessage(
        EventId = 1001,
        Level = LogLevel.Information,
        Message = "User login attempt: email={Email}, ip={IpAddress}")]
    public static partial void LogLoginAttempt(
        ILogger logger,
        [EmailData] string email,
        [IpAddressData] string ipAddress);
}

// Usage
UserLogs.LogLoginAttempt(
    logger,
    "john.doe@example.com",
    "192.168.1.100");

// Output in logs:
// "User login attempt: email=j**e@example.com, ip=192.168.1.x"

Benefits: - Automatic redaction at callsite - Message template and structured state both redacted - Compile-time safety - Type-safe classifications

Pattern 2: DTO with LogProperties

For logging entire DTOs:

public sealed class LoginRequest
{
    [EmailData]
    public string Email { get; set; }

    [SecretData]
    public string Password { get; set; }
}

// Usage
private static void LogLoginRequest(ILogger logger, [LogProperties] LoginRequest request)
{
    logger.LogInformation("Login request: {@Request}", request);
}

// Output in logs:
// "Login request: { Request: { Email: "j**e@example.com", Password: "" } }"

Characteristics: - Redacts structured state automatically - Message template not redacted (uses ToString()) - Useful for complex objects

Pattern 3: Manual Redaction

For custom scenarios:

public class UserService
{
    private readonly IRedactorProvider redactorProvider;

    public UserService(IRedactorProvider redactorProvider)
    {
        this.redactorProvider = redactorProvider;
    }

    public string MaskEmail(string email)
    {
        var redactor = this.redactorProvider.GetRedactor(ConnectSoftTaxonomy.Email);
        Span<char> buffer = stackalloc char[redactor.GetRedactedLength(email)];
        var written = redactor.Redact(email, buffer);
        return new string(buffer[..written]);
    }
}

Use Cases: - Custom masking logic - HTTP headers/query parameters - API responses - Error messages

Pattern 4: Classified Parameters in Controllers

Redact in API logging:

[HttpPost("users")]
public async Task<IActionResult> CreateUser(
    [FromBody] CreateUserRequest request,
    ILogger<UsersController> logger)
{
    // Log with classified parameters
    UserLogs.LogUserCreation(
        logger,
        request.Email,      // Automatically redacted
        request.PhoneNumber); // Automatically redacted

    // ... process request
}

Redaction Profiles

Default Profile

Balanced masking: - Email: Masked (preserves domain) - Phone: Last 4 digits - PAN: Last 4 digits - JWT: Masked (preserves structure) - Secrets: Erased

Audit Profile

More structure preserved: - Email: Domain preserved, local part masked - Phone: Last 4 digits - PAN: Last 4 digits with formatting - JWT: Masked with structure - Secrets: Erased

Strict Profile

Maximum masking: - Email: Erased (or fixed mask) - Phone: Fully masked - PAN: Fully masked - JWT: Erased - Secrets: Erased

Development Strict Override

Development Environment:

When Environment=Development and StrictInDevelopment=true: - Email: Erased (regardless of profile) - JWT: Erased (regardless of profile) - Other types: Follow profile rules

Purpose: Reduce risk during development and debugging

Best Practices

Do's

1. Classify All Sensitive Data

   // ✅ GOOD - Classified properties
   public class UserDto
   {
       [EmailData]
       public string Email { get; set; }
   
       [PhoneData]
       public string Phone { get; set; }
   }
   

2. Use Source-Generated Loggers

   // ✅ GOOD - Automatic redaction
   [LoggerMessage(Message = "User login: {Email}")]
   public static partial void LogLogin(
       ILogger logger,
       [EmailData] string email);
   

3. Enable Redaction in Logging

   // ✅ GOOD - Redaction enabled
   services.AddLogging(loggingBuilder =>
   {
       loggingBuilder.EnableRedaction();
   });
   

4. Use Strict Mode in Development

   {
     "Compliance": {
       "StrictInDevelopment": true
     }
   }
   

5. Never Log Secrets Directly

   // ❌ BAD - Secret in log
   _logger.LogInformation("API key: {ApiKey}", apiKey);
   
   // ✅ GOOD - Classified and redacted
   [LoggerMessage(Message = "API key: {ApiKey}")]
   public static partial void LogApiKey(
       ILogger logger,
       [SecretData] string apiKey);
   

Don'ts

1. Don't Bypass Classification

   // ❌ BAD - Unclassified sensitive data
   public class UserDto
   {
       public string Email { get; set; }  // No [EmailData]
   }
   
   // ✅ GOOD - Classified
   public class UserDto
   {
       [EmailData]
       public string Email { get; set; }
   }
   

2. Don't Use String Interpolation with Sensitive Data

   // ❌ BAD - String interpolation bypasses redaction
   _logger.LogInformation($"User email: {email}");
   
   // ✅ GOOD - Message template with classification
   [LoggerMessage(Message = "User email: {Email}")]
   public static partial void LogEmail(
       ILogger logger,
       [EmailData] string email);
   

3. Don't Log Full Exception Details with Sensitive Data

   // ❌ BAD - Exception may contain sensitive data
   _logger.LogError(ex, "Operation failed");
   
   // ✅ GOOD - Redact exception details
   _logger.LogError(ex, "Operation failed"); // Exception details redacted automatically
   

4. Don't Disable Redaction in Production

   // ❌ BAD - No protection
   {
     "Compliance": {
       "EnableLoggingRedaction": false
     }
   }
   
   // ✅ GOOD - Always enabled
   {
     "Compliance": {
       "EnableLoggingRedaction": true
     }
   }
   

5. Don't Manually Mask Values

   // ❌ BAD - Manual masking (error-prone)
   var maskedEmail = email.Substring(0, 2) + "***" + email.Substring(email.IndexOf('@'));
   
   // ✅ GOOD - Use redactor
   var redactor = redactorProvider.GetRedactor(ConnectSoftTaxonomy.Email);
   var maskedEmail = RedactValue(redactor, email);
   

Common Redaction Scenarios

Scenario 1: User Registration

public sealed class RegisterUserRequest
{
    [EmailData]
    public string Email { get; set; }

    [PhoneData]
    public string PhoneNumber { get; set; }

    [SecretData]
    public string Password { get; set; }
}

[LoggerMessage(Message = "Registering user: {Email}, {PhoneNumber}")]
public static partial void LogUserRegistration(
    ILogger logger,
    [EmailData] string email,
    [PhoneData] string phoneNumber);

// Usage
LogUserRegistration(logger, request.Email, request.PhoneNumber);
// Output: "Registering user: j**e@example.com, ****4567"

Scenario 2: Payment Processing

public sealed class PaymentRequest
{
    [PaymentCardPan]
    public string CreditCardNumber { get; set; }

    [EmailData]
    public string BillingEmail { get; set; }
}

[LoggerMessage(Message = "Processing payment: {CardNumber}, {Email}")]
public static partial void LogPayment(
    ILogger logger,
    [PaymentCardPan] string cardNumber,
    [EmailData] string email);

// Usage
LogPayment(logger, request.CreditCardNumber, request.BillingEmail);
// Output: "Processing payment: **** **** **** 1234, j**e@example.com"

Scenario 3: API Authentication

public sealed class ApiRequest
{
    [JwtData]
    public string AuthorizationToken { get; set; }

    [IpAddressData]
    public string ClientIpAddress { get; set; }
}

[LoggerMessage(Message = "API request: token={Token}, ip={Ip}")]
public static partial void LogApiRequest(
    ILogger logger,
    [JwtData] string token,
    [IpAddressData] string ip);

// Usage
LogApiRequest(logger, request.AuthorizationToken, request.ClientIpAddress);
// Output: "API request: token=<jwt>.<redacted>.<redacted>, ip=192.168.1.x"

Troubleshooting

Issue: Data Not Being Redacted

Symptoms: Sensitive data appears in logs without masking.

Solutions: 1. Verify Redaction is Enabled

{
  "Compliance": {
    "EnableLoggingRedaction": true
  }
}

1. Check Logging Integration

   services.AddLogging(loggingBuilder =>
   {
       loggingBuilder.EnableRedaction(); // Must be enabled
   });
   

2. Verify Classification Attributes

   // Ensure properties are classified
   [EmailData]
   public string Email { get; set; }
   

3. Check Logger Message Format

   // ✅ GOOD - Message template
   [LoggerMessage(Message = "Email: {Email}")]
   public static partial void LogEmail(ILogger logger, [EmailData] string email);
   
   // ❌ BAD - String interpolation
   _logger.LogInformation($"Email: {email}");
   

Issue: Redaction Too Aggressive

Symptoms: Needed information is being erased or masked too heavily.

Solutions: 1. Check Profile Setting

{
  "Compliance": {
    "Profile": "default"  // Use "audit" for more visibility
  }
}

1. Disable Strict Mode in Development

   {
     "Compliance": {
       "StrictInDevelopment": false
     }
   }
   

2. Review Redactor Mappings

3. Check if ErasingRedactor is applied when masking is needed
4. Verify profile-specific redactor selection

Issue: Performance Impact

Symptoms: Logging performance degradation with redaction enabled.

Solutions: 1. Redaction is Optimized - Redactors use Span<char> for zero-allocation redaction - Pre-computed redacted lengths - Minimal overhead in hot paths

1. Profile Performance
2. Measure actual impact (usually < 1% overhead)
3. Use source-generated loggers for best performance
4. Avoid string interpolation in logging

Related Documentation

• Logging: Comprehensive logging documentation including redaction patterns
• HTTP Logging: HTTP request/response logging with redaction
• Exception Handling: Exception logging with redaction

Summary

Data redaction in the ConnectSoft Microservice Template provides:

• ✅ Automatic Protection: Sensitive data automatically redacted in logs
• ✅ Taxonomy-Based Classification: Standardized data classification system
• ✅ Custom Redactors: Specialized redactors for email, phone, JWT, PAN, etc.
• ✅ Policy-Driven: Configurable profiles and strict mode
• ✅ Zero Trust Logging: Credentials and secrets never appear in logs
• ✅ Developer-Friendly: Simple attribute-based classification
• ✅ Performance Optimized: Low-overhead redaction for high-throughput
• ✅ Compliance Ready: Supports GDPR, CCPA, and other privacy regulations

By following these patterns, teams can:

• Protect Sensitive Data: Automatically mask PII and credentials in logs
• Ensure Compliance: Meet privacy regulations with policy-driven redaction
• Maintain Security: Zero trust approach to logging and telemetry
• Simplify Development: Attribute-based classification with automatic enforcement
• Optimize Performance: Minimal overhead with optimized redactors

Data redaction is an essential security feature that ensures sensitive information never leaks into logs, telemetry, or error messages, protecting both users and organizations from data breaches and compliance violations.