Observability Security

This document covers security best practices for observability stacks.

Threat Model

Attack Vectors

1. Data Exfiltration: Unauthorized access to telemetry data
2. Data Injection: Malicious data injected into observability pipeline
3. Denial of Service: Overwhelming observability infrastructure
4. Privilege Escalation: Gaining unauthorized access

Network Security

Network Segmentation

Kubernetes Network Policies:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: observability-isolation
spec:
  podSelector:
    matchLabels:
      component: observability
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: default
      ports:
        - protocol: TCP
          port: 4317
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              name: observability

Firewall Rules

• Restrict access to observability endpoints
• Allow only necessary ports
• Block external access to internal services

Transport Security

TLS/SSL

Collector TLS Configuration:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        tls:
          cert_file: /etc/certs/server.crt
          key_file: /etc/certs/server.key
          client_ca_file: /etc/certs/ca.crt  # mTLS

Exporter TLS Configuration:

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      cert_file: /etc/certs/client.crt
      key_file: /etc/certs/client.key
      insecure_skip_verify: false

Certificate Management

1. Use Certificates: Never use self-signed in production
2. Rotate Certificates: Regular rotation schedule
3. Certificate Authority: Use trusted CA
4. mTLS: Enable mutual TLS where possible

Authentication and Authorization

API Keys

Seq API Key:

exporters:
  otlp/seq:
    endpoint: http://seq:5342/ingest/otlp/v1
    headers:
      X-Seq-ApiKey: ${SEQ_API_KEY}

Best Practices:

• Store in secrets management
• Rotate regularly
• Use least privilege
• Audit key usage

OAuth2 / JWT

Implementation:

• Use service mesh for mTLS
• Implement OAuth2 for API access
• Validate JWT tokens
• Use short-lived tokens

RBAC

Kubernetes RBAC:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: observability-reader
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["otel-collector-config"]
    verbs: ["get"]

Data Protection

PII Redaction

Processor Configuration:

processors:
  attributes:
    actions:
      - key: user.email
        action: delete
      - key: user.credit_card
        action: delete

Best Practices:

• Identify PII in telemetry
• Redact before export
• Document redaction policies
• Regular audits

Data Masking

Example:

processors:
  attributes:
    actions:
      - key: password
        action: update
        value: "***REDACTED***"

Encryption at Rest

Storage Encryption:

• Enable encryption for Elasticsearch
• Encrypt Prometheus storage
• Use encrypted volumes
• Key management

Encryption in Transit

• Always use TLS
• Enable mTLS where possible
• Use strong cipher suites
• Regular cipher suite updates

Access Control

Principle of Least Privilege

1. Minimal Permissions: Grant only necessary permissions
2. Role-Based Access: Use roles, not individual permissions
3. Regular Reviews: Audit and review permissions
4. Remove Unused: Remove unused accounts/permissions

Audit Logging

Enable Audit Logs:

• Log all access to observability data
• Log configuration changes
• Log authentication attempts
• Regular audit log reviews

Secrets Management

Kubernetes Secrets

apiVersion: v1
kind: Secret
metadata:
  name: observability-secrets
type: Opaque
stringData:
  seq-api-key: <api-key>
  azure-connection-string: <connection-string>

Best Practices:

• Use sealed secrets or external secrets operator
• Rotate secrets regularly
• Encrypt secrets at rest
• Limit secret access

External Secrets

Azure Key Vault:

• Store secrets in Key Vault
• Use managed identities
• Rotate secrets automatically

AWS Secrets Manager:

• Store secrets in Secrets Manager
• Use IAM roles
• Enable automatic rotation

Vulnerability Management

Container Security

1. Base Images: Use minimal base images
2. Image Scanning: Scan for vulnerabilities
3. Regular Updates: Keep images updated
4. Non-Root: Run as non-root user

Dependency Management

1. Dependency Scanning: Regular scans
2. Update Dependencies: Keep updated
3. Vulnerability Database: Monitor CVEs
4. Patch Management: Rapid patching

Incident Response

Detection

1. Monitoring: Monitor for anomalies
2. Alerts: Set up security alerts
3. Log Analysis: Regular log reviews
4. Threat Intelligence: Use threat feeds

Response

1. Isolation: Isolate affected systems
2. Investigation: Investigate incidents
3. Remediation: Fix vulnerabilities
4. Documentation: Document incidents

Compliance

GDPR

1. Data Minimization: Collect only necessary data
2. Right to Erasure: Ability to delete data
3. Data Portability: Export data capability
4. Privacy by Design: Build privacy in

SOC 2

1. Access Controls: Implement access controls
2. Audit Logs: Maintain audit logs
3. Encryption: Encrypt data
4. Monitoring: Monitor security events

HIPAA

1. PHI Protection: Protect health information
2. Access Controls: Strict access controls
3. Audit Trails: Comprehensive audit trails
4. Encryption: Encrypt PHI

Security Checklist

Deployment

• TLS enabled for all connections
• Network policies configured
• Secrets stored securely
• RBAC configured
• Non-root containers
• Resource limits set
• Health checks configured

Configuration

• PII redaction configured
• Authentication enabled
• Audit logging enabled
• Retention policies set
• Backup configured
• Monitoring enabled

Operations

• Regular security updates
• Vulnerability scanning
• Access reviews
• Incident response plan
• Security training
• Compliance audits

Further Reading

• Observability Production Patterns
• OpenTelemetry Collector
• Observability Stacks: Observability stack comparison and selection