Skip to content

Observability Security

This document covers security best practices for observability stacks.

Threat Model

Attack Vectors

  1. Data Exfiltration: Unauthorized access to telemetry data
  2. Data Injection: Malicious data injected into observability pipeline
  3. Denial of Service: Overwhelming observability infrastructure
  4. Privilege Escalation: Gaining unauthorized access

Network Security

Network Segmentation

Kubernetes Network Policies: 

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: observability-isolation
spec:
  podSelector:
    matchLabels:
      component: observability
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: default
      ports:
        - protocol: TCP
          port: 4317
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              name: observability

Firewall Rules

  • Restrict access to observability endpoints
  • Allow only necessary ports
  • Block external access to internal services

Transport Security

TLS/SSL

Collector TLS Configuration: 

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        tls:
          cert_file: /etc/certs/server.crt
          key_file: /etc/certs/server.key
          client_ca_file: /etc/certs/ca.crt  # mTLS

Exporter TLS Configuration: 

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      cert_file: /etc/certs/client.crt
      key_file: /etc/certs/client.key
      insecure_skip_verify: false

Certificate Management

  1. Use Certificates: Never use self-signed in production
  2. Rotate Certificates: Regular rotation schedule
  3. Certificate Authority: Use trusted CA
  4. mTLS: Enable mutual TLS where possible

Authentication and Authorization

API Keys

Seq API Key: 

exporters:
  otlp/seq:
    endpoint: http://seq:5342/ingest/otlp/v1
    headers:
      X-Seq-ApiKey: ${SEQ_API_KEY}

Best Practices: - Store in secrets management - Rotate regularly - Use least privilege - Audit key usage

OAuth2 / JWT

Implementation: - Use service mesh for mTLS - Implement OAuth2 for API access - Validate JWT tokens - Use short-lived tokens

RBAC

Kubernetes RBAC: 

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: observability-reader
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["otel-collector-config"]
    verbs: ["get"]

Data Protection

PII Redaction

Processor Configuration: 

processors:
  attributes:
    actions:
      - key: user.email
        action: delete
      - key: user.credit_card
        action: delete

Best Practices: - Identify PII in telemetry - Redact before export - Document redaction policies - Regular audits

Data Masking

Example: 

processors:
  attributes:
    actions:
      - key: password
        action: update
        value: "***REDACTED***"

Encryption at Rest

Storage Encryption: - Enable encryption for Elasticsearch - Encrypt Prometheus storage - Use encrypted volumes - Key management

Encryption in Transit

  • Always use TLS
  • Enable mTLS where possible
  • Use strong cipher suites
  • Regular cipher suite updates

Access Control

Principle of Least Privilege

  1. Minimal Permissions: Grant only necessary permissions
  2. Role-Based Access: Use roles, not individual permissions
  3. Regular Reviews: Audit and review permissions
  4. Remove Unused: Remove unused accounts/permissions

Audit Logging

Enable Audit Logs: - Log all access to observability data - Log configuration changes - Log authentication attempts - Regular audit log reviews

Secrets Management

Kubernetes Secrets

apiVersion: v1
kind: Secret
metadata:
  name: observability-secrets
type: Opaque
stringData:
  seq-api-key: <api-key>
  azure-connection-string: <connection-string>

Best Practices: - Use sealed secrets or external secrets operator - Rotate secrets regularly - Encrypt secrets at rest - Limit secret access

External Secrets

Azure Key Vault: - Store secrets in Key Vault - Use managed identities - Rotate secrets automatically

AWS Secrets Manager: - Store secrets in Secrets Manager - Use IAM roles - Enable automatic rotation

Vulnerability Management

Container Security

  1. Base Images: Use minimal base images
  2. Image Scanning: Scan for vulnerabilities
  3. Regular Updates: Keep images updated
  4. Non-Root: Run as non-root user

Dependency Management

  1. Dependency Scanning: Regular scans
  2. Update Dependencies: Keep updated
  3. Vulnerability Database: Monitor CVEs
  4. Patch Management: Rapid patching

Incident Response

Detection

  1. Monitoring: Monitor for anomalies
  2. Alerts: Set up security alerts
  3. Log Analysis: Regular log reviews
  4. Threat Intelligence: Use threat feeds

Response

  1. Isolation: Isolate affected systems
  2. Investigation: Investigate incidents
  3. Remediation: Fix vulnerabilities
  4. Documentation: Document incidents

Compliance

GDPR

  1. Data Minimization: Collect only necessary data
  2. Right to Erasure: Ability to delete data
  3. Data Portability: Export data capability
  4. Privacy by Design: Build privacy in

SOC 2

  1. Access Controls: Implement access controls
  2. Audit Logs: Maintain audit logs
  3. Encryption: Encrypt data
  4. Monitoring: Monitor security events

HIPAA

  1. PHI Protection: Protect health information
  2. Access Controls: Strict access controls
  3. Audit Trails: Comprehensive audit trails
  4. Encryption: Encrypt PHI

Security Checklist

Deployment

  • TLS enabled for all connections
  • Network policies configured
  • Secrets stored securely
  • RBAC configured
  • Non-root containers
  • Resource limits set
  • Health checks configured

Configuration

  • PII redaction configured
  • Authentication enabled
  • Audit logging enabled
  • Retention policies set
  • Backup configured
  • Monitoring enabled

Operations

  • Regular security updates
  • Vulnerability scanning
  • Access reviews
  • Incident response plan
  • Security training
  • Compliance audits

Further Reading