Skip to content

Observability Scaling

This document covers strategies for scaling observability infrastructure.

Scaling Dimensions

Horizontal Scaling

Add more instances to handle increased load.

Benefits: - Linear scaling - No downtime - Better fault tolerance

Challenges: - State management - Load distribution - Coordination

Vertical Scaling

Increase resources of existing instances.

Benefits: - Simple - No coordination needed - Lower complexity

Challenges: - Limited by hardware - Downtime for scaling - Cost at scale

Collector Scaling

Horizontal Scaling Pattern

Kubernetes Deployment: 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
spec:
  replicas: 3  # Scale based on load
  strategy:
    type: RollingUpdate

Auto-scaling: 

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: otel-collector-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: otel-collector
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Load Balancing

Service Configuration: 

apiVersion: v1
kind: Service
metadata:
  name: otel-collector
spec:
  type: ClusterIP
  ports:
    - port: 4317
      targetPort: 4317
  selector:
    app: otel-collector

Performance Tuning

Batch Processing: 

processors:
  batch:
    timeout: 5s  # Increase for better batching
    send_batch_size: 2048
    send_batch_max_size: 4096

Memory Management: 

processors:
  memory_limiter:
    limit_mib: 400
    spike_limit_mib: 100
    check_interval: 5s

Backend Scaling

Prometheus Scaling

Federation Pattern: 

# Global Prometheus
scrape_configs:
  - job_name: 'federate'
    scrape_interval: 15s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job="otel-collector"}'
    static_configs:
      - targets:
          - 'prometheus-region-1:9090'
          - 'prometheus-region-2:9090'

Thanos for Long-term Storage: - Global query view - Long-term retention - Deduplication - Compression

Elasticsearch Scaling

Cluster Configuration: 

# Elasticsearch cluster with 3 nodes
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
spec:
  replicas: 3
  serviceName: elasticsearch
  template:
    spec:
      containers:
        - name: elasticsearch
          env:
            - name: discovery.seed_hosts
              value: "elasticsearch-0,elasticsearch-1,elasticsearch-2"
            - name: cluster.initial_master_nodes
              value: "elasticsearch-0,elasticsearch-1,elasticsearch-2"

Sharding Strategy: - Primary shards: 3-5 per index - Replica shards: 1-2 per primary - Index per day/week

Jaeger Scaling

Storage Backend Scaling: - Memory: Not scalable (single instance) - Elasticsearch: Scale Elasticsearch cluster - Cassandra: Scale Cassandra cluster

Collector Scaling: 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger-collector
spec:
  replicas: 3

Capacity Planning

Metrics Collection

Estimate: - Metrics per service: 100-1000 - Services: 10-100 - Total metrics: 1,000-100,000 - Sample rate: 15s - Storage: ~1KB per metric per sample

Calculation: 

Storage per day = metrics × samples_per_day × size_per_sample
Storage per day = 10,000 × 5,760 × 1KB = ~55GB

Trace Collection

Estimate: - Traces per request: 1 - Requests per second: 1000 - Spans per trace: 10 - Storage: ~1KB per span

Calculation: 

Spans per day = requests_per_second × spans_per_trace × seconds_per_day
Spans per day = 1,000 × 10 × 86,400 = 864,000,000
Storage per day = 864,000,000 × 1KB = ~824GB

Log Collection

Estimate: - Logs per service: 100-1000 per minute - Services: 10-100 - Storage: ~500 bytes per log

Calculation: 

Logs per day = services × logs_per_minute × minutes_per_day
Logs per day = 50 × 500 × 1,440 = 36,000,000
Storage per day = 36,000,000 × 500 bytes = ~17GB

Scaling Strategies

Proactive Scaling

Based on Predictable Patterns: - Time-based scaling (business hours) - Event-based scaling (releases) - Calendar-based scaling (holidays)

Reactive Scaling

Based on Metrics: - CPU utilization - Memory usage - Queue depth - Error rates

Predictive Scaling

Based on ML Models: - Predict future load - Scale before demand - Optimize resource usage

Cost Optimization

Data Volume Reduction

Sampling: 

processors:
  probabilistic_sampler:
    sampling_percentage: 10.0  # Sample 10%

Filtering: 

processors:
  filter:
    traces:
      span:
        - 'attributes["http.status_code"] == 200'

Aggregation: - Pre-aggregate metrics - Reduce cardinality - Use histograms

Resource Optimization

Right-sizing: - Match resources to workload - Regular reviews - Adjust based on metrics

Auto-scaling: - Scale down during low usage - Scale up during high usage - Use spot instances for non-critical

Monitoring Scaling

Key Metrics

Collector: - otelcol_receiver_accepted_spans - otelcol_exporter_send_failed_spans - otelcol_processor_batch_batch_send_size - CPU and memory usage

Backends: - Ingestion rate - Storage usage - Query latency - Error rates

Alerts

Scale-up Alerts: - High CPU utilization (>80%) - High memory usage (>80%) - Queue depth increasing - Error rates increasing

Scale-down Alerts: - Low CPU utilization (<30%) - Low memory usage (<30%) - Low ingestion rate - No errors

Further Reading