Skip to content

Prometheus + Grafana Setup Example

This example demonstrates how to set up and configure Prometheus and Grafana with the OpenTelemetry Collector for metrics collection and visualization.

Prerequisites

  • Docker Desktop installed and running
  • At least 2GB of available RAM
  • Ports available: 3000 (Grafana), 9091 (Prometheus)

Step 1: Enable Prometheus/Grafana Configuration

Update your application configuration (e.g., appsettings.json):

{
  "ObservabilityStack": {
    "Mode": "OtelCollector",
    "CollectorEndpoint": "http://otel-collector:4317"
  },
  "ObservabilityBackend": {
    "EnabledBackends": ["PrometheusGrafana"],
    "PrometheusGrafana": {
      "PrometheusEndpoint": "http://prometheus:9090",
      "GrafanaEndpoint": "http://grafana:3000"
    }
  }
}

Step 2: Start the Observability Stack

# Navigate to Docker Compose directory
cd <your-docker-compose-directory>

# Start Prometheus, Grafana, and collector
docker-compose up -d prometheus grafana otel-collector

Step 3: Configure Collector

The collector should expose metrics for Prometheus to scrape. The default configuration already includes this:

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

Step 4: Configure Prometheus

Verify Prometheus scrape configuration in your Prometheus configuration file (e.g., prometheus.yml):

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'otel-collector'
    static_configs:
      - targets: ['otel-collector:8889']

  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

Restart Prometheus:

docker-compose restart prometheus

Step 5: Configure Grafana

Add Prometheus Data Source

  1. Open http://localhost:3000 in your browser
  2. Go to Configuration > Data Sources
  3. Click Add data source
  4. Select Prometheus
  5. Set URL to http://prometheus:9090
  6. Click Save & Test

Import Dashboards

Pre-configured dashboards are available for common use cases:

  • OpenTelemetry Collector dashboard - Collector metrics
  • .NET runtime dashboard - .NET runtime metrics
  • ASP.NET Core metrics dashboard - ASP.NET Core metrics
  • Kestrel connections dashboard - Kestrel connection metrics

To import:

  1. Go to Dashboards > Import
  2. Upload the JSON file or paste the content
  3. Select the Prometheus data source
  4. Click Import

Step 6: Verify Setup

Check Prometheus

curl http://localhost:9091/-/healthy

Query metrics:

curl 'http://localhost:9091/api/v1/query?query=up'

Check Grafana

  1. Open http://localhost:3000
  2. Navigate to Dashboards > Browse
  3. View the imported dashboards

Check Collector Metrics

curl http://localhost:8888/metrics

Step 7: Generate Test Metrics

Start your application and generate some metrics:

# Make API calls to generate metrics
for i in {1..10}; do
  curl http://localhost:8081/api/health
  sleep 1
done

Step 8: Create Custom Dashboards

Example: Request Rate Dashboard

  1. Go to Dashboards > New Dashboard
  2. Add a new panel
  3. Set query to: rate(http_server_request_duration_seconds_count[5m])
  4. Set visualization to Graph
  5. Save the dashboard

Example: Error Rate Dashboard

  1. Add a new panel
  2. Set query to: rate(http_server_request_duration_seconds_count{status_code=~"5.."}[5m])
  3. Set visualization to Graph
  4. Add alert: Error rate > 10 requests/second

Configuration Examples

Prometheus Retention

Configure data retention in prometheus.yml:

global:
  external_labels:
    cluster: 'production'
    replica: '0'
  scrape_interval: 15s
  evaluation_interval: 15s
  # Retention: 30 days
  retention: 30d

Grafana Alerting

Configure alert channels in Grafana:

  1. Go to Alerting > Notification channels
  2. Add channels (Email, Slack, PagerDuty, etc.)
  3. Create alert rules in dashboards

Collector Metrics Export

Export collector's own metrics:

service:
  telemetry:
    metrics:
      level: detailed
      readers:
        - pull:
            exporter:
              prometheus:
                host: 0.0.0.0
                port: 8888

Troubleshooting

Prometheus Not Scraping

Problem: No metrics in Prometheus

Solution:

  1. Check targets: http://localhost:9091/targets
  2. Verify collector is exposing metrics: curl http://localhost:8888/metrics
  3. Check Prometheus logs: docker-compose logs prometheus
  4. Verify network connectivity: docker network inspect <network-name>

Grafana Not Loading Dashboards

Problem: Dashboards show "No data"

Solution:

  1. Verify Prometheus data source is connected
  2. Check time range (default is last 6 hours)
  3. Verify metrics exist: Query in Prometheus UI
  4. Check dashboard queries match available metrics

High Cardinality Metrics

Problem: Prometheus using too much memory

Solution:

  • Reduce metric cardinality (fewer label combinations)
  • Increase scrape interval
  • Use recording rules to pre-aggregate metrics
  • Enable metric filtering in collector

Use Cases

Application Performance Monitoring

  • Track request rates and latencies
  • Monitor error rates
  • Identify slow endpoints

Infrastructure Monitoring

  • CPU and memory usage
  • Network throughput
  • Disk I/O

Business Metrics

  • User activity
  • Transaction volumes
  • Revenue metrics

Further Reading