Stackbook Logo
← Back to Tools

Prometheus Monitoring

Monitor services with Prometheus: scrape configuration, metric types, PromQL examples, recording and alerting rules, and common cardinality pitfalls.

4 min read

Prometheus is an open-source monitoring system and time series database, and a graduated CNCF project. It works on a pull model: the server periodically scrapes HTTP endpoints that expose metrics in a plain text format, stores the samples locally, and lets you query them with PromQL. Use it for numeric, dimensional metrics from services, hosts, and containers, particularly in Kubernetes environments.

Quick start

Create prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - rules.yml

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: api
    metrics_path: /metrics
    static_configs:
      - targets: ["api:8080"]

Run it and open http://localhost:9090:

docker run -d --name prometheus -p 9090:9090 \
  -v "$(pwd)/prometheus.yml":/etc/prometheus/prometheus.yml \
  -v "$(pwd)/rules.yml":/etc/prometheus/rules.yml \
  prom/prometheus

In dynamic environments, replace static targets with service discovery such as kubernetes_sd_configs, Consul, or EC2.

Core concepts

  • Metric and labels: a name plus key-value labels, for example http_requests_total{method="GET", status="200"}. Each unique label combination is a separate time series.
  • Counter: a value that only increases, such as requests served. Always query it with rate().
  • Gauge: a value that moves up and down, such as memory in use or queue length.
  • Histogram: counts observations in buckets, enabling percentile estimates across instances.
  • Summary: client-side quantiles that cannot be aggregated across instances, so prefer histograms.
  • Exporter: a process that translates a system's metrics into Prometheus format, such as node_exporter or postgres_exporter.

PromQL examples

# Requests per second per service over 5 minutes
sum by (service) (rate(http_requests_total[5m]))

# Error ratio
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

# 99th percentile latency from a histogram
histogram_quantile(0.99,
  sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))

# CPU usage percent per host (node_exporter)
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory available as a fraction
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes

# Disk predicted to fill within 4 hours
predict_linear(node_filesystem_avail_bytes[1h], 4 * 3600) < 0

# Which targets are down
up == 0

The range selector [5m] is a window, and rate() computes the per-second average increase across it. Use sum by or sum without to aggregate, and make the window at least four times the scrape interval.

Recording and alerting rules

Recording rules precompute expensive expressions. Alerting rules fire when a condition holds. Put both in rules.yml:

groups:
  - name: api
    rules:
      - record: service:http_requests:rate5m
        expr: sum by (service) (rate(http_requests_total[5m]))
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
            / sum(rate(http_requests_total[5m])) > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "Error ratio above 5 percent for 10 minutes"
      - alert: InstanceDown
        expr: up == 0
        for: 2m

Prometheus evaluates alerts but does not send notifications itself. Alertmanager receives them and handles grouping, deduplication, silencing, and routing to email, Slack, or PagerDuty. Validate files with promtool check config prometheus.yml and promtool check rules rules.yml.

Common pitfalls

  • High cardinality. Labels such as user ID, request ID, or full URL create millions of series and exhaust memory. Keep label values bounded.
  • Using rate() on gauges or forgetting it on counters. Raw counters reset on restart and are not meaningful.
  • Averaging percentiles. Aggregate histogram buckets, then compute the quantile.
  • Treating local storage as long-term. The default TSDB suits weeks of data. For long retention or global views use remote write to Thanos, Mimir, or Cortex.
  • Short-lived jobs. Batch jobs may finish before a scrape. Use the Pushgateway sparingly for them.
  • Alerting on causes instead of symptoms, which creates noise.

Where it fits in an architecture

Prometheus is the metrics backbone of an observability stack. Services expose /metrics through client libraries, exporters cover databases and hosts, Prometheus stores and evaluates, Alertmanager notifies, and Grafana visualises. In Kubernetes, the Prometheus Operator and kube-prometheus-stack manage this declaratively through ServiceMonitor resources. Pair it with Loki for logs and Jaeger or Tempo for traces.

Key takeaways

  • Prometheus pulls metrics, stores time series, and queries them with PromQL.
  • Use counters with rate(), gauges directly, and histograms for latency percentiles.
  • Precompute heavy queries with recording rules and alert on symptoms with a for duration.
  • Control label cardinality from day one.
  • Add Alertmanager for notifications and Grafana for dashboards.