Infrastructure Monitoring Basics

Infrastructure monitoring is the practice of collecting, storing, and acting on signals that describe the health of your servers, containers, networks, and the applications running on them. Its purpose is simple: detect problems before users do, and give engineers enough context to fix them quickly. If you run anything in production, you need at least the basics described here.

Monitoring versus observability

Monitoring answers questions you anticipated, such as "is CPU above 90 percent?" Observability lets you ask new questions of a running system without shipping new code. In practice you need both, built from three signal types:

• Metrics: numeric time series, cheap to store and ideal for dashboards and alerts.
• Logs: timestamped event records, rich in detail but expensive at volume.
• Traces: the path of one request across services, essential in distributed systems.

Start with metrics, add structured logs, and introduce tracing once you have several services calling each other.

What to measure

Two frameworks keep you from measuring everything and understanding nothing.

The four golden signals from Google's SRE book apply to any service:

• Latency: how long requests take, tracked separately for successes and errors.
• Traffic: how much demand the system serves, such as requests per second.
• Errors: the rate of failed requests.
• Saturation: how full the system is, such as queue depth or memory pressure.

The USE method applies to resources like CPUs, disks, and network links: Utilisation, Saturation, and Errors. The RED method (Rate, Errors, Duration) is the service-level equivalent.

A starter stack

A widely used open-source setup needs little more than three components:

# Host-level metrics exporter
docker run -d --name node-exporter -p 9100:9100 prom/node-exporter

# Metrics store and alert evaluator
docker run -d --name prometheus -p 9090:9090 \
  -v "$(pwd)/prometheus.yml":/etc/prometheus/prometheus.yml \
  prom/prometheus

# Dashboards
docker run -d --name grafana -p 3000:3000 grafana/grafana-oss

A minimal prometheus.yml that scrapes the exporter:

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["host.docker.internal:9100"]

From there, add Alertmanager for notifications, Loki for logs, and an exporter for each important component such as PostgreSQL, Redis, or NGINX. Managed alternatives like Datadog, New Relic, or cloud-native tools cover the same ground with less operational effort.

Alerting that people trust

Bad alerting is worse than none, because teams learn to ignore it. Follow these principles:

• Alert on symptoms, not causes. Page when users are affected, such as a high error rate, rather than when one CPU spikes.
• Every page must be actionable. If nobody needs to do anything, make it a ticket or a dashboard.
• Use durations. Require a condition to hold for several minutes to avoid flapping.
• Attach a runbook link to every alert.
• Define service level objectives and alert on error budget burn rather than raw thresholds.

An example rule that fires when the 5xx ratio stays high:

groups:
  - name: service-alerts
    rules:
      - alert: HighErrorRate
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "More than 5% of requests are failing"

Common pitfalls

• Collecting everything without a plan, which creates cost and noise.
• High-cardinality labels such as user IDs, which can overwhelm a metrics store.
• Monitoring only hosts and ignoring the user-facing experience. Add synthetic checks that exercise key journeys.
• Alerts with no owner or runbook.
• Letting the monitoring system itself go unmonitored. Add a dead man's switch alert that fires when data stops arriving.

Where it fits in an architecture

Monitoring is a cross-cutting concern. Every service exposes metrics and emits structured logs with a correlation ID, agents or exporters collect them, and a central platform stores and visualises them. It feeds incident response, capacity planning, and release decisions: a deployment is only safe if you can see its effect.

Key takeaways

• Combine metrics, logs, and traces, starting with metrics.
• Measure the golden signals for services and USE for resources.
• Alert on user-visible symptoms and make every page actionable.
• Start simple with Prometheus, Grafana, and Alertmanager.
• Monitor the monitors so silence is never mistaken for health.