Metrics Instrumentation Guide

Good metrics are cheap to store, cheap to query and answer a named question. This guide sets conventions so each service emits comparable signals with bounded cost.

How it works

Every request-handling service exposes rate, errors and duration. Every resource such as a queue or pool exposes utilisation, saturation and errors. Metric types are chosen deliberately: counters for totals that only increase, gauges for values that go up and down, and histograms for distributions. A shared library supplies middleware that records the standard signals so teams add business metrics rather than reinvent plumbing.

Instrumentation should start from questions rather than from available data. For a new service, ask what a responder needs to know in the first five minutes of an incident, which user journeys matter most, and which dependencies could fail. Then ensure each question maps to a metric or a query. This question-first approach avoids both the empty dashboard and the thousand-panel dashboard, and it makes review easier because every metric has a stated purpose that can be checked later.

Implementation steps

1. Define the naming scheme: service prefix, noun, base unit and suffix, such as http_request_duration_seconds.
2. Fix a short list of allowed labels, for example method, route template and status class.
3. Provide middleware for HTTP, gRPC and queue consumers in each language in use.
4. Choose histogram buckets that bracket your SLO threshold.
5. Add a lint step in CI that rejects forbidden labels such as user id or raw URL.
6. Track active series per service and set a budget with alerts at 80 percent.
7. Document each business metric with an owner and the question it answers.

Pitfalls

• Using raw paths or identifiers as label values.
• Averages of pre-computed percentiles, which are mathematically meaningless.
• Mixing units, such as milliseconds in one service and seconds in another.
• Gauge metrics used for events, which loses spikes between scrapes.
• Forgetting to remove metrics for decommissioned code paths.

Metrics to watch

• Active series per service and total ingestion rate.
• Percentage of services emitting standard RED signals.
• Scrape failures and dropped samples.
• Cost per service of monitoring.
• Dashboard reuse across services.

Example instrumentation

from prometheus_client import Counter, Histogram

REQUESTS = Counter(
    "checkout_http_requests_total",
    "Total HTTP requests",
    ["method", "route", "status_class"],
)
DURATION = Histogram(
    "checkout_http_request_duration_seconds",
    "Request latency in seconds",
    ["method", "route"],
    buckets=(0.05, 0.1, 0.2, 0.3, 0.5, 1.0, 2.5),
)

def observe(method, route, status, seconds):
    REQUESTS.labels(method, route, f"{status // 100}xx").inc()
    DURATION.labels(method, route).observe(seconds)

Key takeaways

• Standard signals first, custom metrics second.
• Bound label values; cardinality is the cost driver.
• Use histograms for latency and aggregate before computing percentiles.
• Enforce conventions in CI rather than in documents.