High-Cardinality Management

Overview

Cardinality is the count of unique values a dimension can take. In time-series databases, every unique combination of label values is its own series. Three labels with 100, 50, and 20 values can create 100,000 series, and each series costs memory and index space.

How it works

Manage cardinality in four steps.

series = distinct(service) * distinct(route) * distinct(status) * ...

bad:   http_requests_total{user_id="8841",path="/users/8841/orders"}
good:  http_requests_total{route="/users/:id/orders",status="200"}
detail for one user -> logs and traces, found by trace_id

1. Design: allow only bounded labels, such as route templates, status class, region, and version.
2. Guard: apply relabel rules at scrape or ingest to drop or hash disallowed labels, and set per-metric series limits that reject, not crash.
3. Observe: track active series by metric and by team, and alert on growth rate.
4. Offload: put per-entity detail in logs and traces, and connect to metrics with exemplars.

A weekly top-ten report of series by metric creates healthy pressure, and a pull request check can lint instrumentation for forbidden labels.

Pitfalls

• Labels that look bounded but are not, such as pod name in a fast-churning cluster, or version strings that include build hashes.
• Histogram buckets multiplied by many labels. Each label combination gets every bucket.
• Fixing after the fact only. Deleting series does not stop new ones from appearing.
• Dropping a label globally without telling responders, hiding it when they need it.

Key takeaways

• Series count is the product of label cardinalities.
• Keep metric labels bounded; push detail to logs and traces.
• Enforce with limits, relabeling, and ownership reports.
• Alert on cardinality growth before it becomes an outage.