Designing a Metrics and Monitoring System

A monitoring system collects numeric measurements over time, lets engineers query them and alerts when something is wrong. This design follows Prometheus, Gorilla and M3.

Requirements

Functional

Ingest metrics (counters, gauges, histograms) with labels such as service, region and instance.
Query with aggregation over time ranges and label filters; dashboards.
Alert rules with notification routing, grouping and silencing.

Non-functional

10 million samples per second ingest from 500K targets.
Dashboard queries answer in under 2 seconds for the last 24 hours.
Alert evaluation delay under 30 seconds from event to notification.
Retention: raw 15 days, 5-minute rollups for 1 year.
The monitoring system must be more available than the systems it watches: 99.99 percent for alerting.

Capacity estimation

10 million samples per second times 86,400 is 864 billion samples per day. A naive sample (8-byte timestamp plus 8-byte value) is 16 bytes, or 13.8 TB per day. Gorilla-style compression (delta-of-delta timestamps, XOR floats) averages about 1.4 bytes per sample, giving roughly 1.2 TB per day and 18 TB for 15 days raw. Active series, say 100 million, at 3 KB of index and head memory each, imply roughly 300 GB of RAM across the ingestion fleet.

High-level design

 Targets (apps, exporters)
     ^ pull (scrape)        or push via agent
     |
 Collectors (sharded by target hash)
     |
 Kafka (buffer) --> Ingesters (in-memory head + WAL)
                       | flush 2h blocks
                       v
              Object storage (long-term blocks)
                       ^
 Compactor / downsampler
 Query frontend --> Queriers --> ingesters + storage
 Rule evaluator --> Alertmanager --> pager, chat, email

Data model

A time series is identified by a metric name plus a sorted label set. Samples are (timestamp, float64). An inverted index maps each label pair to the series ids containing it, which makes label selectors fast. Storage is organized in immutable blocks covering two hours, each holding chunk files, an index and metadata. Histograms store bucket counters so quantiles can be computed at query time and aggregated across instances.

Key deep-dives

Pull versus push. Pull (scraping) gives a built-in health signal, since a failed scrape means the target is down, and central control over rate. Push suits short-lived jobs and firewalled networks. Many deployments scrape locally and remote-write to a central store.

Write path. Samples append to an in-memory head chunk and to a write-ahead log for crash recovery. Every two hours the head flushes to a block. Partition by hash of the series identity across ingesters, and replicate three ways with quorum writes to survive node loss.

Compression. Timestamps usually arrive at fixed intervals, so delta-of-delta encoding reduces most to a single bit. Successive float values are often similar, so XORing them yields many leading and trailing zero bits. Together they cut 16 bytes to about 1.4.

Cardinality control. The dominant failure mode is label explosion: putting user id or request id in a label creates millions of series. Enforce per-tenant series limits, reject high-cardinality labels at ingestion, and expose cardinality dashboards.

Downsampling. A compactor merges blocks and produces 5-minute and 1-hour rollups storing min, max, sum and count, which keeps long-range queries fast and cheap. Queries automatically pick the resolution matching the requested range.

Query engine. Parse a query language (PromQL-style), resolve selectors through the inverted index, decode only needed chunks, and evaluate functions such as rate and histogram quantile. A query frontend splits long ranges into daily sub-queries, runs them in parallel and caches results.

Alerting. The rule evaluator runs queries every 15 to 60 seconds. A rule must hold for a duration (for 5 minutes) before firing to avoid flapping. The alert manager deduplicates, groups related alerts into one notification, applies silences and inhibition, and routes by label to the right team. Run the alerting path as an independent, replicated stack, ideally with a heartbeat alert that pages if the pipeline itself goes quiet.

Failure modes and trade-offs

Ingester crash: replay the WAL; replicas keep serving. Short gaps are tolerable, since metrics are approximate by nature.
Backpressure: the Kafka buffer absorbs bursts and lets ingesters catch up.
Query overload: one heavy dashboard can starve others; enforce per-tenant query limits, timeouts and max sample counts.
Alert fatigue: too many noisy alerts desensitize on-call. Alert on symptoms (error rate, latency) rather than causes such as CPU.
Accuracy versus cost: downsampling and sampling reduce cost but lose detail; keep raw data for the incident window.
Percentiles: averaging per-host percentiles is wrong; aggregate histograms instead.

Key takeaways

Time-series compression (delta-of-delta plus XOR) makes billions of samples per day affordable.
Cardinality, not sample rate, is the main scaling hazard; enforce limits early.
Use a WAL plus two-hour immutable blocks, with background compaction and downsampling for cheap long retention.
Keep the alerting path simple, redundant and independent of the data it monitors.
Alert on user-visible symptoms and use multi-window evaluation to keep pages meaningful.