data-stateestablished · medium operational burden

Time-Series Storage

Also known as: tsdb, metrics-storage

Intent

Store timestamped measurements in time-partitioned, compressed form optimized for append-heavy writes and range queries.

Problem

Metrics, sensor readings and events arrive continuously and are almost always queried by time window. General-purpose tables grow without bound, indexes bloat and old data is expensive to delete. Queries typically aggregate recent data and rarely touch the past.

Forces

•Writes are append-only and arrive in near time order
•High-cardinality labels explode index size
•Recent data is queried often, old data rarely
•Retention and downsampling must be cheap

Solution

Overview

Time-series databases such as Prometheus, InfluxDB and TimescaleDB exploit three facts: writes are appends, queries are time ranges, and consecutive values are similar. Storage is shaped around those facts.

How it works

Data is split into time chunks, compressed, and rolled up as it ages.

raw 10s points   -> keep 7 days
1m rollups       -> keep 90 days
1h rollups       -> keep 2 years

-- TimescaleDB style
SELECT create_hypertable('cpu', 'ts', chunk_time_interval => INTERVAL '1 day');
SELECT add_retention_policy('cpu', INTERVAL '30 days');

Dropping a chunk is a file deletion, far cheaper than deleting rows from a big table.

Query shape determines design. Most dashboards ask for a rate or percentile over the last hour for a few series, and alerts evaluate the same expressions repeatedly, so recent chunks are kept in memory and indexed by label sets. Long-range queries read the rolled-up tiers instead of raw data. When you need exact percentiles over time, store histogram buckets or sketches rather than precomputed percentiles, because percentiles cannot be combined across series or windows by averaging.

Pitfalls

Using high-cardinality values (request IDs, user IDs) as labels or tags.
Choosing chunk sizes so large that recent chunks no longer fit in memory.
Keeping raw resolution forever instead of downsampling.
Averaging percentiles during rollup, which gives meaningless results; store histograms instead.

Key takeaways

Time-partitioning makes ingest, queries and retention cheap.
Cardinality is the main scaling limit.
Downsample old data and keep raw data short.
Do not use it for general relational workloads.

✓ When to Use

•Infrastructure and application metrics
•IoT and sensor telemetry
•Financial ticks or event streams queried by time range

✗ When Not to Use

•Data queried mainly by entity rather than time
•Low-volume data a regular table handles easily
•Workloads needing frequent updates to historical points

Pros

+Fast ingestion and strong compression
+Retention by dropping whole partitions
+Efficient time-window aggregation

Cons

−Poor fit for ad hoc relational queries and updates
−High label cardinality can overwhelm memory
−Out-of-order or late data complicates writes

Cost Profile

Infrastructure

Medium - storage grows steadily even when compressed

Operational

Medium - retention, rollups and cardinality governance

Cognitive

Medium - series, labels and downsampling model

Failure Modes

Unbounded label values such as user IDs create a cardinality explosion
Late-arriving data lands in already compacted chunks and is rejected or slow
Retention misconfigured so disks fill or history disappears early

Real-World Examples

Alternatives

Related Patterns

partitioning
observability

Competency Domains

data stateeconomics evolution