Stackbook Logo
data-stateestablished · medium operational burden

Hot-Cold Data Tiering

Also known as: storage-tiering, data-lifecycle-management

Intent

Place data on storage whose cost and speed match how often it is accessed, moving it as it ages.

Problem

Keeping all data on fast, expensive storage is wasteful when most of it is rarely read. Moving data to cheap storage by hand is error prone. Teams need lower cost without breaking access to old data.

Forces

  • •Access frequency usually falls sharply with age
  • •Cheap tiers have higher latency or retrieval fees
  • •Queries should not need to know which tier holds data
  • •Compliance may require retention and fast retrieval for some data

Solution

Overview

Hot-cold tiering matches storage class to access temperature. Recent data lives on fast media, older data migrates to cheaper tiers, and the oldest sits in archives such as S3 Glacier.

How it works

A lifecycle rule moves objects by age.

day 0-30    HOT   SSD / S3 Standard
day 30-365  WARM  S3 Infrequent Access
day 365+    COLD  Glacier (restore 3-12 hours)

Rule: transition after 30d -> STANDARD_IA
      transition after 365d -> GLACIER
      expire after 2555d

In databases, time-based partitions make this easy: detach an old partition and move it to cheaper storage or an external table.

Decide the boundary with data, not intuition. Plot the fraction of reads by object age: if ninety-five percent of reads hit data under thirty days old, that is your hot window. For databases, partition by time so a partition moves as a unit. For object stores, intelligent tiering can move objects automatically based on access, at a small monitoring fee. Keep an index or manifest of cold data in the hot tier so users can search metadata and request a restore only for what they need.

Pitfalls

  • Ignoring minimum storage duration and per-request retrieval charges on cold classes.
  • Archiving data that dashboards or customer queries still touch.
  • No way to rehydrate quickly during an incident or audit.
  • Moving millions of tiny objects, where per-object fees exceed the saving; aggregate first.

Key takeaways

  • Tier by measured access patterns, not guesswork.
  • Include retrieval cost and latency in the decision.
  • Make routing transparent or the restore path explicit.
  • Test the restore procedure before you need it.

✓ When to Use

  • •Logs, events, metrics and audit trails with age-based access
  • •Large media or document stores
  • •Databases with time-partitioned tables

✗ When Not to Use

  • •Data accessed uniformly regardless of age
  • •Small datasets where savings are negligible
  • •Workloads that cannot tolerate cold retrieval latency

Pros

  • +Substantial storage cost reduction
  • +Hot tier stays small and fast
  • +Automated by lifecycle policy

Cons

  • −Cold reads are slow and may incur retrieval fees
  • −Added metadata and routing complexity
  • −Wrong thresholds cause thrashing or surprise bills

Cost Profile

Infrastructure

Low - mainly reduces spend

Operational

Medium - policies, monitoring of tier transitions

Cognitive

Medium - access-pattern analysis

Failure Modes

  • Data moved to cold storage is needed urgently and retrieval takes hours

  • Frequent re-reads of cold data cost more than keeping it hot

  • Lifecycle rule deletes or archives data still referenced by live queries

Real-World Examples

Alternatives

    Related Patterns

    • partitioning
    • cost-optimization

    Competency Domains

    economics evolutiondata state