Log Retention and Cost Control Framework

Logs are valuable when someone is searching them and costly the rest of the time. This framework matches what you pay to how long and how quickly each class of log is needed.

How it works

A log pipeline sits between applications and storage. It parses, tags each event with a class, applies drop and sample rules, and routes the result to a tier. Hot storage holds days of indexed data for incident work. Warm or cold object storage holds months of compressed data searchable on demand. Audit and security classes follow regulatory durations and may be immutable.

Cost control works best when it begins at the producer. Developers who see the volume and monthly cost of their own services tend to remove redundant messages without being asked, particularly when the dashboard breaks cost down by log statement or field. Pair this visibility with guardrails in the pipeline that cap runaway sources, so a bug that loops and logs millions of lines triggers a throttle and an alert instead of a surprise invoice at month end.

Implementation steps

1. Measure current volume per service, level and field to find the heaviest sources.
2. Define log classes and retention: for example debug three days, application fourteen days, audit one year.
3. Enforce structured logging with a mandatory service, environment and class field.
4. Add pipeline rules to drop health checks and sample successful request logs, keeping all errors.
5. Configure tier transitions and lifecycle deletion on object storage.
6. Publish a per-team cost and volume dashboard with a monthly budget.
7. Test restore from cold storage so incident responders know the process.

Pitfalls

• Sampling errors or rare events along with routine traffic.
• Logging full request bodies that carry personal data, extending compliance scope.
• Applying one retention period everywhere.
• Deleting logs without checking legal hold or audit requirements.
• Leaving temporary debug flags enabled after an investigation.

Metrics to watch

• Ingested gigabytes per day per service and per class.
• Cost per gigabyte per tier and total monthly spend.
• Percentage of logs matched to a retention class.
• Query latency on hot and cold tiers.
• Number of incidents where needed logs had already expired.

Example pipeline policy

classes:
  debug:
    hot_days: 1
    total_days: 3
    sample: 0.1
  application:
    hot_days: 7
    total_days: 30
    sample: 1.0
  audit:
    hot_days: 30
    total_days: 365
    immutable: true
rules:
  - match: 'path == "/healthz"'
    action: drop
  - match: 'level == "info" and status < 400'
    action: sample
    rate: 0.2
  - match: 'level in ["warn", "error"]'
    action: keep
lifecycle:
  cold_storage: object-store
  compress: zstd

Key takeaways

• Retention should follow purpose, not habit.
• Reduce at the source before paying to store and index.
• Always keep errors and audit events whole.
• Make teams see their own log bill.