Stackbook Logo
← Back to System Architectures

Designing a Log Aggregation Platform

Design a centralized logging platform with agents, buffered ingestion, indexing, tiered storage and fast search for terabytes of logs per day.

5 min read

When hundreds of services run on thousands of hosts, engineers cannot ssh into machines to read logs. A log aggregation platform collects, stores and searches logs centrally. This design resembles the ELK stack, Loki and Splunk.

Requirements

Functional

  • Collect logs from hosts, containers and cloud services.
  • Parse and enrich logs (timestamp, service, trace id, severity).
  • Full-text and field search with time-range filters, live tail, saved queries and alerts.
  • Retention policies per team; access control and PII redaction.

Non-functional

  • 5 TB of raw logs per day, 2 million events per second at peak.
  • Logs searchable within 10 seconds of emission.
  • Query over the last hour returns in under 3 seconds; over 7 days, under 30 seconds.
  • Retention: 7 days hot, 30 days warm, 1 year cold.
  • Never block or crash the application producing logs; accept bounded loss under extreme overload.

Capacity estimation

5 TB per day averages 58 MB/s, with peaks of three times that, around 175 MB/s. At 500 bytes per event, 2 million events per second is 1 GB/s at the very peak, so ingestion must be sized for bursts. Full-text indexing roughly equals raw size; compression brings stored data to about 20 to 30 percent. Hot tier: 7 days of 5 TB with index and one replica is about 70 TB on SSD. Warm and cold tiers in object storage: 30 days compressed is roughly 40 TB, a year about 450 TB.

High-level design

 Hosts / containers
   [ Agent: tail, parse, batch, compress ]
            |
     Load balancer
            |
   Kafka (partitioned by tenant/service, 3 day retention)
        |                     |
  Stream processors      Archive writer
  (parse, redact,        (batched Parquet/gzip
   enrich, route)         to object storage)
        |
  Indexing cluster (hot) --> warm --> cold (searchable snapshots)
        |
  Query API + UI + alerting

Data model

An event has a timestamp, service, host, severity, message and a map of structured fields (request id, user tier, trace id). Prefer structured JSON logs at the source: it avoids fragile regex parsing. Indices are time-partitioned (hourly or daily) per tenant or log type so retention is a cheap delete of whole indices. Field mappings are limited per index to prevent explosion from unbounded keys.

Key deep-dives

Collection. A lightweight agent per host (Fluent Bit, Vector) tails files or reads the container runtime log, adds metadata, buffers to local disk, batches and compresses (gzip or zstd gives about 8 to 10 times reduction), and ships with retry and backoff. Disk buffering absorbs short outages; a size cap prevents filling the host.

Buffering layer. A durable message queue (Kafka) between agents and indexers decouples bursty producers from slower indexing, lets several consumers (index, archive, alerting) read the same stream and allows replay after a bug. Partition by service to keep related events together and balance load.

Processing. Stream processors parse leftover unstructured lines (grok patterns), normalize timestamps to UTC, redact secrets and personal data (card numbers, emails) before storage, sample noisy debug logs and route by severity.

Indexing and storage choices. Two schools. Full inverted indexing (Elasticsearch, OpenSearch) supports arbitrary text search at the cost of index size and CPU. Label-only indexing (Loki) indexes just a few labels like service and environment and stores log lines as compressed chunks, scanning them at query time: far cheaper storage, slower free-text search. Many platforms index only recent data fully and keep older data as compressed columnar files queried on demand.

Tiering and retention. Hot nodes on NVMe serve recent queries. Index lifecycle policies roll indices to warm (fewer replicas, force-merged segments, cheaper disks) then cold (object storage with on-demand loading), then delete. This reduces cost by an order of magnitude versus keeping everything hot.

Query path. A time range prunes which indices and shards to read; filters on service narrow further; the coordinator merges sorted partial results. Live tail streams from the Kafka topic rather than querying the index.

Failure modes and trade-offs

  • Ingestion spike or log storm: a buggy service can emit 100 times normal volume. Apply per-tenant quotas at the edge, drop or sample lowest severity first, and alert on volume anomalies.
  • Indexing lag: consumer lag in Kafka shows delay; autoscale indexers and prioritize error-level data.
  • Hot node loss: replicas cover reads; rebuild from replicas or replay from Kafka.
  • Mapping explosion: unbounded dynamic field names crash clusters; cap fields and flatten arbitrary maps to key and value pairs.
  • Cost versus searchability: indexing everything is simple but costly; sampling, shorter hot retention and label-only indexes lower cost at the price of slower or less flexible queries.
  • Compliance: redaction must occur before data lands anywhere durable, and deletion requests need an index design that can purge by identifier.

Key takeaways

  • Agents with local disk buffers protect applications from the logging pipeline, and Kafka protects the indexers from bursts.
  • Structured logs and a consistent schema make search faster and parsing unnecessary.
  • Choose between full-text indexing and label-only indexing based on query patterns and budget.
  • Time-partitioned indices with hot, warm and cold tiers make retention cheap and cost predictable.
  • Enforce quotas and redaction at ingestion; they are the controls that keep a shared platform safe and affordable.