Log Aggregation Pipeline

Overview

A log aggregation pipeline moves logs from where they are produced to where they can be queried. Its main jobs are durability (logs outlive the instance), uniformity (common fields), and protection of the backend from bursts.

How it works

app stdout -> node agent -> broker (Kafka) -> processors -> hot index
                                                 |-> object storage (archive)
processors: parse JSON, join multi-line, add k8s labels,
            add trace_id, redact emails and tokens, sample debug

The agent tails container output and attaches metadata such as host, namespace, and pod. The broker decouples producers from the indexer: if the index is slow or down, logs queue instead of being lost. Processors normalize timestamps to UTC, enforce a schema, and tag each event with service name and environment. Tiered storage keeps recent logs fast to search and older logs cheap. Query access is audited, and field-level redaction happens before indexing, not after.

Operationally, track end-to-end lag, agent queue size, dropped events, and index rejection rate, and alert on them. Set per-service volume quotas so one noisy service cannot starve others.

Pitfalls

• Treating logs as free. Indexing all fields of verbose logs can cost more than the compute that produced them.
• Free-form text. Without structure, queries rely on fragile regexes; emit JSON.
• Dropping logs silently on overflow. Make drops visible and prefer to shed low severity first.
• Skipping redaction, which creates a compliance incident waiting to happen.

Key takeaways

• Use an agent plus a durable buffer to survive bursts and restarts.
• Normalize and enrich at the processing tier.
• Tier retention to control cost.
• Monitor the pipeline itself and redact before indexing.