Designing an Audit and Event Log System

An audit log answers who did what, to which resource, when and from where. Unlike debug logs, audit records are evidence: they must not be lost, altered or silently dropped, and they must be searchable years later.

Requirements

Functional

• Capture events from many services: actor, action, resource, outcome, timestamp, source IP, request id, before and after values.
• Search by actor, resource, action and time range; export for auditors.
• Per-tenant views and retention policies.
• Verify integrity of stored records.

Non-functional

• No loss of acknowledged events; at-least-once capture with deduplication.
• Ingest 50,000 events per second at peak; search p95 under 2 seconds over 90 days of data.
• Retention from 1 to 7 years, immutable once written.
• Ingest must not slow down or fail the business operation it describes.

Capacity Estimation

At 50,000 events per second peak and a 10,000 per second average, that is 864M events per day. An event is about 1 KB, so roughly 860 GB per day, 315 TB per year before compression. Compression at 5x brings that to about 63 TB per year. Keep 90 days hot and searchable, about 15 TB compressed, in a search index with replicas. Everything older lives in object storage as columnar files, queried on demand.

High-Level Design

Services -> Audit SDK (local outbox / buffer) -> Ingest API (authn, schema validation)
                                                    |
                                               Kafka (partitioned by tenant_id), replicated
                                  +-----------------+------------------+
                          Indexer (dedupe)     Archiver (batch)    Integrity Service
                               |                    |                    |
                        Search Index (hot)   Object Storage (WORM)   Hash chain / signed roots
                               |
                     Query API (RBAC, tenant scope) -> UI, Export, SIEM

Data Model

• Event: event_id (UUID or ULID from the producer), tenant_id, ts, actor_id, actor_type, action, resource_type, resource_id, outcome, ip, request_id, metadata, schema_version
• Search index fields: tenant_id, ts, actor_id, action, resource_id as keywords, free text restricted
• Archive layout: tenant/yyyy/mm/dd/hour/part-n.parquet with a manifest
• Integrity: chain(tenant_id, segment_id, prev_hash, merkle_root, signature, ts)

Deep Dives

Guaranteed capture

The worst audit bug is a business action that succeeded with no record. For critical actions, write the audit event to a local outbox table in the same database transaction as the change, and let a relay publish it to Kafka. For lower-risk events, a buffered SDK with disk spill and retries suffices. Kafka is configured with replication factor 3, min in-sync replicas 2 and acks all. If the audit pipeline is unavailable, critical operations either queue in the outbox or are rejected, according to a policy per action type: fail closed for permission changes, fail open for page views.

Idempotency and ordering

Producers assign event_id. The indexer writes documents using event_id as the document id, so redelivery overwrites the same document harmlessly. Ordering across services is not guaranteed and not needed; consumers sort by timestamp and sequence per resource. Partitioning by tenant keeps one tenant's events ordered per partition and isolates noisy tenants onto separate partitions when large.

Tamper evidence

Within each segment of, say, 10,000 events, compute a hash for each event that includes the previous hash, forming a chain, and a Merkle root for the segment. Sign the root with a key held in an HSM and publish it to an independent store or notary at intervals. Archived files are written to object storage with write-once retention locks. An integrity job periodically recomputes hashes and alerts on mismatch. This detects tampering, including by administrators, though it does not prevent deletion by anyone with storage-level destruction rights, so separate those permissions.

Query and access

The audit log itself is sensitive. Query API enforces tenant scope and role-based access, and every audit search is itself audited. Free-text search is limited to keyword fields to keep the index small; long-range queries over archives use a query engine over Parquet with partition pruning by tenant and date.

Retention and privacy

Retention is per tenant and event class. Deletion for privacy requests conflicts with immutability, so store personal data as references or encrypted fields with per-subject keys, and delete the key to render data unreadable while preserving the record structure.

Failure Modes and Trade-offs

• Ingest overload: apply backpressure with per-tenant quotas, and let Kafka absorb bursts for hours.
• Index lag: queries may miss the last seconds; show an as-of timestamp.
• Poison events: route schema failures to a dead-letter topic with alerts, never drop them silently.
• Clock skew: record both producer and ingest timestamps, and trust ingest time for ordering evidence.
• Hot index versus cheap archive: a 90-day hot window balances cost against investigator convenience.
• Synchronous versus asynchronous writes: synchronous capture gives the strongest guarantee but couples availability; the outbox pattern gets durability without that coupling.
• Event sourcing versus audit log: an event store is the source of state, while an audit log is a record of actions about state; keep them separate to avoid coupling retention needs.

Key takeaways

• Capture audit events transactionally with the action using an outbox, then ship asynchronously.
• Use producer-generated event ids for idempotent indexing and safe redelivery.
• Hash-chain segments, sign roots in an HSM and archive to write-once storage for tamper evidence.
• Split hot searchable storage from cheap columnar archives.
• Audit the auditors: restrict, scope and log every query.
• Decide fail-open or fail-closed per action type, and reconcile privacy deletion through key destruction.