Stackbook Logo
← Back to System Architectures

Designing a Notification System

Design a multi-channel notification system sending 10B messages a day over push, email, SMS and in-app with preferences, retries and rate limits.

4 min read

A notification system lets product services send messages to users through push, email, SMS and in-app channels. It must respect user preferences, avoid duplicates, absorb bursts and stay reliable when third-party providers fail.

Requirements

Functional

  • Accept notification requests from internal services (transactional and marketing).
  • Deliver over mobile push, email, SMS and in-app inbox.
  • Per-user, per-channel, per-category preferences and quiet hours.
  • Templates with localisation, scheduling, and delivery status tracking.

Non-functional

  • 10B notifications per day, bursts of 10x during campaigns.
  • Transactional (for example one-time codes) delivered within 5 seconds at p99.
  • At-least-once delivery with deduplication, no notification sent twice in practice.
  • Marketing sends can be delayed; transactional must not be.

Capacity estimation

  • 10B per day is roughly 115,000 per second average, up to 1M per second at campaign peak.
  • Event payload about 1 KB: 10 TB per day through queues, retained 3 days is 30 TB.
  • Delivery log: 10B rows per day at 100 bytes is 1 TB per day, so keep 30 days hot and archive the rest.
  • Provider limits: APNs and FCM accept high concurrency over HTTP/2; SMS is costly and slow, so throttle it deliberately.

High-level design

Producer services --> Notification API --> Validate, dedupe, preferences
                                               |
                                               v
                               Priority queues (transactional | bulk)
                                               |
                       +-----------+-----------+-----------+
                       v           v           v           v
                  Push workers Email workers SMS workers In-app writer
                       |           |           |           |
                    APNs/FCM    Email ESP   SMS vendor   Inbox store
                       \           |           /
                        +--> Status events --> Delivery log + Retry queue

Data model

notification(id, user_id, category, template_id, payload, idempotency_key, created_at)
preferences(user_id, category, channel, enabled, quiet_start, quiet_end, timezone)
device_tokens(user_id, token, platform, last_seen)
delivery(notification_id, channel, status, provider_msg_id, attempts, updated_at)
inbox(user_id, notification_id, read, created_at)   -- partitioned by user_id

Deep dives

Idempotency. Producers send an idempotency key. The API writes it with a unique constraint or Redis SET NX with a 24-hour TTL. A retry after a timeout then returns the original result instead of sending again.

Priority isolation. Separate queues and worker pools for transactional and bulk traffic keep a marketing blast from delaying a login code. Bulk consumers are rate limited so they cannot starve shared resources.

Preference and rate checks. Before enqueueing, resolve channels from preferences, apply quiet hours in the user's timezone, and enforce frequency caps (for example at most 3 marketing pushes per day). Legal categories such as security alerts bypass caps. Defer rather than drop quiet-hour messages.

Provider handling. Each channel worker wraps providers with timeouts, circuit breakers and retries using exponential backoff with jitter. Keep two email or SMS vendors; on sustained failure of the primary, shift traffic to the secondary. Process provider feedback such as invalid tokens or bounces to prune device tokens and suppress bad addresses.

Scheduling. For scheduled and timezone-aware sends, store jobs in a time-bucketed store and have a scheduler enqueue them as they come due. Spread large campaigns over a window to avoid thundering herds.

Templates. Render at send time from a template ID plus variables, with locale fallback. Rendering at the worker keeps queue messages small.

Delivery tracking. Providers return acceptance synchronously and delivery or open events asynchronously by webhook. Normalise these into status events written to the delivery log.

Failure modes and trade-offs

  • Queue loss or worker crash: use durable queues with visibility timeouts and acknowledgement after provider acceptance. This is at-least-once, so provider calls need idempotency where supported.
  • Provider outage: retries pile up. Cap retry age (for example drop a one-time code after 5 minutes because it is useless), and fail over vendors.
  • Duplicate on retry: a provider may accept a message but time out. Accept a small duplicate rate for push and email, but for SMS prefer provider idempotency keys given the cost.
  • Preference staleness: caching preferences makes opt-outs lag. Use short TTLs and invalidate on change; for legal opt-outs check the source of truth.
  • Cost: SMS dominates spend, so push first with SMS as fallback only for critical categories.

Key takeaways

  • Separate transactional and bulk paths so campaigns cannot delay critical messages.
  • Idempotency keys at the API plus at-least-once workers give practical exactly-once behaviour.
  • Preferences, quiet hours and frequency caps are enforced before enqueueing.
  • Wrap every provider with timeouts, circuit breakers, retry limits and vendor failover.
  • Process provider feedback to keep device tokens and address lists clean.