Stackbook Logo
← Back to System Architectures

Designing an API Gateway

Design a high-throughput API gateway covering routing, authentication, rate limiting, request transformation, resilience and a dynamic control plane.

5 min read

An API gateway is the single entry point for client traffic to a fleet of backend services. It handles cross-cutting concerns once, so each service does not. This article designs a gateway comparable to Kong, Envoy-based gateways or AWS API Gateway.

Requirements

Functional

  • Route requests to backends by host, path, method and headers.
  • Authenticate (API keys, JWT, OAuth tokens, mutual TLS) and authorize.
  • Rate limiting and quotas per consumer.
  • Request and response transformation, response caching, and protocol translation (REST to gRPC).
  • Observability: access logs, metrics and distributed trace propagation.

Non-functional

  • 200K requests per second peak, scaling horizontally.
  • Added latency under 5 ms at p99 (excluding backend time).
  • 99.99 percent availability; no single point of failure.
  • Configuration changes live within 5 seconds, with no dropped connections.

Capacity estimation

An optimized proxy on a modern 8-core instance handles roughly 20K to 30K requests per second with TLS and plugins enabled. For 200K peak, use about 10 instances; provision 20 across three zones to survive a zone loss and absorb bursts. Memory per instance is small (a few GB), dominated by connection buffers: 100K concurrent connections at 32 KB is about 3 GB. Rate limit counters for 1 million consumers at 100 bytes are only 100 MB.

High-level design

 Clients --> L4 load balancer (anycast / cloud LB)
                  |
         +--------+---------+
         v                  v
   Gateway node       Gateway node  ... (stateless data plane)
   [TLS | authn | rate limit | route | transform | LB | retries]
         |                  |
         +---> Service A / B / C  (via service discovery)
         |
   Redis cluster (shared rate limit counters, cache)
   Control plane: config API --> store --> push to nodes (xDS style)
   Identity provider / JWKS endpoint

Data model

Configuration objects: services (upstream targets, timeouts, retry policy), routes (match rules referencing a service), consumers (id, credentials, plan), plugins (policy attached at global, service, route or consumer level), and certificates. All are versioned and stored in a strongly consistent store such as PostgreSQL or etcd. Gateways keep an immutable in-memory snapshot and swap it atomically on update.

Key deep-dives

Control plane and data plane separation. Data-plane nodes never query the config store per request. The control plane validates changes, then streams incremental updates (the xDS pattern). If the control plane fails, nodes continue serving with their last known config, so a control outage does not become a traffic outage.

Routing. Compile routes into a radix tree or trie for path matching, giving lookups in microseconds regardless of route count. Precedence rules (exact before prefix before regex) must be deterministic.

Authentication. Verify JWT signatures locally using cached JWKS keys, so no network call is needed per request. For opaque tokens or API keys, cache introspection results for 30 to 60 seconds. After authentication, forward identity to backends in a signed header so services trust the gateway, not the client.

Rate limiting. Token bucket per consumer and route. A local-only counter is fast but permits N times the limit across N nodes. A shared Redis counter is accurate but adds about 1 ms and a dependency. A hybrid approach syncs local counters every 100 ms or leases token batches from Redis. Fail open or closed deliberately when Redis is down; most public APIs fail open with a conservative local limit.

Resilience. Per-upstream timeouts, bounded retries for idempotent methods only (with a retry budget of 10 percent of traffic to avoid retry storms), circuit breakers that eject failing instances, and outlier detection. Load balancing by least requests or power-of-two-choices.

Caching and transformation. Cache GET responses keyed by URL and Vary headers with short TTLs. Transformations (header rewrites, JSON field mapping) should be declarative and bounded in cost; for custom logic, run sandboxed plugins (WebAssembly) with strict CPU limits.

Failure modes and trade-offs

  • Gateway as bottleneck: keep it stateless and cheap; push heavy business logic to services.
  • Single point of failure: run across zones behind multiple load balancers; use connection draining on deploys.
  • Bad config push: validate, canary to a small share of nodes, and keep one-click rollback.
  • Plugin cost: every plugin adds latency; budget milliseconds per request and measure.
  • Centralized versus per-service gateways: one gateway simplifies governance but creates a shared blast radius; many team-owned gateways isolate failures at the cost of consistency.
  • Security: the gateway is the first line of defense; enforce request size limits, header validation and slow-client timeouts.

Key takeaways

  • Separate the control plane from a stateless data plane that survives control outages.
  • Local verification (JWT, compiled routes, cached config) keeps added latency in single-digit milliseconds.
  • Distributed rate limiting needs an explicit accuracy versus latency decision and a defined failure policy.
  • Bounded retries, timeouts and circuit breakers protect backends from the gateway itself.
  • Treat configuration changes like code deploys: validate, canary and roll back quickly.