Rate Limiting Framework

Rate limiting bounds how much work a given caller may request in a time window. It is a fairness and protection mechanism: it does not add capacity, it decides who gets the capacity that exists. A good framework defines the dimensions being limited, the algorithm, where enforcement happens, and how clients are told about their budget.

How it works

Each request is mapped to a limit key, such as tenant plus route. The limiter checks the key against its algorithm and either admits or rejects the request.

• Token bucket: tokens refill at a fixed rate up to a bucket size. Bursts are allowed up to the bucket size, then traffic is held to the refill rate.
• Sliding window counter: weights the previous and current fixed windows to approximate a true sliding window with little memory.
• Concurrency limit: caps in-flight requests rather than rate, which suits slow, variable-cost endpoints.

Rejected requests receive HTTP 429, a Retry-After value, and headers showing limit, remaining, and reset time.

Implementation steps

1. Inventory endpoints and rank them by cost; expensive ones get stricter limits or cost-weighted tokens.
2. Pick limit keys. Authenticated identity is preferable to IP, which breaks behind NAT and proxies.
3. Derive numbers from load tests: set the sum of all tenant limits near, but not above, safe capacity, allowing for overcommit.
4. Enforce at the gateway first, then add per-service limits for defense in depth.
5. Use a shared store such as Redis with atomic scripts for global limits, and fall back to local limits if the store is unreachable (fail open for most APIs, fail closed for auth endpoints).
6. Publish limits in documentation and return them in headers.

Common pitfalls

• Failing closed when the limiter store is down, turning a cache outage into a full outage.
• Counting retries from the client as new traffic with no backoff guidance, which amplifies overload.
• Clock skew between nodes when using fixed windows, producing double bursts at window boundaries.
• Applying one global limit that lets a single tenant consume the whole quota.
• Limiting only by IP, which penalizes entire offices and ignores distributed attackers.

Metrics to watch

• Rejection rate by key and route, and the share of tenants hitting limits.
• Limiter decision latency at p99, which should be a small fraction of request latency.
• Limiter store error rate and fallback activations.
• Utilization of each tenant quota versus the limit, to spot misfit tiers.
• Retry volume following 429 responses.

Example configuration

rate_limits:
  - name: public-api-default
    key: [tenant_id, route]
    algorithm: token_bucket
    rate: 100/s
    burst: 200
    on_limit: reject_429
    headers: true
  - name: export-endpoint
    key: [tenant_id]
    algorithm: concurrency
    max_in_flight: 4
  store:
    type: redis
    timeout_ms: 5
    on_error: fail_open

Key takeaways

• Limit by authenticated identity and cost, not only by IP.
• Allow bursts but hold the sustained rate to a measured safe level.
• Always tell clients when to retry; silent drops cause retry storms.
• Decide the limiter failure mode explicitly and test it.
• Rate limiting shares capacity fairly; pair it with load shedding for true overload.