scalingestablished · low operational burden

Batching

Also known as: request-batching, micro-batching, bulk-operations

Intent

Group many small operations into fewer larger ones to amortize per-call overhead.

Problem

Each network call, database statement or message send has fixed overhead for round trips, parsing and acknowledgement. Doing thousands of tiny operations individually wastes throughput and overwhelms downstream systems.

Forces

•Fixed per-call overhead dominates small payloads
•Larger batches raise latency for early items
•Partial failures inside a batch need handling
•Downstream systems enforce batch size limits

Solution

Overview

Batching amortizes fixed costs. A database insert may spend 1 millisecond on the network and 0.05 milliseconds on the row, so inserting 1000 rows one by one takes about 1 second, while one multi-row statement takes roughly 60 milliseconds.

How it works

buffer = []
on item:        buffer.add(item)
                if size(buffer) >= 500: flush()
every 20 ms:    if buffer not empty: flush()

flush(): send(buffer); handle per-item results; clear buffer

Size-or-time triggers keep latency bounded under both heavy and light traffic. Kafka producers expose this as batch.size and linger.ms. Read-side batching appears in the DataLoader pattern, which gathers all keys requested in one tick of the event loop and issues a single multi-get, solving N plus 1 in GraphQL resolvers.

Treat batches as units of retry. Use idempotency keys or upserts so a replay after a timeout does not duplicate data. When the backend returns partial success, retry only the failed items and send repeated offenders to a dead-letter queue.

Bound the buffer. When the producer outpaces the flusher, apply backpressure or drop low-priority items instead of growing memory without limit.

Pitfalls

Choosing batch size by guess. Measure throughput and p99 latency at several sizes.
Hiding failures: a 200 response for a batch may contain item-level errors.
Large transactions that hold locks for a long time. Cap batch size.
Flushing on shutdown without draining the buffer.
Losing ordering when batches are sent in parallel.

Key takeaways

Trade a bounded delay for large throughput gains with size and time triggers.
Make batch operations idempotent and report per-item outcomes.
Bound buffers and drain them on shutdown.
Tune with measurements of throughput and tail latency.

✓ When to Use

•High-volume writes such as logs, events and metrics
•Fan-in of many small reads to one backend (multi-get, DataLoader)
•Throughput-bound pipelines where per-item latency is relaxed

✗ When Not to Use

•Interactive operations where every millisecond counts
•Items that must be committed or ordered individually
•Tiny volumes where batching adds complexity but no gain

Pros

+Large throughput gains per unit of overhead
+Fewer round trips and lower cost
+Smoother load on downstream systems

Cons

−Adds latency while the batch fills
−Partial failure handling is complex
−Buffered items can be lost on crash

Cost Profile

Infrastructure

Lower per-operation cost

Operational

Low

Cognitive

Medium: failure and ordering semantics

Failure Modes

Process crash loses items waiting in the buffer
One poison item causes the entire batch to be retried forever
Batch size grows beyond downstream limits and is rejected

Real-World Examples

Alternatives

Related Patterns

backpressure
dead-letter-queue

Competency Domains

distribution communicationeconomics evolution