Fan-Out Fan-In

Overview

Fan-out fan-in, also called scatter-gather, trades coordination complexity for speed. The job is split, processed concurrently and joined.

The hard part is not the fan-out; it is knowing when the fan-in is done and what to do when it never completes.

The aggregator needs thought. Counting arrivals is fragile under duplicate delivery, so record results by part index in durable storage and complete when all indices are present. Workflow engines can express this directly as parallel steps with a join, saving you from hand-building the tracking logic.

How it works

The coordinator publishes N subtasks with a job id. The aggregator persists a record per job and marks each part as it arrives.

job = new_id(); parts = split(input)
for i, p in enumerate(parts):
    publish("work", {job, i, n: len(parts), p})

# aggregator
state[job].parts[i] = result        # idempotent by index
if len(state[job].parts) == n or past_deadline(job):
    emit(merge(state[job]))

Tail latency matters because the job finishes when the slowest part does. Mitigate with timeouts, hedged retries for stragglers and smaller chunks. Decide the partial-failure contract up front: fail the whole job, return partial results with a flag, or retry only missing parts. Cap concurrency so the fan-out does not overwhelm shared downstream resources.

Pitfalls

• Counting results instead of keying by part index, so duplicates complete the job early.
• No deadline, so a single lost part blocks forever.
• Keeping aggregator progress only in memory.
• Splitting into thousands of tiny tasks where overhead dominates.

Key takeaways

• Key results by part index to make aggregation idempotent.
• Always define a deadline and partial-result policy.
• Persist aggregator state.
• Size subtasks so work outweighs overhead.