Stackbook Logo
architecture-descriptionestablished · medium operational burden

Pipes and Filters

Also known as: pipeline-architecture, filter-chain

Intent

Decompose a processing task into independent steps connected by channels so steps can be reordered, reused, and scaled separately.

Problem

A monolithic processing routine that parses, validates, transforms, and enriches data is hard to test, reuse, or scale. One slow step limits the whole flow and every change risks the entire routine.

Forces

  • •Steps have different performance and scaling needs
  • •Steps should be reusable in different sequences
  • •Data format between steps must be agreed
  • •Errors in one step must not corrupt the whole stream

Solution

Overview

Pipes and filters is one of the oldest architectural styles, familiar from Unix shells: each program does one job and output flows to the next. The same idea powers stream processors, build pipelines, and HTTP middleware.

How it works

Each filter has one input and one output and no knowledge of its neighbors. Pipes carry messages in order, and backpressure keeps a fast producer from overwhelming a slow consumer.

cat access.log | grep 500 | cut -d' ' -f1 | sort | uniq -c

[Source] -> (pipe) -> [Parse] -> (pipe) -> [Validate]
         -> (pipe) -> [Enrich] -> (pipe) -> [Sink]
                         \-> dead-letter queue for failures

When filters are separate services connected by queues, a slow stage can be scaled out by adding consumers, provided messages are independent. Where order matters, partition by a key so each key is handled sequentially.

Pitfalls

  • Unbounded pipes, so a slow stage exhausts memory or disk; use bounded buffers and backpressure.
  • No dead-letter path, so one bad message blocks the stream.
  • Filters that depend on each other's internal state.
  • Assuming exactly-once delivery; design filters to be idempotent.
  • Fine-grained filters whose serialization cost exceeds their useful work.

Key takeaways

  • One responsibility per filter, one agreed message format.
  • Bound the pipes and handle bad messages explicitly.
  • Make filters idempotent and stateless where possible.
  • Scale the bottleneck stage, not the whole pipeline.

✓ When to Use

  • •Data processing pipelines such as ETL, media encoding, or log enrichment
  • •Compilers and request middleware chains
  • •Steps with distinct scaling profiles

✗ When Not to Use

  • •Steps that need heavy shared state or interactive back-and-forth
  • •Simple operations where a single function is clear
  • •Strict low-latency paths where channel overhead matters

Pros

  • +Composable, testable single-purpose steps
  • +Independent scaling and parallelism per filter
  • +Easy to insert or replace steps

Cons

  • −Serialization and channel overhead
  • −Harder end-to-end error handling and tracing
  • −Shared message format couples filters

Cost Profile

Infrastructure

Low in-process; queues needed when distributed

Operational

Medium — monitor each stage and backlog

Cognitive

Low — simple mental model

Failure Modes

  • Slow filter causing unbounded queue growth upstream

  • Poison message blocking or repeatedly failing a stage

  • Message format change breaking downstream filters

Real-World Examples

Alternatives

    Related Patterns

      Competency Domains

      distribution communicationreliability ops