Stackbook Logo
← Back to System Architectures

Designing a Content Moderation Pipeline

Design a moderation pipeline for 100M posts a day combining hash matching, ML classifiers, human review queues, appeals and audit trails.

5 min read

A moderation pipeline decides whether user content violates policy, acting fast on clear violations and routing ambiguous cases to humans. It must balance speed, accuracy, cost and fairness, and it must be auditable.

Requirements

Functional

  • Screen text, images and video at upload and on user report.
  • Actions: allow, label, limit distribution, remove, or escalate to human review.
  • Review tooling with queues, policy guidance, and decision logging.
  • Appeals, and user notification of decisions.

Non-functional

  • 100M items per day, about 1,200 per second average, 5,000 at peak.
  • Automated screening under 2 seconds for text and images; video within minutes.
  • High recall on severe categories such as child safety and terrorism, with precision tuned elsewhere.
  • Every decision traceable to a policy version and a reviewer or model version.

Capacity estimation

  • 100M items per day. Suppose 2 percent are flagged by classifiers as uncertain: 2M per day for humans is too many, so raise automation. Target 0.2 percent human review: 200,000 items per day.
  • A reviewer handles about 400 items per day, so that needs about 500 reviewers per shift pattern, plus reports and appeals.
  • Classifier inference: text at 5 ms and images at 30 ms on GPUs. Peak 5,000 items per second needs a modest GPU fleet with batching, and video needs sampling of frames.
  • Decision log: 100M records per day at 500 bytes is 50 GB per day.

High-level design

Upload -> Content Service -> Event stream (Kafka)
                                  |
                      +-----------+------------+
                      v                        v
            Synchronous fast checks     Async deep analysis
            (hash match, blocklist,     (ML classifiers, OCR,
             regex, spam rules)          video frame sampling)
                      |                        |
                      +----------> Decision Engine <---- user reports
                                       |
          allow / limit / remove / escalate   (policy rules + thresholds)
                                       |
                         Human Review Queues -> Reviewer tool
                                       |
                    Enforcement service + Audit log + Notifications + Appeals

Data model

content_decision(content_id, version, action, reasons, model_versions, policy_version, decided_by, ts)
review_task(task_id, content_id, queue, priority, assigned_to, status, sla_deadline)
hash_list(hash, category, source)         -- known-bad media
user_strikes(user_id, policy, count, last_action_ts)

Decisions are append-only, so history is never overwritten and appeals can reference the exact state at the time.

Deep dives

Layered detection. Cheap checks run first. Exact and perceptual hash matching (such as PhotoDNA-style or pHash) against known-bad databases catches repeat content with near zero cost. Blocklists and rules handle spam patterns. Only content passing through goes to ML classifiers, which produce per-category scores for hate, violence, nudity, harassment and similar.

Thresholds. For each category define two thresholds. Above the high threshold, act automatically; between thresholds, send to a human; below the low one, allow. Set thresholds from measured precision and recall per category, and keep severe categories biased to recall.

Pre-publish versus post-publish. Blocking everything until checked adds latency and hurts experience. Common approach: fast checks run synchronously and can stop obvious violations; deep analysis runs asynchronously and takes content down within seconds to minutes. High-risk surfaces such as live video or new accounts get stricter pre-publish gating.

Human review. Queue by severity and reach: content already spreading or involving child safety jumps to the front, with SLA timers. Show reviewers context and policy text, support double review for borderline items, and measure agreement. Limit exposure to harmful material with blurring, rotation and wellbeing support.

Feedback loop. Reviewer decisions become labelled training data. Periodically retrain, evaluate on a held-out audit set, shadow-deploy new models, and compare against production before switching.

Appeals and transparency. An appeal creates a task routed to a different reviewer. Notify users with the reason and policy cited. Overturns feed back into accuracy metrics.

Reports. User reports raise priority and give extra signal, but report volume can be brigaded, so weight by reporter reliability.

Failure modes and trade-offs

  • Classifier outage or latency spike: fall back to hash and rule checks, and queue content for later scanning rather than failing open entirely on high-risk surfaces.
  • Backlog in review queues: prioritise by severity and reach; temporarily limit distribution of unreviewed flagged content.
  • Model drift and adversarial evasion: monitor score distributions, add new hashes quickly, and retrain on recent evasions.
  • Over-removal versus under-removal: both harm users. Track precision and recall per category and per language, because models perform unevenly across languages and dialects.
  • Policy changes: version policies so old decisions remain explainable, and support bulk re-evaluation of affected content.

Key takeaways

  • Layer cheap deterministic checks before expensive ML, and humans last.
  • Use two thresholds per category to separate auto-action, human review and allow.
  • Combine synchronous gating for obvious violations with asynchronous deep analysis.
  • Prioritise human queues by severity and reach, and feed decisions back as training data.
  • Log every decision with policy and model versions to support appeals and audits.