Stackbook Logo
data-stateestablished · medium operational burden

Inverted Index Search

Also known as: full-text-search-index, search-index

Intent

Map terms to the documents that contain them so that text queries are answered without scanning every document.

Problem

Searching for words across millions of documents with LIKE scans is slow and cannot rank results by relevance. Users expect typo tolerance, stemming and fast filtering. A relational B-tree index cannot answer arbitrary term queries.

Forces

  • •Query latency must stay low as the corpus grows
  • •Relevance ranking matters as much as matching
  • •Index must be updated as documents change
  • •Language handling such as stemming and stop words varies

Solution

Overview

An inverted index flips the document-to-words relationship into word-to-documents. It is the core of Lucene, Elasticsearch, OpenSearch and Solr, and of the full-text features in many databases.

How it works

Indexing and querying with a tiny example.

doc1: "fast database engine"
doc2: "database index"

term       postings
database   [doc1, doc2]
engine     [doc1]
fast       [doc1]
index      [doc2]

query "database AND index" -> [doc1,doc2] intersect [doc2] = [doc2]

Segments are immutable. Updates mark old versions deleted and write new documents, and merges later reclaim space.

Relevance scoring with BM25 weighs how often a term appears in a document against how rare it is across the corpus, with length normalization so long documents do not dominate. Filters and facets use separate doc-value structures, which are columnar, so sorting and aggregations do not need to load documents. Near-real-time visibility comes from a periodic refresh that opens new segments, typically every second, which is why a just-written document may not appear in search immediately.

Pitfalls

  • Treating the search index as the system of record; keep the database authoritative.
  • Dual writing to database and index instead of using events or CDC.
  • Too many small shards, which waste memory and slow queries.
  • Using the default analyzer for every language and field.

Key takeaways

  • Inverted indexes make term search fast and rankable.
  • Search results are eventually consistent with the source.
  • Plan reindexing from day one; mappings change.
  • Rebuild the index from the source of truth when needed.

✓ When to Use

  • •Product, document or log search
  • •Queries needing relevance ranking, faceting and typo tolerance
  • •Corpora too large for scanning relational text columns

✗ When Not to Use

  • •Exact-match lookups served by a normal index
  • •Small datasets where database full-text features suffice
  • •Systems requiring strong transactional consistency on search results

Pros

  • +Fast term and phrase queries at scale
  • +Built-in relevance scoring and facets
  • +Horizontally scalable via sharding

Cons

  • −Near-real-time, not immediately consistent, after writes
  • −Index can be larger than the source data
  • −Analyzer and mapping changes often force reindexing

Cost Profile

Infrastructure

Medium - memory and disk for index segments and caches

Operational

Medium - shard sizing, merges and reindexing

Cognitive

Medium - analyzers, scoring and mappings

Failure Modes

  • Index drifts from the source of truth after missed update events

  • Hot shards or oversized shards degrade query latency

  • Mapping change requires a full reindex that overloads the cluster

Real-World Examples

Alternatives

    Related Patterns

    • cdc
    • polyglot-persistence

    Competency Domains

    data statedistribution communication