Sub-document retrieval

What is sub-document retrieval?

Traditional search returns documents; retrieval-augmented systems often retrieve passages. A 5000-word article rarely matches a user query as a whole, but one well-formed paragraph within it often does. Retrieval-augmented AI engines typically index and rank at the passage or chunk level, with citations drawn from the top-ranked passages. The specific architecture (classic RAG, agentic retrieval, hybrid search variants) varies per engine and is generally not vendor-documented, but most converge on passage-level retrieval as the practical unit of citation¹.

Status in 2026

Passage-level retrieval is the dominant pattern reported in the RAG literature (Lewis et al. 2020¹; earlier passage-retrieval foundations in DPR and ColBERT) and across commercial vector-search vendor docs (Pinecone, Weaviate, Elasticsearch, LangChain, LlamaIndex all document passage-level chunking and ranking). Major AI search engines (ChatGPT search, Perplexity, Claude search, Copilot, AI Overview) have not officially documented their retrieval pipelines at this granularity, but observable behavior is consistent with passage-level retrieval: engines cite specific paragraphs rather than whole documents, and quoted text typically comes from specific page sections rather than from page summaries. The implication for content strategy is the same regardless of which inference path you accept: long-form articles compete passage-by-passage, and a well-shaped paragraph can earn more citation visibility than an equivalent point packed inside a longer unstructured article.

Note on this entry's territory (paired with the hybrid retrieval and LLMO entries' mirror observations): sub-document retrieval sits at the boundary between vendor-canonical and non-vendor-canonical territories. The general passage-retrieval pattern is vendor-canonical (Lewis et al. 2020 plus the major commercial vendor docs are detailed). The application to specific commercial AI search engines is non-vendor-canonical because none of those vendors publish their retrieval pipelines. The leap from "industry-standard pattern" to "this specific engine uses passage-level retrieval" is practitioner inference grounded in observable citation behavior, not vendor-confirmed fact.

How to apply

Sub-document retrieval is the underlying mechanism; your job is to make every section retrievable on its own merits. Three practical moves:

• Audit each article for "section standalone-ness": copy section text into a fresh AI chat with a locked audit prompt. Recommended prompt template: "Summarize this section in one sentence, then list any references to context outside the section that prevent the summary from being self-contained." If the model lists external-context dependencies, those are the rewrite candidates; restore self-contained meaning by inlining the needed context or by tightening the section's scope.
• Use the heading hierarchy as one of your chunking plans: heading-based chunking (LangChain MarkdownHeaderTextSplitter, LlamaIndex HierarchicalNodeParser) maps well to author intent; fixed-token chunking (LangChain RecursiveCharacterTextSplitter ~250 tokens, LlamaIndex SentenceSplitter 1024 tokens) is simpler and used by default in many libraries. Either way, writing H2 boundaries that describe what is underneath (rather than clever marketing phrases) gives both heading-based and fixed-token splitters cleaner edges to work with.
• Avoid mid-paragraph qualifications that fragment claims: a sentence like "X is true, but only when Y, unless Z, in which case actually W" will split awkwardly mid-chunk. Decompose into multiple sentences, each carrying a complete sub-claim.

What to skip: trying to reverse-engineer specific engines' chunking strategies. Chunk sizes change without notice; structural clarity is robust across the variation.

How it relates to other concepts

• Directly motivates section-first content design: each section should be understandable, attributable, and useful when retrieved alone. (Schema markup helps machine readability separately but is not the same as making sections retrievable; see the knowledge graph and defined-term schema entries for the schema-side discussion.)
• Companion to cite-ability: cite-able passages are those that survive chunking and still make sense.
• Architectural pattern within RAG (Retrieval-Augmented Generation), grounded in the Lewis et al. 2020 framework.
• Sister pattern of hybrid retrieval: hybrid retrieval handles the lexical-plus-semantic dimension; sub-document retrieval handles the granularity dimension. Most production RAG stacks combine both.
• Underlies agentic retrieval: agents pull passages iteratively, not entire pages.