Retrieval pipeline

What is the retrieval pipeline?

The retrieval pipeline is the chain of stages an AI search engine runs between a page someone published and the answer it shows: index, retrieve, rerank, assemble, generate, with an optional agentic loop that repeats the retrieve-and-assemble steps. RAG (Lewis et al., 2020) is the canonical two-stage version, retrieve then generate; production AI search adds the indexing that happens before and the reranking and assembly that happen in between.¹

The single most useful thing to know about this pipeline is what you can and cannot touch. You cannot tune any stage of it. The engine operates the chunker, the retriever, the reranker, and the assembler; their exact behavior is undocumented and differs by engine. What you control is one input: the passage you publish. So the most robust publisher-side move in generative engine optimization is not to optimize against a stage, it is to harden the passage so it survives whatever the pipeline does to it.

That reframes GEO advice: ranking-factor claims try to game one stage, but you do not get to pick the stage, so write for robustness to all of them.

The pipeline, stage by stage

Each stage tests a different property of your passage. They are not one property, they are several, but they are all properties of the passage as a single unit, and the same writing discipline produces all of them. That is why the lever is singular even though the tested properties are plural.

Index is the most upstream lever, and it is binary. Before anything else, the engine has to have crawled and indexed your passage; a page that is not in the corpus is one the rest of the pipeline never reaches. Then chunking splits it. Anthropic's contextual-retrieval work found that preserving context around each chunk cut retrieval failures substantially, which is the operator-side version of a publisher-side move: a passage that names its own subject survives the split, one whose subject sits only in a heading three paragraphs up does not.²

Retrieve is about recall across a query population, not one query. Retrievability (Azzopardi & Vinay, 2008) is the information-retrieval measure of exactly this, how easily a document is found across the whole space of queries people might ask, rather than how it ranks for one.³ Query fan-out is why the target is a population: an engine often expands one question into several sub-queries and retrieves for each, so a passage that matches only one phrasing is reached less often than one that answers a cluster of related questions. (BM25 and hybrid retrieval are the methods that do the finding; fan-out is the reason the bar is breadth.)

Rerank and assemble strongly affect whether a retrieved passage is available to be used. Reranking rescores the top candidates; context assembly then orders them and packs them into the token budget. This is where lost in the middle (Liu et al., 2023) bites: models use information at the start and end of a long context more reliably than the middle, so a passage assembled into the middle can be under-used regardless of relevance.⁴ You do not control the order, so the move is a passage that front-loads its own answer and reads as a complete claim even when a single fragment is quoted.

The agentic loop is optional, not a stage. In agentic retrieval and deep research mode, the system may repeat retrieve-and-assemble several times, refining as it goes, and tends to work with passages rather than whole pages (sub-document retrieval). It adds no separate publisher-side lever; it raises the value of the properties already listed. Because these systems pull passages rather than read a page whole, passage-level, self-contained writing matters at least as much in them as in single-pass retrieval, by construction.

Retrievability gates, quality converts

It is tempting to read "the upstream lever is retrievability" as "on-page writing does not matter." That is the wrong conclusion. The honest framing is a sequence, not a competition:

• Retrievability gates. If the engine cannot retrieve your passage, nothing downstream happens. This is a necessary condition, and it is genuinely upstream of any wording choice.
• Quality converts. Once a passage is retrieved, whether it gets used and cited is a question of how it is written: whether it answers the query cleanly, stands on its own, and front-loads its claim. That is editorial quality.

This is where the GEO content methods evidence fits without contradiction. The finding there, that several content methods are weak or null levers, is scoped to micro-tuning knobs: raising word count, padding unique terms, adopting an "authoritative tone." It is not a finding against passage quality. Answer-first, self-contained writing is not a micro-knob; it is a property that helps a passage survive across multiple stages of the pipeline. The two pillars together say: stop tuning micro-knobs on a page the engine cannot retrieve; make the passage retrievable first, then make it convert.

What you cannot tune, and one myth to drop

Because the exact pipeline is undocumented and varies by engine, you cannot optimize for a specific stage's behavior, only for robustness to all of them. Two anti-patterns follow:

• Chasing a stage's quirk. Tuning for one reranker's scoring or one engine's chunk size is fragile; it breaks the next time the engine changes, and it differs across engines you cannot see.
• The front-loading-wins-position myth. "Put your answer in the first 100 words so it ranks" confuses two different positions. Standard lexical and embedding retrieval do not rank a passage higher just because it sits earlier on the page: BM25 scores on term frequency and document length, and while embeddings do encode word order, neither rewards front-of-page placement. Where a sentence sits on the page does affect one thing, which chunk it lands in when the page is split, and that is the self-containment point above, not a ranking weight. The position effect that is real, lost in the middle, is about position within the assembled context window, which the engine sets, not position within your page. Front-loading the answer inside each passage is still worth doing, but for a different reason: so the claim survives being quoted as a fragment, not because early bytes rank higher.

This page describes a common production pattern for retrieval-augmented AI search, not a vendor-confirmed universal architecture: the public RAG literature documents retrieve-then-generate systems and vendor docs expose pieces like search, fan-out, and contextual retrieval, but no engine publishes its full pipeline.

How to apply

You cannot operate the pipeline, so optimize the one input you control to survive all of it:

• Be in the corpus. Make sure the page is crawlable and indexed; an unretrieved passage never enters an answer. This is the most upstream and most binary lever.
• Write self-contained passages. Each passage should carry its own subject and answer, so it survives chunking and still means something when assembly lifts it out alone.
• Be retrievable across a query population. Answer the cluster of related questions a topic attracts, not one keyword phrasing, so query fan-out reaches you more often.
• Front-load the claim inside each passage. Put the answer early in the passage so it reads as complete when a single fragment is quoted, and holds up wherever assembly places it.
• Do not depend on order or proximity. Cross-references like "as described above" break when the pipeline reorders or drops the referenced passage.

What to skip: tuning for a specific engine's stage, and chasing within-page position as a ranking lever.

The retrieval-pipeline cluster

This pillar synthesizes the cluster; each term below covers one mechanism in depth.

• Index: chunking, inverted index, vector embeddings, generative search index
• Retrieve: BM25, hybrid retrieval, retrievability, query fan-out
• Agentic loop: agentic retrieval, deep research mode, sub-document retrieval
• Rerank and assemble: reranking, context assembly, sub-passage extraction
• The architecture and the lever: RAG, passage-level optimization