AI search evaluation

AI search evaluation is the practice of measuring how AI search engines retrieve, ground, and attribute information, and how much influence individual sources have on the answers they generate. The same instruments serve two opposite directions: engine builders and academics evaluate the system (is this engine's retrieval relevant, are its answers grounded, are its citations honest), while publishers and GEO practitioners evaluate their source's standing inside it (is my page in the candidate pool, does it get cited, how much of the answer does it carry). In 2026 three method families coexist without a standard yardstick: academic benchmarks, vendor-internal evaluations (mostly unpublished), and practitioner probing of live consumer engines¹².

The public benchmark trajectory has so far been deflationary: later, more realistic evaluations have generally shrunk the optimization effects that earlier, simplified settings reported. That makes "which evaluation condition produced this number" the first question to ask of any AI-search claim, before the number itself.

Status in 2026

One useful way to read the public academic line is as a sequence of increasingly realistic evaluation conditions. Liu, Zhang, and Liang's 2023 verifiability study introduced citation precision and recall for generative search (do citations support their statements; are statements supported by citations) and found existing engines wanting on both¹. The same year, Aggarwal et al.'s GEO benchmark introduced PAWC (position-adjusted word count) as a source-visibility metric and reported double-digit relative gains for content edits like quotation addition (27.8 versus a 19.5 baseline on its metric)². Then came the re-tests: C-SEO Bench (2025) tested conversational-SEO methods (most derived from the GEO benchmark's, plus two novel ones) under multi-actor conditions where competing documents also optimize, and found effects far smaller than the single-pipeline numbers³. CC-GSEO-Bench (v2, December 2025) reframed the question as source influence, decomposing it into exposure, faithful credit, and causal impact over 1,000+ articles and 5,000+ query-article pairs⁴. SAGEO Arena (February 2026) evaluated the whole retrieval-to-generation pipeline, including structural signals like schema markup that earlier benchmarks omitted; in that benchmark's setting, its authors report that existing optimization approaches "remain largely impractical under realistic conditions," with degradation concentrated in the retrieval and reranking stages that simplified evaluations skip⁵.

Two gaps frame all of this. Vendor-internal evaluations (what OpenAI, Google, or Perplexity actually measure before shipping) are mostly unpublished; public model cards and product benchmarks expose some evaluation categories, but not the live citation-selection and source-ranking criteria that matter here. And academic benchmarks, however realistic, still are not the live product: they evaluate pipelines, not the consumer interface with its personalization, session context, and week-to-week model churn. Practitioner probing fills that last gap, but most published practitioner "studies" are status-grade rather than publication-grade: shifting prompt wording, no recorded denominators, no raw retention, API outputs standing in for consumer surfaces.

How to apply

AI search evaluation is a field to read critically and a discipline to borrow from, in three moves:

• Probe the product, not a proxy, and freeze the design. If your question is "does engine X cite my page," evaluate the consumer interface with a fixed query panel: frozen prompt wording, scheduled rounds, logged-out sessions, every not-cited result recorded as the denominator, raw screenshots retained. The citation probe protocol entry specifies this design; it is the practitioner instance of the evaluation discipline.
• Read paper metrics as designed, not as marketing. PAWC measures share of answer text, not rank; citation precision measures support, not visibility. A method that lifts PAWC in a 2023 single-pipeline benchmark has not thereby been shown to lift citations in a 2026 engine; that is exactly the gap the re-test generation documents.
• Check the evaluation condition before the result. Single-pipeline injection, multi-actor competition, and end-to-end retrieval produce systematically different effect sizes for the same methods (the GEO-bench to C-SEO Bench to SAGEO Arena trajectory). The condition includes the judge: most of these benchmarks score answers with LLM-as-judge setups, so which model judges (and with what rubric) is itself a condition that moves numbers. When two studies disagree, the conditions usually explain it.

What to skip: treating vendor composite "AI visibility scores" as evaluation (opaque weightings, not comparable across tools, no published methodology); evaluating consumer-surface questions through API outputs (different retrieval, different answers); and importing any benchmark's headline number into your planning without reading which condition produced it.

How it relates to other concepts

• Citation probe protocol is the practitioner-side instance of this discipline: fixed panel, frozen prompts, consumer interfaces, denominators. This entry is the map; that one is a working instrument.
• PAWC is the source-visibility metric the GEO benchmark line introduced and later work re-tests; its definition and limits are the cleanest example of why metric design shapes reported results.
• Citation precision carries the verifiability axis (do citations support statements), the system-side quality measure that started the academic line.
• C-SEO Bench is the multi-actor re-test generation: the first systematic shrinkage of single-pipeline GEO effects.
• GEO content methods are the things being evaluated: the content-edit families whose measured effects depend on which benchmark generation is doing the measuring.