← All guides · The agent-ready web

How AI Search Engines Choose Sources (And What You Can Actually Influence)

Part 4 of The agent-ready web · 1,348 words · published 2026-07-26 · updated 2026-07-26 · Markdown variant

What is known, what is claimed and what is speculation about how ChatGPT, Perplexity and AI Overviews pick the pages they cite — and the short list of things a site owner can actually control.

Part of the The Agent-Ready Website: A Complete Guide to AI Visibility, Access Control and Monetization guide.

This is the part of the agent-ready website where the industry is noisiest and the evidence thinnest. So it is worth separating three things: what is mechanically known about how these systems work, what the vendors state, and what is being sold as fact without support.

The mechanism, at the level that is actually known

Every mainstream answer engine is a retrieval-augmented generation system. The details differ per product and change without notice, but the pipeline shape is consistent:

1. Query understanding — the user's question is rewritten, often into several sub-queries.
2. Retrieval — candidate passages are gathered, from a live search index, the vendor's own crawl, a vector store of previously fetched content, or a mix.
3. Reranking — candidates are scored for relevance to the rewritten query, frequently by a cross-encoder or an LLM judge.
4. Generation with attribution — the model writes an answer over the surviving passages and attaches citations to the ones it used.

Two consequences follow directly from that shape, and they are the whole game:

• The retrieved unit is a passage, not a page. Your site is not competing as a domain; individual sections are competing as answers. The engineering behind this — how documents get split and what makes a chunk retrievable — is covered in chunking strategies for RAG and RAG and retrieval for agents. Reading them from the builder's side is the most useful thing a publisher can do, because it shows you exactly how your content will be cut up.
• Citation is a risk decision. A generation step that must attribute a claim prefers passages that make attribution easy and safe: explicit dates, named sources, unhedged specifics. This is not a moral preference, it is a mechanical one — an ambiguous passage produces an unverifiable sentence, which is the failure mode these systems are tuned hardest against.

What you can actually influence

Ordered by confidence. The first three are mechanical; the rest are reasonable inference.

1. Fetchability (mechanical, highest impact)

If the vendor's crawler gets a 403, nothing else on this list matters. Verify per user-agent token against your live origin, and remember that WAF and bot-management rules are evaluated before robots.txt. The token map — which bot indexes for answers versus which collects training data — is in AI crawler policy, and the failure catalogue is why AI agents can't read your site.

The specific mistake to avoid: blocking training crawlers on purpose and search crawlers by accident. GPTBot and OAI-SearchBot are different tokens with different jobs; so are ClaudeBot and Claude-SearchBot.

2. Passage self-containment (mechanical)

Write sections that survive being extracted alone. In practice:

• Question-shaped H2/H3s. A heading that matches how someone asks the question gives the retriever a strong lexical and semantic anchor.
• Answer in the first sentence of the section. Not after three sentences of preamble — the preamble is what gets retrieved when your answer is in sentence four.
• No unresolved pronouns across headings. "It supports three modes" is unretrievable; "The Query API supports three modes" is.
• Repeat the subject. Mild repetition that a stylist would cut is what keeps a chunk intelligible on its own.

3. Machine-cheap representations (mechanical)

A page whose content is only present after JavaScript hydration is unavailable to crawlers that do not render, and expensive for those that do. Markdown twins and JSON endpoints remove that risk entirely: see serving Markdown variants to AI agents and data formats and schema.

4. Verifiability signals (strong inference)

Give the generation step what it needs to defend a citation:

• Dates on claims, not just a publish date on the page. "As of July 2026, X is in closed beta" is quotable a year later because it is scoped; "X is in closed beta" becomes false silently.
• Named primary sources with links. A passage that says "per Cloudflare's September 2025 announcement" is safer to quote than the same claim unattributed.
• Numbers with units and provenance. Vague magnitudes ("many", "most sites") are unusable in an answer; a figure with a source is exactly what an answer needs.
• Explicit uncertainty. Saying "not independently confirmed" costs you nothing and makes the rest of the passage more trustworthy — a model can quote a hedged claim safely.

5. Structural clarity that machines already consume (moderate inference)

Real <table> markup for comparisons, real lists for sequences, real code blocks for code. Comparison tables in particular map cleanly onto the questions people ask answer engines ("X vs Y"), and they extract without ambiguity. Which schema.org types are worth adding on top is in structured data for AI agents.

6. Freshness where freshness is real (moderate inference)

A truthful dateModified on substantively revised pages is useful; bumping dates without changing content is the oldest bad idea in SEO and adds a new failure mode here, because a model that finds a stale claim under a fresh date has been actively misled by you.

What is being sold that is not supported

Being blunt about this saves budget:

• "llms.txt improves AI rankings." No vendor has committed to reading it. Publish it because it is cheap and useful for agents, not for citation lift. See how to write an llms.txt file.
• "Add an FAQ block and you get cited." FAQ markup helps some traditional rich results; there is no published evidence it drives answer-engine citation. Well-written question-shaped sections help retrieval regardless of markup.
• "Mention your brand N times so models learn it." Untestable, unfalsifiable, and indistinguishable from writing badly.
• "Optimise for entity density / semantic triples." Borrowed vocabulary, no published mechanism in any current answer engine.
• "Submit your site to AI engines." Where a vendor offers a real submission surface, use it; otherwise this is a paid list nobody reads.

The pattern is consistent: tactics that mimic 2010s SEO ritual, applied to systems that do not work like a ranked link index. The durable version of the same instinct is make each passage independently useful and independently verifiable.

What to measure instead of guessing

Because vendor behaviour is opaque, treat your own logs as the ground truth:

That last row is the strategic one, and it is where this cluster stops being an SEO conversation. If your content is being consumed to produce answers with no click coming back, your options are to accept it as brand exposure, restrict it, or price it. The restriction path is should you block AI crawlers?; the pricing path is what to charge AI crawlers. The instrumentation for all of it is measuring AI agent traffic.

A short, honest checklist

1. Verify every answer-engine crawler gets a 200 from your origin.
2. Rewrite your top ten pages so every H2 answers one question in its first sentence.
3. Put a date and a source on every factual claim you would mind being misquoted.
4. Ship Markdown twins for those ten pages and link them from llms.txt.
5. Turn comparisons into real tables.
6. Log crawler traffic server-side and check monthly whether fetches turn into referrals.
7. Re-check your own head terms in the two or three answer engines your audience uses, and note who is cited instead of you.

Nothing on that list is a trick, and that is the point: the systems reward being easy to retrieve and safe to quote, which is a property of the content, not of a tag.

#ai-visibility #citations #answer-engines #seo #rag