# How AI Search Engines Choose Sources (And What You Can Actually Influence)

> What is known, what is claimed and what is speculation about how ChatGPT, Perplexity and AI Overviews pick the pages they cite — and the short list of things a site owner can actually control.

Guide: The agent-ready web — part 4
Published: 2026-07-26 · Updated: 2026-07-26 · 1348 words
Canonical: https://changegamer.ai/articles/how-ai-search-engines-choose-sources
JSON: https://changegamer.ai/api/articles/how-ai-search-engines-choose-sources.json
Pillar: https://changegamer.ai/articles/agent-ready-website.md

## In short

- Answer engines are retrieval systems with a generation step. Citation is decided by retrieval (can a passage be found and matched) and by verifiability (can the model defend quoting it) — not by anything resembling PageRank.
- The strongest controllable factor is passage-level self-containment: a section that answers one question completely, without depending on the paragraph above it, is the unit that gets retrieved and quoted.
- Be explicit about facts: dates, numbers, named sources, versions. A model choosing between two passages will prefer the one it can attribute.
- Most "GEO" tactics circulating today are untested. Treat anything without a published mechanism as a hypothesis, and spend the budget on retrievability instead.
- You cannot be cited if you cannot be fetched. Access verification comes before any of this.

---

This is the part of [the agent-ready website](/articles/agent-ready-website) where the industry is noisiest and the evidence thinnest. So it is worth separating three things: what is mechanically known about how these systems work, what the vendors state, and what is being sold as fact without support.

## The mechanism, at the level that is actually known

Every mainstream answer engine is a retrieval-augmented generation system. The details differ per product and change without notice, but the pipeline shape is consistent:

1. **Query understanding** — the user's question is rewritten, often into several sub-queries.
2. **Retrieval** — candidate passages are gathered, from a live search index, the vendor's own crawl, a vector store of previously fetched content, or a mix.
3. **Reranking** — candidates are scored for relevance to the rewritten query, frequently by a cross-encoder or an LLM judge.
4. **Generation with attribution** — the model writes an answer over the surviving passages and attaches citations to the ones it used.

Two consequences follow directly from that shape, and they are the whole game:

- **The retrieved unit is a passage, not a page.** Your site is not competing as a domain; individual sections are competing as answers. The engineering behind this — how documents get split and what makes a chunk retrievable — is covered in [chunking strategies for RAG](/resources/chunking-strategies-for-rag) and [RAG and retrieval for agents](/resources/rag-retrieval-for-agents). Reading them from the *builder's* side is the most useful thing a publisher can do, because it shows you exactly how your content will be cut up.
- **Citation is a risk decision.** A generation step that must attribute a claim prefers passages that make attribution easy and safe: explicit dates, named sources, unhedged specifics. This is not a moral preference, it is a mechanical one — an ambiguous passage produces an unverifiable sentence, which is the failure mode these systems are tuned hardest against.

## What you can actually influence

Ordered by confidence. The first three are mechanical; the rest are reasonable inference.

### 1. Fetchability (mechanical, highest impact)

If the vendor's crawler gets a 403, nothing else on this list matters. Verify per user-agent token against your live origin, and remember that WAF and bot-management rules are evaluated before `robots.txt`. The token map — which bot indexes for answers versus which collects training data — is in [AI crawler policy](/resources/ai-crawler-policy), and the failure catalogue is [why AI agents can't read your site](/articles/why-ai-agents-cant-read-your-site).

The specific mistake to avoid: blocking training crawlers on purpose and search crawlers by accident. `GPTBot` and `OAI-SearchBot` are different tokens with different jobs; so are `ClaudeBot` and `Claude-SearchBot`.

### 2. Passage self-containment (mechanical)

Write sections that survive being extracted alone. In practice:

- **Question-shaped H2/H3s.** A heading that matches how someone asks the question gives the retriever a strong lexical and semantic anchor.
- **Answer in the first sentence of the section.** Not after three sentences of preamble — the preamble is what gets retrieved when your answer is in sentence four.
- **No unresolved pronouns across headings.** "It supports three modes" is unretrievable; "The Query API supports three modes" is.
- **Repeat the subject.** Mild repetition that a stylist would cut is what keeps a chunk intelligible on its own.

### 3. Machine-cheap representations (mechanical)

A page whose content is only present after JavaScript hydration is unavailable to crawlers that do not render, and expensive for those that do. Markdown twins and JSON endpoints remove that risk entirely: see [serving Markdown variants to AI agents](/articles/serving-markdown-variants-to-ai-agents) and [data formats and schema](/resources/data-formats).

### 4. Verifiability signals (strong inference)

Give the generation step what it needs to defend a citation:

- **Dates on claims,** not just a publish date on the page. "As of July 2026, X is in closed beta" is quotable a year later because it is scoped; "X is in closed beta" becomes false silently.
- **Named primary sources with links.** A passage that says "per Cloudflare's September 2025 announcement" is safer to quote than the same claim unattributed.
- **Numbers with units and provenance.** Vague magnitudes ("many", "most sites") are unusable in an answer; a figure with a source is exactly what an answer needs.
- **Explicit uncertainty.** Saying "not independently confirmed" costs you nothing and makes the rest of the passage more trustworthy — a model can quote a hedged claim safely.

### 5. Structural clarity that machines already consume (moderate inference)

Real `<table>` markup for comparisons, real lists for sequences, real code blocks for code. Comparison tables in particular map cleanly onto the questions people ask answer engines ("X vs Y"), and they extract without ambiguity. Which schema.org types are worth adding on top is in [structured data for AI agents](/articles/structured-data-for-ai-agents).

### 6. Freshness where freshness is real (moderate inference)

A truthful `dateModified` on substantively revised pages is useful; bumping dates without changing content is the oldest bad idea in SEO and adds a new failure mode here, because a model that finds a stale claim under a fresh date has been actively misled by you.

## What is being sold that is not supported

Being blunt about this saves budget:

- **"llms.txt improves AI rankings."** No vendor has committed to reading it. Publish it because it is cheap and useful for agents, not for citation lift. See [how to write an llms.txt file](/articles/how-to-write-an-llms-txt-file).
- **"Add an FAQ block and you get cited."** FAQ markup helps some traditional rich results; there is no published evidence it drives answer-engine citation. Well-written question-shaped sections help *retrieval* regardless of markup.
- **"Mention your brand N times so models learn it."** Untestable, unfalsifiable, and indistinguishable from writing badly.
- **"Optimise for entity density / semantic triples."** Borrowed vocabulary, no published mechanism in any current answer engine.
- **"Submit your site to AI engines."** Where a vendor offers a real submission surface, use it; otherwise this is a paid list nobody reads.

The pattern is consistent: tactics that mimic 2010s SEO ritual, applied to systems that do not work like a ranked link index. The durable version of the same instinct is *make each passage independently useful and independently verifiable*.

## What to measure instead of guessing

Because vendor behaviour is opaque, treat your own logs as the ground truth:

| Signal | Where it comes from | What it tells you |
|---|---|---|
| Requests by crawler UA, per path | Server-side logs | Which products are indexing you at all |
| 403/404/429 rate by UA | Server-side logs | Whether your edge is undoing your policy |
| Referrals from answer-engine domains | Referrer header | Whether citations produce clicks |
| Live spot-checks of your own head terms | Manual, monthly | Whether you are cited, and who is cited instead |
| Fetch-to-referral ratio | Both | Crawl without referral means you are feeding answers, not receiving traffic |

That last row is the strategic one, and it is where this cluster stops being an SEO conversation. If your content is being consumed to produce answers with no click coming back, your options are to accept it as brand exposure, restrict it, or price it. The restriction path is [should you block AI crawlers?](/articles/should-you-block-ai-crawlers); the pricing path is [what to charge AI crawlers](/articles/what-to-charge-ai-crawlers). The instrumentation for all of it is [measuring AI agent traffic](/articles/measuring-ai-agent-traffic).

## A short, honest checklist

1. Verify every answer-engine crawler gets a 200 from your origin.
2. Rewrite your top ten pages so every H2 answers one question in its first sentence.
3. Put a date and a source on every factual claim you would mind being misquoted.
4. Ship Markdown twins for those ten pages and link them from llms.txt.
5. Turn comparisons into real tables.
6. Log crawler traffic server-side and check monthly whether fetches turn into referrals.
7. Re-check your own head terms in the two or three answer engines your audience uses, and note who is cited instead of you.

Nothing on that list is a trick, and that is the point: the systems reward being easy to retrieve and safe to quote, which is a property of the content, not of a tag.

## Frequently asked questions

### Is there a ranking algorithm for AI answers I can optimise against?

No public one, and the architectures differ per product: some ground answers in a live search index, some in their own crawl, some in a vector index of previously fetched content, and most combine several. What they share is a retrieval step over passages and a generation step that has to justify a citation — which is why passage quality and verifiability are the durable levers rather than any single trick.

### Does blocking training crawlers stop me being cited?

It depends on which token you block, because training and answering are usually different crawlers. Blocking `GPTBot` (training) does not by itself remove you from ChatGPT Search, which uses `OAI-SearchBot`; blocking the search bot does. Check the token table before assuming a block is safe.

### Do I need to write "AI-friendly" content?

You need to write structurally clear content, which is not a new discipline. Question-shaped headings, one idea per section, real tables, dated claims and named sources all help a retrieval system — and all of them also help human readers. There is no separate register that flatters models.

### How do I tell whether AI answer engines are actually reading my site?

Server-side logs by user agent are the only reliable signal, plus referral traffic from answer-engine domains. Client-side analytics miss almost all of it. See the measurement guide linked below.


---

## The rest of this guide

- [The Agent-Ready Website: A Complete Guide to AI Visibility, Access Control and Monetization](https://changegamer.ai/articles/agent-ready-website.md): The full operator playbook for making a website work for AI agents and AI crawlers: be fetchable, be readable, be controllable, be payable — with a 30-day implementation plan.
- [llms.txt vs robots.txt vs sitemap.xml: Which File Does What](https://changegamer.ai/articles/llms-txt-vs-robots-txt-vs-sitemap.md): The three root-level files every agent-ready site publishes, what each one is actually for, and why publishing one does not substitute for the others.
- [How to Write an llms.txt File (Format, Template, and Maintenance)](https://changegamer.ai/articles/how-to-write-an-llms-txt-file.md): A step-by-step guide to writing a useful llms.txt: the exact format, a copy-paste template, what to put under ## Optional, how to validate it, and how to keep it from rotting.
- [Serving Markdown Variants to AI Agents: The Cheapest Win in AI Visibility](https://changegamer.ai/articles/serving-markdown-variants-to-ai-agents.md): How to publish a .md twin of every page — URL patterns, content negotiation, discovery headers, generation pitfalls — and why it cuts what an agent pays to read you.
- [Should You Block AI Crawlers? A Decision Framework by Business Model](https://changegamer.ai/articles/should-you-block-ai-crawlers.md): Blocking AI crawlers is four separate decisions, not one. A framework that maps each crawler class to what it costs and earns you, by business model, with the exact robots.txt for each answer.
- [What to Charge AI Crawlers: Pricing Models for Machine Buyers](https://changegamer.ai/articles/what-to-charge-ai-crawlers.md): Per-crawl, per-resource, corpus licence or subscription key — the four ways to price AI access, the arithmetic behind each, and why pricing before you have demand data is the standard mistake.
- [Implementing an HTTP 402 Paywall an Agent Can Actually Pay](https://changegamer.ai/articles/http-402-paywall-implementation.md): A working implementation guide for machine-payable content: the 402 response body, Link headers, key issuance and validation, caching rules, and the mistakes that make a 402 gate unpayable.
- [Structured Data for AI Agents: Which Schema.org Types Earn Their Keep](https://changegamer.ai/articles/structured-data-for-ai-agents.md): Most schema.org markup is invisible to machine readers. The types that are worth the effort for AI agents, how to emit them without drift, and what to build instead of more markup.
- [JSON API Design for AI Agents: Endpoints They Prefer Over Scraping](https://changegamer.ai/articles/json-api-design-for-agents.md): How to publish read-only JSON endpoints that agents choose over scraping your HTML: discovery index, stable shapes, freshness signals, bulk exports, and errors a machine can act on.
- [Measuring AI Agent Traffic: Server-Side Telemetry That Answers Real Questions](https://changegamer.ai/articles/measuring-ai-agent-traffic.md): Why client-side analytics miss AI agents entirely, the minimum row schema to log, the five queries worth running, and how to tell a real crawler from a spoofed user agent.
- [Licensing Content for AI Training: RSL, Terms, and Provenance](https://changegamer.ai/articles/licensing-content-for-ai-training.md): How to publish machine-readable licence terms for AI use — what RSL is, what it does and does not do, how it differs from robots.txt and Content Signals, and where provenance standards fit.
- [Running an MCP Server as a Distribution Channel for Your Content](https://changegamer.ai/articles/mcp-server-as-distribution-channel.md): Why a content site should expose an MCP server, which tools to ship, how discovery and authentication work, how to gate paid tools, and the honest limits of the channel.
- [Why AI Agents Can't Read Your Site: Twelve Failure Modes and How to Find Them](https://changegamer.ai/articles/why-ai-agents-cant-read-your-site.md): A diagnostic catalogue of the twelve reasons AI agents and crawlers fail on real sites — from silent WAF blocks to JS-only rendering — each with the command that detects it and the fix.

## Reference resources

- https://changegamer.ai/resources/rag-retrieval-for-agents.md
- https://changegamer.ai/resources/chunking-strategies-for-rag.md
- https://changegamer.ai/resources/ai-crawler-policy.md
- https://changegamer.ai/resources/data-formats.md

All guides: https://changegamer.ai/api/articles.json · Reference corpus: https://changegamer.ai/llms.txt