# Why AI Agents Can't Read Your Site: Twelve Failure Modes and How to Find Them

> A diagnostic catalogue of the twelve reasons AI agents and crawlers fail on real sites — from silent WAF blocks to JS-only rendering — each with the command that detects it and the fix.

Guide: The agent-ready web — part 13
Published: 2026-07-26 · Updated: 2026-07-26 · 1407 words
Canonical: https://changegamer.ai/articles/why-ai-agents-cant-read-your-site
JSON: https://changegamer.ai/api/articles/why-ai-agents-cant-read-your-site.json
Pillar: https://changegamer.ai/articles/agent-ready-website.md

## In short

- The most common cause is not content: it is an edge rule returning 403 to crawlers your robots.txt explicitly allows.
- Diagnose from outside with a spoofed user agent. A page that looks fine in your browser proves nothing about what a crawler receives.
- JavaScript-only content is invisible to crawlers that do not render and expensive for those that do — check the raw HTML response, not the DOM.
- Soft 404s and 200-with-error-page responses are worse than honest errors, because an agent will quote the error page as your content.
- Fix in this order: access, then rendering, then structure, then formats. Each layer is worthless while the one above it is broken.

---

When a site is absent from AI answers, the cause is usually mechanical and locatable in under an hour. This is the diagnostic half of [the agent-ready website](/articles/agent-ready-website): twelve failure modes, in the order they actually occur, each with a detection command and a fix.

Run the whole matrix first, then fix top-down. There is no point improving structure while your edge is returning 403.

## The five-minute baseline

```bash
SITE=https://example.com
for ua in "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" \
          "Mozilla/5.0 (compatible; ClaudeBot/1.0; +https://anthropic.com/aup)" \
          "PerplexityBot/1.0" "curl/8.0"; do
  for p in / /robots.txt /sitemap.xml /llms.txt; do
    printf "%-22s %-14s " "${ua:0:22}" "$p"
    curl -sI -A "$ua" "$SITE$p" | head -1
  done
done
```

Everything below is what to do with the results.

## 1. WAF or bot management is 403ing crawlers you allowed

**Symptom:** `403` for crawler user agents, `200` in your browser. `robots.txt` says `Allow`.

**Why:** edge rules run before `robots.txt` is fetched and have no knowledge of it. Managed "AI bot" rulesets are often enabled by default or turned on during an unrelated security review. On this site, Cloudflare's managed rule silently 403'd `GPTBot`, `ChatGPT-User`, `OAI-SearchBot`, `PerplexityBot`, `CCBot` and `Google-CloudVertexBot` — including on `/` and `/sitemap.xml` — while robots.txt allowed all of them.

**Detect:** the baseline matrix above.

**Fix:** set AI-crawler controls to allow explicitly, then add a rule that skips managed rules for the user agents you want, with logging on. Re-verify afterwards — this is the failure mode most likely to silently return.

## 2. Browser-integrity checks reject non-browser clients

**Symptom:** `403` for `curl` and every non-browser UA, including your own monitoring and sometimes search-console fetches.

**Why:** integrity checks look for a full set of browser-like headers. A legitimate crawler sends few of them. This site's Browser Integrity Check broke Google Search Console's sitemap fetch this way.

**Detect:** `curl -sI -A "curl/8.0" $SITE/` returning 403 while a browser succeeds.

**Fix:** disable on content paths, or scope to authenticated and form-submission routes only.

## 3. Primary content requires JavaScript

**Symptom:** raw HTML contains a shell and a script bundle; your text appears only after hydration.

**Detect:**

```bash
curl -s $SITE/your-page | grep -c "a distinctive sentence from your article"   # expect >= 1
curl -s $SITE/your-page | wc -c                                              # tiny = shell only
```

**Fix:** server-render or statically generate primary content. If that is a large project, ship a [Markdown variant](/articles/serving-markdown-variants-to-ai-agents) of the content immediately — it is a smaller change and it solves the machine-readability problem completely while the rendering work is scheduled.

## 4. Soft 404s and 200-with-error-page

**Symptom:** missing pages return `200` with "not found" in the body.

**Why it is worse for agents than for search:** a crawler deduplicates near-identical pages; an agent quotes what it received. A confident model will happily tell a user that your product page says "Sorry, this page could not be found".

**Detect:** `curl -so /dev/null -w "%{http_code}\n" $SITE/definitely-not-a-real-url` — expect `404`.

**Fix:** honest status codes everywhere: `404`/`410` for gone, `301` for moved, `503` with `Retry-After` for maintenance.

## 5. Redirect chains and canonical drift

**Symptom:** two or more hops to reach content; `http`→`https`→`www`→trailing-slash.

**Why:** every hop costs a round trip against an agent's budget, and chains frequently end in a URL that disagrees with your canonical tag and your sitemap.

**Detect:**

```bash
curl -sIL -o /dev/null -w "%{num_redirects} hops → %{url_effective}\n" $SITE/your-page
```

**Fix:** one hop maximum. Pick a canonical form, serve it directly, and make sitemap, canonical tag and internal links agree with it.

## 6. Rate limits that look like hostility

**Symptom:** `429` or `403` after a handful of requests, no `Retry-After`.

**Why:** an agent that receives a bare block has no protocol for coming back. One with `Retry-After` does, and well-built ones honour it — the client-side contract is in [handling LLM rate limits and retries](/resources/handling-rate-limits-and-retries).

**Detect:** fire 20 sequential requests and watch where the status changes.

**Fix:** graduated limits, a real `Retry-After` header, a JSON body naming the limit, and — the useful part — a pointer to a bulk endpoint so a client fetching 500 pages can fetch one file instead.

## 7. Cookie walls, consent gates and interstitials

**Symptom:** the first meaningful text in your HTML is a consent notice.

**Why:** an agent has no consent to give and cannot dismiss a modal. Worse, the notice becomes the extracted "content" of the page.

**Detect:** read the first 2 KB of the raw response and ask what a summariser would conclude the page is about.

**Fix:** do not block content on consent for read-only requests. Where you must show a notice, keep it out of the top of the document order and out of the Markdown variant.

## 8. Content locked in PDFs, images and iframes

**Symptom:** the substance is a scanned PDF, an infographic, or an embedded third-party widget.

**Why:** extraction from these is lossy at best. Document parsing has improved a great deal but remains a cost and an error source — the state of it is covered in [document extraction and parsing for agents](/resources/document-extraction-for-agents).

**Fix:** publish an HTML or Markdown version of anything you want quoted. Keep the PDF as the printable artefact, not the canonical one.

## 9. Infinite scroll and click-to-reveal

**Symptom:** listings and comments only load on scroll; tabs and accordions hide content behind clicks.

**Why:** a fetch gets the first page and nothing else; even browser-driving agents pay a high cost to keep interacting — see [agentic browsers](/resources/agentic-browsers) and [web data and scraping for agents](/resources/web-data-for-agents).

**Fix:** real paginated URLs with `rel="next"`, or a JSON endpoint that returns the whole list. Render tab and accordion content in the HTML and hide it with CSS rather than withholding it.

## 10. Structure a machine has to guess

**Symptom:** headings are styled `<div>`s; tables are grids of divs; code is a `<pre>` inside three wrappers with syntax spans.

**Why:** every inference is a chance to be wrong, and being wrong about your content is worse for you than not being read.

**Detect:** `curl -s $SITE/page | grep -c "<h2"` and `grep -c "<table"` against what the page visibly contains.

**Fix:** real semantic elements. Tables in particular map directly onto the comparison questions people ask answer engines, so converting a div-grid into a real `<table>` has outsized value — see [data formats and schema](/resources/data-formats).

## 11. No machine-readable variant of anything

**Symptom:** every consumer, human or machine, gets the same heavy HTML.

**Why:** it works, badly. You pay bandwidth, they pay tokens, and structure has to be re-derived on every fetch.

**Fix:** in ascending order of effort — a Markdown twin per page, a JSON index, a bulk export, an llms.txt pointing at all of them. The cheapest three are covered in [serving Markdown variants](/articles/serving-markdown-variants-to-ai-agents), [JSON API design for agents](/articles/json-api-design-for-agents) and [how to write an llms.txt file](/articles/how-to-write-an-llms-txt-file).

## 12. Your policy contradicts itself

**Symptom:** `robots.txt` allows a crawler that your WAF blocks; your sitemap lists `noindex` pages; your canonical tag points somewhere your redirects do not; Content Signals say one thing and your licence another.

**Why:** these files are written by different people at different times, and machines resolve contradictions by distrusting all of it.

**Detect:** read `robots.txt`, your sitemap, your canonical tags and your licence in one sitting and check they describe the same site.

**Fix:** generate what you can from one source, and re-run the baseline matrix after every policy change. Which tokens should say what is worked through in [should you block AI crawlers?](/articles/should-you-block-ai-crawlers) and the reference table is [AI crawler policy](/resources/ai-crawler-policy).

## Fix order

| Priority | Layer | Failure modes |
|---|---|---|
| 1 | Access | 1, 2, 6 |
| 2 | Response honesty | 4, 5, 12 |
| 3 | Rendering | 3, 7, 9 |
| 4 | Structure | 8, 10 |
| 5 | Formats | 11 |

Work down the table. Layer 1 alone accounts for most cases of a site that is technically excellent and entirely invisible — and it is usually a config change, not an engineering project.

## Keep it from coming back

Put four assertions in CI or in a weekly monitor: crawler UAs get `200` on your top pages, a nonexistent URL returns `404`, your first paragraph appears in the raw HTML, and every URL in `llms.txt` resolves. Add server-side logging of requests by user agent and status so a regression shows up as a spike rather than as a slow disappearance — see [measuring AI agent traffic](/articles/measuring-ai-agent-traffic).

## Frequently asked questions

### How do I test what an AI crawler sees?

Request your own pages from outside your network with the crawler's user agent and no browser headers: `curl -sI -A "GPTBot/1.2" https://example.com/`. Compare the raw body to what your browser renders. Any difference in content — not styling — is a failure mode on this list.

### Why would Cloudflare block a crawler I allowed in robots.txt?

Because managed bot rules and browser-integrity checks are evaluated at the edge, before robots.txt is even fetched. They are separate systems with separate defaults; an `Allow` line has no authority over a WAF decision. This is the single most common cause of unexplained invisibility.

### Do AI crawlers execute JavaScript?

Behaviour varies by vendor and is not uniformly documented. Assume some do not, and that those which do treat rendering as a cost that lowers your priority. If primary content is absent from the initial HTML response, treat it as unavailable.

### Is a 403 to an unknown bot ever the right answer?

Yes — for abusive or spoofed clients. The problem is collateral damage: broad rules that catch the crawlers you wanted. Verify vendor identity against published IP ranges and allowlist deliberately rather than blocking by default and hoping.


---

## The rest of this guide

- [The Agent-Ready Website: A Complete Guide to AI Visibility, Access Control and Monetization](https://changegamer.ai/articles/agent-ready-website.md): The full operator playbook for making a website work for AI agents and AI crawlers: be fetchable, be readable, be controllable, be payable — with a 30-day implementation plan.
- [llms.txt vs robots.txt vs sitemap.xml: Which File Does What](https://changegamer.ai/articles/llms-txt-vs-robots-txt-vs-sitemap.md): The three root-level files every agent-ready site publishes, what each one is actually for, and why publishing one does not substitute for the others.
- [How to Write an llms.txt File (Format, Template, and Maintenance)](https://changegamer.ai/articles/how-to-write-an-llms-txt-file.md): A step-by-step guide to writing a useful llms.txt: the exact format, a copy-paste template, what to put under ## Optional, how to validate it, and how to keep it from rotting.
- [Serving Markdown Variants to AI Agents: The Cheapest Win in AI Visibility](https://changegamer.ai/articles/serving-markdown-variants-to-ai-agents.md): How to publish a .md twin of every page — URL patterns, content negotiation, discovery headers, generation pitfalls — and why it cuts what an agent pays to read you.
- [How AI Search Engines Choose Sources (And What You Can Actually Influence)](https://changegamer.ai/articles/how-ai-search-engines-choose-sources.md): What is known, what is claimed and what is speculation about how ChatGPT, Perplexity and AI Overviews pick the pages they cite — and the short list of things a site owner can actually control.
- [Should You Block AI Crawlers? A Decision Framework by Business Model](https://changegamer.ai/articles/should-you-block-ai-crawlers.md): Blocking AI crawlers is four separate decisions, not one. A framework that maps each crawler class to what it costs and earns you, by business model, with the exact robots.txt for each answer.
- [What to Charge AI Crawlers: Pricing Models for Machine Buyers](https://changegamer.ai/articles/what-to-charge-ai-crawlers.md): Per-crawl, per-resource, corpus licence or subscription key — the four ways to price AI access, the arithmetic behind each, and why pricing before you have demand data is the standard mistake.
- [Implementing an HTTP 402 Paywall an Agent Can Actually Pay](https://changegamer.ai/articles/http-402-paywall-implementation.md): A working implementation guide for machine-payable content: the 402 response body, Link headers, key issuance and validation, caching rules, and the mistakes that make a 402 gate unpayable.
- [Structured Data for AI Agents: Which Schema.org Types Earn Their Keep](https://changegamer.ai/articles/structured-data-for-ai-agents.md): Most schema.org markup is invisible to machine readers. The types that are worth the effort for AI agents, how to emit them without drift, and what to build instead of more markup.
- [JSON API Design for AI Agents: Endpoints They Prefer Over Scraping](https://changegamer.ai/articles/json-api-design-for-agents.md): How to publish read-only JSON endpoints that agents choose over scraping your HTML: discovery index, stable shapes, freshness signals, bulk exports, and errors a machine can act on.
- [Measuring AI Agent Traffic: Server-Side Telemetry That Answers Real Questions](https://changegamer.ai/articles/measuring-ai-agent-traffic.md): Why client-side analytics miss AI agents entirely, the minimum row schema to log, the five queries worth running, and how to tell a real crawler from a spoofed user agent.
- [Licensing Content for AI Training: RSL, Terms, and Provenance](https://changegamer.ai/articles/licensing-content-for-ai-training.md): How to publish machine-readable licence terms for AI use — what RSL is, what it does and does not do, how it differs from robots.txt and Content Signals, and where provenance standards fit.
- [Running an MCP Server as a Distribution Channel for Your Content](https://changegamer.ai/articles/mcp-server-as-distribution-channel.md): Why a content site should expose an MCP server, which tools to ship, how discovery and authentication work, how to gate paid tools, and the honest limits of the channel.

## Reference resources

- https://changegamer.ai/resources/ai-crawler-policy.md
- https://changegamer.ai/resources/web-data-for-agents.md
- https://changegamer.ai/resources/agentic-browsers.md
- https://changegamer.ai/resources/handling-rate-limits-and-retries.md
- https://changegamer.ai/resources/document-extraction-for-agents.md
- https://changegamer.ai/resources/data-formats.md

All guides: https://changegamer.ai/api/articles.json · Reference corpus: https://changegamer.ai/llms.txt