Why AI Agents Can't Read Your Site: Twelve Failure Modes and How to Find Them
A diagnostic catalogue of the twelve reasons AI agents and crawlers fail on real sites — from silent WAF blocks to JS-only rendering — each with the command that detects it and the fix.
- The most common cause is not content: it is an edge rule returning 403 to crawlers your robots.txt explicitly allows.
- Diagnose from outside with a spoofed user agent. A page that looks fine in your browser proves nothing about what a crawler receives.
- JavaScript-only content is invisible to crawlers that do not render and expensive for those that do — check the raw HTML response, not the DOM.
- Soft 404s and 200-with-error-page responses are worse than honest errors, because an agent will quote the error page as your content.
- Fix in this order: access, then rendering, then structure, then formats. Each layer is worthless while the one above it is broken.
When a site is absent from AI answers, the cause is usually mechanical and locatable in under an hour. This is the diagnostic half of the agent-ready website: twelve failure modes, in the order they actually occur, each with a detection command and a fix.
Run the whole matrix first, then fix top-down. There is no point improving structure while your edge is returning 403.
The five-minute baseline
SITE=https://example.com
for ua in "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" \
"Mozilla/5.0 (compatible; ClaudeBot/1.0; +https://anthropic.com/aup)" \
"PerplexityBot/1.0" "curl/8.0"; do
for p in / /robots.txt /sitemap.xml /llms.txt; do
printf "%-22s %-14s " "${ua:0:22}" "$p"
curl -sI -A "$ua" "$SITE$p" | head -1
done
done
Everything below is what to do with the results.
1. WAF or bot management is 403ing crawlers you allowed
Symptom: 403 for crawler user agents, 200 in your browser. robots.txt says Allow.
Why: edge rules run before robots.txt is fetched and have no knowledge of it. Managed "AI bot" rulesets are often enabled by default or turned on during an unrelated security review. On this site, Cloudflare's managed rule silently 403'd GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, CCBot and Google-CloudVertexBot — including on / and /sitemap.xml — while robots.txt allowed all of them.
Detect: the baseline matrix above.
Fix: set AI-crawler controls to allow explicitly, then add a rule that skips managed rules for the user agents you want, with logging on. Re-verify afterwards — this is the failure mode most likely to silently return.
2. Browser-integrity checks reject non-browser clients
Symptom: 403 for curl and every non-browser UA, including your own monitoring and sometimes search-console fetches.
Why: integrity checks look for a full set of browser-like headers. A legitimate crawler sends few of them. This site's Browser Integrity Check broke Google Search Console's sitemap fetch this way.
Detect: curl -sI -A "curl/8.0" $SITE/ returning 403 while a browser succeeds.
Fix: disable on content paths, or scope to authenticated and form-submission routes only.
3. Primary content requires JavaScript
Symptom: raw HTML contains a shell and a script bundle; your text appears only after hydration.
Detect:
curl -s $SITE/your-page | grep -c "a distinctive sentence from your article" # expect >= 1
curl -s $SITE/your-page | wc -c # tiny = shell only
Fix: server-render or statically generate primary content. If that is a large project, ship a Markdown variant of the content immediately — it is a smaller change and it solves the machine-readability problem completely while the rendering work is scheduled.
4. Soft 404s and 200-with-error-page
Symptom: missing pages return 200 with "not found" in the body.
Why it is worse for agents than for search: a crawler deduplicates near-identical pages; an agent quotes what it received. A confident model will happily tell a user that your product page says "Sorry, this page could not be found".
Detect: curl -so /dev/null -w "%{http_code}\n" $SITE/definitely-not-a-real-url — expect 404.
Fix: honest status codes everywhere: 404/410 for gone, 301 for moved, 503 with Retry-After for maintenance.
5. Redirect chains and canonical drift
Symptom: two or more hops to reach content; http→https→www→trailing-slash.
Why: every hop costs a round trip against an agent's budget, and chains frequently end in a URL that disagrees with your canonical tag and your sitemap.
Detect:
curl -sIL -o /dev/null -w "%{num_redirects} hops → %{url_effective}\n" $SITE/your-page
Fix: one hop maximum. Pick a canonical form, serve it directly, and make sitemap, canonical tag and internal links agree with it.
6. Rate limits that look like hostility
Symptom: 429 or 403 after a handful of requests, no Retry-After.
Why: an agent that receives a bare block has no protocol for coming back. One with Retry-After does, and well-built ones honour it — the client-side contract is in handling LLM rate limits and retries.
Detect: fire 20 sequential requests and watch where the status changes.
Fix: graduated limits, a real Retry-After header, a JSON body naming the limit, and — the useful part — a pointer to a bulk endpoint so a client fetching 500 pages can fetch one file instead.
7. Cookie walls, consent gates and interstitials
Symptom: the first meaningful text in your HTML is a consent notice.
Why: an agent has no consent to give and cannot dismiss a modal. Worse, the notice becomes the extracted "content" of the page.
Detect: read the first 2 KB of the raw response and ask what a summariser would conclude the page is about.
Fix: do not block content on consent for read-only requests. Where you must show a notice, keep it out of the top of the document order and out of the Markdown variant.
8. Content locked in PDFs, images and iframes
Symptom: the substance is a scanned PDF, an infographic, or an embedded third-party widget.
Why: extraction from these is lossy at best. Document parsing has improved a great deal but remains a cost and an error source — the state of it is covered in document extraction and parsing for agents.
Fix: publish an HTML or Markdown version of anything you want quoted. Keep the PDF as the printable artefact, not the canonical one.
9. Infinite scroll and click-to-reveal
Symptom: listings and comments only load on scroll; tabs and accordions hide content behind clicks.
Why: a fetch gets the first page and nothing else; even browser-driving agents pay a high cost to keep interacting — see agentic browsers and web data and scraping for agents.
Fix: real paginated URLs with rel="next", or a JSON endpoint that returns the whole list. Render tab and accordion content in the HTML and hide it with CSS rather than withholding it.
10. Structure a machine has to guess
Symptom: headings are styled <div>s; tables are grids of divs; code is a <pre> inside three wrappers with syntax spans.
Why: every inference is a chance to be wrong, and being wrong about your content is worse for you than not being read.
Detect: curl -s $SITE/page | grep -c "<h2" and grep -c "<table" against what the page visibly contains.
Fix: real semantic elements. Tables in particular map directly onto the comparison questions people ask answer engines, so converting a div-grid into a real <table> has outsized value — see data formats and schema.
11. No machine-readable variant of anything
Symptom: every consumer, human or machine, gets the same heavy HTML.
Why: it works, badly. You pay bandwidth, they pay tokens, and structure has to be re-derived on every fetch.
Fix: in ascending order of effort — a Markdown twin per page, a JSON index, a bulk export, an llms.txt pointing at all of them. The cheapest three are covered in serving Markdown variants, JSON API design for agents and how to write an llms.txt file.
12. Your policy contradicts itself
Symptom: robots.txt allows a crawler that your WAF blocks; your sitemap lists noindex pages; your canonical tag points somewhere your redirects do not; Content Signals say one thing and your licence another.
Why: these files are written by different people at different times, and machines resolve contradictions by distrusting all of it.
Detect: read robots.txt, your sitemap, your canonical tags and your licence in one sitting and check they describe the same site.
Fix: generate what you can from one source, and re-run the baseline matrix after every policy change. Which tokens should say what is worked through in should you block AI crawlers? and the reference table is AI crawler policy.
Fix order
| Priority | Layer | Failure modes |
|---|---|---|
| 1 | Access | 1, 2, 6 |
| 2 | Response honesty | 4, 5, 12 |
| 3 | Rendering | 3, 7, 9 |
| 4 | Structure | 8, 10 |
| 5 | Formats | 11 |
Work down the table. Layer 1 alone accounts for most cases of a site that is technically excellent and entirely invisible — and it is usually a config change, not an engineering project.
Keep it from coming back
Put four assertions in CI or in a weekly monitor: crawler UAs get 200 on your top pages, a nonexistent URL returns 404, your first paragraph appears in the raw HTML, and every URL in llms.txt resolves. Add server-side logging of requests by user agent and status so a regression shows up as a spike rather than as a slow disappearance — see measuring AI agent traffic.
Frequently asked questions
- How do I test what an AI crawler sees?
- Request your own pages from outside your network with the crawler's user agent and no browser headers: `curl -sI -A "GPTBot/1.2" https://example.com/`. Compare the raw body to what your browser renders. Any difference in content — not styling — is a failure mode on this list.
- Why would Cloudflare block a crawler I allowed in robots.txt?
- Because managed bot rules and browser-integrity checks are evaluated at the edge, before robots.txt is even fetched. They are separate systems with separate defaults; an `Allow` line has no authority over a WAF decision. This is the single most common cause of unexplained invisibility.
- Do AI crawlers execute JavaScript?
- Behaviour varies by vendor and is not uniformly documented. Assume some do not, and that those which do treat rendering as a cost that lowers your priority. If primary content is absent from the initial HTML response, treat it as unavailable.
- Is a 403 to an unknown bot ever the right answer?
- Yes — for abusive or spoofed clients. The problem is collateral damage: broad rules that catch the crawlers you wanted. Verify vendor identity against published IP ranges and allowlist deliberately rather than blocking by default and hoping.