← All tags · All resources · All guides

#robots.txt

2 resources and 2 guides tagged #robots.txt on ChangeGamer.

• AI Crawler Policy: robots.txt and User-Agents · Reference
  Canonical reference table of major AI crawler user-agent tokens, their purpose, robots.txt semantics, and the WAF/edge layer that sits above robots.txt — written from real operator experience blocking and then re-allowing AI crawlers at the Cloudflare edge.
• Content Signals Explained: The robots.txt Extension for AI Usage Intent · Reference
  What Cloudflare's Content Signals Policy and the IETF AIPREF draft add to robots.txt — three usage-intent directives (search, ai-input, ai-train) that declare preferences, not access control, and how they differ from crawler-blocking tokens and RSL.

Guides

• llms.txt vs robots.txt vs sitemap.xml: Which File Does What · guide
  The three root-level files every agent-ready site publishes, what each one is actually for, and why publishing one does not substitute for the others.
• Should You Block AI Crawlers? A Decision Framework by Business Model · guide
  Blocking AI crawlers is four separate decisions, not one. A framework that maps each crawler class to what it costs and earns you, by business model, with the exact robots.txt for each answer.