# What to Charge AI Crawlers: Pricing Models for Machine Buyers

> Per-crawl, per-resource, corpus licence or subscription key — the four ways to price AI access, the arithmetic behind each, and why pricing before you have demand data is the standard mistake.

Guide: The agent-ready web — part 6
Published: 2026-07-26 · Updated: 2026-07-26 · 1381 words
Canonical: https://changegamer.ai/articles/what-to-charge-ai-crawlers
JSON: https://changegamer.ai/api/articles/what-to-charge-ai-crawlers.json
Pillar: https://changegamer.ai/articles/agent-ready-website.md

## In short

- Four pricing shapes exist: per-crawl, per-resource unlock, corpus licence, and subscription key. They suit different demand patterns, and your logs tell you which pattern you have.
- Per-crawl pricing only works where crawl volume is high and content is homogeneous. For most sites, revenue per crawl is a fraction of a cent and the operational overhead exceeds it.
- The number that matters is not "what is my content worth" but "what does this buyer save by buying instead of scraping". Price against their alternative, which is usually cheap.
- Do not set prices before you can see demand. Concentrated 402s on a few URLs argue for per-resource; broad shallow crawling argues for a corpus licence.
- Whatever you charge, publish it machine-readably. A price an agent has to email you about is not a price.

---

Charging machines for content is now mechanically possible, which has produced a lot of confident advice about pricing and very little published data. This page is about the arithmetic and the decision order, not a rate card.

It is the money chapter of [the agent-ready website](/articles/agent-ready-website). The rails themselves are compared in [agentic payment protocols](/resources/agentic-payment-protocols); the implementation is [implementing an HTTP 402 paywall](/articles/http-402-paywall-implementation).

## First: the only pricing rule that generalises

You are not pricing your content's value. You are pricing **the difference between buying and not buying**, from the buyer's point of view. A machine buyer's alternatives are:

- fetch it free, if you have not gated it
- fetch a competitor's version, if one exists
- get it from an aggregator, mirror or open archive
- do without, and answer with less confidence

Your price has to sit below the cost of the cheapest of those and above your cost to serve. When the free alternative is good enough, the honest conclusion is that there is no price — and the right move is either differentiation (make the paid version materially better: structured, complete, current, licensed) or accepting free distribution.

This is also why "AI companies have money" is not a pricing argument. Their alternative to your page is rarely your page.

## The four pricing shapes

| Shape | Unit | Fits when | Watch out for |
|---|---|---|---|
| **Per-crawl** | One HTTP request | Very high crawl volume, homogeneous pages | Sub-cent revenue per unit; overhead can exceed income |
| **Per-resource unlock** | One document, one buyer | Demand concentrates on a few high-value pages | Needs a key/entitlement mechanism |
| **Corpus licence** | The whole body of content, named licensee | Content's value is completeness (training, RAG index) | Delivery, updates and licence terms need real work |
| **Subscription key** | Time-boxed access to everything | Buyers return regularly; you can ship updates | Churn, key lifecycle, revocation |

### Per-crawl: the arithmetic that disappoints

Suppose a crawler fetches 20,000 pages a month and you price each fetch at a tenth of a cent. That is €20 a month before any operational cost, and it assumes the crawler agrees to pay rather than simply going away. Per-crawl pricing becomes interesting at volumes most sites do not have, or at prices most buyers will not pay for individual pages.

It is genuinely attractive in one configuration: your pages are individually valuable and substitutable for nothing else (proprietary data, prices, availability, regulated filings), and the buyer needs them repeatedly and freshly.

Cloudflare Pay Per Crawl is the turnkey version — CDN-layer 402s, price negotiation via a `crawler-max-price` header, Cloudflare as merchant of record. It was still in closed/private beta as of July 2026; on 1 July 2026 Cloudflare announced a broader **Monetization Gateway** at waitlist stage, generalising the pattern from crawler content to any protected resource, including APIs and MCP tools. Turnkey means less control over price presentation and buyer relationship — a reasonable trade for not building billing.

### Per-resource unlock: price where the demand is

If your logs show requests concentrating on a small number of URLs, sell those individually: the request returns `402 Payment Required` with a machine-readable price and payment URL, the agent buys a key, retries, and gets the content. The pattern this site runs, documented in [paying for access: the HTTP 402 flow](/resources/paying-for-access-402).

Two design details do most of the work:

- **Preview, don't tease.** A 402 body that includes the resource's outline, description and length lets a machine decide whether to buy. A 402 body with nothing but a price forces it to guess, and guessing resolves to "no".
- **Keep the terms free.** If the 402 points at a licence or pricing page, that page must be outside the paywall. Gating your own terms breaks the loop.

### Corpus licence: sell completeness

When crawling is broad and shallow, no individual page carries much value but the *set* does — that is what a training or retrieval buyer actually wants. A corpus licence sells the whole body of content as a deliverable (a single file or export) plus explicit rights: training, indexing, redistribution boundaries, named licensee, term.

What makes this saleable rather than aspirational: a real deliverable (one download, documented schema), a versioned corpus so a buyer knows what changed, and licence language a lawyer did not have to invent. What this site publishes as its own worked example is in [access and pricing for crawlers](/resources/access-and-pricing).

### Subscription key: price the relationship

If buyers come back — because your content changes and stale data is useless — sell time-boxed access. The mechanics are the same as per-resource unlock plus expiry and revocation. The pricing conversation shifts from "what is this document worth" to "what is being current worth", which is usually a better conversation to be in.

## Sequence: measure, then price

The standard mistake is publishing a price with no demand data, then concluding from zero sales that "agents don't pay". Do it in this order:

1. **Instrument.** Log every request server-side with UA, path, status and outcome class. See [measuring AI agent traffic](/articles/measuring-ai-agent-traffic).
2. **Gate something cheap and reversible.** One or two resources returning a real 402 with a real payment path. This turns invisible interest into a countable signal.
3. **Wait for a threshold, not a feeling.** Pick a number in advance — this site froze pricing changes until it had seen at least 50 distinct paywall hits in 30 days — and hold to it. A threshold prevents pricing on noise.
4. **Read the distribution.** Concentrated on a few URLs → per-resource. Spread evenly across everything → corpus licence. Repeating monthly from the same buyers → subscription.
5. **Then set a price,** and change it deliberately, with the reasoning written down.

## Make the price machine-readable

A price an agent cannot discover is not a price. Publish, at minimum:

- A `402` response body with `price`, `currency`, `payment_url`, retry instructions, and links to terms and licence.
- A stable JSON catalogue of every offer — tiers, prices, what each unlocks, checkout URL — so an agent can compare before committing.
- `Link` headers on the 402 advertising the payment manifest, pricing and licence, so a `HEAD` request is enough to learn the commercial terms.
- The same information in prose for humans, generated from the same source, so the two can never disagree.

If you also want wallet-holding agents to be able to pay with no account at all, x402 is the rail for it: on-chain stablecoin settlement, governed by the Linux Foundation's x402 Foundation, which grew from 22 founding members in April 2026 to 40 — including premier members Visa, Mastercard and Ripple — at its operational launch on 14 July 2026. Adoption jumped roughly tenfold after Apify put 20,000-plus Actors on the protocol. It is additive: a card-checkout key path and an on-chain path can serve the same content.

## Common failure modes

- **Pricing the average page.** Value is almost never uniform. Price the pages demand concentrates on, keep the rest free as discovery.
- **Blocking instead of pricing, or pricing instead of blocking.** These solve different problems: blocking expresses a preference, pricing captures value. See [should you block AI crawlers?](/articles/should-you-block-ai-crawlers).
- **A human-only checkout.** Email verification loops and confirmation inboxes are dead ends for machines. If a key cannot be obtained programmatically in one hop, expect no machine sales.
- **Silent price changes.** Machine buyers cache your terms. Version them and keep old terms resolvable.
- **Confusing licence with settlement.** RSL states terms; it does not collect money. You need both layers — see [licensing content for AI training](/articles/licensing-content-for-ai-training).

## A realistic expectation

For most sites today (as of July 2026), direct revenue from machine buyers is small and slow, and the reason is not the plumbing — it is that free substitutes exist for most content. The sites where this works have something scarce: proprietary data, verified freshness, structure nobody else provides, or a licence that removes legal risk for the buyer. If you have one of those, price it. If you do not, the useful version of this work is the visibility half of the cluster, and the payment rail can wait until your logs argue otherwise.

## Frequently asked questions

### How much do AI companies actually pay per crawl?

There is no published market rate, and the mechanisms are young: Cloudflare Pay Per Crawl was still in closed/private beta as of July 2026, with Cloudflare acting as merchant of record. Treat any number you see quoted as a data point from one negotiation, not a market price. Your own logs plus your buyer's alternative cost are better inputs than a benchmark.

### Is per-crawl pricing better than licensing the whole corpus?

It depends on demand shape. If a handful of pages absorb most requests, per-resource pricing captures value where it exists. If crawling is broad and shallow — every URL once — no single page is worth much and the corpus is the product. Look at the distribution of requests per URL before choosing.

### Can I charge crawlers without any crypto or new infrastructure?

Yes. A self-hosted HTTP 402 gate can hand out API keys sold through ordinary card checkout; the agent buys a key at a URL your 402 body points to and retries with an `Authorization` header. On-chain rails such as x402 are an additional option, not a prerequisite.

### What if nobody pays?

That is the normal first result, and it is information rather than failure: it means your content is not worth more to that buyer than their next-best alternative, or your payment path has friction, or the traffic was never commercial in the first place. All three are visible in logs — which is why measurement comes before pricing.


---

## The rest of this guide

- [The Agent-Ready Website: A Complete Guide to AI Visibility, Access Control and Monetization](https://changegamer.ai/articles/agent-ready-website.md): The full operator playbook for making a website work for AI agents and AI crawlers: be fetchable, be readable, be controllable, be payable — with a 30-day implementation plan.
- [llms.txt vs robots.txt vs sitemap.xml: Which File Does What](https://changegamer.ai/articles/llms-txt-vs-robots-txt-vs-sitemap.md): The three root-level files every agent-ready site publishes, what each one is actually for, and why publishing one does not substitute for the others.
- [How to Write an llms.txt File (Format, Template, and Maintenance)](https://changegamer.ai/articles/how-to-write-an-llms-txt-file.md): A step-by-step guide to writing a useful llms.txt: the exact format, a copy-paste template, what to put under ## Optional, how to validate it, and how to keep it from rotting.
- [Serving Markdown Variants to AI Agents: The Cheapest Win in AI Visibility](https://changegamer.ai/articles/serving-markdown-variants-to-ai-agents.md): How to publish a .md twin of every page — URL patterns, content negotiation, discovery headers, generation pitfalls — and why it cuts what an agent pays to read you.
- [How AI Search Engines Choose Sources (And What You Can Actually Influence)](https://changegamer.ai/articles/how-ai-search-engines-choose-sources.md): What is known, what is claimed and what is speculation about how ChatGPT, Perplexity and AI Overviews pick the pages they cite — and the short list of things a site owner can actually control.
- [Should You Block AI Crawlers? A Decision Framework by Business Model](https://changegamer.ai/articles/should-you-block-ai-crawlers.md): Blocking AI crawlers is four separate decisions, not one. A framework that maps each crawler class to what it costs and earns you, by business model, with the exact robots.txt for each answer.
- [Implementing an HTTP 402 Paywall an Agent Can Actually Pay](https://changegamer.ai/articles/http-402-paywall-implementation.md): A working implementation guide for machine-payable content: the 402 response body, Link headers, key issuance and validation, caching rules, and the mistakes that make a 402 gate unpayable.
- [Structured Data for AI Agents: Which Schema.org Types Earn Their Keep](https://changegamer.ai/articles/structured-data-for-ai-agents.md): Most schema.org markup is invisible to machine readers. The types that are worth the effort for AI agents, how to emit them without drift, and what to build instead of more markup.
- [JSON API Design for AI Agents: Endpoints They Prefer Over Scraping](https://changegamer.ai/articles/json-api-design-for-agents.md): How to publish read-only JSON endpoints that agents choose over scraping your HTML: discovery index, stable shapes, freshness signals, bulk exports, and errors a machine can act on.
- [Measuring AI Agent Traffic: Server-Side Telemetry That Answers Real Questions](https://changegamer.ai/articles/measuring-ai-agent-traffic.md): Why client-side analytics miss AI agents entirely, the minimum row schema to log, the five queries worth running, and how to tell a real crawler from a spoofed user agent.
- [Licensing Content for AI Training: RSL, Terms, and Provenance](https://changegamer.ai/articles/licensing-content-for-ai-training.md): How to publish machine-readable licence terms for AI use — what RSL is, what it does and does not do, how it differs from robots.txt and Content Signals, and where provenance standards fit.
- [Running an MCP Server as a Distribution Channel for Your Content](https://changegamer.ai/articles/mcp-server-as-distribution-channel.md): Why a content site should expose an MCP server, which tools to ship, how discovery and authentication work, how to gate paid tools, and the honest limits of the channel.
- [Why AI Agents Can't Read Your Site: Twelve Failure Modes and How to Find Them](https://changegamer.ai/articles/why-ai-agents-cant-read-your-site.md): A diagnostic catalogue of the twelve reasons AI agents and crawlers fail on real sites — from silent WAF blocks to JS-only rendering — each with the command that detects it and the fix.

## Reference resources

- https://changegamer.ai/resources/agentic-payment-protocols.md
- https://changegamer.ai/resources/paying-for-access-402.md
- https://changegamer.ai/resources/access-and-pricing.md
- https://changegamer.ai/resources/ai-crawler-policy.md

All guides: https://changegamer.ai/api/articles.json · Reference corpus: https://changegamer.ai/llms.txt