What to Charge AI Crawlers: Pricing Models for Machine Buyers
Per-crawl, per-resource, corpus licence or subscription key — the four ways to price AI access, the arithmetic behind each, and why pricing before you have demand data is the standard mistake.
- Four pricing shapes exist: per-crawl, per-resource unlock, corpus licence, and subscription key. They suit different demand patterns, and your logs tell you which pattern you have.
- Per-crawl pricing only works where crawl volume is high and content is homogeneous. For most sites, revenue per crawl is a fraction of a cent and the operational overhead exceeds it.
- The number that matters is not "what is my content worth" but "what does this buyer save by buying instead of scraping". Price against their alternative, which is usually cheap.
- Do not set prices before you can see demand. Concentrated 402s on a few URLs argue for per-resource; broad shallow crawling argues for a corpus licence.
- Whatever you charge, publish it machine-readably. A price an agent has to email you about is not a price.
Charging machines for content is now mechanically possible, which has produced a lot of confident advice about pricing and very little published data. This page is about the arithmetic and the decision order, not a rate card.
It is the money chapter of the agent-ready website. The rails themselves are compared in agentic payment protocols; the implementation is implementing an HTTP 402 paywall.
First: the only pricing rule that generalises
You are not pricing your content's value. You are pricing the difference between buying and not buying, from the buyer's point of view. A machine buyer's alternatives are:
- fetch it free, if you have not gated it
- fetch a competitor's version, if one exists
- get it from an aggregator, mirror or open archive
- do without, and answer with less confidence
Your price has to sit below the cost of the cheapest of those and above your cost to serve. When the free alternative is good enough, the honest conclusion is that there is no price — and the right move is either differentiation (make the paid version materially better: structured, complete, current, licensed) or accepting free distribution.
This is also why "AI companies have money" is not a pricing argument. Their alternative to your page is rarely your page.
The four pricing shapes
| Shape | Unit | Fits when | Watch out for |
|---|---|---|---|
| Per-crawl | One HTTP request | Very high crawl volume, homogeneous pages | Sub-cent revenue per unit; overhead can exceed income |
| Per-resource unlock | One document, one buyer | Demand concentrates on a few high-value pages | Needs a key/entitlement mechanism |
| Corpus licence | The whole body of content, named licensee | Content's value is completeness (training, RAG index) | Delivery, updates and licence terms need real work |
| Subscription key | Time-boxed access to everything | Buyers return regularly; you can ship updates | Churn, key lifecycle, revocation |
Per-crawl: the arithmetic that disappoints
Suppose a crawler fetches 20,000 pages a month and you price each fetch at a tenth of a cent. That is €20 a month before any operational cost, and it assumes the crawler agrees to pay rather than simply going away. Per-crawl pricing becomes interesting at volumes most sites do not have, or at prices most buyers will not pay for individual pages.
It is genuinely attractive in one configuration: your pages are individually valuable and substitutable for nothing else (proprietary data, prices, availability, regulated filings), and the buyer needs them repeatedly and freshly.
Cloudflare Pay Per Crawl is the turnkey version — CDN-layer 402s, price negotiation via a crawler-max-price header, Cloudflare as merchant of record. It was still in closed/private beta as of July 2026; on 1 July 2026 Cloudflare announced a broader Monetization Gateway at waitlist stage, generalising the pattern from crawler content to any protected resource, including APIs and MCP tools. Turnkey means less control over price presentation and buyer relationship — a reasonable trade for not building billing.
Per-resource unlock: price where the demand is
If your logs show requests concentrating on a small number of URLs, sell those individually: the request returns 402 Payment Required with a machine-readable price and payment URL, the agent buys a key, retries, and gets the content. The pattern this site runs, documented in paying for access: the HTTP 402 flow.
Two design details do most of the work:
- Preview, don't tease. A 402 body that includes the resource's outline, description and length lets a machine decide whether to buy. A 402 body with nothing but a price forces it to guess, and guessing resolves to "no".
- Keep the terms free. If the 402 points at a licence or pricing page, that page must be outside the paywall. Gating your own terms breaks the loop.
Corpus licence: sell completeness
When crawling is broad and shallow, no individual page carries much value but the set does — that is what a training or retrieval buyer actually wants. A corpus licence sells the whole body of content as a deliverable (a single file or export) plus explicit rights: training, indexing, redistribution boundaries, named licensee, term.
What makes this saleable rather than aspirational: a real deliverable (one download, documented schema), a versioned corpus so a buyer knows what changed, and licence language a lawyer did not have to invent. What this site publishes as its own worked example is in access and pricing for crawlers.
Subscription key: price the relationship
If buyers come back — because your content changes and stale data is useless — sell time-boxed access. The mechanics are the same as per-resource unlock plus expiry and revocation. The pricing conversation shifts from "what is this document worth" to "what is being current worth", which is usually a better conversation to be in.
Sequence: measure, then price
The standard mistake is publishing a price with no demand data, then concluding from zero sales that "agents don't pay". Do it in this order:
- Instrument. Log every request server-side with UA, path, status and outcome class. See measuring AI agent traffic.
- Gate something cheap and reversible. One or two resources returning a real 402 with a real payment path. This turns invisible interest into a countable signal.
- Wait for a threshold, not a feeling. Pick a number in advance — this site froze pricing changes until it had seen at least 50 distinct paywall hits in 30 days — and hold to it. A threshold prevents pricing on noise.
- Read the distribution. Concentrated on a few URLs → per-resource. Spread evenly across everything → corpus licence. Repeating monthly from the same buyers → subscription.
- Then set a price, and change it deliberately, with the reasoning written down.
Make the price machine-readable
A price an agent cannot discover is not a price. Publish, at minimum:
- A
402response body withprice,currency,payment_url, retry instructions, and links to terms and licence. - A stable JSON catalogue of every offer — tiers, prices, what each unlocks, checkout URL — so an agent can compare before committing.
Linkheaders on the 402 advertising the payment manifest, pricing and licence, so aHEADrequest is enough to learn the commercial terms.- The same information in prose for humans, generated from the same source, so the two can never disagree.
If you also want wallet-holding agents to be able to pay with no account at all, x402 is the rail for it: on-chain stablecoin settlement, governed by the Linux Foundation's x402 Foundation, which grew from 22 founding members in April 2026 to 40 — including premier members Visa, Mastercard and Ripple — at its operational launch on 14 July 2026. Adoption jumped roughly tenfold after Apify put 20,000-plus Actors on the protocol. It is additive: a card-checkout key path and an on-chain path can serve the same content.
Common failure modes
- Pricing the average page. Value is almost never uniform. Price the pages demand concentrates on, keep the rest free as discovery.
- Blocking instead of pricing, or pricing instead of blocking. These solve different problems: blocking expresses a preference, pricing captures value. See should you block AI crawlers?.
- A human-only checkout. Email verification loops and confirmation inboxes are dead ends for machines. If a key cannot be obtained programmatically in one hop, expect no machine sales.
- Silent price changes. Machine buyers cache your terms. Version them and keep old terms resolvable.
- Confusing licence with settlement. RSL states terms; it does not collect money. You need both layers — see licensing content for AI training.
A realistic expectation
For most sites today (as of July 2026), direct revenue from machine buyers is small and slow, and the reason is not the plumbing — it is that free substitutes exist for most content. The sites where this works have something scarce: proprietary data, verified freshness, structure nobody else provides, or a licence that removes legal risk for the buyer. If you have one of those, price it. If you do not, the useful version of this work is the visibility half of the cluster, and the payment rail can wait until your logs argue otherwise.
Frequently asked questions
- How much do AI companies actually pay per crawl?
- There is no published market rate, and the mechanisms are young: Cloudflare Pay Per Crawl was still in closed/private beta as of July 2026, with Cloudflare acting as merchant of record. Treat any number you see quoted as a data point from one negotiation, not a market price. Your own logs plus your buyer's alternative cost are better inputs than a benchmark.
- Is per-crawl pricing better than licensing the whole corpus?
- It depends on demand shape. If a handful of pages absorb most requests, per-resource pricing captures value where it exists. If crawling is broad and shallow — every URL once — no single page is worth much and the corpus is the product. Look at the distribution of requests per URL before choosing.
- Can I charge crawlers without any crypto or new infrastructure?
- Yes. A self-hosted HTTP 402 gate can hand out API keys sold through ordinary card checkout; the agent buys a key at a URL your 402 body points to and retries with an `Authorization` header. On-chain rails such as x402 are an additional option, not a prerequisite.
- What if nobody pays?
- That is the normal first result, and it is information rather than failure: it means your content is not worth more to that buyer than their next-best alternative, or your payment path has friction, or the traffic was never commercial in the first place. All three are visible in logs — which is why measurement comes before pricing.