# Measuring AI Agent Traffic: Server-Side Telemetry That Answers Real Questions

> Why client-side analytics miss AI agents entirely, the minimum row schema to log, the five queries worth running, and how to tell a real crawler from a spoofed user agent.

Guide: The agent-ready web — part 10
Published: 2026-07-26 · Updated: 2026-07-26 · 1221 words
Canonical: https://changegamer.ai/articles/measuring-ai-agent-traffic
JSON: https://changegamer.ai/api/articles/measuring-ai-agent-traffic.json
Pillar: https://changegamer.ai/articles/agent-ready-website.md

## In short

- Agents mostly do not execute JavaScript, so JS-based analytics record almost none of this traffic. Everything useful is server-side.
- Six fields are enough: timestamp, path, user agent, status, outcome class, and a bot-verified flag. Resist logging more, especially anything credential-shaped.
- The decisive metric is the fetch-to-referral ratio per crawler: high fetches with no referrals means you are supplying answers, not receiving visitors.
- User agents are trivially spoofed. Verify against published vendor IP ranges before you treat a row as a real vendor crawler.
- Set a decision threshold in advance — "N distinct paywall hits in 30 days" — so that pricing and blocking decisions are made on evidence rather than on the first interesting week.

---

You cannot make any of the decisions in [the agent-ready website](/articles/agent-ready-website) without data, and the data you already have is probably wrong: it comes from a JavaScript tag that agents do not run.

## The row schema

Log one row per request that reaches your origin or edge. Six fields carry every insight worth having:

| Field | Example | Why |
|---|---|---|
| `timestamp` | `2026-07-26T09:14:03Z` | Trends, bursts, crawl cadence |
| `path` | `/blog/agent-traffic.md` | What is actually wanted |
| `user_agent` | `GPTBot/1.2` (truncate to ~100 chars) | Which client class |
| `status` | `200`, `403`, `402`, `429` | Whether you served or blocked it |
| `outcome` | `served`, `blocked`, `payment_required`, `upgrade`, `not_found` | Business meaning of the status |
| `verified` | `true`/`false` | Did the source IP match the vendor's published range |

Two rules on what **not** to log: never record API keys, session identifiers, or tokens — a log store containing credentials is a credential store — and keep user agents truncated so a hostile client cannot inflate your storage with a megabyte UA.

If you run at an edge platform, this is a handful of lines in the request path. This site writes exactly this shape into a Cloudflare Analytics Engine dataset from its Worker; the same pattern works with any log sink you can query.

```ts
function logAccess(env, req, res, outcome) {
  const ua = (req.headers.get("user-agent") ?? "").slice(0, 100);
  env.ANALYTICS.writeDataPoint({
    indexes: [new URL(req.url).pathname],
    blobs: [classOf(ua), outcome, ua, res.status.toString()],
    doubles: [1],
  });
}
```

One caveat worth designing around: requests served straight from a CDN cache or static-asset layer may never reach your logging code. Decide deliberately which surfaces are counted, and write it down — otherwise you will later read a zero as "no demand" when it means "not instrumented". Index files such as `/llms.txt` and `robots.txt` are commonly in this category.

## The five queries that drive decisions

### 1. Who is visiting, and are they real?

```sql
SELECT user_agent, COUNT(*) AS n, SUM(verified) AS verified_n
FROM access_log WHERE timestamp > now() - interval 30 day
GROUP BY user_agent ORDER BY n DESC LIMIT 25
```

Look for the tokens that matter: training crawlers, answer-engine indexers, live user fetchers. A large gap between `n` and `verified_n` for a vendor token means somebody is spoofing it — usually a scraper hoping you allowlisted that name. The token reference is [AI crawler policy](/resources/ai-crawler-policy).

### 2. Am I accidentally blocking the crawlers I want?

```sql
SELECT user_agent, status, COUNT(*) AS n
FROM access_log WHERE timestamp > now() - interval 7 day
  AND status IN (403, 429, 503)
GROUP BY user_agent, status ORDER BY n DESC
```

This is the query that pays for the whole setup. A wall of 403s against a crawler your `robots.txt` allows means bot management or a browser-integrity check is overriding your published policy — the exact failure this site hit, where a managed rule silently 403'd six named AI crawlers including on `/` and `/sitemap.xml`. Everything in that family is catalogued in [why AI agents can't read your site](/articles/why-ai-agents-cant-read-your-site).

### 3. What do they actually want?

```sql
SELECT path, COUNT(*) AS n, COUNT(DISTINCT user_agent) AS clients
FROM access_log WHERE outcome = 'served' AND timestamp > now() - interval 30 day
GROUP BY path ORDER BY n DESC LIMIT 30
```

Read the *distribution*, not the top row. Concentrated demand on a few URLs means those pages are a product; flat demand across everything means your corpus is the product. That distinction decides your entire pricing shape — see [what to charge AI crawlers](/articles/what-to-charge-ai-crawlers).

### 4. Does the traffic come back as visitors?

```sql
-- referrals from answer-engine domains, from the same log
SELECT referer_host, COUNT(*) AS visits
FROM access_log WHERE timestamp > now() - interval 30 day
  AND referer_host IS NOT NULL
GROUP BY referer_host ORDER BY visits DESC LIMIT 20
```

Then compute the ratio that matters: **fetches by answer-engine crawlers ÷ referrals from that engine's domain.** A high ratio means your content is being consumed to produce answers that do not send anyone your way. That is not necessarily bad — brand presence in answers has value — but it is the number that turns "should we monetize or restrict this?" from a philosophical question into an arithmetic one.

### 5. If you gate anything, does the wall convert?

```sql
SELECT outcome, COUNT(*) AS n
FROM access_log WHERE timestamp > now() - interval 30 day
  AND outcome IN ('payment_required', 'upgrade', 'served_paid')
GROUP BY outcome
```

A funnel with many `payment_required` rows and no paid fetches means the price is wrong, the payment path has friction, or the buyers were never commercial. `upgrade` rows — valid keys of insufficient tier — are the cleanest upsell signal that exists, because someone already paid you once.

## Verifying crawlers: three options

| Method | Effort | Strength |
|---|---|---|
| UA string match | Trivial | None — trivially spoofed |
| Reverse DNS + forward confirmation | Low | Good for vendors that support it |
| Vendor published IP ranges | Moderate (refresh periodically) | Strong, and what vendors recommend |
| Signed requests (cryptographic bot auth) | Depends on adoption | Strongest; removes IP lists entirely |

The last row is where this is heading: instead of maintaining allowlists of addresses, the client proves its identity by signing the request. The state of that work is covered in [web bot auth](/resources/web-bot-auth). Until it is universal, IP-range verification is the practical answer, and it belongs in your logging path so `verified` is a field rather than a later investigation.

## Decide in advance what would change your mind

The most common analytical failure here is not a missing metric, it is acting on a week of noise. Write the thresholds down before you look:

- *"If a crawler exceeds N requests/day against paths we do not want indexed, we add a Disallow."*
- *"If we see at least 50 distinct paywall hits in 30 days, we revisit pricing; below that, pricing is frozen."* (This site uses exactly that rule, and it has kept several tempting-but-unsupported price changes from shipping.)
- *"If fetch-to-referral for an engine exceeds X, we treat it as a licensing conversation, not a traffic channel."*

A pre-committed threshold converts an argument into a measurement.

## A one-screen dashboard

Four panels, refreshed weekly, are enough to run this:

1. **Requests per crawler per week, stacked by status.** Catches blocking and bursts.
2. **Top 20 paths by served requests.** Shows what your product actually is.
3. **Fetch-to-referral ratio per answer engine.** The strategic number.
4. **Outcome funnel** (`served` → `payment_required` → `served_paid`). Only if you gate anything.

If you also run agents of your own — and most sites building anything with LLMs now do — the tracing discipline on that side is a different problem with a mature toolset, covered in [agent observability and tracing](/resources/agent-observability).

## Privacy, briefly

This is machine traffic, but the same log can carry human requests, so apply the usual discipline: no credentials, truncate user agents, avoid full query strings where they can carry personal data, keep retention short and documented, and aggregate rather than hoard. Nothing in this guide requires identifying a person, and a schema that cannot identify one is a schema you never have to defend.

## Frequently asked questions

### Why does Google Analytics not show AI crawler traffic?

Because it is a client-side tag: it needs JavaScript to execute in order to record a hit. Most crawlers and many agents never run it. Whatever your JS analytics shows for AI bots, treat it as a lower bound that is close to zero rather than as data.

### How do I know a request claiming to be GPTBot really is?

Check the source IP against the vendor's published ranges — every major vendor publishes them. UA strings are a claim, not evidence. An emerging alternative is cryptographic bot authentication, where the client signs its requests, which removes the need for IP lists entirely.

### How long should I keep this data?

30 to 90 days is enough for every decision in this cluster and keeps the storage question boring. Aggregate older data into monthly counts rather than retaining raw rows.

### What is the single most useful chart?

Requests per crawler per week, split by HTTP status. It surfaces the two failures that matter — an edge rule blocking crawlers you meant to allow, and a crawler hammering a path you did not expect — long before anything else does.


---

## The rest of this guide

- [The Agent-Ready Website: A Complete Guide to AI Visibility, Access Control and Monetization](https://changegamer.ai/articles/agent-ready-website.md): The full operator playbook for making a website work for AI agents and AI crawlers: be fetchable, be readable, be controllable, be payable — with a 30-day implementation plan.
- [llms.txt vs robots.txt vs sitemap.xml: Which File Does What](https://changegamer.ai/articles/llms-txt-vs-robots-txt-vs-sitemap.md): The three root-level files every agent-ready site publishes, what each one is actually for, and why publishing one does not substitute for the others.
- [How to Write an llms.txt File (Format, Template, and Maintenance)](https://changegamer.ai/articles/how-to-write-an-llms-txt-file.md): A step-by-step guide to writing a useful llms.txt: the exact format, a copy-paste template, what to put under ## Optional, how to validate it, and how to keep it from rotting.
- [Serving Markdown Variants to AI Agents: The Cheapest Win in AI Visibility](https://changegamer.ai/articles/serving-markdown-variants-to-ai-agents.md): How to publish a .md twin of every page — URL patterns, content negotiation, discovery headers, generation pitfalls — and why it cuts what an agent pays to read you.
- [How AI Search Engines Choose Sources (And What You Can Actually Influence)](https://changegamer.ai/articles/how-ai-search-engines-choose-sources.md): What is known, what is claimed and what is speculation about how ChatGPT, Perplexity and AI Overviews pick the pages they cite — and the short list of things a site owner can actually control.
- [Should You Block AI Crawlers? A Decision Framework by Business Model](https://changegamer.ai/articles/should-you-block-ai-crawlers.md): Blocking AI crawlers is four separate decisions, not one. A framework that maps each crawler class to what it costs and earns you, by business model, with the exact robots.txt for each answer.
- [What to Charge AI Crawlers: Pricing Models for Machine Buyers](https://changegamer.ai/articles/what-to-charge-ai-crawlers.md): Per-crawl, per-resource, corpus licence or subscription key — the four ways to price AI access, the arithmetic behind each, and why pricing before you have demand data is the standard mistake.
- [Implementing an HTTP 402 Paywall an Agent Can Actually Pay](https://changegamer.ai/articles/http-402-paywall-implementation.md): A working implementation guide for machine-payable content: the 402 response body, Link headers, key issuance and validation, caching rules, and the mistakes that make a 402 gate unpayable.
- [Structured Data for AI Agents: Which Schema.org Types Earn Their Keep](https://changegamer.ai/articles/structured-data-for-ai-agents.md): Most schema.org markup is invisible to machine readers. The types that are worth the effort for AI agents, how to emit them without drift, and what to build instead of more markup.
- [JSON API Design for AI Agents: Endpoints They Prefer Over Scraping](https://changegamer.ai/articles/json-api-design-for-agents.md): How to publish read-only JSON endpoints that agents choose over scraping your HTML: discovery index, stable shapes, freshness signals, bulk exports, and errors a machine can act on.
- [Licensing Content for AI Training: RSL, Terms, and Provenance](https://changegamer.ai/articles/licensing-content-for-ai-training.md): How to publish machine-readable licence terms for AI use — what RSL is, what it does and does not do, how it differs from robots.txt and Content Signals, and where provenance standards fit.
- [Running an MCP Server as a Distribution Channel for Your Content](https://changegamer.ai/articles/mcp-server-as-distribution-channel.md): Why a content site should expose an MCP server, which tools to ship, how discovery and authentication work, how to gate paid tools, and the honest limits of the channel.
- [Why AI Agents Can't Read Your Site: Twelve Failure Modes and How to Find Them](https://changegamer.ai/articles/why-ai-agents-cant-read-your-site.md): A diagnostic catalogue of the twelve reasons AI agents and crawlers fail on real sites — from silent WAF blocks to JS-only rendering — each with the command that detects it and the fix.

## Reference resources

- https://changegamer.ai/resources/ai-crawler-policy.md
- https://changegamer.ai/resources/agent-observability.md
- https://changegamer.ai/resources/web-bot-auth.md
- https://changegamer.ai/resources/access-and-pricing.md

All guides: https://changegamer.ai/api/articles.json · Reference corpus: https://changegamer.ai/llms.txt