← All guides · The agent-ready web

Measuring AI Agent Traffic: Server-Side Telemetry That Answers Real Questions

Part 10 of The agent-ready web · 1,221 words · published 2026-07-26 · updated 2026-07-26 · Markdown variant

Why client-side analytics miss AI agents entirely, the minimum row schema to log, the five queries worth running, and how to tell a real crawler from a spoofed user agent.

Part of the The Agent-Ready Website: A Complete Guide to AI Visibility, Access Control and Monetization guide.

You cannot make any of the decisions in the agent-ready website without data, and the data you already have is probably wrong: it comes from a JavaScript tag that agents do not run.

The row schema

Log one row per request that reaches your origin or edge. Six fields carry every insight worth having:

Two rules on what not to log: never record API keys, session identifiers, or tokens — a log store containing credentials is a credential store — and keep user agents truncated so a hostile client cannot inflate your storage with a megabyte UA.

If you run at an edge platform, this is a handful of lines in the request path. This site writes exactly this shape into a Cloudflare Analytics Engine dataset from its Worker; the same pattern works with any log sink you can query.

function logAccess(env, req, res, outcome) {
  const ua = (req.headers.get("user-agent") ?? "").slice(0, 100);
  env.ANALYTICS.writeDataPoint({
    indexes: [new URL(req.url).pathname],
    blobs: [classOf(ua), outcome, ua, res.status.toString()],
    doubles: [1],
  });
}

One caveat worth designing around: requests served straight from a CDN cache or static-asset layer may never reach your logging code. Decide deliberately which surfaces are counted, and write it down — otherwise you will later read a zero as "no demand" when it means "not instrumented". Index files such as /llms.txt and robots.txt are commonly in this category.

The five queries that drive decisions

1. Who is visiting, and are they real?

SELECT user_agent, COUNT(*) AS n, SUM(verified) AS verified_n
FROM access_log WHERE timestamp > now() - interval 30 day
GROUP BY user_agent ORDER BY n DESC LIMIT 25

Look for the tokens that matter: training crawlers, answer-engine indexers, live user fetchers. A large gap between n and verified_n for a vendor token means somebody is spoofing it — usually a scraper hoping you allowlisted that name. The token reference is AI crawler policy.

2. Am I accidentally blocking the crawlers I want?

SELECT user_agent, status, COUNT(*) AS n
FROM access_log WHERE timestamp > now() - interval 7 day
  AND status IN (403, 429, 503)
GROUP BY user_agent, status ORDER BY n DESC

This is the query that pays for the whole setup. A wall of 403s against a crawler your robots.txt allows means bot management or a browser-integrity check is overriding your published policy — the exact failure this site hit, where a managed rule silently 403'd six named AI crawlers including on / and /sitemap.xml. Everything in that family is catalogued in why AI agents can't read your site.

3. What do they actually want?

SELECT path, COUNT(*) AS n, COUNT(DISTINCT user_agent) AS clients
FROM access_log WHERE outcome = 'served' AND timestamp > now() - interval 30 day
GROUP BY path ORDER BY n DESC LIMIT 30

Read the distribution, not the top row. Concentrated demand on a few URLs means those pages are a product; flat demand across everything means your corpus is the product. That distinction decides your entire pricing shape — see what to charge AI crawlers.

4. Does the traffic come back as visitors?

-- referrals from answer-engine domains, from the same log
SELECT referer_host, COUNT(*) AS visits
FROM access_log WHERE timestamp > now() - interval 30 day
  AND referer_host IS NOT NULL
GROUP BY referer_host ORDER BY visits DESC LIMIT 20

Then compute the ratio that matters: fetches by answer-engine crawlers ÷ referrals from that engine's domain. A high ratio means your content is being consumed to produce answers that do not send anyone your way. That is not necessarily bad — brand presence in answers has value — but it is the number that turns "should we monetize or restrict this?" from a philosophical question into an arithmetic one.

5. If you gate anything, does the wall convert?

SELECT outcome, COUNT(*) AS n
FROM access_log WHERE timestamp > now() - interval 30 day
  AND outcome IN ('payment_required', 'upgrade', 'served_paid')
GROUP BY outcome

A funnel with many payment_required rows and no paid fetches means the price is wrong, the payment path has friction, or the buyers were never commercial. upgrade rows — valid keys of insufficient tier — are the cleanest upsell signal that exists, because someone already paid you once.

Verifying crawlers: three options

The last row is where this is heading: instead of maintaining allowlists of addresses, the client proves its identity by signing the request. The state of that work is covered in web bot auth. Until it is universal, IP-range verification is the practical answer, and it belongs in your logging path so verified is a field rather than a later investigation.

Decide in advance what would change your mind

The most common analytical failure here is not a missing metric, it is acting on a week of noise. Write the thresholds down before you look:

• "If a crawler exceeds N requests/day against paths we do not want indexed, we add a Disallow."
• "If we see at least 50 distinct paywall hits in 30 days, we revisit pricing; below that, pricing is frozen." (This site uses exactly that rule, and it has kept several tempting-but-unsupported price changes from shipping.)
• "If fetch-to-referral for an engine exceeds X, we treat it as a licensing conversation, not a traffic channel."

A pre-committed threshold converts an argument into a measurement.

A one-screen dashboard

Four panels, refreshed weekly, are enough to run this:

1. Requests per crawler per week, stacked by status. Catches blocking and bursts.
2. Top 20 paths by served requests. Shows what your product actually is.
3. Fetch-to-referral ratio per answer engine. The strategic number.
4. Outcome funnel (served → payment_required → served_paid). Only if you gate anything.

If you also run agents of your own — and most sites building anything with LLMs now do — the tracing discipline on that side is a different problem with a mature toolset, covered in agent observability and tracing.

Privacy, briefly

This is machine traffic, but the same log can carry human requests, so apply the usual discipline: no credentials, truncate user agents, avoid full query strings where they can carry personal data, keep retention short and documented, and aggregate rather than hoard. Nothing in this guide requires identifying a person, and a schema that cannot identify one is a schema you never have to defend.

#analytics #observability #crawlers #measurement #ai-visibility