{
  "slug": "web-data-for-agents",
  "title": "Web Data and Scraping for Agents",
  "description": "Tool landscape for agent web-data pipelines: reader/URL-to-Markdown APIs, crawl/scrape services, and search APIs — with MCP exposure, OSS/SaaS classification, and practical guidance.",
  "category": "Reference",
  "tags": [
    "web-scraping",
    "crawling",
    "search-api",
    "markdown",
    "rag",
    "agents",
    "mcp",
    "tools"
  ],
  "updated": "2026-07-02",
  "premium": false,
  "canonical": "https://changegamer.ai/resources/web-data-for-agents",
  "markdown": "https://changegamer.ai/resources/web-data-for-agents.md",
  "outline": [
    {
      "depth": 2,
      "text": "Key facts",
      "anchor": "key-facts"
    },
    {
      "depth": 2,
      "text": "Three jobs — pick the right tool for each",
      "anchor": "three-jobs-pick-the-right-tool-for-each"
    },
    {
      "depth": 2,
      "text": "Job 1: Reader / URL-to-Markdown",
      "anchor": "job-1-reader-url-to-markdown"
    },
    {
      "depth": 2,
      "text": "Job 2: Crawl / scrape at scale",
      "anchor": "job-2-crawl-scrape-at-scale"
    },
    {
      "depth": 2,
      "text": "Job 3: Search APIs",
      "anchor": "job-3-search-apis"
    },
    {
      "depth": 2,
      "text": "The agent angle",
      "anchor": "the-agent-angle"
    },
    {
      "depth": 2,
      "text": "Practical guidance",
      "anchor": "practical-guidance"
    },
    {
      "depth": 2,
      "text": "Verified sources",
      "anchor": "verified-sources"
    },
    {
      "depth": 2,
      "text": "Content freshness",
      "anchor": "content-freshness"
    }
  ],
  "related": [
    {
      "slug": "nlweb-explained",
      "title": "NLWeb Explained: Microsoft's Natural-Language Query Protocol for Websites",
      "description": "What NLWeb is: the open, MIT-licensed protocol that lets a site answer natural-language questions over its own Schema.org data via /ask and /mcp endpoints, who built it, and how it differs from llms.txt, AGENTS.md, and a generic MCP server.",
      "url": "https://changegamer.ai/resources/nlweb-explained"
    },
    {
      "slug": "building-mcp-servers",
      "title": "Building an MCP Server",
      "description": "Implementation guide for MCP servers: architecture roles, the three server primitives, stdio vs Streamable HTTP transports, official SDKs, server lifecycle, remote-server concerns, testing with MCP Inspector, and publishing to the official registry.",
      "url": "https://changegamer.ai/resources/building-mcp-servers"
    },
    {
      "slug": "mcp-server-discovery",
      "title": "Finding and Evaluating MCP Servers",
      "description": "How to discover, assess and safely integrate MCP servers into agent pipelines.",
      "url": "https://changegamer.ai/resources/mcp-server-discovery"
    },
    {
      "slug": "agent-frameworks-compared",
      "title": "AI Agent Frameworks Compared",
      "description": "Vendor-neutral comparison table of the major agent-orchestration frameworks — language, license, multi-agent model, MCP/A2A support — plus a how-to-choose guide for agent builders.",
      "url": "https://changegamer.ai/resources/agent-frameworks-compared"
    }
  ],
  "furtherReading": [
    {
      "slug": "mcp-server-as-distribution-channel",
      "title": "Running an MCP Server as a Distribution Channel for Your Content",
      "description": "Why a content site should expose an MCP server, which tools to ship, how discovery and authentication work, how to gate paid tools, and the honest limits of the channel.",
      "url": "https://changegamer.ai/articles/mcp-server-as-distribution-channel"
    },
    {
      "slug": "mcp-server-failure-modes",
      "title": "Common MCP Server Failure Modes and How to Fix Them",
      "description": "A runtime playbook for the two MCP server failure modes with no dedicated deep-dive elsewhere: unrecoverable state after a mid-call crash, and malformed or hallucinated tool calls that reach the handler despite upstream validation.",
      "url": "https://changegamer.ai/articles/mcp-server-failure-modes"
    }
  ],
  "body": "Agents cannot use raw HTML efficiently: it bloats the context window, embeds navigation noise, and costs 3–10x more tokens than clean Markdown of the same content. A web-data layer transforms live web content into agent-consumable form. Three distinct jobs require different tools.\n\n## Key facts\n\n- Feeding an agent raw HTML instead of a converted, cleaned-up text format multiplies its token bill several times over for the same underlying content, which is the core reason a dedicated web-data transformation layer exists at all.\n- Web-data tooling splits into three distinct jobs — reading a single known URL, crawling/scraping at scale, and searching the web — each best served by a different class of tool rather than one do-everything product.\n- For single-URL reading, running your own open-source extractor costs nothing and keeps data in-house, but it has no way to get past a CAPTCHA, which is exactly what the hosted reader/scrape services are built to handle automatically.\n- Some crawl/scrape services now expose payment or MCP integration directly — for example, tens of thousands of Apify Actors became payable per-run over x402 with no account needed, and several providers ship their own MCP servers.\n- Search APIs take different approaches: some return ranked links and snippets, some are built for semantic/embedding-based retrieval, and some return LLM-generated answers with citations — alongside server-side search tools now built natively into the major model APIs.\n- When passing scraped content into an agent's context, Markdown rather than raw HTML is the default choice across this whole tooling landscape, purely to keep token cost down.\n- Fetched web content should be treated as untrusted input given prompt-injection risk, and an agent acting as a crawler should respect robots.txt and identify itself with a recognizable, contactable User-Agent.\n\n## Three jobs — pick the right tool for each\n\n| Job | What you need | Tools |\n|---|---|---|\n| 1. Read a known URL → clean text | URL-to-Markdown conversion | Jina Reader, Firecrawl /scrape, trafilatura, Mozilla Readability |\n| 2. Crawl a site / scrape at scale | Multi-URL crawl + JS rendering + anti-bot | Firecrawl /crawl, Apify, Crawlee, Browserbase, Bright Data, ScrapingBee |\n| 3. Search the web | Search query → ranked URLs + snippets or answers | Tavily, Exa, Brave Search API, Serper, Perplexity Sonar API, built-in provider tools |\n\n## Job 1: Reader / URL-to-Markdown\n\n**Jina Reader** (SaaS + OSS) — prefix any URL with `https://r.jina.ai/` and receive clean Markdown optimized for LLMs. No key required for basic usage; optional API key for higher rate limits. The extraction model is ReaderLM-v2 (1.5B). Supports PDF and MS Office documents via direct POST. Free tier available; OSS branch at github.com/jina-ai/reader. Exposes an MCP server via the Jina AI MCP (smithery.ai registry).\n\n**Firecrawl /scrape** (SaaS + AGPL self-host) — one-URL scrape endpoint returning Markdown, HTML, or structured JSON. Handles JS-rendered pages, proxy rotation, and CAPTCHA. Free tier (500 credits/month); paid from $19/month. GitHub: github.com/mendableai/firecrawl. Exposes an official MCP server.\n\n**Self-hosted options (OSS):**\n\n- *trafilatura* (Apache 2.0, Python) — extracts main text and metadata from HTML with high accuracy; outputs TXT, Markdown, CSV, JSON, or TEI-XML. Command-line and library. Used by HuggingFace, IBM, and Microsoft Research. Docs: trafilatura.readthedocs.io.\n- *Mozilla Readability* (Apache 2.0, JavaScript) — the parser behind Firefox Reader Mode; strips nav/ads and returns article DOM. GitHub: github.com/mozilla/readability. Pair with Playwright or Puppeteer for JS-rendered pages.\n\n**When to self-host vs use a service:** self-hosted options are free and private but require infrastructure and cannot solve CAPTCHAs. Services handle anti-bot at scale out of the box.\n\n## Job 2: Crawl / scrape at scale\n\n**Firecrawl /crawl** (SaaS + AGPL self-host) — crawls an entire site and returns all pages as Markdown. Same service as the /scrape endpoint; the /crawl endpoint accepts a root URL and traverses all sub-URLs. Handles JS rendering, rate limiting, and proxy rotation automatically.\n\n**Apify** (SaaS) — managed cloud platform with tens of thousands of community-built Actors (preconfigured scrapers for common targets; the store advertises 46,000+) plus a proxy network and storage layer. Actors run serverlessly; pricing is pay-per-compute-unit. Since June 2026, **20,000+ public Actors are also payable per run over x402** (USDC on Base, in partnership with Coinbase) — an agent can call an Actor with no Apify account and no API key; see /resources/agent-wallets-x402. Two Actors worth knowing by name: **Website Content Crawler** (site → clean, LLM-ready Markdown for RAG pipelines; Apify's June 2026 update announcement reports a faster, cheaper build with no config changes needed) and **AI Web Scraper**, whose **\"Agentic\" extraction mode** takes a plain-language prompt, reads the site's sitemap, plans its own crawl, and returns structured data — steerable with navigation hints in the prompt (\"only follow product pages and pagination\"), with no selectors to break on layout changes. Homepage: apify.com.\n\n**Crawlee** (Apache 2.0, OSS by Apify) — open-source TypeScript/JavaScript (and Python) web-scraping library. Supports Cheerio, JSDOM, Playwright, and Puppeteer crawlers with auto proxy rotation, fingerprinting, and autoscaling. Can run locally or deploy to Apify. Python port stable since September 2025. GitHub: github.com/apify/crawlee.\n\n**Browserbase** (SaaS) — managed cloud headless browsers (Playwright/Puppeteer API) optimized for AI agents. Handles CAPTCHA, stealth, and session recording. Priced per session. Homepage: browserbase.com.\n\n**Bright Data** (SaaS) — enterprise proxy + scraping stack. Web MCP server (free tier: 5,000 requests/month) exposes Web Unlocker, SERP API, and Scraping Browser directly to MCP-compatible agents. Homepage: brightdata.com.\n\n**ScrapingBee** (SaaS) — headless browser scraping API; handles JS rendering and proxy rotation. Acquired by Oxylabs in 2025; operates as an independent brand. Homepage: scrapingbee.com.\n\n## Job 3: Search APIs\n\n**Tavily** (SaaS) — agent-native search API: Search, Extract, Map, and Crawl endpoints. Returns structured results optimized for RAG. Sub-200ms p50 latency; 100M+ monthly requests. Joined Nebius (AI infrastructure) in February 2026. MCP server available. Docs: docs.tavily.com.\n\n**Exa** (SaaS) — formerly Metaphor; neural/embedding-based search designed for AI agents. Retrieves pages by semantic meaning, not keyword matching. Raised $85M at $700M valuation (September 2025). APIs: Search, Contents, Answer, Find Similar, Websets. Contents (up to 10 results) included free with each Search call as of March 2026. Docs: exa.ai/docs.\n\n**Brave Search API** (SaaS) — REST API over Brave's own independent web index (30B+ pages). Does not license from Google or Bing. SOC 2 Type II attested (October 2025). Supplies real-time search data to several major LLMs. Docs: brave.com/search/api.\n\n**Serper** (SaaS) — fast Google SERP API. Returns real-time Google results (web, news, images, maps) in JSON. ~2.87s latency; $0.30–$1.00/1k queries at scale. 2,500 free queries/month. MCP server available. Homepage: serper.dev.\n\n**Perplexity Sonar API** (SaaS) — LLM-generated answers with inline web citations. Four model tiers: Sonar, Sonar Pro, Sonar Reasoning, and Deep Research. $14–$22 per 1,000 Pro Search queries. Docs: docs.perplexity.ai.\n\n**Built-in provider search tools** — all three major providers expose native web-search tools that run server-side (no extra API key needed):\n\n- *Anthropic Claude* — `web_search_20260209` server tool in the Messages API; supports domain filtering, `max_uses` cap, and dynamic result filtering via code execution. $10 per 1,000 searches plus token costs. Docs: platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool.\n- *OpenAI Responses API* — `{\"type\": \"web_search\"}` built-in tool; supports `external_web_access`, filters, and `return_token_budget` controls. Docs: platform.openai.com/docs/guides/tools-web-search.\n- *Google Gemini* — `google_search` grounding tool; can be combined with custom function calling in a single API call. Docs: ai.google.dev/gemini-api/docs/google-search.\n\n## The agent angle\n\nSeveral services expose **MCP servers** (Jina, Firecrawl, Bright Data, Tavily, Serper), letting any MCP-compatible agent call web-data tools without custom integration. Check each provider's MCP docs or the registry at registry.modelcontextprotocol.io.\n\nClean **Markdown is the standard interchange** between web-data tools and agent context. Prefer it over raw HTML to minimize token cost.\n\nWhen your agent is the crawler, respect robots.txt and AI-crawler policies: see /resources/ai-crawler-policy.\n\nTreat all fetched web content as untrusted — prompt injection is a real attack surface. See /resources/agentic-security-checklist, sections 1 and 4.\n\nWeb data that feeds a retrieval system connects to the RAG layer: see /resources/rag-retrieval-for-agents.\n\n## Practical guidance\n\n- **Prefer reader/Markdown over raw HTML** — token cost difference is often 5–10x.\n- **Cache aggressively** — a 1-hour TTL covers most agent use cases and cuts cost and latency substantially.\n- **JS-rendered vs static** — static pages work with lightweight extractors (trafilatura, Readability). JS-heavy sites require a headless browser (Playwright, Browserbase, Bright Data Scraping Browser).\n- **Rate-limit and identify your crawler honestly** — set a recognizable `User-Agent` with a contact URL; back off on 429; honor `Crawl-delay` in robots.txt.\n- **For scale** — managed services (Firecrawl, Apify, Bright Data) handle proxy rotation and anti-bot. Self-hosted stacks (Crawlee + Playwright) give more control at higher ops cost.\n\n## Verified sources\n\n- Jina Reader API: https://jina.ai/reader/\n- Jina Reader GitHub (jina-ai/reader): https://github.com/jina-ai/reader\n- Firecrawl homepage: https://www.firecrawl.dev/\n- Firecrawl GitHub (mendableai/firecrawl, AGPL-3.0): https://github.com/mendableai/firecrawl\n- Crawlee GitHub (apify/crawlee, Apache 2.0): https://github.com/apify/crawlee\n- Crawlee Python GitHub: https://github.com/apify/crawlee-python\n- Apify platform: https://apify.com/\n- Apify x402 launch — 20,000+ Actors payable per run (June 2026): https://blog.apify.com/introducing-x402-agentic-payments/\n- Apify Website Content Crawler: https://apify.com/apify/website-content-crawler\n- Apify AI Web Scraper (Agentic extraction mode): https://apify.com/apify/ai-web-scraper\n- Trafilatura docs: https://trafilatura.readthedocs.io/\n- Mozilla Readability GitHub: https://github.com/mozilla/readability\n- Bright Data Web MCP blog: https://brightdata.com/blog/ai/web-scraping-with-mcp\n- Tavily docs: https://docs.tavily.com/\n- Exa Search API docs: https://exa.ai/docs/reference/search-api-guide\n- Brave Search API: https://brave.com/search/api/\n- Brave Search API growth announcement: https://brave.com/blog/search-api-growth/\n- Perplexity Sonar API docs: https://docs.perplexity.ai/\n- Anthropic web_search tool docs: https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool\n- OpenAI web search (Responses API): https://platform.openai.com/docs/guides/tools-web-search\n- Google Gemini grounding with Search: https://ai.google.dev/gemini-api/docs/google-search\n\n## Content freshness\n\nSeveral figures on this page are the kind that age quickly: Apify's 20,000+ x402-payable Actors, Firecrawl's free-tier credit allowance, Tavily's 100M+ monthly request volume, Exa's September 2025 $85M/$700M funding round, and Anthropic's dated `web_search_20260209` tool-model string are all provider-specific numbers and version identifiers that get revised, expanded, or superseded on each vendor's own schedule. Treat them as most reliable as of this page's `updated` date (2026-07-02); if you're choosing a provider after roughly Q4 2026, independently re-verify current pricing, quotas, and tool-model identifiers rather than this snapshot.",
  "sources": [
    "https://jina.ai/reader/",
    "https://github.com/jina-ai/reader",
    "https://www.firecrawl.dev/",
    "https://github.com/mendableai/firecrawl",
    "https://github.com/apify/crawlee",
    "https://github.com/apify/crawlee-python",
    "https://apify.com/",
    "https://blog.apify.com/introducing-x402-agentic-payments/",
    "https://apify.com/apify/website-content-crawler",
    "https://apify.com/apify/ai-web-scraper",
    "https://trafilatura.readthedocs.io/",
    "https://github.com/mozilla/readability",
    "https://brightdata.com/blog/ai/web-scraping-with-mcp",
    "https://docs.tavily.com/",
    "https://exa.ai/docs/reference/search-api-guide",
    "https://brave.com/search/api/",
    "https://brave.com/blog/search-api-growth/",
    "https://docs.perplexity.ai/",
    "https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool",
    "https://platform.openai.com/docs/guides/tools-web-search",
    "https://ai.google.dev/gemini-api/docs/google-search"
  ]
}