← All resources

Web Data and Scraping for Agents

Reference · updated 2026-07-02 · Markdown variant

Tool landscape for agent web-data pipelines: reader/URL-to-Markdown APIs, crawl/scrape services, and search APIs — with MCP exposure, OSS/SaaS classification, and practical guidance.

Agents cannot use raw HTML efficiently: it bloats the context window, embeds navigation noise, and costs 3–10x more tokens than clean Markdown of the same content. A web-data layer transforms live web content into agent-consumable form. Three distinct jobs require different tools.

Key facts

• Feeding an agent raw HTML instead of a converted, cleaned-up text format multiplies its token bill several times over for the same underlying content, which is the core reason a dedicated web-data transformation layer exists at all.
• Web-data tooling splits into three distinct jobs — reading a single known URL, crawling/scraping at scale, and searching the web — each best served by a different class of tool rather than one do-everything product.
• For single-URL reading, running your own open-source extractor costs nothing and keeps data in-house, but it has no way to get past a CAPTCHA, which is exactly what the hosted reader/scrape services are built to handle automatically.
• Some crawl/scrape services now expose payment or MCP integration directly — for example, tens of thousands of Apify Actors became payable per-run over x402 with no account needed, and several providers ship their own MCP servers.
• Search APIs take different approaches: some return ranked links and snippets, some are built for semantic/embedding-based retrieval, and some return LLM-generated answers with citations — alongside server-side search tools now built natively into the major model APIs.
• When passing scraped content into an agent's context, Markdown rather than raw HTML is the default choice across this whole tooling landscape, purely to keep token cost down.
• Fetched web content should be treated as untrusted input given prompt-injection risk, and an agent acting as a crawler should respect robots.txt and identify itself with a recognizable, contactable User-Agent.

Three jobs — pick the right tool for each

Job 1: Reader / URL-to-Markdown

Jina Reader (SaaS + OSS) — prefix any URL with https://r.jina.ai/ and receive clean Markdown optimized for LLMs. No key required for basic usage; optional API key for higher rate limits. The extraction model is ReaderLM-v2 (1.5B). Supports PDF and MS Office documents via direct POST. Free tier available; OSS branch at github.com/jina-ai/reader. Exposes an MCP server via the Jina AI MCP (smithery.ai registry).

Firecrawl /scrape (SaaS + AGPL self-host) — one-URL scrape endpoint returning Markdown, HTML, or structured JSON. Handles JS-rendered pages, proxy rotation, and CAPTCHA. Free tier (500 credits/month); paid from $19/month. GitHub: github.com/mendableai/firecrawl. Exposes an official MCP server.

Self-hosted options (OSS):

• trafilatura (Apache 2.0, Python) — extracts main text and metadata from HTML with high accuracy; outputs TXT, Markdown, CSV, JSON, or TEI-XML. Command-line and library. Used by HuggingFace, IBM, and Microsoft Research. Docs: trafilatura.readthedocs.io.
• Mozilla Readability (Apache 2.0, JavaScript) — the parser behind Firefox Reader Mode; strips nav/ads and returns article DOM. GitHub: github.com/mozilla/readability. Pair with Playwright or Puppeteer for JS-rendered pages.

When to self-host vs use a service: self-hosted options are free and private but require infrastructure and cannot solve CAPTCHAs. Services handle anti-bot at scale out of the box.

Job 2: Crawl / scrape at scale

Firecrawl /crawl (SaaS + AGPL self-host) — crawls an entire site and returns all pages as Markdown. Same service as the /scrape endpoint; the /crawl endpoint accepts a root URL and traverses all sub-URLs. Handles JS rendering, rate limiting, and proxy rotation automatically.

Apify (SaaS) — managed cloud platform with tens of thousands of community-built Actors (preconfigured scrapers for common targets; the store advertises 46,000+) plus a proxy network and storage layer. Actors run serverlessly; pricing is pay-per-compute-unit. Since June 2026, 20,000+ public Actors are also payable per run over x402 (USDC on Base, in partnership with Coinbase) — an agent can call an Actor with no Apify account and no API key; see /resources/agent-wallets-x402. Two Actors worth knowing by name: Website Content Crawler (site → clean, LLM-ready Markdown for RAG pipelines; Apify's June 2026 update announcement reports a faster, cheaper build with no config changes needed) and AI Web Scraper, whose "Agentic" extraction mode takes a plain-language prompt, reads the site's sitemap, plans its own crawl, and returns structured data — steerable with navigation hints in the prompt ("only follow product pages and pagination"), with no selectors to break on layout changes. Homepage: apify.com.

Crawlee (Apache 2.0, OSS by Apify) — open-source TypeScript/JavaScript (and Python) web-scraping library. Supports Cheerio, JSDOM, Playwright, and Puppeteer crawlers with auto proxy rotation, fingerprinting, and autoscaling. Can run locally or deploy to Apify. Python port stable since September 2025. GitHub: github.com/apify/crawlee.

Browserbase (SaaS) — managed cloud headless browsers (Playwright/Puppeteer API) optimized for AI agents. Handles CAPTCHA, stealth, and session recording. Priced per session. Homepage: browserbase.com.

Bright Data (SaaS) — enterprise proxy + scraping stack. Web MCP server (free tier: 5,000 requests/month) exposes Web Unlocker, SERP API, and Scraping Browser directly to MCP-compatible agents. Homepage: brightdata.com.

ScrapingBee (SaaS) — headless browser scraping API; handles JS rendering and proxy rotation. Acquired by Oxylabs in 2025; operates as an independent brand. Homepage: scrapingbee.com.

Job 3: Search APIs

Tavily (SaaS) — agent-native search API: Search, Extract, Map, and Crawl endpoints. Returns structured results optimized for RAG. Sub-200ms p50 latency; 100M+ monthly requests. Joined Nebius (AI infrastructure) in February 2026. MCP server available. Docs: docs.tavily.com.

Exa (SaaS) — formerly Metaphor; neural/embedding-based search designed for AI agents. Retrieves pages by semantic meaning, not keyword matching. Raised $85M at $700M valuation (September 2025). APIs: Search, Contents, Answer, Find Similar, Websets. Contents (up to 10 results) included free with each Search call as of March 2026. Docs: exa.ai/docs.

Brave Search API (SaaS) — REST API over Brave's own independent web index (30B+ pages). Does not license from Google or Bing. SOC 2 Type II attested (October 2025). Supplies real-time search data to several major LLMs. Docs: brave.com/search/api.

Serper (SaaS) — fast Google SERP API. Returns real-time Google results (web, news, images, maps) in JSON. ~2.87s latency; $0.30–$1.00/1k queries at scale. 2,500 free queries/month. MCP server available. Homepage: serper.dev.

Perplexity Sonar API (SaaS) — LLM-generated answers with inline web citations. Four model tiers: Sonar, Sonar Pro, Sonar Reasoning, and Deep Research. $14–$22 per 1,000 Pro Search queries. Docs: docs.perplexity.ai.

Built-in provider search tools — all three major providers expose native web-search tools that run server-side (no extra API key needed):

• Anthropic Claude — web_search_20260209 server tool in the Messages API; supports domain filtering, max_uses cap, and dynamic result filtering via code execution. $10 per 1,000 searches plus token costs. Docs: platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool.
• OpenAI Responses API — {"type": "web_search"} built-in tool; supports external_web_access, filters, and return_token_budget controls. Docs: platform.openai.com/docs/guides/tools-web-search.
• Google Gemini — google_search grounding tool; can be combined with custom function calling in a single API call. Docs: ai.google.dev/gemini-api/docs/google-search.

The agent angle

Several services expose MCP servers (Jina, Firecrawl, Bright Data, Tavily, Serper), letting any MCP-compatible agent call web-data tools without custom integration. Check each provider's MCP docs or the registry at registry.modelcontextprotocol.io.

Clean Markdown is the standard interchange between web-data tools and agent context. Prefer it over raw HTML to minimize token cost.

When your agent is the crawler, respect robots.txt and AI-crawler policies: see /resources/ai-crawler-policy.

Treat all fetched web content as untrusted — prompt injection is a real attack surface. See /resources/agentic-security-checklist, sections 1 and 4.

Web data that feeds a retrieval system connects to the RAG layer: see /resources/rag-retrieval-for-agents.

Practical guidance

• Prefer reader/Markdown over raw HTML — token cost difference is often 5–10x.
• Cache aggressively — a 1-hour TTL covers most agent use cases and cuts cost and latency substantially.
• JS-rendered vs static — static pages work with lightweight extractors (trafilatura, Readability). JS-heavy sites require a headless browser (Playwright, Browserbase, Bright Data Scraping Browser).
• Rate-limit and identify your crawler honestly — set a recognizable User-Agent with a contact URL; back off on 429; honor Crawl-delay in robots.txt.
• For scale — managed services (Firecrawl, Apify, Bright Data) handle proxy rotation and anti-bot. Self-hosted stacks (Crawlee + Playwright) give more control at higher ops cost.

Verified sources

• Jina Reader API: https://jina.ai/reader/
• Jina Reader GitHub (jina-ai/reader): https://github.com/jina-ai/reader
• Firecrawl homepage: https://www.firecrawl.dev/
• Firecrawl GitHub (mendableai/firecrawl, AGPL-3.0): https://github.com/mendableai/firecrawl
• Crawlee GitHub (apify/crawlee, Apache 2.0): https://github.com/apify/crawlee
• Crawlee Python GitHub: https://github.com/apify/crawlee-python
• Apify platform: https://apify.com/
• Apify x402 launch — 20,000+ Actors payable per run (June 2026): https://blog.apify.com/introducing-x402-agentic-payments/
• Apify Website Content Crawler: https://apify.com/apify/website-content-crawler
• Apify AI Web Scraper (Agentic extraction mode): https://apify.com/apify/ai-web-scraper
• Trafilatura docs: https://trafilatura.readthedocs.io/
• Mozilla Readability GitHub: https://github.com/mozilla/readability
• Bright Data Web MCP blog: https://brightdata.com/blog/ai/web-scraping-with-mcp
• Tavily docs: https://docs.tavily.com/
• Exa Search API docs: https://exa.ai/docs/reference/search-api-guide
• Brave Search API: https://brave.com/search/api/
• Brave Search API growth announcement: https://brave.com/blog/search-api-growth/
• Perplexity Sonar API docs: https://docs.perplexity.ai/
• Anthropic web_search tool docs: https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool
• OpenAI web search (Responses API): https://platform.openai.com/docs/guides/tools-web-search
• Google Gemini grounding with Search: https://ai.google.dev/gemini-api/docs/google-search

Content freshness

Several figures on this page are the kind that age quickly: Apify's 20,000+ x402-payable Actors, Firecrawl's free-tier credit allowance, Tavily's 100M+ monthly request volume, Exa's September 2025 $85M/$700M funding round, and Anthropic's dated web_search_20260209 tool-model string are all provider-specific numbers and version identifiers that get revised, expanded, or superseded on each vendor's own schedule. Treat them as most reliable as of this page's updated date (2026-07-02); if you're choosing a provider after roughly Q4 2026, independently re-verify current pricing, quotas, and tool-model identifiers rather than this snapshot.

#web-scraping #crawling #search-api #markdown #rag #agents #mcp #tools

Category: Reference