Scraping APIs for AI agents

Bright Data Web Unlocker

Diffbot — Extract STRUCTURED, typed data from a URL: it classifies the page (article / product / discussion / image) and returns parsed fields — title…

Diffbot — Same structured extraction as diffbot/analyze, but over HTML YOU already fetched rather than a URL Diffbot fetches itself. This is the pairing for…

CRW — Start an ASYNC multi-page crawl from a seed URL, following links. Pass `url`; optional `maxPages` (1-100, default 100 — vendor bills 1 credit/page so…

CRW — Poll an async crawl started by crw/crawl. Pass `id` (from the crawl response). Returns `{ status: scraping|completed|failed|cancelled, completed, total…

CRW — Structured extraction over up to 10 URLs using an LLM. Pass `urls` plus `prompt` (natural language) and/or `schema` (JSON schema); optional `basis…

CRW — Poll an async extraction started by crw/extract, on the rare occasions it returns an `id` instead of inline results. Pass `id`. Returns `{ status…

CRW — Discover the URLs of a website without scraping content (sitemap + crawl fallback). Pass `url`; optional `maxDepth`, `useSitemap` (default true)…

CRW — Scrape a single URL to clean markdown/HTML/JSON (Firecrawl-compatible). Pass `url`; optional `formats`…

CRW — Search the web and optionally scrape the hits in one call. Pass `query`; optional `limit` (1-20, default 5), `tbs` (freshness: qdr:h|d|w|m|y), `sources`…

Firecrawl — Crawl a website starting from a URL, following links.

Firecrawl — Extract structured data from URLs using a schema.

Firecrawl — Map all URLs on a website without scraping content.

Firecrawl — Scrape a single URL and return clean markdown/HTML.

Firecrawl — Search the web and return scraped results.

Jina Reader — fetch a URL and return LLM-ready markdown (r.jina.ai). Pass `url`. Handles JS rendering and boilerplate stripping automatically; returns `{…

Jina Search — web search that returns the top hits WITH their full reader-processed page content in one call (s.jina.ai). Pass `q`; optional `num` (result…

Scrape.do — Fetch a page through a rotating datacenter-proxy pool with anti-bot handling. Surprisingly strong for the price: it returned the REAL page on a…

Scrape.do — The heavy rung: RESIDENTIAL/mobile proxy pool plus full JS rendering (`super` + `render`). For pages that bounce the plain scrapedo/scrape call…

ScrapingAnt — AI data extraction WITHOUT a schema: describe the fields in plain English and get structured JSON back. Pass `url` and `extract_properties` — a…

ScrapingAnt — Scrape a URL and return LLM-ready markdown (rendered in headless Chrome, then converted). Pass `url`; optional…

ScrapingAnt — Scrape a URL through a managed headless-Chrome cluster (datacenter proxies). Pass `url`; optional `browser` (default true — set false for plain…

ScrapingAnt — Scrape a HARD page through the 3M+ residential-proxy pool + headless Chrome: Cloudflare and anti-bot walls, geo-fenced content, sites that block…