vaaya / docs / reference

Web scraping

Scrape, crawl, and extract content from the web in vendor tiers: Exa (cheapest, no JS), Firecrawl and CRW (JS rendering plus a full scrape/crawl/map/extract platform — interchangeable request shapes), Tavily extract (the batch rung for known URLs), ScrapingAnt (headless Chrome behind a residential-proxy pool for hard pages), and Apify (typed rows from named platforms). Start with the cheapest option that works; escalate only when you need JS, multi-page, structured extraction, or the residential-proxy rung.

When to use which

The escalation ladder within scraping: exa/contents → firecrawl/crw scrape → scrapingant/scrape → scrapingant/scrape_residential → a browser session (only if you must click or log in). For a managed version of this ladder in one call, pass facets: ["scrape"] to SuperSearch.

Measured performance

From a 12-page, 5-vendor benchmark (August 2026; pages included JS SPAs and bot-hostile marketing pages):

• Known-static pages → exa/contents (0.1¢); anything possibly JS-rendered → firecrawl/scrape.
• Slow-tail pages differ per vendor — racing firecrawl + crw on latency-critical scrapes cut p95 to 0.2–0.6s vs 2–26s single-vendor tails.
• CRW is coverage-equal but never faster — use it when Firecrawl errors, not for speed.
• ScrapingAnt is for Cloudflare / anti-bot / geo-fenced pages the cheap rungs bounce off — send it one page at a time and retry on 409, never batch through it.

exa/contents

Price

0.1¢ per (URL + id) × per content field, min 1¢

Latency

fast (~0.1s warm)

JS rendering

no

Fetch and clean text from URLs (or Exa document IDs) you already have — the cheapest text-extraction option in the catalog when JS rendering isn’t needed. Use it for pulling article bodies for summarization, extracting docs from URLs returned by a previous exa/search, or batch-fetching many pages in one call.

curl -X POST https://vaaya.ai/api/run/exa/contents \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com/article", "https://example.com/about"],
    "text": true
  }'

Provide urls (full URLs) or ids (Exa document IDs from exa/search), or both. Each content field you request — text, highlights, summary — multiplies the cost, ceiling-rounded to whole cents: 1 URL with text → 1¢ (minimum charge); 10 URLs with text → 1¢; 10 URLs with text + highlights + summary → 3¢.

Gotchas

• Asking for highlights and summary on top of text triples the cost without much marginal value if you’re feeding the page into an LLM anyway — pick the minimum field set.
• Cold fetches of JS-heavy pages can return thin content (the extraction is cache-dependent) — escalate those to firecrawl/scrape.

firecrawl/scrape

Price

1¢

Latency

medium (~0.3s p50)

Output

clean markdown or HTML

Scrape a single URL and return clean markdown or HTML. Handles JS rendering, anti-bot bypassing, and content cleaning automatically — the go-to when you need clean, structured output from one page.

curl -X POST https://vaaya.ai/api/run/firecrawl/scrape \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "formats": ["markdown"],
    "onlyMainContent": true,
    "waitFor": 3000
  }'

• url — required.
• formats — output formats, default ["markdown"]; also html.
• onlyMainContent — strips navigation, footers, sidebars.
• waitFor — milliseconds to wait for JS to finish before scraping.

Vs. Exa: use exa/contents (0.1¢/URL) when you don’t need JS rendering; use Firecrawl when you want boilerplate-stripped markdown or JS / anti-bot handling.

firecrawl/crawl, firecrawl/map, firecrawl/search

Price

1¢ each

Latency

medium (map, search) · slow (crawl, multi-page)

firecrawl/crawl crawls an entire website from a seed url, following links — good for indexing documentation sites or scraping product catalogs. limit caps total pages; maxDepth controls link hops from the seed.

curl -X POST https://vaaya.ai/api/run/firecrawl/crawl \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://docs.stripe.com/payments", "limit": 20, "maxDepth": 2}'

• Always set limit to avoid runaway crawls on large sites — start with 10–20 pages, check results, then increase.
• firecrawl/map ({ url }) discovers all URLs on a site without scraping content — a reconnaissance step: map first, then firecrawl/scrape the specific pages you need.
• firecrawl/search ({ query, limit }) combines web search with scraping in one call — full page content from the results, not just snippets. Use exa/search (1¢) for quick semantic search with snippets; use this when you need the content with the hits.

firecrawl/extract

Price

1¢

Latency

medium

Extract structured data from URLs using a natural-language prompt or a JSON schema (or both) — clean, typed data instead of raw HTML. urls (array) is required. Use cases: comparing competitor pricing, pulling product specs from e-commerce pages, extracting contact info from company pages, building datasets from structured web pages.

curl -X POST https://vaaya.ai/api/run/firecrawl/extract \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com/pricing", "https://competitor.com/pricing"],
    "prompt": "Extract pricing tiers: name, monthly price, annual price, key features"
  }'

crw — the Firecrawl-compatible second vendor

Price

1¢ scrape/search/map · 10¢ crawl · 5¢ extract

Latency

sync (scrape/search/map) · async (crawl, extract)

Polling

crw/crawl_status · crw/extract_status — 1¢ per poll

CRW mirrors the Firecrawl request shapes (url, formats, onlyMainContent, waitFor) at the same 1¢ scrape price, so the two are interchangeable for routine scraping. Use CRW as the fall-through when Firecrawl errors — or first when you need its extra knobs: stealth: true (anti-bot browser), country (2-letter proxy egress), renderJs (null = auto-detect), jsonSchema (structured output straight from a scrape).

• crw/scrape (1¢, sync) — single URL → markdown/HTML/JSON.
• crw/search (1¢, sync) — web search, optionally scraping the hits in one call (scrapeOptions). tbs (qdr:h|d|w|m|y) filters by freshness; sources picks web, news, or images.
• crw/map (1¢, sync) — discover a site’s URLs without scraping (sitemap + fallback).
• crw/crawl (10¢, async) — multi-page crawl from a seed URL. One 10¢ price covers the whole job (≤100 pages). This is the crawl whose results you can actually retrieve — prefer it over firecrawl/crawl when the crawled pages themselves are the deliverable.
• crw/extract (5¢, async) — LLM extraction over up to 10 URLs with a prompt and/or JSON schema; basis: true adds per-field source evidence. Read results[].data when complete.

# 1) start the crawl — returns { id } immediately
curl -X POST https://vaaya.ai/api/run/crw/crawl \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://docs.example.com", "maxPages": 20, "maxDepth": 2}'

# 2) poll (1¢ per poll) until status is "completed" — data holds the pages
curl -X POST https://vaaya.ai/api/run/crw/crawl_status \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"id": "crawl_..."}'

Gotchas

• Crawl and extract bill at launch — the status polls are separate 1¢ calls, so don’t poll more than every few seconds.
• The vendor bills per page on crawls — always set maxPages small first and keep it tight.

scrapingant — residential proxies + headless Chrome for hard pages

Price

1¢ (scrape, markdown) · 4¢ (scrape_residential) · 20¢ (extract)

Latency

slow (~5s p50, long tail)

Proxy pool

3M+ rotating IPs, 100+ countries

The escalation rung when normal scraping bounces: Cloudflare or anti-bot walls, geo-fenced content, sites that block datacenter IPs. A managed headless-Chrome cluster over a rotating residential-proxy pool. All actions share the same knobs: url (required), browser (default true), wait_for_selector, js_snippet (base64 JS run on the page), proxy_country (2-letter), timeout (5–60s), cookies, return_page_source.

• scrapingant/scrape (1¢) — Chrome-rendered page over datacenter proxies → { raw: "<html>…" }. The cheap rung; try it before residential.
• scrapingant/scrape_residential (4¢) — the same call through the residential pool. Escalate here only after a plain scrape (Firecrawl / CRW / ScrapingAnt datacenter) fails.
• scrapingant/markdown (1¢) — scrape → LLM-ready markdown, when you want the proxy pool behind a clean-text fetch.
• scrapingant/extract (20¢) — AI extraction without a schema: pass extract_properties as plain English, e.g. product title, price(number), reviews(list: review title, review content) → JSON. The priciest extraction rung — prefer crw/extract (5¢, schema-driven) unless you specifically want the no-schema form or the proxy pool.

curl -X POST https://vaaya.ai/api/run/scrapingant/scrape_residential \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://blocked-site.com", "proxy_country": "de"}'

Gotchas

• scrape and scrape_residential return raw HTML, not cleaned markdown — use the markdown action when feeding an LLM.
• It serializes browser renders and returns 409 under load — send one page per call and retry serially, never batch through it.

tavily/extract — batch content from known URLs

Price

1¢ per call (≤5 URLs)

Latency

very fast on repeat URLs (vendor cache)

Batch content from URLs you already have: pass urls (≤5) and an optional format (markdown default, or text). The cheapest batch rung after exa/contents (0.1¢/URL, no JS) when the pages need JS handling; for bigger batches, make multiple calls.

curl -X POST https://vaaya.ai/api/run/tavily/extract \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com/a", "https://example.com/b"]}'

apify — platform scraping catalog

Price

≈ maxItems × per-result rate, 1¢ minimum

Latency

synchronous, ~10–15s per run

When the target is a specific platform’s structured data — social posts, profiles, comments, followers, marketplace listings, reviews, map places — route to an apify action rather than Firecrawl or Exa. Firecrawl is for arbitrary web pages; Exa is for search; Apify actors return typed rows for a named platform. All actions take an identifier param plus an optional maxItems cap (bounded per action; the cap also sets the price), run synchronously, and return the dataset rows.

curl -X POST https://vaaya.ai/api/run/apify/tweets \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"searchTerms": ["ai agents"], "maxItems": 25}'

Gotchas

• Keep maxItems small — it directly sets both cost and latency.
• Feed the right identifier per platform: URLs vs usernames vs search terms — see the table.
• Fewer rows may return than requested; you pay for the requested cap.