vaaya / docs / reference

Web scraping

Scrape, crawl, and extract content from the web in vendor tiers: Exa (cheapest, no JS), Firecrawl and CRW (JS rendering plus a full scrape/crawl/map/extract platform, with interchangeable request shapes), Tavily extract (the batch rung for known URLs), ScrapingAnt (headless Chrome behind a residential-proxy pool for hard pages), Oxylabs (geo-targeted scraping by country, state, or city), and Apify (typed rows from named platforms). 2Captcha solves the captchas the ladder runs into. Start with the cheapest option that works; escalate only when you need JS, multi-page, structured extraction, or the residential-proxy rung.

scraping/scrape: the category endpoint

Price

the resolved provider’s price (1¢ today)

Providers

crw, firecrawl

Resolution

deterministic. Pin a provider, or priority order. No model in the path

scraping/scrape gives you one request shape that works across scraping providers. Vaaya picks the provider unless you pin one with provider. Unpinned calls go to the first configured provider, crw then firecrawl. The response names the provider that served the call in data.provider.

curl -X POST https://vaaya.ai/api/run/scraping/scrape \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["markdown"]}'

Automatic provider routing is planned. Until then, resolution follows the fixed order above. Every vendor action below stays directly callable with its full native params.

When to use which

The escalation ladder within scraping: exa/contents → firecrawl/crw scrape → scrapingant/scrape → scrapingant/scrape_residential → a browser session (only if you must click or log in). When the block is regional rather than anti-bot, oxylabs/scrape with geo_location is the parallel rung: it exits from a chosen country, state, or city. For a managed version of this ladder in one call, pass facets: ["scrape"] to OneSearch.

Documented prices are defaults or ranges. Check GET /api/run/{service}/{action} before relying on a price in production.

Measured performance

From a 12-page, 5-vendor benchmark (August 2026; pages included JS SPAs and bot-hostile marketing pages):

• Known-static pages → exa/contents (0.1¢); anything possibly JS-rendered → firecrawl/scrape.
• Slow-tail pages differ per vendor. Racing firecrawl + crw on latency-critical scrapes cut p95 to 0.2–0.6s vs 2–26s single-vendor tails.
• CRW is coverage-equal but never faster. Use it when Firecrawl errors, not for speed.
• ScrapingAnt is for Cloudflare / anti-bot / geo-fenced pages the cheap rungs bounce off. Send it one page at a time and retry on 409, never batch through it.

exa/contents

Price

0.1¢ per (URL + id) × per content field, min 1¢

Latency

fast (~0.1s warm)

JS rendering

no

Fetch and clean text from URLs (or Exa document IDs) you already have. It is the cheapest text-extraction option in the catalog when JS rendering isn’t needed. Use it for pulling article bodies for summarization, extracting docs from URLs returned by a previous exa/search, or batch-fetching many pages in one call.

curl -X POST https://vaaya.ai/api/run/exa/contents \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com/article", "https://example.com/about"],
    "text": true
  }'

Provide urls (full URLs) or ids (Exa document IDs from exa/search), or both. Each content field you request (text, highlights, summary) multiplies the cost, ceiling-rounded to whole cents: 1 URL with text → 1¢ (minimum charge); 10 URLs with text → 1¢; 10 URLs with text + highlights + summary → 3¢.

Gotchas

• Asking for highlights and summary on top of text triples the cost without much marginal value if you’re feeding the page into an LLM anyway. Pick the minimum field set.
• Cold fetches of JS-heavy pages can return thin content (the extraction is cache-dependent). Escalate those to firecrawl/scrape.

firecrawl/scrape

Price

1¢

Latency

medium (~0.3s p50)

Output

clean markdown or HTML

Scrape a single URL and return clean markdown or HTML. Handles JS rendering, anti-bot bypassing, and content cleaning automatically. It is the go-to when you need clean, structured output from one page.

curl -X POST https://vaaya.ai/api/run/firecrawl/scrape \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "formats": ["markdown"],
    "onlyMainContent": true,
    "waitFor": 3000
  }'

• url is required.
• formats sets the output formats, default ["markdown"]; also html.
• onlyMainContent strips navigation, footers, sidebars.
• waitFor is milliseconds to wait for JS to finish before scraping.

Vs. Exa: use exa/contents (0.1¢/URL) when you don’t need JS rendering; use Firecrawl when you want boilerplate-stripped markdown or JS / anti-bot handling.

firecrawl/crawl, firecrawl/map, firecrawl/search

Price

1¢ each

Latency

medium (map, search) · slow (crawl, multi-page)

firecrawl/crawl crawls an entire website from a seed url, following links. Good for indexing documentation sites or scraping product catalogs. limit caps total pages; maxDepth controls link hops from the seed.

curl -X POST https://vaaya.ai/api/run/firecrawl/crawl \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://docs.stripe.com/payments", "limit": 20, "maxDepth": 2}'

• Always set limit to avoid runaway crawls on large sites. Start with 10–20 pages, check results, then increase.
• firecrawl/map ({ url }) discovers all URLs on a site without scraping content. It is a reconnaissance step: map first, then firecrawl/scrape the specific pages you need.
• firecrawl/search ({ query, limit }) combines web search with scraping in one call: full page content from the results, not just snippets. Use exa/search (1¢) for quick semantic search with snippets; use this when you need the content with the hits.

firecrawl/extract

Price

1¢

Latency

medium

Extract structured data from URLs using a natural-language prompt or a JSON schema (or both). You get clean, typed data instead of raw HTML. urls (array) is required. Use cases: comparing competitor pricing, pulling product specs from e-commerce pages, extracting contact info from company pages, building datasets from structured web pages.

curl -X POST https://vaaya.ai/api/run/firecrawl/extract \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com/pricing", "https://competitor.com/pricing"],
    "prompt": "Extract pricing tiers: name, monthly price, annual price, key features"
  }'

crw: the Firecrawl-compatible second vendor

Price

1¢ scrape/search/map · 10¢ crawl · 5¢ extract

Latency

sync (scrape/search/map) · async (crawl, extract)

Polling

crw/crawl_status · crw/extract_status, 1¢ per poll

CRW mirrors the Firecrawl request shapes (url, formats, onlyMainContent, waitFor) at the same 1¢ scrape price, so the two are interchangeable for routine scraping. Use CRW as the fall-through when Firecrawl errors, or first when you need its extra knobs: stealth: true (anti-bot browser), country (2-letter proxy egress), renderJs (null = auto-detect), jsonSchema (structured output straight from a scrape).

• crw/scrape (1¢, sync): single URL → markdown/HTML/JSON.
• crw/search (1¢, sync): web search, optionally scraping the hits in one call (scrapeOptions). tbs (qdr:h|d|w|m|y) filters by freshness; sources picks web, news, or images.
• crw/map (1¢, sync): discover a site’s URLs without scraping (sitemap + fallback).
• crw/crawl (10¢, async): multi-page crawl from a seed URL. One 10¢ price covers the whole job (≤100 pages). This is the crawl whose results you can actually retrieve. Prefer it over firecrawl/crawl when the crawled pages themselves are the deliverable.
• crw/extract (5¢, async): LLM extraction over up to 10 URLs with a prompt and/or JSON schema; basis: true adds per-field source evidence. Read results[].data when complete.

# 1) start the crawl: returns { id } immediately
curl -X POST https://vaaya.ai/api/run/crw/crawl \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://docs.example.com", "maxPages": 20, "maxDepth": 2}'

# 2) poll (1¢ per poll) until status is "completed": data holds the pages
curl -X POST https://vaaya.ai/api/run/crw/crawl_status \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"id": "crawl_..."}'

Gotchas

• Crawl and extract bill at launch. The status polls are separate 1¢ calls, so don’t poll more than every few seconds.
• The vendor bills per page on crawls. Always set maxPages small first and keep it tight.

scrapingant: residential proxies + headless Chrome for hard pages

Price

1¢ (scrape, markdown) · 4¢ (scrape_residential) · 20¢ (extract)

Latency

slow (~5s p50, long tail)

Proxy pool

3M+ rotating IPs, 100+ countries

The escalation rung when normal scraping bounces: Cloudflare or anti-bot walls, geo-fenced content, sites that block datacenter IPs. A managed headless-Chrome cluster over a rotating residential-proxy pool. All actions share the same knobs: url (required), browser (default true), wait_for_selector, js_snippet (base64 JS run on the page), proxy_country (2-letter), timeout (5–60s), cookies, return_page_source.

• scrapingant/scrape (1¢): Chrome-rendered page over datacenter proxies → { raw: "<html>…" }. The cheap rung; try it before residential.
• scrapingant/scrape_residential (4¢): the same call through the residential pool. Escalate here only after a plain scrape (Firecrawl / CRW / ScrapingAnt datacenter) fails.
• scrapingant/markdown (1¢): scrape → LLM-ready markdown, when you want the proxy pool behind a clean-text fetch.
• scrapingant/extract (20¢): AI extraction without a schema. Pass extract_properties as plain English, e.g. product title, price(number), reviews(list: review title, review content) → JSON. The priciest extraction rung; prefer crw/extract (5¢, schema-driven) unless you specifically want the no-schema form or the proxy pool.

curl -X POST https://vaaya.ai/api/run/scrapingant/scrape_residential \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://blocked-site.com", "proxy_country": "de"}'

Gotchas

• scrape and scrape_residential return raw HTML, not cleaned markdown. Use the markdown action when feeding an LLM.
• It serializes browser renders and returns 409 under load. Send one page per call and retry serially, never batch through it.

oxylabs/scrape: geo-targeted scraping

Price

pass-through, capped at 25¢

Params

url · geo_location · render

Settlement

MPP, no vendor API key

Scrape a public URL through Oxylabs with a chosen egress location. Pass url, an optional geo_location (country, state, or city targeting), and render: "html" when the page needs JS rendering. Use it when a page serves different content per region, or when the block is geographic rather than bot detection. The price is dynamic pass-through: you pay what the request actually costs upstream, capped at 25¢.

curl -X POST https://vaaya.ai/api/run/oxylabs/scrape \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/pricing", "geo_location": "Germany", "render": "html"}'

twocaptcha: solve captchas

Price

~0.3¢ per solve, pass-through

Flow

twocaptcha/solve submits · twocaptcha/result polls

Settlement

MPP, no vendor API key

Submit a captcha task and poll for the answer. Supports reCAPTCHA, Turnstile, hCaptcha, and image captchas. twocaptcha/solve takes a task object in the 2Captcha createTask shape and returns a taskId. Poll twocaptcha/result with that taskId until the solution is ready.

# 1) submit the task, returns { taskId }
curl -X POST https://vaaya.ai/api/run/twocaptcha/solve \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"task": {"type": "RecaptchaV2TaskProxyless", "websiteURL": "https://example.com", "websiteKey": "..."}}'

# 2) poll until status is "ready"
curl -X POST https://vaaya.ai/api/run/twocaptcha/result \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"taskId": 123456789}'

tavily/extract: batch content from known URLs

Price

1¢ per call (≤5 URLs)

Latency

very fast on repeat URLs (vendor cache)

Batch content from URLs you already have: pass urls (≤5) and an optional format (markdown default, or text). The cheapest batch rung after exa/contents (0.1¢/URL, no JS) when the pages need JS handling; for bigger batches, make multiple calls.

curl -X POST https://vaaya.ai/api/run/tavily/extract \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com/a", "https://example.com/b"]}'

apify: platform scraping catalog

Price

≈ maxItems × per-result rate, 1¢ minimum

Latency

synchronous, ~10–15s per run

When the target is a specific platform’s structured data (social posts, profiles, comments, followers, marketplace listings, reviews, map places), route to an apify action rather than Firecrawl or Exa. Firecrawl is for arbitrary web pages; Exa is for search; Apify actors return typed rows for a named platform. All actions take an identifier param plus an optional maxItems cap (bounded per action; the cap also sets the price), run synchronously, and return the dataset rows.

curl -X POST https://vaaya.ai/api/run/apify/tweets \
  -H "Authorization: Bearer $VAAYA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"searchTerms": ["ai agents"], "maxItems": 25}'

Gotchas

• Keep maxItems small. It directly sets both cost and latency.
• Feed the right identifier per platform: URLs vs usernames vs search terms. See the table.
• Fewer rows may return than requested; you pay for the requested cap.