# The best way to scrape a website with an AI agent

_By Apoorv Khanna, November 16, 2026_

Start with `vaaya/onescrape`. It costs 2¢ per URL for up to five URLs and tries a no-JavaScript fetch, then a JavaScript renderer, then a low-cost anti-bot fetch, so one call covers ordinary pages and JavaScript pages. Send pages that still come back blocked, longer lists and whole sites to `vaaya/onescrape-deep`, which adds harder scrapers under a budget you set and charges only for pages it reads.

## What best means for scraping

- **You get the real page.** A blocked scrape can return HTTP 200 with a challenge page in the body, so judge the content.
- **You pay for the rung the page needs.** Ordinary pages do not need an unblocker. Start on the cheap rungs and climb only when a page bounces.
- **A crawl has a ceiling.** A cap on pages and a cap on spend, set before it starts.

## The scrapers in the catalog

Prices are as listed on 16 November 2026. The [catalog](/catalog) has current prices.

| Tool | What it returns | Price per call | When to pick it |
| --- | --- | --- | --- |
| `vaaya/onescrape` | A row per URL: title, content as markdown or HTML, and the provider that read it | 2¢ per URL, 1 to 5 URLs | The default for URLs you already have |
| `vaaya/onescrape-deep` | The same rows for up to 50 URLs, or a site of up to 50 pages. Asynchronous | Holds `budgetCents` (10 to 500), or 10¢ per URL by default. Charges only the rung that read each page | Blocked pages, long lists and whole sites |
| `exa/contents` | Text for URLs you already have. No JavaScript rendering | 0.1¢ per URL per content field, rounded up to a whole cent, 1¢ minimum | Static pages in bulk |
| `firecrawl/scrape` | Clean markdown or HTML for one URL, JavaScript rendered | 1¢ | One page, with format options such as links or a screenshot |
| `crw/scrape` | Markdown, HTML or JSON for one URL | 0.069¢ | When Firecrawl errors, or for `stealth` and `country` |
| `scrapedo/scrape` | Raw HTML, fetched through a rotating proxy pool with anti-bot handling | 1¢ | The first escalation when a reader comes back blocked |
| `scrapedo/scrape_super` | Raw HTML through residential and mobile proxies, with full JavaScript rendering | 2¢ | The plain `scrapedo/scrape` call bounced |
| `scrapingant/scrape_residential` | Raw HTML through a residential proxy pool and headless Chrome | 2.375¢ | A second opinion, as the last scraper rung |
| `olostep/crawl` | An asynchronous crawl of up to 100 pages, read back with `olostep/crawl-pages` | 1¢ per 4 pages of `max_pages`, rounded up. Polls and page reads are free | A crawl priced by its page cap |

## The default path

**URLs you already have.** Call `vaaya/onescrape` with up to five. A row no cheap rung could read comes back with `content: null` and an `error` such as `blocked`.

**Blocked pages and longer lists.** Call `vaaya/onescrape-deep`. It runs the same cheap rungs, then the unblock rungs, which cost 1¢ to 4¢ a page and include `scrapingant/scrape_residential` and `crw/scrape` with `stealth`. The call returns a `job_id`, and the agent polls `result` for the table. Calling it again to check starts a second job and a second hold.

**One vendor on purpose.** Pick a single scraper for something OneScrape does not expose, such as a screenshot (`firecrawl/scrape`) or a proxy country (`crw/scrape` with `country`). Escalating by hand, go `exa/contents`, then `firecrawl/scrape` or `crw/scrape`, then `scrapedo/scrape`, then `scrapedo/scrape_super`, with `scrapingant/scrape_residential` last.

**Pages no scraper can read.** A `content: null` row from the deep tier means every rung bounced. Use a browser session, as in [how to give an agent a browser](/blog/how-to-give-an-agent-a-browser-with-vaaya). Pages that need a click or a login go there too.

## How to cap a crawl

Map first. `crw/map` (0.069¢) or `firecrawl/map` (1¢) lists a site's URLs, so you can see how large a section is. Then set two caps.

**Pages.** In `vaaya/onescrape-deep`, pass `site` with `max_pages` (1 to 50, default 20) and `include` or `exclude` URL patterns. In `olostep/crawl`, `max_pages` is required and sets the price.

**Money.** `budgetCents` is the most a deep job can charge. Without it, the job reserves 10¢ per page plus 1¢ for the map, and that hold has to fit in your balance. The unused part is released when the job finishes, and pages the budget could not cover come back marked `over budget`.

Start with 10 to 20 pages, check the rows, then raise the cap.

## A worked example

This is arithmetic at listed prices on 16 November 2026. The job: read five competitor pricing pages, then pull the guides section of one docs site.

```json
{
  "service": "vaaya",
  "action": "onescrape",
  "params": {
    "urls": [
      "https://a.example/pricing",
      "https://b.example/pricing",
      "https://c.example/pricing",
      "https://d.example/pricing",
      "https://e.example/pricing"
    ]
  },
  "max_cost_cents": 10
}
```

Five URLs at 2¢ is 10¢. Suppose four rows come back with content and one comes back blocked. The price is still 10¢, because the flat price is per URL sent.

Send the blocked URL to `vaaya/onescrape-deep` with `budgetCents: 10` and `max_cost_cents: 10`. The job holds 10¢ and charges for the rung that reads the page: 1¢ to 4¢, or 0¢ if no rung can.

Then the docs section, capped at 20 pages and 60¢:

```json
{
  "service": "vaaya",
  "action": "onescrape-deep",
  "params": {
    "site": { "url": "https://docs.example.com", "max_pages": 20, "include": ["/guides/*"] },
    "budgetCents": 60
  },
  "max_cost_cents": 60
}
```

The map is 1¢. If every page reads on a 1¢ rung, the 20 pages cost at most 20¢, so the job is 21¢ or less. It cannot charge more than 60¢.

If the blocked page needs the 4¢ rung and every docs page reads at 1¢, the whole job comes to 10¢ + 4¢ + 21¢ = 35¢.

## How to run it

Connect your agent with the setup guides at [/install](/install), then ask in plain English:

> Read these five pricing pages and list each plan name and price as shown. If a page comes back blocked, retry it with the deep scraper and a 10 cent budget. Do not crawl beyond these URLs.

The agent can ask Vaaya's `consult` tool to plan the calls, and it gets back the calls to run with their prices. Every call carries `max_cost_cents`. For a deep job that number has to cover the hold. For a tool charged below a whole cent, such as `crw/scrape`, Vaaya still checks and holds the whole-cent price first (1¢, or 4¢ for `scrapingant/scrape_residential`), so `max_cost_cents` must be at least that.

## What to check, and where this falls short

- **The body.** Treat a body of only a few kilobytes, or one containing `Just a moment...` or `DataDome`, as a failed scrape. OneScrape checks this before it counts a rung as a hit. With a single scraper, check it yourself.
- **The row.** `provider` and `hops` show which rung read the page and what each attempt cost. `error` says why a row is empty, for example `blocked` or `over budget`.
- **Refused URLs.** OneScrape refuses social platform URLs and documents such as PDFs, without charge.

For typed fields instead of a page, `diffbot/analyze` (0.1¢) returns fields such as title, author and date, but it does not get past bot protection. The [extraction walkthrough](/blog/how-to-extract-web-content-with-vaaya) covers pulling fields from one page and checking them. Scrape public content only.

## Why pay per call suits scraping

Scraping cost is uneven: ordinary pages read on a 1¢ rung, bot-protected ones can need a 4¢ one, and you cannot tell which until you try. Paying per page means only the hard pages cost more, and on the deep tier a page no rung could read costs nothing. A crawl is also the job most likely to run away, so stack the caps: a page limit, `budgetCents` on the job, and a daily, weekly or monthly limit on the agent's key (see [spending controls](/spending-controls)).

## Questions

**What should an agent try first when it needs the content of a page?**

vaaya/onescrape. It takes 1 to 5 URLs at a flat 2¢ per URL and tries a no-JavaScript fetch, then a JavaScript renderer, then a low-cost anti-bot fetch, so one call covers ordinary pages and JavaScript pages.

**How does an agent scrape a page that blocks bots?**

Send the URL to vaaya/onescrape-deep with a budget. It adds the unblock rungs, which cost 1¢ to 4¢ a page, and charges only for the rung that read the page. A page no rung could read costs nothing, and the next step is a browser session.

**How do I cap what a crawl can cost?**

Set a page cap and a money cap. In vaaya/onescrape-deep, site.max_pages limits the pages (1 to 50) and budgetCents is the most the job can charge. In olostep/crawl, max_pages is required and sets the price at 1¢ per 4 pages.

**Am I charged for a page the scraper could not read?**

On vaaya/onescrape-deep, no: a blocked row costs 0¢. On vaaya/onescrape the price is 2¢ per URL sent, and the call is not charged if every URL comes back blocked. Prices are as listed on 16 November 2026.
