Best way seriesNovember 16, 2026
The best way to scrape a website with an AI agent
Start with vaaya/onescrape at 2¢ per URL. Send blocked pages and whole sites to vaaya/onescrape-deep with a budget, and pay only for the pages it reads.
Start with vaaya/onescrape. It costs 2¢ per URL for up to five URLs and tries a no-JavaScript fetch, then a JavaScript renderer, then a low-cost anti-bot fetch, so one call covers ordinary pages and JavaScript pages. Send pages that still come back blocked, longer lists and whole sites to vaaya/onescrape-deep, which adds harder scrapers under a budget you set and charges only for pages it reads.
What best means for scraping
- You get the real page. A blocked scrape can return HTTP 200 with a challenge page in the body, so judge the content.
- You pay for the rung the page needs. Ordinary pages do not need an unblocker. Start on the cheap rungs and climb only when a page bounces.
- A crawl has a ceiling. A cap on pages and a cap on spend, set before it starts.
The scrapers in the catalog
Prices are as listed on 16 November 2026. The catalog has current prices.
| Tool | What it returns | Price per call | When to pick it |
|---|---|---|---|
vaaya/onescrape |
A row per URL: title, content as markdown or HTML, and the provider that read it | 2¢ per URL, 1 to 5 URLs | The default for URLs you already have |
vaaya/onescrape-deep |
The same rows for up to 50 URLs, or a site of up to 50 pages. Asynchronous | Holds budgetCents (10 to 500), or 10¢ per URL by default. Charges only the rung that read each page |
Blocked pages, long lists and whole sites |
exa/contents |
Text for URLs you already have. No JavaScript rendering | 0.1¢ per URL per content field, rounded up to a whole cent, 1¢ minimum | Static pages in bulk |
firecrawl/scrape |
Clean markdown or HTML for one URL, JavaScript rendered | 1¢ | One page, with format options such as links or a screenshot |
crw/scrape |
Markdown, HTML or JSON for one URL | 0.069¢ | When Firecrawl errors, or for stealth and country |
scrapedo/scrape |
Raw HTML, fetched through a rotating proxy pool with anti-bot handling | 1¢ | The first escalation when a reader comes back blocked |
scrapedo/scrape_super |
Raw HTML through residential and mobile proxies, with full JavaScript rendering | 2¢ | The plain scrapedo/scrape call bounced |
scrapingant/scrape_residential |
Raw HTML through a residential proxy pool and headless Chrome | 2.375¢ | A second opinion, as the last scraper rung |
olostep/crawl |
An asynchronous crawl of up to 100 pages, read back with olostep/crawl-pages |
1¢ per 4 pages of max_pages, rounded up. Polls and page reads are free |
A crawl priced by its page cap |
The default path
URLs you already have. Call vaaya/onescrape with up to five. A row no cheap rung could read comes back with content: null and an error such as blocked.
Blocked pages and longer lists. Call vaaya/onescrape-deep. It runs the same cheap rungs, then the unblock rungs, which cost 1¢ to 4¢ a page and include scrapingant/scrape_residential and crw/scrape with stealth. The call returns a job_id, and the agent polls result for the table. Calling it again to check starts a second job and a second hold.
One vendor on purpose. Pick a single scraper for something OneScrape does not expose, such as a screenshot (firecrawl/scrape) or a proxy country (crw/scrape with country). Escalating by hand, go exa/contents, then firecrawl/scrape or crw/scrape, then scrapedo/scrape, then scrapedo/scrape_super, with scrapingant/scrape_residential last.
Pages no scraper can read. A content: null row from the deep tier means every rung bounced. Use a browser session, as in how to give an agent a browser. Pages that need a click or a login go there too.
How to cap a crawl
Map first. crw/map (0.069¢) or firecrawl/map (1¢) lists a site's URLs, so you can see how large a section is. Then set two caps.
Pages. In vaaya/onescrape-deep, pass site with max_pages (1 to 50, default 20) and include or exclude URL patterns. In olostep/crawl, max_pages is required and sets the price.
Money. budgetCents is the most a deep job can charge. Without it, the job reserves 10¢ per page plus 1¢ for the map, and that hold has to fit in your balance. The unused part is released when the job finishes, and pages the budget could not cover come back marked over budget.
Start with 10 to 20 pages, check the rows, then raise the cap.
A worked example
This is arithmetic at listed prices on 16 November 2026. The job: read five competitor pricing pages, then pull the guides section of one docs site.
{
"service": "vaaya",
"action": "onescrape",
"params": {
"urls": [
"https://a.example/pricing",
"https://b.example/pricing",
"https://c.example/pricing",
"https://d.example/pricing",
"https://e.example/pricing"
]
},
"max_cost_cents": 10
}
Five URLs at 2¢ is 10¢. Suppose four rows come back with content and one comes back blocked. The price is still 10¢, because the flat price is per URL sent.
Send the blocked URL to vaaya/onescrape-deep with budgetCents: 10 and max_cost_cents: 10. The job holds 10¢ and charges for the rung that reads the page: 1¢ to 4¢, or 0¢ if no rung can.
Then the docs section, capped at 20 pages and 60¢:
{
"service": "vaaya",
"action": "onescrape-deep",
"params": {
"site": { "url": "https://docs.example.com", "max_pages": 20, "include": ["/guides/*"] },
"budgetCents": 60
},
"max_cost_cents": 60
}
The map is 1¢. If every page reads on a 1¢ rung, the 20 pages cost at most 20¢, so the job is 21¢ or less. It cannot charge more than 60¢.
If the blocked page needs the 4¢ rung and every docs page reads at 1¢, the whole job comes to 10¢ + 4¢ + 21¢ = 35¢.
How to run it
Connect your agent with the setup guides at /install, then ask in plain English:
Read these five pricing pages and list each plan name and price as shown. If a page comes back blocked, retry it with the deep scraper and a 10 cent budget. Do not crawl beyond these URLs.
The agent can ask Vaaya's consult tool to plan the calls, and it gets back the calls to run with their prices. Every call carries max_cost_cents. For a deep job that number has to cover the hold. For a tool charged below a whole cent, such as crw/scrape, Vaaya still checks and holds the whole-cent price first (1¢, or 4¢ for scrapingant/scrape_residential), so max_cost_cents must be at least that.
What to check, and where this falls short
- The body. Treat a body of only a few kilobytes, or one containing
Just a moment...orDataDome, as a failed scrape. OneScrape checks this before it counts a rung as a hit. With a single scraper, check it yourself. - The row.
providerandhopsshow which rung read the page and what each attempt cost.errorsays why a row is empty, for exampleblockedorover budget. - Refused URLs. OneScrape refuses social platform URLs and documents such as PDFs, without charge.
For typed fields instead of a page, diffbot/analyze (0.1¢) returns fields such as title, author and date, but it does not get past bot protection. The extraction walkthrough covers pulling fields from one page and checking them. Scrape public content only.
Why pay per call suits scraping
Scraping cost is uneven: ordinary pages read on a 1¢ rung, bot-protected ones can need a 4¢ one, and you cannot tell which until you try. Paying per page means only the hard pages cost more, and on the deep tier a page no rung could read costs nothing. A crawl is also the job most likely to run away, so stack the caps: a page limit, budgetCents on the job, and a daily, weekly or monthly limit on the agent's key (see spending controls).
Questions
What should an agent try first when it needs the content of a page?
vaaya/onescrape. It takes 1 to 5 URLs at a flat 2¢ per URL and tries a no-JavaScript fetch, then a JavaScript renderer, then a low-cost anti-bot fetch, so one call covers ordinary pages and JavaScript pages.
How does an agent scrape a page that blocks bots?
Send the URL to vaaya/onescrape-deep with a budget. It adds the unblock rungs, which cost 1¢ to 4¢ a page, and charges only for the rung that read the page. A page no rung could read costs nothing, and the next step is a browser session.
How do I cap what a crawl can cost?
Set a page cap and a money cap. In vaaya/onescrape-deep, site.max_pages limits the pages (1 to 50) and budgetCents is the most the job can charge. In olostep/crawl, max_pages is required and sets the price at 1¢ per 4 pages.
Am I charged for a page the scraper could not read?
On vaaya/onescrape-deep, no: a blocked row costs 0¢. On vaaya/onescrape the price is 2¢ per URL sent, and the call is not charged if every URL comes back blocked. Prices are as listed on 16 November 2026.