Crawl site for AI agents
Start from a URL and follow links to collect pages from a website.
from 1¢
What it's for
Crawl a whole site starting from one seed URL, following its links.
Good for indexing docs, a product catalog, or a competitor site.
Cap how many pages it follows so it doesn't run away.
Comes back as clean markdown per page, ready to read or search.
How to ask your agent
Connect Vaaya, then just say it in plain language. For example:
❯Use Vaaya to crawl docs.stripe.com/payments, up to 20 pages.
❯Use Vaaya to crawl our competitor's whole docs site and summarize what changed.
What you get back
{
"status": "completed",
"total": 14,
"data": [
{ "markdown": "# Payments\n\n…", "metadata": { "sourceURL": "https://docs.stripe.com/payments", "title": "Payments" } },
{ "markdown": "# Payment methods\n\n…", "metadata": { "sourceURL": "https://docs.stripe.com/payments/payment-methods", "title": "Payment methods" } }
]
}
Pricing
from ~1¢ per page crawled. Charges are shown in cents and recorded in your transaction history. Set a maximum cost for each call. Failed calls are not charged.
Try it with your agent.
Connect Vaaya and describe what you want to do. Your agent can find a suitable service and call it within your budget.
npx @vaaya/mcp install