# Jev × Vaaya: frozen routing test

All prompts are synthetic and authored for this test. There is no private inbox,
customer data, or answer generated by a model in the fixture. All inference,
including Jev, goes through Vaaya's public catalog routes.

Read protocol.json before interpreting results. The 50 prompts run three times
per arm. The rubric measures explicit task constraints, not general model quality.
The test's rule is lower total customer charges and at most a 5 percentage point
pass-rate loss. Repeated prompts do not provide 150 independent tasks.

## Reproduce

Requires Node 20+, an existing Vaaya account with sufficient balance, and a Vaaya
access token in VAAYA_ACCESS_TOKEN (keep it private). The runner can instead use
an installed Vaaya auth module via VAAYA_AUTH_MODULE; its getAccessToken export
uses the normal account authorization and refresh path. No OpenRouter API key is
needed by the runner. Authentication goes only to https://vaaya.ai.

1. Copy these files to a local directory.
2. Review protocol.json and prompts.json; change run_id for your own run.
3. Run `node --test benchmark.test.mjs`.
4. Set JEV_OUTPUT_DIR to a new output directory, then `node benchmark.mjs`.
5. Keep calls.jsonl, answers.jsonl, manifest.json, and complete.json.

The run refuses to append to an existing calls file. Idempotency keys identify
repeat, prompt, arm, and stage; do not change keys when reconciling a timeout.
Do not restart a partial run blindly. The $12 stop threshold is a client-side
check; concurrent in-flight calls can finish above it. Individual chat calls
have a $1 cap, and Jev calls a 2¢ caller cap. The original run used the pricing
available at execution time. Current Jev pricing is metered usage + 3%;
`charged_cents` alone no longer describes its accrued price. Review limits before
running. Four independent paired-prompt workers run concurrently.

## What is recorded

Every call: run ID, prompt ID, arm, stage, time, elapsed milliseconds, HTTP status,
Vaaya transaction ID and charged cents, plus the provider's returned payload.
Account balance and credentials are deliberately omitted. Answer rows add final
JSON, individual rubric checks, policy decision, and any fallback. Provider
usage.cost is a separate number from Vaaya's customer charge.

Dashboard filter: /transactions?agent=jev-20260919-50x3-v1, signed in to the
account that made the calls. The run ID is passed in X-Vaaya-Agent. The dashboard
shows its latest 100 matching rows; the downloadable log contains the whole run.
A ledger receipt proves a recorded call/charge, not that its answer is correct.

## Analyze the measured run

Run `node analyze.mjs` to reproduce summary.json from the recorded calls and
answers. Provider-reported usage totals $0.167287 for treatment versus $0.250420
for control. The recorded customer bill was $1.64 versus $0.18, so the original
acceptance rule failed. Some answer calls returned zero-cent charges through
MPP; provider usage and customer charges are separate measurements.

The original runner and protocol are preserved as executed. For a future live
run, account for `data.vaaya_billing.price_microusd` on Jev responses as well
as the answer calls' `charged_cents`; count aggregate settlement only once.
Use a new run ID and freeze the accounting before running.

## Agreed Jev pricing

Vaaya Jev pricing is provider usage + 3%, with no per-call cent minimum.
`node price-jev.mjs` reproduces jev-pricing.json: $0.005151 for the 150 saved
router calls, averaging $0.00003434 per call. This router-only calculation is
not a newly recorded transaction or a full customer-bill comparison.
The separate live verification in pricing-verification.json accrued $0.000032.
It is not part of the benchmark. No full benchmark rerun at this rate is claimed.

To regenerate the chart, run `python render-results.py` with matplotlib installed.
Three panels show measured results; the pricing panel shows the separate live
verification and the router-only calculation at the agreed rate.

## Demonstration artifacts

- explore.html: interactive walkthrough of actual saved prompts, decisions,
  model answers, grades, provider usage and calculated Jev pricing.
  Historical receipts remain accessible in the full responses and downloads.
- dashboard.png: original benchmark dashboard capture, with historical amounts.
- results.png / results.svg: measured provider usage, p95 latency and frozen
  checks, alongside the separately labeled current Jev pricing.
- pricing-dashboard.png: actual separate $0.000032 metered-price verification.

- story.mp4 / story-poster.png: narrated motion graphic of the measured results,
  with the actual dashboard capture.

No synthetic model answers or fabricated dashboard visuals are used.