Jev routing benchmark: lower provider usage cost, slower answers

Jev routing reduced provider-reported usage cost by 33.2% ($0.167287 vs $0.250420). The recorded customer bill was $1.64 vs $0.18 under the old Jev charge; illustrative repricing under mixed historical billing gives $0.145151 vs $0.18. Pass rates were 80% vs 76% on frozen deterministic checks; p95 latency was 20.07s vs 12.47s (61% slower).

All comparisons above are routed treatment versus frontier-only control. We ran 50 synthetic prompts three times per arm. The recorded run failed its original lower-bill acceptance rule. The provider-usage result includes Jev; the customer repricing is a separate calculation on saved usage. We did not rerun the benchmark at the new price.

Treatment waits for Jev, then calls the answer model. That serial step adds work to the request path. The measured p95 includes both calls and gateway overhead; this test does not isolate how much of the delay came from each component. The pass rates measure formatting and required fields, not general answer quality.

Explore the actual prompts, decisions, answers and receipts. The walkthrough ends on a capture of the real Vaaya dashboard filtered by the run ID.

What does Jev do in a model router?

Jev makes a structured decision about a task before an answer model handles it. It returns choices and probabilities that code can inspect. It does not write the final answer. TypeSafe describes this as a System One model: a focused decision layer around a larger workflow. TypeSafe's explanation also makes a useful distinction: calibration across predictions does not guarantee that an individual decision is correct.

Where Jev’s claims come from

TypeSafe describes Jev’s training method as reinforcement learning for calibrated decisions (RLCD). Its launch post attributes the model’s behavior to that training and a parallel sampler.

LangChain’s Jev introduction reports TypeSafe’s claims of up to 200× faster inference and 400× lower cost than comparable LLMs on classification tasks. Those are vendor claims; our router-plus-answer experiment did not test or confirm those multipliers.

Jev supports choice (select an option), score (rate against ordered levels), and noul (estimate the probability a statement is true). LangChain describes questions within one request as evaluated in parallel and exposes them through TypeSafeClassifier. This test used Vaaya’s OpenRouter integration. Parallel questions inside Jev do not make the later answer-model call parallel with Jev.

Our classifier asks five questions in one request:

We requested typesafe/jev-1.13; the provider returned typesafe/jev-1.13-20260917. Jev's OpenRouter integration uses POST /api/alpha/decisions, with state and questions. Sending its model name to chat completions is the wrong integration. See the official OpenRouter Decisions example.

For this demonstration, we added a bounded Vaaya openrouter/decisions action; the published runner shows the actual request. It uses Vaaya’s existing OpenRouter connection and meters provider usage cost plus 3%, with no per-call cent minimum. The benchmark client needs a Vaaya credential; it never receives the provider key.

How does code keep control of the decision?

Jev recommends; deterministic policy selects the model. We ranked the lanes using their published token rates, then declared eligibility rules before running:

These are the experiment's rules, not claims about a model's guaranteed capabilities. Actual per-call charges can differ from token-rate rankings.

The policy applies high stakes OR low confidence once after the eligibility floor. Both conditions together still raise the lane by only one tier. Frontier is the ceiling. Invalid router output goes directly to frontier.

// Simplified excerpt; the download contains validation and the full policy.
let tier = laneIndex[decision.route];
if (decision.complexity === "deep") tier = 2;
else if (decision.complexity === "multi-step" ||
         decision.needs_tools || decision.needs_verification) {
  tier = Math.max(tier, 1);
}
if (decision.high_stakes || confidence < 0.8) {
  tier = Math.min(2, tier + 1);
}

The full policy treats missing confidence as low confidence. It uses the lower of the route and complexity confidence values, falling back to the selected-choice probability when confidence is absent. The yes/no fields use a 0.5 probability threshold. Inspect the exact runner and policy.

A verification flag raises the lane in this experiment; it does not perform an independent check. Tool permissions, spending ceilings and approval requirements still belong in code. No real tools or financial actions executed in these tests.

Why aren't separate provider bills enough to prove savings?

A routing claim needs to connect the same prompt to the routing decision, every answer attempt, its result and its charge. A monthly provider total cannot establish that connection on its own. Router overhead and failed attempts are easy to omit when the numbers live in separate places.

Vaaya gave this run a common transaction trail. Every request carried X-Vaaya-Agent: jev-20260919-50x3-v1; the local log also recorded the repeat, prompt, arm and stage. That makes each comparison inspectable. A receipt establishes a recorded call and charge. The separate grader establishes whether its answer passed the declared checks.

Provider usage cost and customer price answer different questions. OpenRouter reported $0.250420 of usage for the frontier control and $0.167287 for treatment, including Jev: 33.2% less provider usage cost. Every call returned a usage-cost field. These are provider-reported figures, not the amount debited from a Vaaya account.

The recorded customer bill was $1.64 for treatment versus $0.18 for control. The old flat Jev charge accounted for $1.50 of treatment’s bill. That historical amount remains in the receipts and the original summary.

Illustrative repricing under mixed historical billing gives $0.145151 for treatment versus $0.18 for control, or 19.4% less. It replaces only Jev’s old charge with its current usage-based price. Some answer calls returned a 0¢ customer charge through MPP, and the same frozen control work was billed $0.12, $0.03 and $0.03 across repeats. Those settlement differences affect the comparison. The 19.4% figure cannot establish routing-only customer savings or predict a clean live rerun.

Is Jev really cheaper than one cent per call?

At Vaaya’s current provider-usage-plus-3% rate, the 150 saved Jev calls price at $0.005151 total, averaging $0.00003434 per call (0.0034¢). This recalculates recorded usage; it does not change the original receipts.

OpenRouter lists Jev at $0.042 per million input tokens and $0 per million output tokens. Vaaya rounds provider cost and the marked-up price up to whole micro-dollars, then accumulates them for settlement.

const costMicrousd = Math.ceil(providerUsageCostUsd * 1_000_000);
const priceMicrousd = Math.ceil(costMicrousd * 1.03);
const priceUsd = priceMicrousd / 1_000_000;

What exactly did we test?

We froze 50 synthetic prompts in five buckets, ten each: lookup, rewriting, tool planning, multi-step reasoning and high-stakes policy exercises. The saved run manifest records the start time and SHA-256 hashes of the prompts and deterministic checks, protocol, and runner.

Those hashes match the downloadable files. They let readers verify the benchmark contents against our execution record. This is an author-published record, not an independently timestamped public preregistration.

The control sent each prompt to Claude Opus 4.6. The treatment made one five-question Jev call, applied policy and called the chosen answer model. A failed call, truncated answer or unusable JSON could trigger one higher-lane fallback, except at frontier. Grader failures could not trigger a fallback.

Both arms used the same task text, a 512-token output limit and temperature zero. We ran three repeats with seeded prompt order and randomized arm order within each pair, using four concurrent paired-prompt workers. End-to-end latency includes the router and any fallback; it excludes local queue wait and grading.

Before running, we declared lower total Vaaya charges and no more than a five-percentage-point pass-rate loss as the acceptance rule. The recorded run failed the charge condition. Lower reported provider usage is a separate result; post-run repricing does not turn the original test into a pass. The quality tolerance is descriptive, not a statistical non-inferiority finding. The three repeats reuse 50 tasks; they are not 150 independent tasks.

Read the frozen protocol, download all prompts and checks, or reproduce the run. Four access preflights cost 3¢ in total and are excluded from both arms; their receipts are separate.

What did the recorded experiment show?

Cost per passing answer divides all costs in an arm, including failed answers, by its passing answers. The illustrative mixed-billing repricing is $0.180000 versus $0.145151 overall, or $0.001579 versus $0.001210 per passing answer. It is excluded from the chart’s measured results.

The treatment initially selected 29 fast, 23 balanced and 98 frontier calls. The published thresholds kept substantial traffic on the frontier lane. Download the repricing analysis and recorded metrics.

What failed, and what did escalation miss?

The four-point pass-rate difference includes formatting failures. The control had 33 final answers that could not be parsed as the requested JSON object; treatment had 16. Some contained correct arithmetic followed by JSON, but the consuming program required the entire response to be JSON. We kept the frozen parser and scores. The multi-step bucket passed 0/30 in control and 9/30 in treatment. These checks do not establish a general reasoning advantage. A future test with enforced structured output would be a different experiment.

A valid JSON object can still be wrong. In repeat 1 of multi-step-09, GPT-4.1 Mini selected plan B but returned $7.80. The supplied prices give $6.80: $6 plus 80 extra requests at $0.01. Because the response was parseable, the policy did not call another model. Inspect that actual comparison.

The graders checked exact fields for 40 prompts. For ten rewrites, they checked required strings, forbidden strings and word limits. They did not evaluate every aspect of tone or factual faithfulness. No live research, real tool execution, adversarial prompt suite or production traffic was tested. All failed answers and checks are published.

Can you inspect the actual Vaaya receipts?

Yes. The comparison CSV places each original transaction amount beside the current-price calculation. The original receipt CSV includes each model, provider generation ID and Vaaya transaction ID. Historical receipts retain the amounts recorded when the calls ran; they have not been rewritten to match the repricing. The raw responses and final answers with individual checks are also available. We also cross-checked the latest 200 receipts against the authenticated transactions API; every returned amount and status matched.

Actual dashboard capture after the recorded run, with the account sidebar excluded. Displayed amounts are historical; the current-price analysis is in the linked CSV. The authenticated run filter shows the latest 100 matching rows; the downloadable log contains the full experiment. These are transaction records and provider response IDs, not cryptographically signed attestations.

Questions

Does Jev replace an LLM?

Jev doesn't replace an LLM, it makes the fast typed decisions around it. In this test, Jev classified the task; deterministic code selected an answer model, which generated the response.

Can I use Jev through OpenRouter?

Yes. Jev uses OpenRouter's alpha Decisions API, with model typesafe/jev-1.13. It does not use the chat-completions endpoint. This test called it through Vaaya's openrouter/decisions action.

How much does Jev cost through Vaaya?

Vaaya prices Jev at OpenRouter’s reported usage cost plus 3%, with cost and price rounded up to whole micro-USD and aggregated for settlement. Applying that formula to our 150 saved Jev calls gives $0.005151 total, averaging $0.00003434 per call. This is repricing of recorded usage, not a new benchmark run.

How was answer quality measured?

We froze 50 synthetic prompts and deterministic graders before running. All required checks had to pass, including JSON formatting. The control passed 114 of 150 responses and treatment passed 120 of 150. This measures the published constraints, not general model quality.

Did this Jev benchmark demonstrate a lower customer bill?

No. Provider-reported usage cost was 33.2% lower, but the recorded customer bill was $1.64 for treatment versus $0.18 for control, so the original acceptance rule failed. Applying current Jev pricing gives an illustrative $0.145151 versus $0.18 under mixed historical billing, not an observed customer saving.

Who controls model escalation?

Ordinary code. After applying the task's minimum lane, high stakes or confidence below 0.8 raises it one tier, capped at frontier. A failed call or unusable JSON can trigger one higher-lane fallback. A wrong but parseable answer does not automatically trigger one.