Jev is TypeSafe's model for decisions rather than prose. You send it a state and typed questions, and it returns answers constrained to the options you defined, each with a probability. One of the first things people pointed it at was choosing which LLM should handle a request.
There is a repository called jev-model-router that picks a model from a catalog under quality, cost and latency constraints. Its default setting never calls Jev. On that path the ranking is a deterministic local heuristic over the catalog, and Jev enters only when you switch providers, at which point it acts as a judge over the same catalog. The README does not claim the two paths perform alike, and its bundled benchmark does not test the Jev one, so this settles nothing about which routes better.
What it does is put the division of labour in plain sight. A router has to work out what a request needs and match that against what your models can supply. Jev participates in both, since the criteria you hand it are what it matches against. What it cannot do is supply those descriptions, or tell you whether they are accurate.
What the model decides
LangChain's routing middleware, announced on 17 September, is the cleanest statement of the pattern. You declare named choices, each with a criteria string describing the work it suits, plus an instruction to choose the least costly model that can complete the task safely. One Choice question per request, and the answer with its probabilities and confidence lands in agent state for your code to read.
Udit Goenka, writing up his own migration, counted forty-one decisions in one product going through a text model with a prompt ending in some version of "respond only with JSON", each followed by code whose job was to survive the model ignoring that instruction. A typed interface removes the parsing and the recovery from malformed output. It does not remove semantic validation, network retries or a fallback policy, which are all still yours to write.
The catalog is where the assumptions live
The second job does not come from the request at all. Jev cannot know your models, and TypeSafe says so itself: the same weights serve every account, there is no fine-tuning on customer data, and everything the model knows about your situation arrives in the request. The router repository puts the same constraint in its own terms, saying capabilities come only from the catalog and that the task cannot add them.
So the catalog is where assumptions about model capability enter the system. A criterion like "architecture, novel root-cause reasoning, and high-stakes decisions" tells the router what you intend a model to handle. It does not establish that the model handles it reliably on your workload. Those descriptions can be informed by benchmarks and by what you have seen in production, and they still need testing against outcomes. Every model release gives you a reason to test again. What the public evidence does not tell us is how much that maintenance costs in practice, or how often a description that has drifted out of date produces a misroute.
One documented constraint shapes how you write them. A Choice question accepts up to 255 options, while a gateway's inventory runs to thousands of models. Your candidate set does not have to be the whole inventory, and the same page describes chaining questions level by level for large taxonomies. Routing to tiers and letting code resolve tier to model on price and availability is the simplest way through, and it separates the semantic judgment from the procurement decision.
What the one routing experiment shows
Routing is not a new subject. RouteLLM and the work around it have been measuring routing strategies against public benchmarks since 2024. For Jev specifically, one community directory listed 409 projects on 17 September, mostly demos, with a benchmarks and research section of about seventy entries by my count. Exactly one is a routing evaluation.
TokenTrim's jev-routing-experiment reports two runs. A small pilot routes 100 cached RouterArena queries across three models, where Jev given a frozen difficulty rubric scored 69.5 per cent at a median 311 milliseconds per decision, beating the same setup given bare capability labels. None of the routers tested achieved a net improvement over the best fixed model on that pool.
The larger run matters more. On LLMRouterBench, 5,835 held-out queries across thirteen flagship models, a router combining Jev's difficulty rating with retrieval evidence about how each model performed on similar queries scored 62.4 per cent against 60.3 for the best single model, at about 15 per cent lower cost. The benchmark ships precomputed scored answers, so this is measured by lookup with no inference, which is what makes a comparison of that size affordable. The Jev calls cost about thirteen cents. Reported costs cover the selected model's answer and exclude the routing overhead.
Then the ablation, which the author runs and reports himself. Take Jev out and the same pipeline scores 62.4 per cent again, 62.442 against 62.425 in the saved results. Be precise about what was removed. The no-Jev arm still embeds the request and retrieves similar examples, and its retrieval index is per dataset, so it also knows which benchmark the query came from. What goes is Jev's difficulty score, which in the selection code scales how much cost the router tolerates. The finding is therefore narrow: this implementation performed about as well without adding Jev's difficulty judgment to the retrieval machinery it already had.
The saved results for each arm, with and without Jev, go further than the README says. Taking the cheapest sampled setting in each arm that reaches or exceeds the best single model's accuracy, the Jev arm gets there at about 33 per cent below that model's cost and the ablation at about 43 per cent. These use different cost-weight settings; they do not compare the arms at an identical sweep weight, and the ablation is not consistently more accurate when the weight is held fixed. The README calls the difference slight.
Three limits matter more than that wording. The eval queries stay out of the retrieval evidence, but the highlighted operating points are picked from the eval half after sweeping it, with no separate final test, so the headline figures carry some unmeasured optimism. The retrieval index knows each query's source dataset, which is information you will not have on live traffic. And I read the code and the saved results without reproducing the run. Anyone leaning on these numbers should recompute from the cached answers and recorded decisions, freeze the selected settings and test them on a separate evaluation set, and see how much of the advantage survives not knowing which benchmark a query came from.
What this supports is modest and useful. On this benchmark, a router using performance on similar requests scored about the same with or without Jev's difficulty judgment on top, which suggests the extra judgment contributed little in this design. It does not show that interpreting requests is unnecessary, since the retrieval system already uses the request text. The lesson points the same way as the catalog argument: measure what an additional routing model contributes beyond the information and machinery you already have.
The latency tax
Three early reports measure Jev's latency using different methods. A developer on r/AI_Agents ran two routing pipelines in parallel and saw roughly one second against four to fourteen for an LLM with structured output, informally and without a published method. The FGV Direito SP study measured a 0.32 second median on twelve questions per document, with confidence intervals and published code. Browser Use logged 178 milliseconds inside its agent loop, from three matched pairs its authors call thin. The first covers a whole pipeline and the others a single model call, so combining them into a range would mislead.
The money is trivial, at roughly two thousandths of a cent per call on Goenka's gateway receipts. The time is not. Put a 320 millisecond routing call in front of a one-second answer and routing is about a quarter of the elapsed time, and the overhead falls hardest on the cheap fast requests the router exists to protect. Within a single run there is a second cost: LangChain's middleware classifies the latest human message and keeps that model for every call in the run, so a task that turns out to be harder three tool results later is never reconsidered. Reclassifying before each model call allows the choice to be reconsidered, while adding latency at each iteration.
Give the policy an escape hatch
A routing policy needs somewhere to put uncertainty. Langfuse's writeup frames this as Jev cannot abstain, though the primitive is not really the issue: a Choice returns a probability for every option and a confidence, and your code is free to decline the result. TypeSafe's Choice page recommends an other option so the model can say that none of the alternatives fit, and its worked example sends a low-confidence answer to a person rather than acting on it.
In routing, not knowing has a correct answer, and that answer is escalate. Decide where the threshold sits, hard-code the fail-safe direction so uncertainty resolves upward, and keep difficulty and risk as separate questions, because their errors and thresholds are easier to evaluate apart than combined in one number.
The leak that inverted an AUC
Goenka's write-up contains the most useful warning in this material, and it cost him a night to find it. He scored cold emails with seven Jev questions weighted into a composite and got an AUC of 0.889 on 113 emails. On 90 days of production, 2,106 emails, the same composite scored 0.237. Inverted, worse than chance in the wrong direction. Replies in the first sample had clustered in threads whose subject line began with Re:. A bare Re: flag on its own scored 0.816, and on cold email with no prior thread all seven questions landed between 0.47 and 0.56.
His summary is that calibration is about the question you asked and signal is about whether the question matters. That carries straight over to routing. A router that looks excellent in evaluation may be keying on request length, channel, customer tier or time of day, every one of which correlates with which model you currently send things to. And that correlation is exactly the label you would be scoring against if you evaluate the router by asking whether it agrees with your existing rules.
Shadowing is not enough
Running the router in shadow tells you how often it agrees with your live system. It does not tell you whether the cheap model would have produced an acceptable answer, which is the thing you wanted to know.
What you need is a comparable outcome for both choices on the same request, scored against criteria you wrote down beforehand. Fresh inference is one way and not the only one. The routing experiment above compares thirteen models across 5,835 queries without calling any of them, because the benchmark ships scored answers to look up, and outputs you already have from the model you would have used anyway count the same way. For an agent that takes actions, controlled replay is safer than live traffic. Then split the results on whatever field is most likely to be carrying the outcome for free.
A commenter in that Reddit thread who builds a routing gateway put the metrics better than I would have. Routing accuracy, abstention rate, fallback frequency, and the cost of decisions that were wrong and confident. To which I would add the framing from LangChain's small evaluator experiment, which multiplies oracle agreement by repeatability, so a router that consistently sends hard work to the cheap model cannot look excellent on stability alone. Log the pinned Jev version alongside all of it, since the jev-latest alias moves when a release ships and a threshold tuned against one version is evidence about that version only.
The failures you never see
Routing is usually described as a safe place for a cheap model because a wrong answer is recoverable: you catch the failure and retry on the bigger one.
The arithmetic of that is fine. Say every request goes to the cheap model first, the router costs 0.01, the cheap model 0.10 and the expensive one 1.00. If a fifth of requests then need the expensive model, the average is 0.31 against 1.00 for always paying full price. That excludes whatever the verification step costs and assumes the retry succeeds. Fallbacks can be common and the economics still work.
The problem is the word catch. Measuring routed quality offline, the way that routing experiment does, is tractable when you have a benchmark with scored answers. Deciding in production, on one request, that the answer you just got back is worse than the one the bigger model would have given is a different problem, and nothing in the material I read addresses it. A malformed response or a failed tool call is easy to spot. An answer that is merely worse is not.
LangChain's caveat about evaluators transfers without modification: low cost amplifies mistakes, because something consistently wrong produces bad output at scale. And a router is unusually good at hiding its mistakes. Nobody files a ticket to say the cheap model answered a question the expensive one would have handled better. They just get the worse answer, while the dashboard shows the saving.
Unlock the Future of Business with AI
Dive into our immersive workshops and equip your team with the tools and knowledge to lead in the AI era.