Which model should run this task? The question has a correct answer. It arrives after you have stopped needing it.
You find out how hard a task was by doing it. The small model takes the request, the output looks reasonable, and the problem only shows up later when something downstream breaks. By then you have paid twice, once to generate the wrong answer and once to work out that it was wrong.
A router bets against that. It reads the prompt, guesses which model can handle it, and commits before a single token exists. Vendors sell this as cost optimisation. It is a quality decision made blind, priced like a discount.
How much that matters depends on who reads the output. In a chat product, someone reads every response and can push back on a bad one in the next turn. Routing errors are cheap there. The user absorbs them, one turn at a time, and mostly never knows.
An agent run removes all of that. It runs for minutes or hours. Nobody reads the intermediate steps, and each step feeds the next. A bad choice at step four does not produce an obviously wrong answer. It produces a plausible one, and the next nineteen steps build on it. The error itself is cheap. What gets stacked on top of it is not.
Two different problems. The same products are sold into both.
The clearest demonstration happened in public. When OpenAI launched GPT-5 in August 2025 it shipped a router that chose between fast and reasoning variants per request, and it removed the model picker that had let users choose for themselves. The day after launch, Sam Altman conceded that the autoswitcher had broken and spent much of the day out of commission, and that the model had seemed far weaker as a result.
Look at what users complained about. Not that routing had gone wrong. That the model had got worse. Inside a single conversation those are the same thing, and nobody could tell them apart.
It surfaced at all because millions of people had a year of experience to compare against and could feel the drop. Your agent has no such baseline. It never sees how the other model would have answered, it forms no impression of a bad week, and it reports success in the same tone either way.
That does not make the problem unmeasurable. The counterfactual is for sale. Send a sample of traffic through both candidates, judge the outputs, and you get an estimate of what routing is costing you in quality. The catch is what this does to the arithmetic. The advertised saving becomes a gross figure, and evaluation, drift monitoring and the downstream cost of compounded errors all come out of it.
The vendors buy it too. Not Diamond's documentation for training a custom router asks for your inputs, every candidate model's response to each input, and evaluation scores for all of them. Minimum fifteen samples, maximum ten thousand per job. That is the counterfactual, paid for upfront, before the router exists. Their own product cannot be built without it. The trouble starts when the buying stops. The router ships calibrated against a fixed snapshot, then runs for a year against traffic and a model pool that have both moved.
So the industry publishes the number it gets for free. Cost shows up in the bill. AWS advertises that intelligent prompt routing cuts costs by up to thirty per cent without compromising accuracy, a figure it reports about its own service and supports with no published method for the second half. Your finance team audits the savings. Nobody audits the compromise.
The good news is that routing products vary far less than the market suggests. Take the branding off a dozen of them and you find five mechanisms recombined. Each one has a limitation that belongs to the mechanism, not to the vendor.
The first is a score and a threshold. RouteLLM, the open reference implementation, puts it in plain sight: one method takes a prompt and returns a predicted win rate for the strong model, and anything over your threshold routes strong. Every learned router does some version of this. The method name gives away the limitation. That score predicts a comparison that never runs, so how confident the router is and how right it is stay separate numbers, and you only ever see one.
The second is how you set the threshold. RouteLLM calibrates against the public Chatbot Arena dataset, and what you specify is the share of traffic going to the strong model. Not a quality level. You cannot ask for a quality level. OpenRouter's Auto Router ships the same idea as a parameter called cost_quality_tradeoff, an integer from zero to ten, where zero always takes the strongest model and ten always takes the cheapest. It defaults to seven. You are being asked to state a preference over something nobody has defined, on a scale with no units, and the factory setting leans towards cost.
The third is restricting the candidate set, and it explains why so much of this field is pairwise. AWS Bedrock routes between two models from one family. Not Diamond's open-source router works in pairs. Comparing one model against one other is tractable. Ranking twelve is not. So the problem keeps getting solved at the size where it can be evaluated rather than the size where it would help. Commercial frameworks do train over larger candidate sets, so this is a gradient rather than a wall, but it runs one way. Restriction earns its keep elsewhere: OpenRouter lets you fence the pool by wildcard, which is real control over who ever sees your traffic. It does nothing for accuracy.
The fourth is pinning. A per-request router would pick a different model every turn, so OpenRouter pins model and provider for the length of a conversation, implicitly once the provider reports cache usage, explicitly from the first successful response if you pass a session identifier. In a chat interface that is correct and fixes an obvious problem. Now put it in an agent. The model that will run for four hours gets chosen from the opening exchange, before a tool is called or a file is read, before any of the work that determines the difficulty has happened. Minimum information, maximum commitment. The pin also expires after five minutes of inactivity, so a run that stalls on a slow dependency can come back on a different model with nothing in the transcript to show it.
The fifth skips prediction altogether: rules over things you can observe. Keywords, declared task type, context length, whether tools are needed. LiteLLM documents heuristic and keyword routing deciding in under a millisecond, its own figure about its own software. Speed is not the point. A rule you can read lets you find the branch that fired, once you already know the output was bad. That is diagnosis, and it is worth having. On whether the choice was right, a rule is as silent as a neural net.
Seeing the mechanisms rather than the products makes two things obvious. Transparency and evaluation are different features, and the market has shipped one of them. OpenRouter returns the identity of the model that served each request, the exact thing whose absence turned a flagship launch into a trust crisis, and it charges nothing for the routing decision. It still cannot tell you whether the choice was right. The response tells you what happened. It says nothing about whether what happened was any good.
Your own logs have the same limit. A gateway records the model, the latency, the tokens and the cost. All of that describes behaviour. None of it describes quality, and without task outcomes coming back, no amount of logging will produce it. Which is why, as of July 2026, plenty of vendors publish routing benchmarks and cost-quality curves and none publishes an audited error rate from a customer's production traffic. Competition usually forces benchmarks out into the open, because a vendor with a number that wins will not shut up about it. What we get instead is each vendor's own score on each vendor's own dataset.
The number everyone quotes makes the point. RouteLLM reports cost reductions above eighty-five per cent while keeping around ninety-five per cent of the strong model's performance, measured on MT-Bench. A real result, reported without inflation, on a public benchmark of multi-turn questions scored by another model. It is the best public evidence that routing works, and it comes from a distribution that looks like nobody's production traffic. RouteLLM says so itself, warning that the share actually routed will differ from the calibration because your queries differ from the dataset, and telling you to calibrate on data resembling your own. That warning is the argument of this piece, published by the people who wrote the reference implementation.
So what do you do on Monday.
Ask whether a human reads every output. If one does, route away. Your error handler is sitting there at every step, the mechanisms above are working for you, and a sample of shadowed traffic will tell you what you need about aggregate quality. Chat products have the easy version of this problem and keep treating it as the hard one.
If nobody reads the output, ask how wide the quality gap between your candidates really is. Routing pays in proportion to that gap, and at the mid-tier the gap has been closing. If your candidates are close, the saving is small and the evaluation overhead is not. Stop here. Pick one model per workload class and skip the machinery. No vendor in this market can afford to give you that advice, and it fits more teams than the market lets on.
If the gap is wide and nobody is reading, ask whether success can be checked by something other than the model that produced it. A passing test. A schema match. A retrieval that either found the document or did not. Where a check like that exists, route the subtask and escalate when the check fails, which gives you a prediction you can correct. Where no check exists, keep the strong model. Planning, irreversible actions and anything consumed rather than inspected belong to the expensive model whatever a router thinks.
Log the selected model on every call, because that field is the only record of what the router did and the only way a mid-run switch ever becomes visible. Shadow one or two per cent of production traffic through both candidates and judge the difference, using a model as the first pass and humans on a sample of that. Recalibrate on triggers rather than dates: when the candidate pool changes, when your traffic mix shifts, and otherwise on whatever cadence you will actually keep.
Keep availability fallbacks out of all of it. Ordered fallbacks across providers, retries with cooldowns, circuit breakers, budgets, allowlists, regional pinning. You can inspect all of it and test it with fault injection, and it earns its keep the afternoon a provider starts returning 503s. It predicts nothing, which is why you can trust it. OpenRouter ships its fallbacks as a separate feature from its router, which is the right call. AWS gives its prompt router something called a "fallback model," which sounds like the thing that catches you when a provider dies. It is the model used when the router's predicted quality difference comes in under your threshold. A quality baseline wearing the name of a resilience feature is the confusion to design against.
Routing pays in proportion to the quality gap it exploits, and as the mid-tier converges that gap narrows and takes the saving with it. Meanwhile the workloads burning the most tokens are getting longer, less supervised and more compounding, which multiplies the cost of one bad choice by however many steps follow it.
Unlock the Future of Business with AI
Dive into our immersive workshops and equip your team with the tools and knowledge to lead in the AI era.