AI Model Routing Is Turning “Which Model?” Into a Per-Request Decision

The immediate development

The less flashy NVIDIA release is the more interesting one

On August 11, Reuters reported two NVIDIA developments that point in different directions. The headline-grabbing one is Nemotron 4, a model family reportedly still in development, with no release date and no completed final training. The more concrete release is NeMo Switchyard, an open-source routing layer that can direct requests across different model backends.

The Nemotron 4 report is interesting, but it remains a report about an unfinished product. Switchyard already exposes a more durable architectural idea: an AI application does not have to decide once that it is an “OpenAI application,” a “Claude application” or a “Nemotron application.” It can decide repeatedly, at runtime, which model should handle the next piece of work.

That changes the unit of model selection. Instead of choosing one model for an application or one provider for a contract period, the system can choose a model for each request—or, in a long-running agent, for each stage of a task.

This is not entirely new. Research systems have explored model routing for several years. What is changing is that routing is moving into mainstream agent infrastructure. NVIDIA’s Switchyard repository describes a proxy and library that translates among OpenAI and Anthropic API formats, routes traffic across multiple backends and records latency, token and error metrics. It supports simple random routing, an LLM classifier, stage-based routing and escalation from a weaker model to a stronger one.

There is also an important caution in the same documentation: Switchyard is explicitly described as pre-alpha experimental software, not for production use. That makes it evidence of direction, not proof that model routing has become a solved production problem.

A router makes model selection part of inference

Most AI applications still treat the model name as a configuration value. A developer selects a model, sends prompts to it and changes that choice manually when price, quality or product requirements change.

A model router inserts a decision layer before the generation call. It examines information about the request and chooses from a pool of candidate models. The simplest version asks a question such as: Is this request easy enough for the cheaper model, or difficult enough to justify the stronger one?

The mechanism can be implemented in several ways. A lightweight classifier can predict request difficulty. An embedding model can compare the new request with examples that different models handled well. A rules engine can route coding tasks, image tasks and routine extraction to different specialists. An escalation design can let a cheap model answer first and call an expensive model only when a judge decides the first answer is inadequate.

Switchyard adds another practical element: protocol translation. A client that speaks Anthropic’s Messages API or an OpenAI-compatible API can sit in front of a router that forwards the request to a different compatible backend. This matters because model choice is not only an intelligence problem. API shape, tooling and provider-specific integration can create switching costs even when another model is technically suitable.

Routing therefore sits between the application and the models. The application asks for a capability. The router decides where that capability should come from.

“Use the cheap model for easy questions” is too simple

Cost is the obvious reason to route. A system that sends every request to its most capable model may pay frontier-model prices for work that a much smaller model could complete reliably.

But production routing is a multi-objective problem. At minimum, a useful router may need to consider quality, price and latency. In many environments it must also consider context-window length, tool support, modality, geographic availability, privacy requirements, rate limits, vendor outages, licensing, safety controls and whether a model can be self-hosted.

Latency is particularly easy to underestimate. A cheaper model is not automatically faster, and a normally fast endpoint can become slow under load. Recent routing research therefore treats runtime conditions as part of the decision. FLARE, presented at ACL 2026, estimates per-query latency and cost rather than optimizing accuracy alone. Its reported experiments show large reductions in latency and cost while maintaining competitive accuracy, but those results are benchmark-specific rather than a guarantee for arbitrary production traffic.

The same applies to reasoning effort. AIUpdateWatch has previously explained why reasoning effort is itself an inference variable. A router could eventually choose not only which model handles a task, but how much computation that model is allowed to spend. Research such as R2-Router already explores this combined model-and-budget decision.

The practical implication is that “best model” becomes conditional. The right choice for a high-stakes legal synthesis may not be the right choice for extracting invoice fields, classifying support tickets or generating a first-pass summary. Model selection starts to look more like workload scheduling than product loyalty.

Agents make routing a sequential problem rather than a one-shot classification

Routing a single chat request is comparatively clean: inspect the request, select a model, generate an answer.

An agent can make dozens or hundreds of model calls while completing one task. It may plan, retrieve information, write code, inspect tool output, recover from errors and verify its own work. Different stages can require different strengths.

A cheap model may be adequate for formatting a tool result but poor at diagnosing why the tool failed. A coding specialist may be useful while editing a repository but unnecessary when summarizing the completed change. A local model may handle private document extraction while a stronger hosted model performs a de-identified reasoning step. The route can therefore change within one user request.

This is why NVIDIA’s stage-routing concept is more interesting than a simple weak-model/strong-model switch. Switchyard can use signals already present in a conversation—tool results, errors or stage state—to select a tier without adding another model call solely to make the routing decision.

Recent academic work is moving in the same direction. MTRouter treats routing across a long-horizon interaction as a history-aware decision rather than classifying each prompt independently. In its published experiments, the system reduced cost substantially on ScienceWorld and Humanity’s Last Exam while remaining competitive with, or in one setting surpassing, a GPT-5 baseline. Those numbers are useful evidence that sequential routing can work, but they remain results on specific tasks and model pools.

This also connects to the broader shift in evaluation that AIUpdateWatch examined in ARC-AGI-3. Once memory, tools, retries and orchestration affect performance, the model is no longer the whole system. A router becomes another component whose decisions can materially change both capability and cost.

The evidence supports model complementarity, not magical routing

The core premise behind routing is that models have different strengths. If one model dominated every other model on every relevant task while also being cheaper and faster, routing would be unnecessary. Real model portfolios are more heterogeneous.

RouteLLM, published at ICLR 2025, showed that learned routers could reduce cost substantially while retaining most of a stronger model’s benchmark performance. That work helped establish the modern routing pattern: predict when the expensive model is actually needed rather than calling it by default.

Newer evidence is more sobering. LLMRouterBench, published in Findings of ACL 2026, evaluates more than 400,000 instances across 21 datasets and 33 models. It confirms strong model complementarity, but it also reports that many sophisticated routing methods perform similarly under unified evaluation and that several recent methods—including commercial routers—do not consistently beat a simple baseline.

The benchmark also finds a substantial gap between current routers and an “oracle” that always knows which model will succeed. One reason is model recall: routers fail to identify the right specialist often enough to leave meaningful performance on the table. Adding more models does not automatically fix this; the paper reports diminishing returns from larger ensembles compared with careful model curation.

That is an important check on the sales pitch. A router can save money only if it knows when not to save money. Sending a difficult request to a cheap model can create a false economy if the answer has to be redone, escalated later or—worse—accepted despite being wrong.

The benchmark question should change

Instead of asking only “Which model scores highest?”, a routed system needs another evaluation: How often does the router choose the model that is good enough for this particular request under the required cost and latency constraints?

Routing creates a new operational control plane

Once a router can decide which provider receives a request, that router becomes a consequential piece of infrastructure. Its configuration can change cost, privacy exposure, latency, failure behavior and output quality without any change to the user-facing application.

That creates several operational requirements.

First, routing decisions need observability. Operators should be able to reconstruct which model handled a request, why it was selected, what it cost, how long it took and whether the request later escalated. Switchyard’s support for request, error, latency, token and routing-overhead metrics points in this direction.

Second, router evaluation must use the organization’s real traffic. A router trained on public preference data may behave differently on medical notes, insurance claims, software incidents or multilingual customer service. The workload distribution matters.

Third, the model pool must be governed as carefully as the router. Model versions change. Prices change. Providers alter limits. A local model may receive a new quantization. A once-cheap model may become slower under load. Routing thresholds that were sensible last month can become wrong without any code change in the application.

Fourth, failure has to degrade safely. If the classifier or routing service is unavailable, what happens? Does the application fail closed, fall back to a known model or spray requests across providers? A routing layer adds resilience options, but it also adds another dependency.

For organizations comparing models through published API prices, routing changes the economics from a single price-per-token comparison to a workload-weighted mix. The relevant question becomes the average cost of achieving an acceptable result across the actual distribution of tasks.

NVIDIA’s interest is larger than selling another Nemotron model

NVIDIA’s model strategy increasingly looks like a portfolio rather than a single flagship. Its Nemotron materials describe specialized reasoning, safety, speech, retrieval and multimodal models working inside agent systems. The company also supplies the GPUs, serving software, training stack and orchestration tools around those models.

Switchyard fits that strategy because a heterogeneous model world is not a threat to NVIDIA in the same way it might be to a vendor whose economics depend on every request reaching one proprietary model. A router can send some work to Nemotron, some to another open model and some to a closed frontier API while keeping NVIDIA software in the orchestration layer.

This does not mean NVIDIA is neutral. It has obvious incentives to make its own models and infrastructure attractive. But the architectural direction is notable: compete not only for the model call, but for the software layer that decides where the model call goes.

The reported Nemotron 4 project should be viewed through that lens. Reuters says final training is incomplete and NVIDIA has not announced a release date. Even if Nemotron 4 eventually becomes a strong open model, the more durable question is whether organizations will still want to send every task to it. Routing assumes the answer is increasingly no.

That is also why the recent shift toward smaller local models matters. A local specialist becomes much more useful when an application can invoke it selectively rather than asking it to replace the frontier model for every task.

The next routing benchmarks need to look like operations, not trivia contests

Model routing has a credible technical case, but the next step is not another chart showing that a router saved tokens on a static benchmark.

The stronger evidence would come from long-running production-like workloads where task difficulty changes over time, provider latency varies, prices differ, model versions drift and failed routes have a measurable cost. Evaluation should include the overhead of the router itself and the cost of escalation after a poor first choice.

Three signals are especially worth watching.

History-aware routing: whether routers can use tool results, earlier errors and task progress without repeatedly paying for a separate classifier call.
Calibrated reliability: whether operators can set a measurable failure tolerance for cheap-model routes rather than relying on a vague “quality” score. Recent conformal-routing research is beginning to address this problem.
Portable model pools: whether routing layers can switch among cloud APIs and self-hosted models without breaking tools, structured outputs, safety policies or observability.

If those pieces mature, the familiar question “Which AI model should we use?” will become less useful. The better question will be: Which model should this system use for this step, under these constraints, and how do we know the router made a good decision?

That is a systems problem. NVIDIA’s Switchyard makes it visible, but the research suggests the market is still early in solving it.

Sources