Nvidia’s NeMo Switchyard routes AI queries to cheaper models, but overhead and cost swings persist

Nvidia has released NeMo Switchyard, an open-source software router that sits between an application and a pool of language models, choosing which model should handle each request or conversation turn. The goal is to curb rising AI costs by pairing simple tasks with smaller, cheaper models instead of always using frontier systems.

Enterprise customers are already struggling with escalating expenses as AI adoption grows. Nvidia is not alone in targeting this problem: the field is attracting major investment, evidenced by Stripe’s planned acquisition of OpenRouter for more than $7 billion.

Switchyard functions as a proxy that uses a routing algorithm to decide where to send queries, while returning responses in the format the calling application expects. It accepts OpenAI, Anthropic, and Responses API requests, translates between them, and records the chosen model, decision reasoning, token usage, and latency for each call. This transparency allows tuning over time. Nvidia points to Nemotron Parse, a one-billion-parameter model designed to extract structure from PDFs, as an example of a small model that can handle specific tasks.

However, routing comes at a cost. In a LangChain test of Nemotron 3.5 Lightning, the judge model—which checks whether the agent is still on track after each turn—consumed up to 21.2% of total cost, second only to what was spent on Claude’s Opus 4.8. A more optimized judge could reduce overhead, but might also make the system less effective for users who only need a basic AI model, since the judge reads or checks output at every turn.

Cost variance is another concern. While average spending was noticeably lower than running only Opus 4.8, the test costs fluctuated between $2.16 and $3.61—a 67% swing driven by when the router escalated queries to a larger model. This makes costs harder to predict: routing lowers the average but widens the distribution around it.

LangChain also advises against using Switchyard for short or latency-sensitive workloads, estimating it adds about 700ms of latency because the judge model must read all output.

Nvidia’s strategy is to offer a packaged, open-source solution with broad integrations in a fragmented space. Routing work to smaller open-weight models pushes inference toward hardware that enterprises own—hardware Nvidia sells. Historically, cheaper inference has led to more inference, not less, which aligns with the interests of a company selling AI accelerators as many enterprises remain hesitant to invest directly in the hardware.