Most of Your AI Traffic Does Not Need the Expensive Model

A team that has put a model call into production usually finds that the bill arrives before the pattern does. The first month is small enough to ignore. The third is a line item somebody asks about, and the answer is nearly always the same: every request in the application, from a two-word classification to a multi-document synthesis, goes to one expensive model, because that is the model somebody picked in week one and wrote into configuration.

The cost curve bends when the default path becomes a small model and the expensive one becomes the exception. You can have that without writing any routing logic of your own.

Three Routing Modes, and What Each One Gives Up

Microsoft ships a model router in Foundry that you deploy like any other model and call like any other chat deployment: the deployment name goes in the model parameter and nothing else in your calling code changes. It reads the whole request - system message, tool definitions, conversation history - and selects an underlying model for that one request.

What it optimises for is a setting, and all three options are published with the quality bands they work inside. Balanced is the default: it considers every underlying model within a small quality range, given as 1 to 2 percent compared with the highest-quality model for that prompt, and picks the most cost-effective of those. Cost mode widens the band to 5 to 6 percent and again picks the cheapest inside it. The design write-up is blunter about what that buys: Cost mode aggressively favors cheaper models, accepting slightly lower quality on complex prompts. Quality mode picks the highest rated model for the prompt and ignores cost.

So the mode is a number describing how much quality you will trade for price, and it belongs to the deployment: a nightly enrichment job and a customer-facing chat surface can be two deployments with two modes, and neither has to compromise for the other. One operational note before anybody tunes this in a hurry - a change to the routing mode takes up to five minutes to take effect.

The Smallest Model in the Pool Sets Your Context Limit

Here is the constraint that changes the design, and it is published in two places on the overview page because it catches people. The effective context window of a model router deployment is limited by the smallest underlying model. Not the largest. Not an average.

The Limitations section spells out what that means at runtime: other underlying models are compatible with larger context windows, which means an API call with a larger context will succeed only if the prompt happens to be routed to the right model. Read that twice. A long prompt does not fail cleanly and it does not fail predictably. It fails when the router picks a model that cannot hold it, and it works when the router picks one that can, on two inputs that look identical to whoever ends up filing the bug.

That moves prompt-size control out of the optimisation column. In a single-model deployment a prompt that fits today fits tomorrow, and trimming it saves money. Under a router the size of the prompt is part of whether the call works at all, so a retrieval step that returns four chunks on Tuesday and eleven on Wednesday because somebody uploaded a long document has become an availability question.

There are two documented answers and they are complements. The first is to keep the prompt inside the floor: the page names summarising the prompt before passing it on, truncating it to the more relevant parts, and using document embeddings so the model retrieves the relevant sections rather than receiving everything. That is the same discipline a retrieval-augmented application needs anyway, which is the argument for doing it first.

The second is to raise the floor, by deploying with an explicit model subset containing only models that support the context length you need. The guidance states it as an instruction: design for the smallest context window, or raise the floor. A subset then carries a second obligation that is easy to miss, because the page documents it two sections away, under failover. Your configured subset is also your failover set, and the guidance is to select at least two models for failover. A one-model subset is not a cautious choice, it is listed as a practice to avoid, and the stated reason is exact: you lose routing optimization, cost savings, and failover, effectively using model router as an expensive passthrough.

The other thing a subset does is freeze the pool. New base models are not included in your selection unless you explicitly add them to the inclusion list, which is what makes a subset usable as a compliance gate and is also why it needs a review date on it.

The Parameters That Quietly Stop Working

Three more constraints show up after the first week, and each one is documented.

The first is that some of your inference parameters stop being honoured, depending on where a request lands. Reasoning models in the o-series do not support Temperature and Top_P, and if model router selects a reasoning model for your prompt, it ignores the Temperature and Top_P input parameters. The same goes for stop, presence_penalty, frequency_penalty, logit_bias and logprobs, which are similarly dropped for o-series models but used otherwise. An application that relies on a stop sequence to keep a response parseable now relies on it conditionally, and the condition is a routing decision it does not make.

The second is prompt caching. It works, because the underlying models support it, and the benefit is conditional in the same way: caching applies only when the same model handles consecutive requests with overlapping prompt prefixes. A cache hit rate that looks poor under a router may be a routing distribution rather than a caching problem, and those two get diagnosed very differently.

The third is deployment mechanics. You do not deploy the underlying models yourself, with one exception: Claude models go onto the same account first, and a routing subset naming one that is not deployed fails the deployment request outright. Content filters and the tokens-per-minute limit go on the router deployment, not on the underlying models.

Which Model Answered, and How You Find Out

The router discloses its decision. The selected model is always disclosed in the API response via the model field, and the response body is otherwise identical to a standard chat completion - so on a real response that field reads something like gpt-5-mini-2025-08-07.

That one field is most of the observability story, and the guidance is direct: do not ignore the model field in responses, because it is your primary observability signal. Log it, build dashboards, and track which models are handling your traffic. Logging it costs one column. Skipping it means that when somebody reports answers got worse last Thursday, you cannot tell whether the model mix moved underneath the application. At the platform level the same signal arrives twice more - Azure Monitor metrics for the deployment split by underlying model, and Cost analysis filtered on the Deployment tag.

Before committing production traffic, benchmark the router against your current model on quality, cost and latency, and know one gap first: the Foundry Evaluations service does not integrate with model router directly. The sizing guidance is specific. Use at least 100 prompts from your actual workload for statistically reliable results, and fewer than 30 prompts gives only directional signal. On the cost side, account for the router markup on input tokens on top of the underlying model input and output pricing, since usage is charged for input prompts at the rate listed on the pricing page.

Where IKRC Fits

Routing is a configuration decision wrapped around an application decision. The configuration part is small: deploy the router, choose a mode, decide whether you need a subset. The application part is the work - knowing which of your calls are classification and extraction and which genuinely need reasoning, keeping prompt size under control so the floor never bites, and having the numbers that say whether it was worth doing. IKRC does that as part of building and running AI features inside .NET applications, on systems carrying real traffic where somebody has to answer for the bill.

A first conversation is short with two things in hand: a rough split of what your AI calls actually do, even if it is a guess from reading the code, and the current monthly spend. Call IKRC at 646-783-1441 or email info@ikrc.co.

Related Reading

For the shape of the application these calls live inside, read Adding AI to the .NET Application You Already Run. For scoping what a model is asked to do in the first place, read Why a Task-Specific Agent Beats a Chatbot Bolted Onto Your App.