Take a hypothetical customer-service system where an AI feature reads each incoming support email, picks a category such as billing, delivery or cancellation, and copies the customer's account number into the new ticket. The feature calls a Standard deployment of gpt-4o, version 2024-05-13, in Azure OpenAI. As of October 1, 2026, Microsoft's model retirement schedule lists December 9, 2026 as that version's retirement date, and Microsoft's lifecycle page adds that the date and the replacement are "subject to change". If nobody changes the deployment before then, what Azure does to it is decided by one property.
Three upgrade settings and when each one changes the model
The property is versionUpgradeOption, and Microsoft's model management page defines its three values. OnceNewDefaultVersionAvailable moves the deployment to a new default version within two weeks of Microsoft designating it. OnceCurrentVersionExpired keeps the current version until the retirement date and then moves to whatever the default is on that day. NoAutoUpgrade never moves automatically: "Once the retirement date is reached the model deployment stops working." The lifecycle and support policy says that a retired version answers every inference request with 410 Gone, and that "Retirement dates aren't extendable."
A deployment created without choosing a value still has one. When the property is unset, the REST response leaves it out, and the model management page says "null is equivalent to OnceCurrentVersionExpired". The hypothetical triage deployment, where nobody set the property, is therefore upgraded on its retirement date to whatever the default version is that day.
The upgrade does not reach every region on the same day. Microsoft says automatic upgrades for Standard, Data Zone Standard and Global Standard deployments are "scheduled on a rolling, region-by-region basis", and that they "can occur even if the new model version isn't yet separately available in that region". A business that runs the same feature from deployments in two regions can have two model versions answering its customers while the rollout is in progress. Provisioned deployments sit outside all of this: "Provisioned deployments are NOT auto-upgraded", and their customers migrate by hand to the replacement.
Setting NoAutoUpgrade keeps the 2024-05-13 version answering until December 9, the currently published retirement date, and from then the triage feature's calls fail until someone points it at a supported deployment. Under OnceNewDefaultVersionAvailable the change can come earlier, within two weeks of a new default being designated. Under either upgrade setting, Azure moves the deployment to another version and the feature's requests go to it. Microsoft's model versions page warns that after an upgrade "you might notice some changes in the model behavior and compatibility." The date to plan against is the earliest change the deployment's setting allows, and the saved cases have to be replayed against the new version before it.
Which cases belong in the evaluation set?
Microsoft's advice for choosing a replacement is to "evaluate candidate models against your own application, prompts, and representative data". For the triage feature, representative data is a set of past support emails whose correct category and account number a person has already confirmed, such as tickets an agent moved to a different category after the feature filed them. Each case stores the email text, the expected category and the expected account number. The set also needs emails that carry no account number, where the expected result is an empty field. In the hypothetical, a new version that fills that field with an order number it found in the email fails the expected-empty check. In production, that ticket would have carried a wrong account reference.
The Microsoft.Extensions.AI.Evaluation libraries run under dotnet test with MSTest, xUnit or NUnit. A custom check is a class that implements the IEvaluator interface, and Microsoft's reporting tutorial builds one that "isn't based on AI" and marks its metric as failed when the value is out of range, and the test then asserts that the metric did not fail. For the triage set, the check compares the returned category and account number with the confirmed values, and any mismatch fails the test.
We would leave the built-in quality evaluators out of this set. They "use an LLM to perform the evaluation", so each score brings a second model into the test, and the tutorial notes that scores can change over time as the models used change. The replay is meant to measure one model change at a time.
A cached replay can still return the old model's answers
The evaluation libraries can cache model responses, and Microsoft's tutorial turns that on. With caching enabled, a later run whose prompt and model are unchanged is answered from the cache, according to the libraries page, and the tutorial keeps a cache entry for "14 days by default".
With Azure OpenAI, that model is a deployment name: the tutorial creates its chat client with GetChatClient(deploymentName: model). An automatic upgrade changes the version behind the deployment and leaves the name alone, which is how the model versions page describes deployments that "switch to the new version automatically". The API reference for CachingKeys lists what goes into the default cache key: the request content, the chat options and "metadata such as model name and endpoint", plus the scenario and iteration names, which are always included. After an in-place upgrade, a replay of the same scenario and iteration sends the same emails to the same deployment name at the same endpoint, so a run inside the cache lifetime can report a pass from responses the previous version produced.
The same reference documents the fix. An extra caching key can be "the version of the AI model being invoked", and changing that key invalidates every cached response that depends on it. DiskBasedReportingConfiguration.Create accepts such keys through its cachingKeys parameter. The deployments endpoint returns the model version for each deployment, so the run should read it there at the start, and the key then describes the deployment being replayed. The execution name needs the version and a build or run number. With the default name, Default, the tutorial says "results from one run are overwritten by the next", and a name holding only the version would do the same to a second run against that version. A run that straddles an upgrade would mix two versions, so the run should read the version again at the end and discard its results if the two values differ.
Who hears about a retirement, and how early?
For a generally available model, the normal notice on the lifecycle page is an email at least 60 days before retirement, sent to "subscription owners with active deployments". The same page reserves an emergency retirement "with shortened notice" for a model found to have compliance or security issues. The person who owns the Azure subscription may not be the developer who maintains the triage feature.
An Azure Service Health alert reaches whoever the alert rule names. The page says to open the health advisories, "filter by Azure OpenAI Service", and create an alert rule that sends email, a text message or a webhook call. Service Health has no separate Microsoft Foundry entry.
A script can also read the Models API, whose fields include lifecycleStatus and deprecation.inference. The names do not line up with the documentation's stages. A model the docs call Deprecated, still serving existing customers, reports Deprecating, and the page warns that the API value Deprecated "means the model is retired and no longer serves inference." A check that alerts on the API value Deprecated hears about a model version only after it has been retired, when a deployment still pinned to it is already failing and one set to upgrade has already moved.
How IKRC would set up the replay for a feature already in production
On an AI services engagement for a feature already in production, we would first read the deployment's versionUpgradeOption and find the date that applies to it. For the two settings that wait for retirement, that is the retirement date in Microsoft's published table. For OnceNewDefaultVersionAvailable, it is a default-version change, which the model versions page says Microsoft announces at least two weeks ahead. The confirmed cases would go into the application's existing test project, next to the code that calls the model, as described in adding AI to the .NET application you already run, with the model version in the caching keys and the version plus a build number in the execution name. A feature that routes requests across several models also needs the record of which model answered, covered in the model routing article.
Microsoft advises against waiting for the official replacement, which is declared only about 90 to 120 days before retirement and can be tested in Global Standard about 90 days before. We would replay the set against a candidate deployment wherever one is available, even when the production region will get the new version only through the upgrade. Each replay's report lists every case where a returned category or account number differs from its confirmed value, and the developer who maintains the feature reviews that list to decide whether the deployment is ready to move before its date.
Related Reading
Routing Routine AI Requests to Smaller Models covers router modes and how to find out which model answered. Adding AI to the .NET Application You Already Run covers where a feature like the triage step sits in the application.