Not All Model Upgrades Are Upgrades [Evals]
Bottom Line
A newer model with a cheaper rate card can still be more expensive. Microsoft measured Claude Sonnet 5 vs 4.6 on agent tasks and saw median token multipliers up to ~12× on architecture work — enough to erase a 33% per-token discount.
Key Takeaways
- ›Sonnet 5 rate card in the study: $2/$10 vs Sonnet 4.6 $3/$15 (input/output per 1M) — Sonnet 5 cheaper per token.
- ›Architecture tasks: ~12× more tokens at the median for Sonnet 5; some scenarios much higher.
- ›Code upgrade tasks: ~10× token gap; example cost $2.01 vs $0.55 per run in the write-up.
- ›Evaluate upgrades with fixed harness + fixed tasks; never with “feels smarter.”
- ›Gate production model flips on cost-per-success, not benchmark screenshots.
Microsoft’s Agent Experience team published “Not all model upgrades are upgrades” with hard numbers from Claude Sonnet 4.6 vs Sonnet 5 agent runs.
This piece unpacks what actually changed, how the system works, who feels it first, and what to verify before you treat Microsoft Developer Blog's account as an action item.
What shipped
A versioned cut is a contract with anyone who pinned the last one. Not All Model Upgrades Are Upgrades [Evals] should be read as a changelog first and a launch second. If you cannot find the changelog, you do not have enough to upgrade.
Microsoft’s Agent Experience team published “Not all model upgrades are upgrades” with hard numbers from Claude Sonnet 4.6 vs Sonnet 5 agent runs. This piece unpacks what actually changed, how the system works, who feels it first, and what to verify before you treat Microsoft Developer Blog's account as an action item.
What changed for builders
Builders should diff the release notes for APIs, defaults, and removed flags. That list is the migration. Anything not on it is a rumor until it shows up in a follow-up patch.
A versioned cut is a contract with anyone who pinned the last one. Not All Model Upgrades Are Upgrades [Evals] should be read as a changelog first and a launch second.
How to install or upgrade
Install via the vendor's documented channel. Snapshot config, roll through staging, keep a one-command rollback. Time-box the canary. If the release has no documented rollback, that is the first risk you escalate.
If you cannot find the changelog, you do not have enough to upgrade. Builders should diff the release notes for APIs, defaults, and removed flags.
Gotchas and compatibility
Gotchas hide in transitive deps, license files, and anything that touches auth or storage. Read those sections twice. Then grep your own repo for the old flag names so you are not surprised in prod.
Anything not on it is a rumor until it shows up in a follow-up patch. Snapshot config, roll through staging, keep a one-command rollback.
What to watch next
Watch the first patch release. If it arrives inside a week, the original cut was not as boring as the announcement implied. Pin to the patch, not the day-zero tag, unless you have a reason.
If the release has no documented rollback, that is the first risk you escalate. Microsoft’s agent evals show Claude Sonnet 5 can burn far more tokens than Sonnet 4.6 despite lower rates.
A 3–5 minute news post is a briefing, not a runbook. Keep Microsoft Developer Blog and the vendor's primary page in another tab, quote only what they printed, and write down the single decision this story forces (upgrade, wait, or ignore) before you Slack it to the rest of the team. If you need more than that decision, you want the primary docs or a later engineering deep-dive — not another recap of Not All Model Upgrades Are Upgrades [Evals].
When you brief someone else on Not All Model Upgrades Are Upgrades [Evals], lead with the surface that moved and the decision you need from them. Do not paste the whole thread. If you cannot name the surface — API, policy, model, hardware, or commercial terms — you are not ready to brief. Go back to Microsoft Developer Blog and the vendor page until you can. That extra ten minutes is cheaper than a wrong upgrade or a missed exposure.
Treat day-one coverage of Not All Model Upgrades Are Upgrades [Evals] as a pointer, not a specification. Microsoft Developer Blog is useful for names, dates, and the claim as stated; it is not a substitute for the changelog, the advisory, or the contract clause that actually binds you. If those artifacts are not public yet, wait. Acting on a paraphrase is how teams ship the wrong flag or miss the one dependency that was actually in scope.
If Not All Model Upgrades Are Upgrades [Evals] touches something you operate, open a ticket with the primary link, the owner, and the verification step — not a Slack emoji reaction. If it does not touch you, write that down too so the next person does not re-open the question. Either way, the artifact is the point. Recaps of recaps of Microsoft Developer Blog do not help the on-call.