Cutting LLM Costs Without Cutting Quality [Testμ 2026]

Two Unsolved Problems

For two years the question was how to get an LLM into production at all. That has largely been answered, and production volume brought a different bill with it.

The questions she sees now are defined by three things: token efficiency, robust evaluation, and diversification across models rather than commitment to one.

Her diagnosis of why teams overspend is that most are paying for capability they never measured. Without an evaluation set there is no way to know whether the expensive model is earning its price on your particular task.

Model Shelf Life Collapsed

She opened with a chart tracking model announcements from 2022 to 2026, where each point marks a provider taking the lead. The spacing between those points is the story.

In 2022 and 2023 a model stayed on top for roughly 12 to 18 months. By 2026 the interval had compressed to about two and a half months.

That timeline turns an architectural convenience into a liability. Coupling application logic directly to one provider means committing to a leader whose lead expires roughly every ten weeks.

Her structural answer is separation. The application layer should be decoupled from the model serving layer, with evaluation and observability strong enough to tell you which model you should be using at any given moment.

The Cursor Case Study

Open-weight models now trail the frontier by about the same two and a half months, which is what makes them viable rather than merely cheap. Her worked example is Cursor, the IDE with automated coding features.

Cursor built its architecture on top frontier APIs and charges by subscription, so scaling production volume made the cost per task unsustainable. The economics were structural rather than a pricing accident.

What they did was analyse their own prompt and response loops, build a rigorous validation suite, and move to a fine-tuned variant of the open-weight Kimi model. She cited a figure from their own site: an 86 percent drop in cost per task.

The order of operations is the transferable part. The validation suite came before the model switch, which is what made the switch defensible rather than a gamble.

Three Model Archetypes

Before the demo she set out the landscape, starting with a claim about market structure: committing to a single model provider is becoming structurally obsolete.

Open-weight models now account for 38 percent of enterprise token volume globally, and the price spread she showed runs to roughly 5,000x between the most capable model in the comparison and the cheapest. China currently leads the open-weight race.

• Proprietary - you call a provider and pay per token, with no control beyond the API surface.
• Open-weight - the same pay-per-call option is available, and you can also host the model on your own tenant.
• Fine-tuned - the most control, because the model is tuned on enterprise data, registered in a catalog, and becomes your own intellectual property.

Matching Size to Task

Cutting across those archetypes is size, and each tier has a job it is genuinely better at.

• Frontier - unmatched at reasoning and highly complex logic, with cost and latency penalties to match.
• Mid-tier - the throughput and large context tier, which is where a model like Sonnet sits.
• Low-latency - small and highly specialised, such as Gemma at 12 billion parameters, delivering sub-second time to first token.

She was clear that this is a starting point rather than a decision. Which tier your task actually needs is a question the evaluation answers.

Her demo makes the point with two tasks that look similar and are not. A soccer chat needs messages classified as spam, ads, harassment or jokes, and it needs the ambiguous ones, such as a threat disguised as a joke, escalated to a human reviewer with a written brief.

The Evaluation Baseline

Everything else in the session depends on knowing what good looks like, which means building the baseline first. She used MLflow, the open source platform that began as experiment tracking for MLOps and now spans evaluation, model registry and serving.

• Define the evaluation set - the examples for your use case with their expected labels, which for moderation meant eight categories, including the edge cases where the right label is genuinely hard to pin down.
• Define the scorers - a Python function doing exact matching where that works, or an LLM judge where it does not. For the escalation brief she defined five criteria for whether the brief was correct.
• Run it - execute through MLflow and collect a trace for every single call.
• Analyze - read pass and fail, accuracy, speed and cost per model, which is the basis for any trade-off.

She ran 16 models through that set. Every run is stored, so adding a model later means comparing it against the same bar rather than a remembered impression.

Two feedback modes run alongside each other. Automatic scoring provides speed and scale, while human review supplies ground truth on exactly the examples where the model was least confident, and those corrected labels feed back into the set.

The Gateway Layer

Evaluating models is only half the problem, because the application still has to reach them. Her position is that a model-agnostic layer is a competitive differentiator rather than plumbing.

A gateway routes and balances traffic across providers, whether self-hosted or called by API, and it is not limited to models. The same layer can route to edge agents, MCP servers, external agents and coding skills.

Budgets and policies attach at that layer too, per model and per agent, along with who is allowed to call what. One endpoint replaces provider APIs hard-coded throughout the codebase.

Start Small, Escalate on Failure

With a baseline and a gateway in place, her selection procedure is short. Start with the smallest model, run the evaluation, and only increase model intelligence when the small one fails.

The stronger model then faces the identical evaluation, which is what keeps the comparison honest. Models can also be run in parallel rather than in sequence.

Traffic splitting handles the case everyone hits when a new model lands. Send 10 percent of production traffic to it, collect the traces, and evaluate those instead of guessing from the launch benchmarks.

Her insistence on this point is the practical core of the talk. Evaluation is not one and done against a fixed set; it continues as new models arrive, using live traces from production.

The Results

The dashboard from her 16-model run plots open-weight models in green and proprietary ones in red, and the classification task produced a result worth sitting with.

• Gemma 12B - accuracy 0.85 at a run cost of 23 dollars, with markedly better latency.
• GPT-5 nano - the same 0.85 accuracy at a run cost of 912 dollars.
• Sonnet - accuracy 0.9, at a cost she described as much higher again.

A 12-billion-parameter model matched a proprietary one on accuracy while its cost bar looked almost flat by comparison. Whether the extra 0.05 from Sonnet is worth buying is a question only the use case answers.

For some applications that gap is worth millions of dollars and the frontier model is the correct purchase. For high-volume moderation, where the task is straightforward classification at enormous scale, the small open-weight model wins on arithmetic.

Fine-Tuning and Its Three Signals

She was direct that fine-tuning should not be the default, because a gateway plus good prompting handles most cases. It becomes justified when the mistakes repeat, the labelled data exists, and the cheaper options have already been tried.

• The mistakes repeat - the model keeps missing the same way, with the same wrong category, format or tone.
• You have labelled examples - a few thousand good input and output pairs to tune against.
• You have genuinely tried prompting and RAG - because a fine-tuned model carries an ongoing maintenance cost that prompting does not.

On her classification task it paid off, lifting accuracy from 0.85 to 0.91 and closing much of the gap to Opus at a small fraction of the cost.

On the human escalation task it failed outright, scoring 1.5 out of 5. Writing a brief that explains what happened in a conversation needs the complex reasoning that frontier models still hold.

The finding she flagged as most instructive came from the same run. Sonnet outperformed Opus on that escalation task, even though Opus was acting as the judge.

Smart Routing

She was candid about where the framework stops. It works for repeatable tasks, and a great deal of real work is something new where nobody knows in advance which model fits.

The direction she sees is smart routing, where the system classifies the task and picks the model against cost, quality and performance objectives. Her framing of the goal is worth keeping: not to spend less, but to maximise the value of every AI interaction.

Her closing demo showed it working in Omni, an open-source meta harness. Asked to build an app, it classified planning as a moderate task and routed to Sonnet, at a visible cost of 21 cents.

Budgets are enforced mid-run rather than reported afterwards. She set a threshold, several agents launched in parallel, and at the limit the run paused for approval so she could change the model or intervene if something had got stuck in a loop.

The routing decision changed when the task did. Asking it to investigate what had been built, an audit and debugging job, was classified as complex and routed automatically to Opus 4.5.

Q & A Session

The session ran to time, so two questions were taken at the close.

• For a team staring at a shocking first LLM bill, what are the quick wins?

  Viktoria Semaan: Put everything through an AI gateway first, whatever the sources are, and set your limits there. The value is having one place to control it all, because the alternative is spend scattered across services and separate bills arriving from everywhere, which becomes a nightmare to manage. Treat the gateway as your control plane before optimising anything else.

• What saves more, switching models or caching and prompt trimming?

  Viktoria Semaan: Models, and by a distance, because the price difference between frontier and open-weight models is striking enough to dominate any other lever. After that come the framework-level changes, prompting and RAG. The third area she pointed to is memory, which is caching by another name: with long-term and short-term memory in place, similar questions do not have to reach the model at all, which reduces cost and improves quality at the same time.

Her final takeaways stayed close to where she began. Build the evaluation set, access the latest models through a gateway rather than through hard-coded APIs, and fine-tune only when the three signals genuinely appear.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.