Cloud Assumes You Know What a Request Will Cost [Testμ 2026]

The Assumption Under Cloud

Ojus Save opened with the claim that every piece of cloud infrastructure anyone has ever used makes one assumption about the workload, and makes it before the code runs: that the cost of a request is knowable at the moment the request arrives.

Four separate systems rest on that single belief. You size a timeout against it, forecast capacity from its average, rate limit on it, and bill by it.

The belief has held, he said, for as long as we have been putting things behind an HTTP endpoint.

Agent workloads break it, and they break it right at the source rather than at some later failure point. An agent decides what work to do while it is executing, so tool calls, model invocations and retries are only settled once the run has finished.

One Request, Many Decisions

He described a slide he said shows the whole problem in one picture. At the edge on the left there is one request, one prompt, one HTTP call.

The load balancer counts it as one. The rate limiter counts it as one. The billing system counts it as one.

Inside the runtime it is not one thing. It fans out into a sequence of decisions, and those decisions get made during execution rather than before it.

His worked example of the dependency: the agent picks its next tool call based on what the last tool call returned. It could read a file, and what is in that file determines whether it reads three more or none at all.

He called that mismatch the entire talk, with everything else following from it.

The Admission Event

He named the instant infrastructure accepts a request the admission event, arguing the failure originates earlier than most people expect.

Four commitments are made in that same millisecond: assign a timeout, take a slot in a worker pool, open a billing record, decrement a rate limit bucket.

What the system actually knows then is thin. There is one prompt. The tasks are unknown and the compute required is unknown.

He rejected reading unknown as hard to estimate. It is undefined. No number is sitting somewhere that a smarter system could have looked up, because the workload has not been discovered yet, and the only thing that discovers it is running it.

The cruel part is that the request returns a 200 and looks great. Nothing in your monitoring tells you a guess was made, and nothing in your invoice tells you which guess was wrong.

The Audience Bet

He asked attendees to commit a number, in their heads and in chat, so they would be wrong on the record: how many steps would show up in the execution graph, four, ten, twenty?

Guesses read out during the session included 20, more than 20, and 100. They are unvetted audience guesses rather than data about agent step counts.

The instruction he gave the agent had five parts: investigate the incident, find the cause, fix it safely, deploy a preview, and verify the result.

The actual prompt was “Investigate why checkout latency spiked after the last deploy and ship a safe fix”, which he said he ran on a Tuesday, twice, against the same repository with the same agent.

Five Nodes, Then Seven

Run one is the one that ends up in the demo, in the capacity model, and in whatever spreadsheet somebody used to price the product. The run you would typically design for.

He was explicit that run two is not the error case. The agent encountered evidence and responded to it, which is exactly the behaviour you want. It found more work because there was more work to find.

No dollar, token or duration figures were shown for either run, so the cost comparison in this session is qualitative and rests on node counts alone.

Node Count Misleads

He then told the audience not to walk away with seven either, because node count is misleading, and the way it misleads is the part that matters.

The first extra node is a patch, which means re-reading the surrounding code plus everything the agent learned in the previous four steps, so it enters the model carrying the accumulated context of the entire run so far.

The second extra node is a full verification pass that reruns the suite from the top.

Both are more expensive than the equivalent step in the five-node graph, for the same reason: the context they drag along has grown.

Cost is therefore not a function of node count. It grows faster than node count, because every decision is made against everything that came before it, and the pile keeps getting bigger.

His three closing questions for the section: which run was your timeout sized for, which one was your worker pool sized for, and which one was somebody picturing when they priced the product? The admission event made you pick before you knew.

The Run as the Unit

He flagged his argument as bigger than “agents are unpredictable” and invited the audience to disagree, saying he likes making unpopular takes.

The argument is that a request is the wrong thing to price and the wrong thing to bound. A request works as a unit of accounting because of one property, that the cost is knowable at the start, and once that is gone there is little left to work with.

He used deliberately hypothetical caps to show the break. Say run one must execute in one minute: what happens when the path fails and needs longer? Say the request should only cost $5: what happens when something fails, it makes more runs and goes past the budget? Would you rather your request simply fail?

What you want instead is the run and its budget. Something bounded that owns a body of work and holds state across a duration rather than an instant, carrying an allowance set at the top: a token ceiling, a dollar ceiling, a wall clock ceiling.

When the run exceeds its budget the run stops, because the run is the thing the system can actually tell exceeded.

He conceded that many platforms he has seen do not give you this cleanly today, and framed the rest of the talk as the three problems that follow, plus what you can do about each while the industry catches up.

Compute and the Fat Tail

He split one execution into three separate infrastructure problems, noting that solving one does not get you the others. The execution graph asks what work exists, compute asks what can run right now, environment asks what can run safely, and context asks what has to persist. Most agent frameworks answer the first thoroughly and hand you the rest, which he called fair enough, since frameworks are not infrastructure.

Problem one is compute, shown as a slide state: every slot in the pool occupied, discovery tasks sitting in the queue, and no traffic spike. The request rate did not change. The same number of prompts came in, and they turned out to be bigger than the ones before.

Capacity plans miss it because they forecast from an average, and an average is only a good summary of a distribution with a thin tail.

Agent run cost has no thin tail. A branch does not add a fixed amount, it multiplies what follows it, and a branch can contain another branch. That is a compounding process, and compounding processes produce a body of cheap runs and a small population of expensive ones rather than a tidy bell curve around a mean.

That leaves a bad trade. Provision for the mean and you queue exactly when the work matters most; provision for the worst and you pay for idle capacity nearly all the time. His answer is not a bigger pool but a pool without a fixed size.

The action you can take today, requiring no infrastructure change: pull your tokens, tool calls, duration and dollars per run, plot the distribution, and look at P95 and P99 rather than the mean. He predicted many teams would find the gap between their median run and their worst run larger than anything their capacity plan accounts for.

Invisible Failures

He spent a minute on a boundary case he called where failure becomes invisible, and invisible failures are the expensive kind.

The scenario: your model provider is degraded, so the next step in the graph cannot run. You get ready work sitting in the queue alongside low active compute at the same time, which should feel wrong, and adding workers does nothing, because more workers cannot unblock a dependency. Meanwhile the run still owes an outcome to whoever asked for it.

Put a timeout sized for a normal request on top of that and it fires partway through a legitimate run. The caller gets a response and it reads as a completion, as if the agent got through the investigation, wrote a partial patch, and got cut off before verifying.

What surfaces upstream is a plausible summary of work that did not finish, because summarising is one thing these systems are reliably excellent at.

Every token up to the cut is on the bill, nothing usable came back, and the accounting probably recorded it as a success. Error rate stays flat while cost per useful outcome climbs, and those two numbers live in different dashboards, so this can run for months without anybody connecting them.

His two fixes: separate the deadlines, because per-tool-call timeouts and per-run budgets do different jobs and most teams have one number doing both badly; and make the run’s completion state explicit and machine-checkable, so a truncated run is distinguishable from a finished one without a human reading prose and forming an impression.

Environment and Context

Problem two is environment, which gets skipped because it does not look like an infrastructure problem until you try to run something. Scheduling a task is not the same as giving it somewhere to run.

Going back to the patch step of his own runs, the agent needed a checked-out repository, a shell, git, package installs, and a process that can build the project, run the test suite and drive a browser. That is a machine, not a function invocation. And because the agent decided mid-run that it needed to run tests, that machine could not have been provisioned at admission either.

The environment requirements pull against each other. It has to be isolated, because the agent is writing the code and then executing the code it wrote. It has to be disposable, because you will create a lot of them and idle ones put you back to paying for a pool you could not size.

Problem three is context, whatever survives when the model forgets, and it has three layers that get conflated constantly. Model context is the working set for one call, what the model can see at that instant. Workflow state is the plan, the outputs and the status, what was decided and what came back. Execution state is files, processes and artifacts sitting on disk.

That makes the naive fix of putting everything in the window the most expensive available way to store state, paying model prices for something a database would hold for a rounding error.

It also does not work. When the run is retried or resumed the window is gone, so if the agent’s plan only existed in the prompt, what you are calling a retry is a fresh run that merely shares the instruction with the old one. It will make its own decisions and you will pay for both sets of them.

Budgets and Authority

He offered three questions for drawing the state line, calling state a product decision as much as an engineering one.

Plan: what can be reconstructed? If you can rebuild it cheaply from something you already have, do not store it. He noted most people over-persist here.

Evidence: what has to be durable? Anything a human will be asked to trust later. If an agent claims it verified a fix, the evidence of that verification needs to outlive the context window, because an unverifiable claim sitting in a deployed pipeline is worse than no claim at all.

Authority: what may continue without approval? People leave this one implicit, and it is the one that costs money, because a run that discovers more work is a run expanding its own scope, and scope is spend.

He was blunt that per-request rate limits do not bind agent workloads in any meaningful way. One admitted request can consume unbounded downstream resources while the limiter counted it as one and moved on half a second earlier. Per-request billing has the same hole from the other direction, charging a fixed amount for an event whose cost you never measured.

The concrete version: give every run a budget at admission in tokens, tool calls and dollars; enforce it inside the run and not at the edge, because the edge cannot see any of it; and rate limit on what gets consumed rather than what gets accepted. He said none of that requires any platform to ship anything, which is his estimate of the effort rather than a measured claim.

Workflows and Sandboxes

The closing section is a vendor pitch mapped onto the problem taxonomy he had just built, and he presented it as such.

Two primitives carry it. Workflows owns execution order, meaning the graph, the state, dynamic compute and recovery. Sandboxes owns the environment, meaning isolation, file system, tools and lifecycle.

He said Render is solving two of the three problems that way, and described Sandboxes as still in development rather than generally available.

The third problem, context, he handed back to the audience, cutting the section short for time and recommending people look at what durable workflows are and how they can solve some of this.

He overran the slot, with the host noting around the 28-minute mark that one minute remained before inviting him to continue, so the final argument was compressed rather than fully delivered.

Q & A Session

One audience question was answered on air. The host said the remaining questions would be shared with the speaker after the session, with responses published later.

• How do I validate agentic AI workflows?

  Ojus Save: That is a very good question with a very broad answer, because it depends on the workflow you are optimising for and what the outcome is. My approach is to choose a framework for what I am building. Mastra is a favourite of mine, for its TypeScript SDK and because it is open source, and that is a hot take that will not work for everyone. Most of the time I know what my input is and what my output is going to be, and I judge the outcome against that. Use something that makes sure the workflow actually runs too, whether background jobs, retries or queues, or one of the many open-source and paid tools that promise durable workflows.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.