Testing LLM Responses You Cannot Predict [Testμ 2026]

A Different Answer Each Time

Almost every product now has a chat box attached to it. Far fewer teams can say with confidence that it gives correct answers, stays civil with users, and will not explain how to do something dangerous.

His example is the population question, which people have been using informally as a probe. The five cities come back differently on each run, which raises three separate questions rather than one.

Are these the correct five, given that the answer legitimately changes over time? Do they appear in the right order? And if the order is wrong, is the answer still true?

Anyone testing an application that uses a model as the origin of something needs a way to answer that. Two years ago the word for the failure mode was hallucination, and while the models have improved, so has the demand placed on them.

The Filter Agents Lack

Gil Zilberfeld’s framing for why this feels manageable in practice is that people are running a filter without noticing. Reading a chatbot answer, your brain marks one part as fine and another as dodgy, discards the second, and sometimes steers the conversation back with a follow-up question.

That works well enough as a user, and it is why the problem stays invisible. It stops working the moment nobody is reading.

So the standard has to be higher than what a person will tolerate on screen. Testing a chatbot, or anything relying on an external brain, means being able to evaluate how good it is well enough to answer the only question that matters at the end.

Someone will ask whether this can be released, whether it is ready, and whether it is good enough. That answer has to be qualified rather than felt.

A Funnel of Finer Sieves

The shape he proposed for the answer is a funnel. Results pass through a sieve, each stage finer than the last, and the more stages a result survives the better it is.

What the funnel replaces is the habit the industry spent two decades building. A hundred green tests used to mean everything was fine, and that is no longer the case.

He was blunt that this is not a passing inconvenience. The non-determinism is not going away, so the testing model has to adapt to it rather than wait it out.

The Test Planning Agent

Rather than argue it abstractly, he demonstrated on an agent he built. It takes an API endpoint and returns a test plan, and the code is on GitHub for anyone who wants to follow along.

The architecture is deliberately ordinary. A client sends a request such as give me a plan for this API, a Python server holds the agent logic, that logic wraps the input in a prompt and sends it to the model, and the result comes back to the caller. The demo runs on Gemini.

The prompt casts the model as an expert automation engineer and requires four sections: an API explanation, happy path, unhappy path, and edge cases. Those four section names matter later, because they are the first thing a test can look for.

He also generated the test plan for the agent itself by asking a different model, which returned API tests as black box work, agent unit tests as white box work, and suggestions on performance and reliability. His aside on that is worth keeping: a coding agent with access to the code produces better tests than one working from a description.

System Tests Prove the Code, Not the Answer

The API tests do what API tests do. Send a GET or a PUT to an endpoint that only accepts POST and it returns a 405; send malformed JSON and the error handling catches it.

All of that confirms the code behaves correctly, and none of it says anything about the response itself. That gap is the entire subject of the session.

He showed the limit with a relevance check that reads well and proves little. A generated plan for a POST users API says it should focus on creating users, which is true and empty, because the agent has no context about what will actually be sent to it.

Supplying that context is a human job, done by review and iteration. His point about timing is the one testers tend to resist: you usually cannot state at the beginning what correct means, so you define it as you go, the same way software gets built.

Scaffolding and Plumbing

Around every model call there is ordinary deterministic code, and he argued it deserves ordinary deterministic tests. Run the model twice and the answers differ; run this code twice and it does not.

The payoff is diagnostic. If the surrounding code is covered by tests that pass repeatedly, a failure narrows to the part nobody can pin down, which is where you want to be spending attention.

• Scaffolding - the logic that runs before and after the model call, tested with mocks and a fake key so the tests never reach the model at all.
• Plumbing - the integration layer, including what happens when the call returns an error and how that error is handled.

He anticipated the objection that this is developer work. His answer inverts it: if testers are not writing these tests, they are expecting developers to, and for developers to be able to, the code has to be built testable in the first place.

Sanity Tests as the Lowest Bar

The first stage that actually touches the model is the smoke test, and its definition has not changed. If this fails, nothing beyond it is worth testing.

For his agent that means checking the response contains the four sections the prompt demanded. A plan missing the API explanation or the edge cases goes back to the developers rather than forward into deeper analysis.

A second check looks for the entities in the request. Asked about a get user by ID API, a plan that never mentions a user or an ID has failed the lowest bar there is.

He includes speed in the same tier, on the grounds that good enough can be defined on any axis you care about. If a response has to arrive within a certain time and does not, it is not good enough.

There is a budget argument underneath all of it. Tests that call the model cost tokens, tokens cost money in CI, and stopping early is how that bill stays sane.

Golden Data Sets

Past sanity, the question turns semantic, and he made it concrete with a generated plan for a get user by ID endpoint. Shown that plan, some people would call it fine and others would say things are missing.

The instability makes the point for him. Maybe two test cases is not enough and three is; run it again and you get three happy paths but only two unhappy ones, because it changes.

A golden data set is his answer: a small written record of what a correct answer looks like, section by section. The API explanation should say the endpoint retrieves single user profile information; the happy path should include retrieving an existing user with a valid ID.

The qualification rule is yours to set. A section might need at least two named examples, or one of two acceptable ones, and a plan that comes back without them is a failure by your own definition rather than by a machine’s.

Scorecards, Not Pass or Fail

With a golden data set in place, a response can be scored rather than judged. His scorecard records the endpoint, the date, whether the explanation was correct, whether the happy path section held up, and a rating per criterion.

He was direct that the rating is subjective, because a human is giving it. Averaging across criteria produces a score, and every run adds a line.

Enough lines make a baseline. If the average clears the bar the release goes; if it does not, the system gets tweaked.

Parts of the card can be automated even though the definition cannot. Checks that a plan for a read endpoint never proposes delete or update cases, and that the sections appear in order with happy path above unhappy path above edge cases, run without a human in the loop.

The PII Bug and Prompt Fixes

The most useful stretch of the session is a bug story that starts with a reasonable request. A product manager wants the plans to be more practical, so a developer edits the prompt to require concrete example values in the test cases.

The next run produces a plan containing something that looks like a credit card number. The sections are all present and the output is worse than before.

He fixed it the way bugs get fixed, by writing tests first: a mocked unit test asserting the value is absent, and an integration test asserting nothing resembling a credit card appears anywhere in the response. Then the prompt gained an instruction to identify whether the API deals with personal data and to avoid real-looking examples of it.

Whether that counts as a fix is where he pushed back on his own demo.

A prompt-level fix works most of the time, and most of the time is not what a fix means. Something in the code has to enforce it as well.

The new requirement then rippled into the scoring. He wrote a fresh golden data set saying responses must not contain sensitive non-public user data, and added a security and privacy section to the expected structure.

Run against the new criteria, the same output scores lower, because it carries no security cases at all. He described it as the same result seen through new glasses, and it was a good plan two days earlier.

Snapshots Versus Trends

A score answers whether the system is good enough now. It does not answer whether it will be good enough tomorrow, and that is the question he ended on.

Run the same evaluation for a week and the numbers land somewhere in a band rather than on a point. If the scores sit between nine and eleven, someone has to decide whether that band is acceptable.

Expecting a full score every time is the wrong target, because the ground moves underneath. Models change between versions, and behaviour shifts within a version across a day.

Three more concerns close the picture. Token cost has to be understood and managed, prompt hijacking widens the attack surface, and an AI judge is available as a final stage where a stronger model evaluates the response your system produced.

His summary of the whole shift is that it is still what testing was before, expanded. The same discipline applied across a much wider space.

Q & A Session

Two audience questions closed the session, and he flagged that each deserved half an hour of its own.

• What is the best approach to test a chatbot against prompt injection and jailbreak attacks?

  Gil Zilberfeld: Security used to arrive as outside expertise, and thinking about it is now part of the testing job. The sequence he gave is to work out what can happen, understand how the system is built, and place guardrails where that understanding says they belong. His own repository contains an example where the agent can be hijacked into returning a joke instead of a test plan, which makes the concrete question how you would even notice. That leads to looking for suspicious words in the incoming prompt and cleaning them before they reach the model, then checking that whatever leaves the agent and the chatbot is presentable in terms of safety and bias. His warning is about imagination rather than tooling: if you have never heard of prompt injection or history poisoning, you will not test for them.

• Which parts of an agentic system should be deterministic to make failures reproducible?

  Gil Zilberfeld: Every part you cannot allow to be wrong. He described teaching a coding agent to follow test-driven development, writing the failing test before the code, which worked most of the time because it was only a prompt; occasionally it skipped the step and apologised politely for it. The exercise is to identify what is not permitted to be non-repeatable, then either pull that logic out of the model or wrap it in guards that make it behave deterministically. Underneath sits the same asymmetry he opened with: people ignore what is obviously wrong because they recognise it, while an agent has to be told. Since prompts are the new programming language and bug fixes inside them are not reproducible either, this becomes a design question about how much of the system must be deterministic. For testers it is also a testability question, because a black box with moving parts makes red and green meaningless when a rerun gives a different answer.

He closed on the role rather than the tooling. The prediction that AI replaces testers reads backwards to him, because the work moves toward understanding risk and defining what good enough means.

Manual testing and automation testing are the old division in his view, and the job title he offered instead is quality architect. Do that work for long enough, he said, and you can actually trust your chatbots.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.