ML-Driven Test Intelligence at Scale [Testμ 2026]

Generation Is Cheap Now

She opens on the observation that generating tests is getting cheaper. You can ask a model for test cases, API scenarios, Gherkin, edge cases, even automation code, and that is useful. It is also, in her view, no longer the hard problem.

The hard problem she names is what to test next, when the system changes continuously, model behaviour is probabilistic, and prompts and retrieval shift underneath you.

She defines her subject narrowly and sticks to it: not more test generation, but better testing decisions.

She separates it from prompting explicitly. Asking a model for twenty test cases is generation, and generation is useful but different.

Test intelligence, in her framing, is a layer that continuously asks three questions she puts on screen: what to test next, how strongly the expected behaviour can be asserted, and how much the thing doing the evaluation can be trusted.

The Activation Experiment

The origin story is a small experiment she ran herself with GitHub Copilot before the session. In one context she asked it to fetch a user’s account balance by user ID and it returned a parameterised query, which she calls a very good secure pattern.

In essentially the same code base she added a little more surrounding complexity and asked for a date filter, and the assistant fell back to string concatenation.

Her reading of it is careful, and worth preserving exactly. She does not call the second answer wrong, and never uses the words insecure, vulnerable or injection anywhere in the talk. The point is that this was not a knowledge problem but an activation problem: the model has the capability, and the surrounding context decided whether that capability showed up in the output.

Both results appeared as static code on a slide. The experiment itself happened off camera, with no prompt text, model version or date given, so it is a single unreproduced anecdote rather than a property of the product.

She says she also ran an eleven-case battery against the same assistant to ask a different question, how stable a known capability is when context changes. No scores, table or results from it appeared on screen, and the rubric she names is not recoverable from the recording.

She then puts a show of hands to the audience about seeing an assistant produce the right pattern in one place and forget it twenty lines later, and answers it herself with the assertion that this happens to everyone. That is a rhetorical device, not a measurement.

Assumptions AI Breaks

Summarising AI behaviour as non-determinism is too shallow, she argues, and instead names at least four specific assumptions that traditional automation has depended on. She hedges the count herself.

Output is unstable, which she says everyone already knows. Less discussed is that the oracle is also unstable, because sometimes there is no single exact expected sentence or answer.

The change surface is bigger. Behaviour can move because the model changed, the system prompt changed, the retrieved context changed, or because more than one tool call is involved.

Coverage becomes uncomfortable. Her rhetorical question is what 80% coverage even means when context combinations are effectively unbounded, and that figure is an illustration rather than a number from her environment.

The resulting shift she names is from verifying output to measuring behaviour, which she concedes sounds subtle while arguing it changes the whole strategy.

She makes it concrete with an API contrast. For a traditional API she would assert a 200 status and a known schema. For an AI workflow she also has to ask whether the tool selection was right, whether the retrieved evidence was actually used, whether the model preserved a policy constraint, and whether it did so consistently across context variations.

Scale Beyond Test Counts

She is careful in her phrasing: she is not talking only about thousands of test cases. Scale widens rather than narrows, and she lists four ways it shows up.

Variance comes first, across different prompts, contexts, tool calls and user journeys.

Change comes second, across model versions, prompts and retrieval. The recording garbles her label for this one, so it is given here in her own terms rather than as a single word.

Privacy is third, and she frames it as a constraint rather than a preference: the best test data may exist in production, and production logs cannot simply be thrown into an external model.

Auditability is fourth. In regulated systems somebody eventually asks a boring but important question about why a particular test became a release gate, and you need a real answer.

Her conclusion from the four is that execution is no longer the bottleneck. The bottleneck is deciding which signals matter and which assertions deserve the trust.

The Closed Loop

She presents a closed-loop diagram she calls the centre of the talk, with two entry points feeding one pipeline.

The first entry point is known risk, and she names secure coding, hallucination, tool misuse, authorisation and compliance. That becomes the offline evaluation set.

The second is a production signal that tells her something she did not already know: strange user journeys, a rare sequence, a new failure pattern, or behaviour that never appeared in the authored regression suites.

Both signals run the same pipeline as she narrates it: observe, prioritise, reconstruct enough context to reproduce the behaviour, define the assertions, validate them, let CI or a human make the decision, and feed what was learned back in. Her own closing mnemonic later is shorter, dropping three of those steps, so both versions are given here rather than merged.

Her test for whether the loop is genuinely closed is the line the talk turns on. If a production incident teaches her something once and her test system learns nothing from it, the loop is not really closed.

Her worked example, flagged as hypothetical, is a customer journey where login succeeds, a downstream service retries twice, and the transaction enters a state never represented in pre-production. The intelligent response is not just fixing the incident but asking whether that observed journey should now join the permanent evaluation set.

Prioritising By Risk

She says plainly that she likes models and uses them, and that she does not want an opaque model making an uninspectable release-risk decision. That is the reason she derived an explicit prioritisation approach, which she introduces haltingly and hedges as a framework of a kind rather than a named method.

The things she weighs are novelty, meaning whether this is genuinely unusual; exposure, meaning how often users could hit it; impact, meaning how bad the consequence is if it fails; and reproducibility, meaning whether she can reproduce it reliably enough to act on it.

She allows a model to help estimate some of those signals, while insisting the decision itself stays inspectable.

Her rejection line is the memorable part. If the answer to why something was classified high risk is that a model said so, that is not a risk model, it is just a number.

She contrasts two candidates to show why one score is not enough. A rare formatting variation can carry very high novelty and almost zero impact, while a slightly unusual authentication sequence can be less novel and very high impact, and the two should not be handled the same way.

Three Layers Of Evaluation

Because no single signal mechanism catches every failure mode, she argues for a layered strategy: manual review with a human spot-checking, automated evaluation in CI, and production monitoring.

She compares it to the Swiss cheese idea applied to AI evaluation. That is a borrowed safety-engineering analogy and she attributes it to no author.

Each layer has a stated weakness. Manual review catches nuance and does not scale. Automated evaluation scales much better and inherits the weaknesses of its assertions and its judges. Production monitoring shows reality, and by definition shows it late.

The design constraint she draws out is that the blind spots have to differ across layers, and she calls that the important part.

Her failure case makes it concrete. If every layer ultimately asks the same model the same question, you have not built three layers, you have built one dependency three times.

Redact Before Intelligence

She introduces LogMiner QA as an open-source tool she has worked on, and hedges its scope herself: one implementation of part of the closed loop, not her definition of test intelligence.

The problem it was built around is stated first-hand. Production was showing behaviour the test suite did not know about, carrying real user journeys that teams rarely see in lower environments.

She names three properties of production logs that make this hard. They are noisy, they are very sensitive, and unusual does not automatically mean important.

Her first challenge was not machine learning at all, it was privacy, and she says she took it first because she feels strongly about it. Her criticism of common practice is that many AI systems sanitise data only after it has already passed through the intelligence part of the pipeline, which she calls uncomfortable in regulated environments.

Her rule is to redact first. Raw telemetry enters, sensitive values are replaced before anything else happens, using regex and structured patterns, domain validators and spaCy NER. Only then are features or embeddings created and anomaly analysis run, and that ordering is the whole point of the slide.

Her example for stable tokens is correlation without exposure. If the same account identifier appears across several events in one customer journey, she does not need the real account number, but she does need to know those events belong together.

Anomaly Is Not Defect

The next trap she names is treating anomaly detection as defect detection, and she presents it as an assumption she personally had to unlearn.

A rare event might be a defect. It might equally be a very important customer, month-end processing, a new feature rollout, or a perfectly legitimate edge case.

For journey-level sequence anomalies she says she has used isolation forest or embedding distance, where it makes sense. No thresholds, volumes, precision figures or tuning detail are given for either.

She is explicit about what the detector is allowed to conclude. None of it declares anything a defect. Anomalies are candidates for test generation, and risk plus reproducibility decides whether a candidate deserves attention.

Her worked example is a payments flow, and she opens it with the word imagine. Suppose the overwhelming majority of payment journeys follow one path and a single journey retries and lands in manual review. The percentage she uses is an illustration, not a measured distribution.

Even for that interesting journey you still have to establish whether it represents a failure, an accepted recovery path, or a rare legitimate workflow before it becomes a regression test.

Weak Tests And Judges

This is the part she says AI testing discussions underestimate most. Models are very good at producing convincing-looking tests, with tidy names and tidy given-when-then, and sometimes the test proves almost nothing.

The hard part is the oracle. You have to ask what invariant should have held before generating anything, then validate syntax and semantics, replay, remove duplicates, and only then promote a test into regression.

Her weak-versus-meaningful example is a security one. Production shows repeated failed logins followed by a successful session. A weak generated test asserts that the user is logged in successfully, which proves almost nothing. The meaningful invariant is that after a number of failed attempts the expected security control must trigger and an auditable security event must be recorded. At that point the test has a reason to exist.

She generalises it into one of her unlearned assumptions: a synthetically valid test is not test coverage. The same oracle problem reappears when one model judges another.

On the judge pattern she concedes it is extremely useful and objects to the framing, because the word judge grants a level of authority it has not necessarily earned. Before letting one gate a release she wants human-labelled cases from her own problem space.

She then measures disagreement by slice rather than by average, across security, ambiguous responses, long-context cases and edge conditions, and defines a confidence threshold plus a human fallback zone. An unevaluated judge, in her opinion, is just another untested dependency. Her illustration of why averages mislead is hypothetical: a judge agreeing with humans most of the time sounds good until the disagreement turns out to sit entirely in the safety cases you bought it for.

The Small Next Step

She closes with the three assumptions she carried in from traditional automation and had to drop. Capability does not equal reliability, because seeing a model perform correctly once proves capability rather than stable activation. Anomaly detection does not equal defect detection. Generated tests do not equal coverage.

She adds a fourth on the spot, flagging it as an addition: calling something a judge does not make it objective.

Her advice is deliberately small. She would not tell anyone to build everything she just showed. Pick one AI-assisted flow where getting the decision wrong actually matters.

Then instrument it, capturing enough lineage that when behaviour changes you can tell whether the model changed, the prompt changed, the context changed or the tool path changed.

Then close exactly one loop. Find one production-derived behaviour, sanitise it, replay it, define the invariant, and promote it into the evaluation suite, keeping a human in that loop.

Her warning about automating judgement is the line worth carrying out of the session. The tendency in AI projects is to automate ambiguous decisions simply because a model can now produce a score, and in her opinion uncertainty is not a defect. Sometimes it is the most honest state the system can return.

Q & A Session

Time ran short and the host read out two questions. No attendee was named or appeared on camera.

• How do you stop log-driven test generation producing redundant, overlapping Gherkin test cases as log volume scales?

  Tanvi Mittal: So the question is really about generation producing unwanted test cases. One possible way is to review the generated cases manually, compare them against what already exists in the repository, and filter. Or use another model as an agent to compare candidates against the existing repository and filter the duplicates out. Both mechanisms are hedged. She names no deduplication technique, similarity measure or threshold, does not connect the answer to the duplicate-removal step in her own pipeline, and the answer trails off mid-sentence.

• What governance model works best for agentic QA?

  Tanvi Mittal: I have worked on a framework. It stands for two of these things, and there are two more. I can share more on my GitHub. That is a deferral rather than an answer: no governance model, control set, approval flow or comparison with any other framework is described, and the acronym is not recoverable from the recording, so it is not reproduced here.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.