Redefining Test Data Strategy in the Gen AI Era [Testμ 2026]

Execution Is Fast, Its Input Is Not

He opens on the neglected surface. Test data has traditionally received far less attention than automation, CI/CD or infrastructure, and teams invested heavily in execution capability while the data needed to exercise those systems is still created by hand.

His hook sits on a question slide: AI has changed, but has your test data strategy?

His illustration is explicitly hypothetical, and he flags it as such. A regression suite might complete in twenty minutes, preceded by hours spent finding accounts, adjusting records, coordinating with other teams or refreshing databases. No measured figure is offered anywhere in the talk.

The redefinition the rest of the session rests on is that cycle time is the entire span from test intent to usable data, rather than the runtime of the suite.

He argues AI-assisted development widens the gap rather than closing it, because AI can generate code, tests and scenarios quickly while the data processes underneath stay static and manual.

Three Forces At Once

Speed comes first. Release cycles keep shortening, which means test data provisioning has to become far more dynamic.

Complexity comes second. Enterprise applications are no longer one application talking to one database, and a single customer journey may involve multiple APIs, event streams, identity platforms, external integrations and several microservices, so a valid scenario needs coordinated state across many systems.

AI comes third. Traditional applications are largely deterministic, taking input through business logic to a validated output, while generative systems introduce prompts, context, retrieved knowledge, conversation history and probabilistic responses.

The consequence he draws is that the data you need to test is no longer limited to the rows in the tables.

His summing-up is that speed demands faster data, complexity demands richer data and AI demands context-aware data, which is why he treats test data as a strategic engineering capability rather than something you provision before a run. No organisation, survey or dataset is cited for any of the three.

The Limits Of Production Copies

He is fair to the traditional model before criticising it. Production data is copied, masked, moved into lower environments and consumed by automation, and it provides realistic relationships, realistic distributions and realistic business states.

The costs he lists are slow and expensive refreshes, carefully controlled access, and masking as another operational process.

His sharper objection is that production reflects what already happened, so it does not necessarily contain every scenario you want to test. His miss case is needing a specific combination of customer type, account status, transaction risk and failure, which can be hard to find intact.

The framing worth leading with is not the cost list. It is that the limitation is not that production data is inconvenient, but that a production-first model makes it difficult to engineer coverage intentionally.

The four workarounds he says teams fall back on are hardcoded values because they are predictable, small generators for known scenarios, SQL put directly into test code because it is fastest at scale, and datasets managed through spreadsheets and fed into automation.

Their cost is that the business meaning of the data ends up scattered across test code, SQL, spreadsheets, framework utilities and individual engineers’ heads. He explicitly declines to call any of them wrong: the issue is that none provides a common intelligence layer capable of understanding what scenario is required.

Testing A Generative System

The deterministic baseline is that you usually know the business rule and can predict what the output should be.

Generative systems break that, because the response is influenced by enterprise knowledge, conversation history, system instructions, user context and model settings.

His non-determinism claim is hedged rather than absolute: even with the same input, the output may not be exactly the same on every run.

The testing shift that follows is that you are no longer testing a single request, you are testing the full context around the request.

He adds a second category of test asset alongside customers, accounts, orders, payments and policies: prompts, context, conversations, knowledge, memory and adversarial scenarios. All of it is asserted as a design consequence on a slide, with no application, evaluation harness or adversarial suite shown or named.

Records Versus Scenarios

A record tells you what data exists. A scenario tells you what situation you are trying to reproduce, and he self-corrects mid-sentence to land on that phrasing.

His example carries the point. A payment record by itself does not tell you whether you are testing insufficient funds, fraud detection, a timeout, a race condition or an unusually high-risk transaction.

The payoff line is that the scenario gives the data meaning.

The starting question changes accordingly, from what data do we have, and can we find a customer or account that approximately matches the test, to what scenario do we need.

He deliberately undersells the change, calling it small but important. The scenario rather than the existing data becomes the starting point of the test, which makes the scenario the unit of testing.

Context Without Database Access

He poses the obvious question and answers it flatly. How does AI know which data is relevant, and does it magically know? It does not.

His point is that a general-purpose model may know what an account is without knowing that in your platform an account in a particular state cannot initiate a particular transaction, or that one API requires a relationship that exists in another system. The captions garble this passage badly, so it is paraphrased rather than quoted.

What has to be supplied is your data models, relationships, business rules, API contracts and existing testing knowledge, so that a request expressed in business terms can be translated into a technically valid scenario.

The second question gets the same flat answer. Should you simply give the model all your databases? No, because giving the system more intelligence does not mean giving it unrestricted access.

The split is that the model works against a metadata and context layer explaining schemas, business rules, APIs and domain knowledge, while any real data access or change happens through controlled data services enforcing permissions, read-only access, masking and auditing.

His division of labour follows from that: the model determines what it needs, and the governed system determines how the request can be safely carried out, without giving up security, governance or operational control.

The Worked Example

The intent he starts from is to validate a high-risk payment failure, which he notes makes sense to a tester or a product owner while being too broad to execute directly.

Context comes first: what kind of customer, what account, what payment risk profile, and the business rules determining when a payment should fail.

Reasoning then turns that intent into a specific scenario, namely a high-value payment from a high-risk customer that is expected to fail because of a particular business rule.

Only then is the synthetic data created to represent that scenario and sent to the system under test.

The ordering claim is the point. Nobody is generating random customers or payment data and hoping it proves useful, because the scenario defines exactly what needs to exist. The list of entities he assembles is partly lost in the captions, so the walkthrough is described here rather than reproduced item by item.

Volume Is Not Coverage

The design principle he wants remembered is to not generate data first.

The trap is that generative AI makes it easy to create large amounts of synthetic information, and volume does not improve testing.

The concrete failure mode is asking a model for ten thousand customers, transactions or test users and ending up with thousands of records representing essentially the same business condition.

The correct order is to identify the happy path, the boundaries, the failure conditions, the risk conditions and the meaningful combinations first, then let the generation layer produce exactly the records each case requires.

The benefit he claims is traceability. You can trace a generated record back to the scenario it supports, and determine whether important areas of the system remain untested, which he argues is a more useful engineering model than measuring success by the volume of synthetic records produced.

The LLM Plans, Systems Execute

His guardrail is that the model should provide the intelligence without becoming the entire platform.

What the model is for is understanding natural language requests, assembling relevant context, reasoning through possible scenarios and producing a clear blueprint of what needs to happen.

Once that blueprint exists, deterministic systems take over wherever possible. A generator creates a million records more reliably than a model producing them one by one, an API creates an account more safely than letting the model reach the database directly, and a validation engine enforces schemas and business rules more consistently than asking the model to check its own work. All three are hypothetical illustrations.

The factory he describes begins with the scenario and then chooses the appropriate generator, using direct AI generation for natural-language or conversational context and traditional tooling for large structured datasets.

He is explicit about not replacing what teams already own. Mature test data frameworks, approved APIs and existing services should be reused and orchestrated rather than replaced, because the intelligence is not in generating the data but in selecting the right operator for the scenario.

Validation And The Feedback Loop

He calls validation probably the most important control in the entire design, and his reasoning is a direct shot at demos. It is easy to demonstrate AI generating realistic-looking test data, and difficult to guarantee that data can be trusted, because a generated record can look perfectly reasonable and still violate a schema.

What has to be checked is schemas, referential integrity, business rules, data distribution and scenario coverage.

The loop draws on production signals showing what happened in the real world, test execution showing what is covered, and quality engineering showing where defects occur or coverage is weak. Those create new scenarios, which generate new data, which drives another round, so the test data stops being a static asset.

The full architecture he narrates runs from intent through the context layer, the reasoning layer, the scenario engine, quality engineering intelligence drawn from past defects and coverage gaps, the data factory, and then validation and governance, with results feeding back into future intent. It is a described architecture, not a running system.

Success is two metrics down and three up. Provisioning time and production dependency fall. Scenario coverage, reusability across teams and layers, and defect detection rise, especially for defects the current approach misses. No baselines, targets or measured results accompany any of them.

One sentence in this stretch loses its negation in the captions and reads as the opposite of his argument. His actual point is that generating millions of records is worthless if provisioning stays slow and production dependency stays high.

The Incremental Roadmap

He explicitly de-risks adoption, saying organisations do not need to build the whole ecosystem on day one and that the progression is intentionally incremental.

Metadata comes first, because a system cannot reason about enterprise data it does not understand, so the first move is building visibility into schemas, APIs and business rules.

Synthetic data follows in the areas where production dependency causes the most difficulty, then generative AI for scenario generation rather than for operational control, then governance and validation so generated output can be trusted.

Agentic workflows come last, and this deserves foregrounding because it inverts the usual framing of the genre. Orchestrating request understanding, generator selection, output validation and automation triggering only arrives once the underlying controls are mature enough to support a continuous learning loop.

The five shifts he closes on are production-first to synthetic-first, records to scenarios, static data to data intelligence, manual provisioning to governed agentic provisioning, and standalone test data management to a connected ecosystem. The first is hedged rather than absolute: production data may still have a role, but it should no longer be the default answer to every testing requirement, which is a demotion rather than the replacement the published description describes.

Q & A Session

Questions arrived in the attendee box and were read aloud by the host, four in all. No questioner is named, and the speaker asked for two of the four to be repeated, which is worth knowing as context for how thin three of the answers are.

• If AI can generate infinite test data, how do we decide which data actually matters?

  Manideep Singireddy: The governance layer is what decides what matters. Generating hundreds of records does not make them all useful, so you need validation points in between. He misses the question and asks for it again first, never describes how a governance layer ranks or selects data, and the sentence trails off before the host moves on. This exchange carries no chapter marker in the published list.

• Does generative AI created data fully solve the privacy problem, or just move it?

  Manideep Singireddy: Not fully. It just moves it. That is the entire answer, delivered in roughly ten seconds, with nothing on re-identification risk, residual exposure or how masking fits, and it is a notably narrower position than the session description, which lists masking sensitive information among the framework’s capabilities.

• How do you prevent defects from happening?

  Manideep Singireddy: That is a very hard question. A tester’s first job is to find a defect, and defects are hard to find. Then there are false failures: automation tests that fail because of the test data and get logged as tickets. He pivots there, and prevention itself goes unaddressed.

• How often should test data be updated to stop model drift?

  Manideep Singireddy: It should not be frequent. He says that twice, so it is not a caption error and is reported here as stated. No cadence, trigger, signal or reasoning accompanied it.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.