Quality in the Age of AI-Written Code [Testμ 2026]

How Much Code Is AI

Somu Suryanarayanan framed the remaining 10% as the critical part: the human-in-the-loop step where the team reviews the code and makes sure it is written well before it goes to production.

Rahul Nethi kept the QE remit constant regardless of authorship. His team does not really care whether code was written by AI, by a developer or by a combination: they understand the requirements, identify the risk, build the right test and make sure the product works as expected. He expects the volume and speed of change to force that to shift eventually.

Reviews Now Outlast Writing

Somu Suryanarayanan gave the clearest before-and-after. Previously his team wrote automation, pushed a PR with very minimal review, and merged within three days.

Now the AI generates the code much faster, in less than half a day, and the code reviews take more than a day. The cheap step got cheaper and the expensive step got more expensive.

He named over-engineering as the specific burden. It is too much, and reviewers end up asking whether they really need this many assertions.

Nandini Srinivasan rebuilt the point into a general mechanism. The review takes longer because a human has to verify that AI understood what needed to be developed, and separately that the test agent understood that the corresponding test matches the code that was written.

Her conclusion is that a human parsing both sides is what keeps instability bounded. When someone does that, the instability spikes stay manageable.

Guardrails Versus The Spike

The host opened this question by citing recent industry research showing delivery speed and instability rising together. No study, author, publisher or date was named at any point, so it functions as framing rather than evidence.

Rahul Nethi, told by the host that she was putting him on the spot, declined to confirm it from experience. Little Caesars had not come across such issues yet, since they are early in the AI journey, and he described his team as the gatekeepers of what gets sent out.

Nandini Srinivasan accepted the premise as arithmetic. If code is generated exponentially faster than humans can digest it, instability will go through a natural spike.

She stressed that the guardrail has to start at the requirement rather than the test. If the agent chain runs from product requirements through design to code and then tests, the agent has to have understood the requirement correctly and know what customers expect before any of the downstream steps mean anything.

She also hedged her own speed figure in the same breath she gave it, offering 10x and immediately saying she was just giving a number and did not want to be taken literally.

Coverage Out, Risk In

Nandini Srinivasan argued the shift raises rather than lowers quality engineering’s standing. This is where QE gains prominence, and where its responsibility increases.

Her concrete prescription is a metric swap. Move away from traditional test coverage metrics and start talking in terms of risk mitigation.

The mechanism she named for capturing speed safely is a deterministic test harness plus proper automated contractual checks, specifically so AI-generated features do not compromise system availability.

She also pointed out that “test” is not one thing in this pipeline. Unit, integration, regression and what the end user actually receives are different questions, with a ton of services under the hood, so an agent told to write the tests has to be told which kind.

Later she demoted the old metrics without discarding them. Automation and test coverage have not gone anywhere; they are just a side artefact now.

The Research Test

Asked for an example from her own organisation, Nandini Srinivasan did not give one. She substituted a consumer-level demonstration instead, and the host accepted it without pressing for the organisational case.

Her illustration: open a brand new project with a general-purpose assistant, ask it to research an unfamiliar topic, and watch it report back that it is surfing and sifting and collaborating.

Then judge the output. If you have subject matter expertise on the topic, she says you will sift through what it produced and find that a large share of it is nonsense. That share is an offhand figure about an unnamed topic rather than a measurement.

She described the model’s response to correction as agreeable rather than reliable. Push back and you get a note conceding the point, acknowledging it missed something or overlooked it.

The point she extracted is that the safety property lives in the human rather than the tool. You can catch the error because you have the expertise; without it, you are blind, the model is blind, and you will simply accept the answer.

200 Cases, 70% Failing

Somu Suryanarayanan recounted the assignment first-hand. CookUnity ships a mobile and a web release every two weeks, and his manager told him to build the entire mobile automation suite, around 200-plus test cases, with no additional resource.

His first move was to ask Claude, and week one looked like a win. He had a framework, everything was working, and engineers could run it.

Week two inverted it. His manager came back and said 70% of the tests were failing, and the debugging did not converge, because it kept failing again and again. That figure is what he was told, relayed inside his own account.

His diagnosis was a knowledge gap rather than a model gap. He knows his app; the model does not, and neither does the tooling around it. So the question became how to teach that first.

He described the first-generation output as structurally unusable. It generated tests in multiple patterns, all of them flaky, none of them useful. He was explicit that this is a general lesson rather than his own mistake, and something everyone has to take forward before starting.

Teaching The Agent First

His fix was documentation the agent is compelled to read before generating anything. Write a good architecture document, a standards document and the patterns, then instruct the agent to go back and read all of it before doing any work.

The explicit anti-pattern he named is unconstrained regeneration. Do not keep generating it multiple times.

The second half of the teaching loop is negative examples. The agents need to learn the product, and to be taught the failures and the mistakes, before they start doing well.

His honest ceiling is around 70% accuracy, with reaching 100% still difficult and flakiness still arriving daily. He does not define what accuracy means there, so it should not be read as a pass rate or a coverage figure.

He named the productivity trade directly. Automating ten test cases in a day is not possible with a human, and at that rate some will be flaky, some unstable, and reviews still miss them. His overall verdict was positive but qualified: it has been pretty decent, and it comes with a big learning curve.

Debugging The Test First

This is the sharpest before-and-after in the session. Before AI, a failing test meant a product bug, a user-flow bug or an infrastructure bug, with very little possibility of a bug in the test code itself, because a human wrote it and it had been through multiple reviews.

After AI, the order of suspicion flips. The first instinct is to check whether the test code is correct, because Somu Suryanarayanan says he cannot trust AI 100%. Only after clearing the test does he ask whether it is a product, infrastructure or data bug.

The cost is triage time. It used to be easy and now it is messier, because the bugs cannot be pinpointed, so more and more time goes into analysing failures.

His response was to put an agent on the agents. The team is building a failure analysis agent, asking AI to triage the failures, because reviewing everything at once is too much for a human.

His reported distribution is the part worth carrying away. Most failures come from the test bug rather than the product: the agents are failing, and that is why the test failures appear. His bottom line is that he keeps fixing the agents so the tests do not leak product bugs, which is more work for QA rather than less.

Specification Beats Syntax

Ankur Agarwal, joining late, reframed the object of testing. Teams used to test the code, and now they want to test the process behind the code generation.

His illustration is explicitly hypothetical rather than an incident at his employer: a story to disable users inactive for 90 days. Without a clear definition of inactive, you may end up disabling users who were inactive and paying.

His failure mode for AI code is convincing correctness. AI can easily generate something that looks correct and misses the business intent, syntactically fine and not solving the problem you were trying to solve.

His ordering is unambiguous. The definition, the specification and the process behind how the code is generated now matter more than the code itself.

Nandini Srinivasan closed the same loop from the other end. If the spec is giving a problem, the product team did not understand what the customer wants. If the tests are giving a problem, the agent did not understand the spec that was generated.

Agents In Production

Asked whether QE shipping production fixes is where the field is heading, Nandini Srinivasan answered why not, grounding it in what AI is genuinely good at: automating things that are mundane or take a long time.

Her structural claim is that the lines between a developer and a quality engineer have blurred, and a QE should be able to read product code and test code and land a production fix.

Her headline numbers are self-reported with no artefact shown: more than 25 AI agents running in production on her team, and more than 300 production fixes generated in the last three months. She names Claude plus a second platform the captions do not render reliably enough to print, and gives no definition of what counts as a production fix.

She credited pre-existing capability rather than the tooling. They did not wait for AI to show up to build automation expertise; they already had it and took AI as an augmentation.

Her precondition for earning that mandate is product depth. Quality teams are the subject matter experts on their product because they understand how the customer receives it, how the developer built it and where the gaps are.

Her practical habit for building that depth is worth stealing: do not just ask what changed, ask how it changed and why, which repositories were affected, and understand the architecture properly. Landing production fixes then becomes the easiest artefact to come out of it.

Shipping Past Adoption

Nandini Srinivasan argued release velocity is being copied from companies with entirely different consumption patterns. Everybody wants to ship multiple times and have multiple releases, and not everybody is building a platform people live inside.

She contrasted a fashion retailer that does not need product updates ten times a day with social and streaming platforms that might. Those comparisons are rhetorical illustrations about companies with no involvement in the session, offered without a source.

Her ranking of assets is clear. Specs and tests have always been the asset, and nothing trumps customer success and customer adoption, because this is a human building a product for another human to consume rather than catering to a bot.

Somu Suryanarayanan made the same case from inside CookUnity, naming the delivery competitors whose features they mirror. Whether they need an update or not, they see it and want to bring the same thing into their platform.

He was openly sceptical of his own company’s AI roadmap, saying nobody wants to search by dietary restriction, people go to the app and search for the food they like and order it, and yet the team is pushed to build an agent, a model and chatbots.

His usage figure is the punchline, and it is an unsourced internal number: customer usage of those features is under 10%, nobody is going into that flow, and QE still has to test it.

Skills Worth Building Now

Ankur Agarwal explicitly rejected the host’s framing that code is no longer a moat. He does not agree that code stopped mattering; it has become easier to reproduce, while what stays hard to reproduce is the knowledge built up over time.

His recommendation is strong problem-solving and systems thinking, understanding how a system behaves, over knowing Python or Java. He also rejected the replacement narrative, arguing AI takes the redundant and monotonous work.

Nandini Srinivasan told engineers to stop investing in automation craft itself, using an elevator analogy: in an era without elevators you climbed the stairs, and once you have one you do not take the stairs to the 95th floor.

Her substitute focus is risk and scale. A hundred to a thousand users is scale, a thousand to a hundred thousand is also scale, and the days of running everything every time are gone. Her concrete ask is whether you understand distributed systems and how they fail at scale.

Rahul Nethi supplied the panel’s most quotable warning, that AI lets us generate code much faster and therefore build the wrong things faster. He described Little Caesars onboarding as domain-first, walking new joiners through the whole ordering flow from web or app through the kitchen and back to the customer, because a framework can be taught in weeks while the business and customer domain takes much longer.

Somu Suryanarayanan closed on that distinction, and argued QEs need little new beyond prompt engineering basics plus keeping the QA hat on to interrogate AI output. He opens that answer by saying quality engineers do not need to learn anything, then names a skill one sentence later, so the two halves belong together.

One structural note on this session: every question came from the moderator, and no audience questions were reached. The host apologised at the close for not getting to the questions posted during the panel.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.