Scaling Quality in a Decision Intelligence Platform [Testμ 2026]

The Breaking Point

Hammad Ahmed frames the session around what he calls the breaking point. The application his team tests is not defined on conventional inputs and outputs, so the define-input and assert-output contract has nothing left to grip.

He states it flatly: the behaviour of the application itself is not predictable. He contrasts that with the e-commerce work earlier in his career, where the checkout process is very defined and you know exactly what steps need to be done.

The second break is pace. Every day and every month there is some new change to adapt to, and as a result even the acceptance criteria of what is being tested become vague.

Coverage then stops being definable at all. Asked what the test coverage of something is, he says it is very hard to define 100% coverage of anything, and especially so in an application whose behaviour cannot be predicted.

One attribution worth separating: the lines about there being no fixed output to assert and no script to anticipate emergent behaviour were spoken by the session host in the introduction, not by Hammad Ahmed, although the video description reuses them as though they were his.

From Excel To Decisions

The status quo the product replaced was decision intelligence based on a lot of manual Excel file reading plus analytics. Entirely human-driven, slow, and error-prone, because humans are subjective.

His scaling objection is concrete. For a Fortune 500 company taking decisions across a sector and against competitors, manual Excel analysis does not scale well.

The product vision he describes is to use AI to compile all the data into one place and show executives and managers the patterns and the decisions they can take, on the phone, in a few seconds.

He characterises it as a shift from humans to agents and tools controlling both decisions and the actions taken on them. Then he adds the line that defines his job: that is the product we were testing.

The Scaling Challenge

The initiative did not start in QA. Hammad Ahmed says the CEO set the team a challenge about scaling fast, and that a conversation with the CEO produced the decision to build a quality super agent internally. He dates that pressure inconsistently across the talk, so the timing is best treated as approximate.

He frames it as a good problem rather than a crisis, since the mandate arrived because the company was scaling fast.

The stakes he attaches are absolute in his telling: dealing with Fortune 500 companies, the team cannot afford mistakes and there is no room for error. No client is named anywhere in the session and no incident is described.

He calls the platform mission critical, because if it is down, customers cannot run their analysis and cannot predict or act on it.

Breadth multiplies that risk twice over. The product serves different verticals inside different Fortune 500 companies, so no single domain’s knowledge suffices, and it runs on web, mobile, iPad and watch, growing on all of them.

Seven Agents, One Super Agent

The design decision he emphasises is plurality. The idea was not to build one agent but multiple agents that correlate and talk to one agent, which is the quality super agent.

On the critic agent he offers a translation for people using different vocabulary: what AI-native companies might call evals and models, his team calls the critic agent.

The end state he describes is swarming, with all the agents brought together and, in future, allowed to talk to each other inside a workflow supporting autonomous QA.

Conventional Tools Kept

This is the point where the session diverges most sharply from how it was billed. The video description says the team rebuilt QA from the ground up. Hammad Ahmed says the opposite: all the conventional QA tools were never disregarded and were not put to one side, and the team actually used all of them to build on.

The automation stack he names is Playwright, Appium, Behave, visual regression, and cloud devices. He mentions TestMu AI’s own cloud devices among them, at TestMu AI’s conference, so treat that as a vendor mention inside a vendor’s event.

For documentation and context the sources are Jira and Confluence, where a lot of documentation context already lives.

For observability he names Datadog, Sentry and Langfuse, which let the team see how the application performs in production, including errors, warnings and performance.

For collaboration it is Slack and GitHub, used internally, and these later become the delivery surface for the agents themselves.

His rationale for reuse is speed of learning as much as capability. They are building agents, and they want to use the tools they already have to scale faster and learn faster.

Grounding In Product Context

His premise for grounding is blunt. AI hallucinates, and without grounding or context it will hallucinate a lot.

He acknowledges the newer generation of models has improved without conceding the problem: with new models arriving it is somewhat less, and it can still hallucinate a lot.

The first grounding move was feeding product specs by connecting the agents to Confluence and Jira, which he describes as enriched data already sitting in the team’s arsenal.

A second problem surfaced that specs did not solve. The agents were not speaking the product’s language, so they did not describe the product in the company’s own terms.

The fix was documenting business rules by hand. The platform is industry-agnostic, and in Fortune 500 companies there are things very specific to those companies. He concedes that LLMs are experts in general industry knowledge but not in that company-specific layer.

His stated goal for business rules is testing hygiene, avoiding false positives and false negatives. He undercuts the absolute himself a sentence earlier by admitting it does hallucinate sometimes and cannot be predicted 100%.

Test Cases As Documentation

The most portable tactic in the session is this one. The team fed their existing automated test cases to the agents as product documentation, not as tests to run.

His figure is an arsenal of around 10,000 test cases. He hedges it on air, and it is never broken down by platform or suite, so the number is his approximation rather than an audited count.

The argument behind it is a claim about where QA sits in an organisation. QA engineers are the closest to the product, they know each and every end of it, and they are the ones who created those test cases.

He generalises it into advice worth trying anywhere. If you want a product description, go into a repository of keyword test automation and you will learn how the product works.

The effect is reported only qualitatively, that it allowed the team to make the agents more concrete. No accuracy or defect-detection numbers are attached.

Skill Gating The Context

Here is the failure mode most teams hit second. After loading Confluence, Jira, test automation scripts and business rules, the context became bigger and bigger, and at some point the bigger the context, the more it hallucinated.

The fix he names is skill gating, adopted quickly after Anthropic released skills at around the same time. He gives no date for either the release or the adoption, and the ordering is his recollection.

Mechanically, the team built an internal agent that decides access, gating which parts of the product, the repository and the context each skill can reach.

He names an unglamorous second benefit alongside accuracy. You use fewer tokens, because if there is very specific knowledge you want, you go to that part and take only that.

The knowledge was split into what he calls small silos of information by domain: financial knowledge, R&D and organisational knowledge, manufacturing, and supply.

He calls the result a eureka moment, because it made the agents consistent and persistent in their predictions. In his framing, automated exploratory testing is a prediction of whether there is an issue or not.

Building A Quality Pulse

The trigger was round-the-clock operation. Automation runs 24/7 and the agent does not sleep anymore, which raised the question of how anyone knows something is wrong in production while the agents work through the night.

Three pulse dimensions are named: AI accuracy of the application, defect leakage, and tenant health. No values, thresholds or trends are shown for any of them.

The pulse is fed by signals from the agents, and which agents feed it is not settled in the recording. On air he names a security agent and the PR reviewer agent, telling the team how much code is being shipped and at what quality, while the published chapter list instead names exploratory, PR review and root cause analysis.

The root cause agent’s contribution is historical pattern rather than incident alerting, telling the team what the major root causes have been across the application.

Distribution is deliberately wide, going to the executive team, the QA engineers and the leads, and openable on demand whenever anyone wants to look.

Living Inside Slack

He treats adoption as a separate problem from capability. Having a quality pulse was not enough, because it also needed adoption and actual use.

The mechanism is embedding rather than launching a new dashboard. The agents were integrated into Slack and GitHub, the tools living alongside the team, on the reasoning that he spends a lot of his time in Slack anyway.

The behavioural shift he names is reactive to proactive. Initially the AI was reactive and something had to be triggered to get anything out, which he judged could not scale because it depended on a human.

Daily briefings were the first proactive instance and, in his account, the first visible value. Everyone responsible for a team got a briefing about their team, their customer, what was happening, the major issues and how they were fixed.

Prioritisation is the second use, aimed at leads and managers, and he justifies it from scale-up experience where everything is important and nothing is less important.

The third surface is on-demand runs opened to the whole company. Developers, managers and the executive team can trigger security testing, a bug fix or a PR review, which he frames as QA providing a service to the company.

Toward Agent Chains

This part is explicitly fresh and unpolished. He says it happened very recently, after a conversation with one of his senior QA engineers while travelling, which is why it is not in the slides.

The chain he describes runs root cause analysis to bug fixing to test creation. Connect the RCA agent to the bug fixing agent and the flow follows, then connect the test creation agent and it creates the tests.

The concrete result he claims is a security testing agent that went on to fix a bug in their environment somewhat autonomously. No example bug, no metric and no demo accompanied it, and he hedges the maturity himself, saying there are things they can make more perfect.

Next on the roadmap, all unshipped: self-healing tests, and adding more and more agents to the swarm.

He names an open problem rather than claiming it solved. With so many agents talking to each other, how do you reach out to a specific agent? That is something the company is still figuring out.

Three Closing Takeaways

His first takeaway is to give quality a pulse. Quality cannot be reduced to one specific KPI or metric, his team uses a combination of quality drivers, and which drivers matter depends on each company, so the instruction is to identify the metrics tied to your own challenge.

The second is to build and orchestrate agents rather than single-purpose tools. He argues the build barrier has collapsed and creating agents is now straightforward in a way it was not eight or ten years ago.

He extends that into orchestration with a team analogy. Take a team of five people: if they talk to each other and bring different ideas in, good things come out of it, and agents should be treated the same way.

The third is to adopt AI into the workflow rather than treat it as a shortcut. He reports that in one year the team achieved what he thinks would have taken four or five years otherwise, which is his impression rather than a measured result, offered without a baseline or a definition of what was achieved.

He positions AI against the job-loss framing, saying he does not see it as a challenge but as a tool any person can use to improve their skills, productivity and efficiency.

The mechanism he credits for adoption is small and continuous rather than a big launch. Small nudges from AI agents inside the workflows are what let a team become more adaptive.

Q & A Session

Three audience questions closed the session, read aloud by the host. The askers’ names are not reliably recoverable from the captions, so they are not reproduced here.

• How can the latency and monetary cost of using LLMs as evaluators be optimised to match the speed of modern high-frequency development cycles?

  Hammad Ahmed: Right-size the model. We use task-specific models, and with mixture-of-experts models you do not need the largest available model for everything. Smaller models handle smaller questions and smaller tasks, which cuts cost and, because they are faster, cuts latency. We are also building small language models internally to keep up with the frequency of the development cycle. This is my own view. He gives no cost figures, latency numbers or before-and-after comparison.

• If your triage agent mis-routes, or your root cause agent gets it wrong, how do you catch that?

  Hammad Ahmed: Good question, because it has happened to us. The mechanism is a human feedback loop: a QA engineer, developer or manager who sees something wrong leaves feedback as a Jira comment or in Slack, we gather that and feed it into the agent so it does not happen next time. Any agent or AI should have a feedback loop. That concession is the session’s clearest admission of agent failure, and he describes no automated detection, confidence scoring or measurement of how often the agents are wrong.

• If AI can generate software faster than the organisation can understand it, are we creating an accountability gap?

  Hammad Ahmed: Yes. But the best field to be in right now is QA, because the volume of AI-generated code is growing so fast. I have friends who work at GitHub and the volume of code arriving is enormous, and we are creating a lot of tech debt. The gap cannot be closed by humans, so agents have to counter-check AI-generated code, and above all guardrails come first. Eventually context will be large enough and the feedback loop accurate enough that mistakes will not happen as often. The GitHub anecdote is hearsay from unnamed sources, and that closing optimism is offered without evidence.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.