AI Architecture Thinking: From Idea to Impact [Testμ 2026]

The Idea-To-Impact Gap

The host opens by asking how many AI projects the audience has seen start with a genuinely brilliant demo and then stop, never reach production, or fail over a few months without anyone noticing. It is a rhetorical question rather than a poll, though he asserts that almost everyone in the room has those stories.

His framing is that the model is not what failed. What failed was everything around it: the data, the context, the integrations, security, cost, governance, and whether a real person could actually use the thing.

He calls those the unglamorous parts nobody put on the slides with the demo, and names the distance between them and the demo as the gap between idea and impact.

Safeena Banu anchors her half on one real artifact rather than an abstraction, taking the quality engineering agentic stack her team built and working through the architectural lessons and the mindset shifts she had to go through.

Her governing metaphor is a layer cake, where a weak base cannot support what sits on it, paired with a cost argument she returns to repeatedly: correcting a mistake at the beginning is cheap, and it gets more expensive as the trajectory widens.

Inside The QE Agentic Stack

The team started with a single agent, thought it was groundbreaking because everyone was building one, and found it seemed like magic until it was not. The scaling question that changed her framing was whether the system could survive the task.

One agent became several: planning, authoring, a deterministic runner, a healer and a deployment monitor, with an observability stack she calls the most important element because the evals cut across the layers from test planning through to deployment monitoring.

The base is a Playwright BDD TypeScript stack for UI tests, with equivalent agents for APIs and non-functional testing implemented as skills the feature planner can call on demand depending on the scenario.

MCP is used heavily. The feature planner reads requirements from Jira through an MCP server, and the locator inspector uses Playwright MCP, which she stresses is read-only: it inspects the page, interacts with elements, takes a screenshot, and passes on what it collected.

Model routing splits by job. The locator inspector runs on a cheaper model, having moved up from a smaller one that was not giving her the results she wanted, while the feature planner uses a frontier model.

Test execution is deterministic and never touches the model. The tests are run, results collected, and only then is the failure analysed. Failure routing is conditional: a test failure goes to the healer, anything else informs the user. The workflow orchestration lives outside the code in a chat-based agent with a Slack trigger, and the platform name is not recoverable from the recording.

Inside The Recording

The run executes inside Codex. She explains her personal split, using Claude Code and GitHub Copilot heavily for authoring and Codex mainly as a reviewer, and says she used it here because that harness surfaces sub-agent activity visibly.

A human-approval guardrail is wired into the locator inspector. She has instructed it to ask her every time it navigates to a URL, and the approval prompt appears and is granted inside the recording. That is her concrete example of rules and constraints becoming guardrails.

The locator inspector completes and hands off to the feature planner, which writes the BDD scenarios and the step definitions. She stresses the planner works from a deliberately rich context, including prior review comments on earlier tests and explicit rules about what not to do.

The quality reviewer runs last as the final gate, and the gate is not just about passing. It checks conventions, a playbook and reusable framework steps before artifacts are produced.

She pauses the recording to open per-agent traces showing every tool called and every tool failure, which the team logs for later failure analysis.

Three artifacts appear on screen out of a candidate pack: a feature file, a locator file and step definitions. She says the system also identifies which steps should be promoted back into a shared framework pack sitting alongside per-microservice packs.

The Personal Learning Map

The second thing she shows is unrelated to quality engineering: something she built for personal productivity, a monthly AI learning map where she supplies her top priorities for the month and it generates a fresh focus.

It pulls from three lenses she names on screen, a founder lens, a builder lens and deep technical, drawing on official documentation and specific blogs she has pointed it at.

She expands an entry to show the underlying resources and describes opening one to start learning from it, adding that a user can put their own resources into the map. Whether this segment was live or recorded is never stated.

She has been running it since June, without naming a year, and uses it to track what she has completed and to write her notes, having moved off a note-taking tool she previously used because she wanted a use case to build against.

She offers it explicitly as ideation rather than a product, framing it as her way of being intentional about learning AI in a field that keeps changing faster than anyone can stay on top of.

Splitting An Agent

Her definition of the object under discussion is that an agentic QA system is not just a model in a wrapper. It is goal-based, and for every touch point you need to understand the failure modes and build around them.

Her reframing of the discipline is the sentence worth carrying away. You are no longer prompt engineering, you are building a distributed system with a probabilistic component, which is what adds the complexity, because deterministic systems tell you what to expect and therefore what to test.

The splitting rule is negative first. You do not split because multi-agent sounds more advanced, and if something can be a tool, a deterministic script or a function, it should be that rather than an agent.

The positive test has five conditions: split only when a capability has a distinct responsibility, context, policy, evaluation surface or failure domain. If you cannot answer all of them, it should not be an agent. Her compressed version is that agent boundaries follow ownership and failure domains rather than prompt boundaries.

She showed a table of every agent in her stack with what it owns, its boundaries and what it must not own, noting the agent owns observability, traces, evals, cost and feedback but should not take action, because deciding where the boundaries sit is the human’s job. The table’s rows were only partly narrated, so they are not reconstructed here.

Her cost framing is that multi-agent is a cost rather than an achievement, to be paid only where there is a return and where the failure domains are genuinely different. Her fallback advice is to keep it simple, iterate, fail often and learn before building the next layer.

Six Layers, Bottom-Up

You cannot ice a cake with no base, and the build order runs upward from governance rather than downward from the model.

She separates guidance from enforcement, saying you guide where needed and enforce when needed, and argues critical controls need a real enforcement layer because a prompt cannot enforce anything against a probabilistic system.

Her reason prompts cannot be controls comes from an analogy she credits to an unnamed source: a model’s output is like a weather forecast, so you cannot control it. What you control is the harness and everything around it, so the system produces a predictable outcome.

Across all six she tells the audience to identify the edges in the system that can mutate state and never ignore them, and to push anything deterministic out of the model. Running tests and capturing results does not need a model, so that became a deterministic script.

Autonomy As Delegated Control

Her one-line definition is that autonomy is just delegated control flow. Once you accept that, choosing a model becomes your responsibility and follows from the task the agent performs.

Guardrails are set by access and risk. Read access is the least, with no state change and audit as the only control. Write paths get a closed schema plus a path allow-list, idempotency keys, timeouts, typed failure handling and audit records. The exact list is partly garbled in the recording and is paraphrased here.

Sub-agent context isolation was a later fix rather than the original design. The locator inspector generated a lot of output, and each sub-agent has its own context window, so isolating it stopped those snapshots poisoning the context of the main agent running on the frontier model.

The same change was a cost lever. The read-only inspector got a smaller model and read-only tools because it does not need to do much, while the planner kept the larger one.

She compared three shapes, a micro pipeline, a router with workers, and a multi-agent system, across control flow, which states get modified, where each goes wrong and how to debug it.

On debugging multi-agent specifically, the trace is distributed, so you do not write it at the end. You build evals across the system and write the trace as you go, so you know at which point it failed. Her line on the pattern is that complexity should not be a reason to use an agent.

Tools Are API Contracts

Tools are like APIs and should be treated as API contracts. Decide what input you give, how much of it, and what output shape you expect, because if you do not know what output you want the model will not know either. Her examples of an output contract are three lines, something not too extensive, or JSON.

Permissions belong in the harness rather than the prompt. She points to pre- and post-tool-use hooks and to giving explicit permissions in the harness configuration, adding that ignoring files in version control is not sufficient, because she has repeatedly seen a model touch files it should not have.

Her sharpest security formulation is that a tool accepting natural language authority is not good, because you have moved your security boundary into the model.

Do not hand over the whole toolbox by default. As you would import only the libraries you need, give only the tools you need, and be careful about the tool signature.

Her takeaway for this layer is that a production tool should make the safe action easy and the dangerous action impossible, which is her design principle rather than an industry standard.

Her structural statement of the same idea is that the model proposes a patch and cannot apply one, which is a property of the tool signature rather than of the prompt. She also frames the tool layer as a place to save context by injecting the right context at the right time.

Context And Memory Discipline

Context is a finite resource. More context does not mean good context, and a shared source of truth is not the same thing as a shared context window.

She names her own team’s mistake directly. They started by giving the model everything, which produced context bloat, later context poisoning, model drift, and a model that does not focus on what it needs to focus on. Giving it everything feels generous and is the fastest way to make an agentic system both expensive and unexplainable.

Her personal working rule is to compact or start a new session once the context window crosses fifty or sixty percent, a habit she needed more before the agentic layer existed.

Context should be structured per stage. Planning gets requirements, documentation links, change diffs, risk history and prior defects. Authoring gets the API and UI structure, examples and the plan. Healing gets the failing trace and the failing test.

For execution data the team pulls from LambdaTest APIs because runs happen on a web grid, and extracts narrowly: take just enough, search for the error part, pass only the error-specific detail, and let the model ask if it wants more. LambdaTest is the platform behind the publisher of this recap.

Memory placement depends on access frequency and can be object storage, a Markdown file in the repository, the harness’s own memory or user preferences, with repository memory taking highest priority in her setup. Her hard boundary is that retrieved context is untrusted input carrying source identity, freshness and trust, while state is trusted, and the two must never be merged. She prunes aggressively, keeping carried-forward failure analysis in a docs folder and discarding one-off findings.

Believing The Heal

She showed a slide of escalating healing autonomy, running from suggesting a locator change at one end to changing a test in a branch, committing it and merging it while the organisation trusts the result at the other. Every step to the right increases the blast radius.

Her framing is that this is the same model with six different risk profiles and six different architectures. The model is constant and the granted authority is the variable. Only the two endpoints were described aloud, so the intermediate postures exist on the slide and are not reconstructed here.

Reads are recoverable and writes alter state, so the more irreversible the action the less autonomy you delegate, and that is a policy decision rather than a model decision.

On placing the human, she references a four-part model of AI fluency and singles out discernment, arguing you decide where the human enters because humans are the ones accountable. Planning is where it matters most, and the first stage is where the human in the loop counts. She names only one of the four parts and does not attribute the framework.

The approval loop she describes runs from the model proposing, through code validation and policy checks, to approval, tool execution and the outcome being logged. Part of that chain is garbled in the recording, so it is paraphrased rather than quoted.

Evals, Cost And Security

You cannot build the eval at the end and call a passing test a good eval. Evals should check requirement coverage, gaps and missed non-functional requirements, and while the planner chooses the regression scope, the decision about how much testing is needed stays hers.

Evaluate the trajectory rather than only the final answer. Failing at an early step beats running all the way to a late one before discovering the problem, which costs tokens and adds human review load, because AI generates so much that reviewing it can become impossible.

Instrument before you tune the prompt. She says prompt engineering feels like a long time ago and is still important, tells the audience to engineer system prompts too, and keeps agents deliberately lean, holding only rules, tools, model, non-negotiables and gotchas, with everything else becoming a progressively disclosed skill. An agent, in her framing, is just a way of orchestrating things.

Cost and latency are architecture decisions rather than a bill, decided up front. The trap she names is that a cheaper model in a decision-making step is not cheaper, because it keeps asking questions and generating results that are not good enough, so you pay the same in the end. She names specific models here and self-corrects mid-sentence, so the versions are not reproduced.

Measure cost per transaction including model calls and retries, on an outcome basis, comparing a smaller model against a larger one for the same full outcome, and choose a reasoning level rather than only a model.

On security she is direct. Do not use a skill because it has more stars on GitHub and assume it is safe, and treat any natural language as code. Sandbox anything you did not write or run it in a container, and better still develop your own workflow and convert it into a skill. She adds that her company does not encourage MCP servers its security team has not approved. For quality engineering specifically, test artifacts are attacker-influenced inputs, dependencies and test fixtures are injection surfaces, and everything reaching the model should be treated as attacker-controlled until proven otherwise.

The Closing Synthesis

Her final slide is an end-to-end agentic quality engineering system showing where security, observability, traces, evaluation, cost, audit and a kill switch are built in. The point is that all of them are architectural rather than bolt-ons.

She describes the framework she works from as questions grouped under framing, feasibility, bounded autonomy, architecture, evaluation, piloting and scaling. Once she has the answers she writes them down, looks at where trade-offs can be made, and then builds.

Her readiness test for any layer is ownership. If you cannot name the contract, the owner, the failure mode and the success metric for a layer, it is not ready, and she applies that across model, skills, tools, memory and control. The published chapter list compresses those four items into three by merging the first two.

She is candid that none of this came from getting it right first time, saying the team did not start out thinking this way and learned it the hard way, and that she failed at every stage before recovering from the errors and asking how to improve further.

She encourages reuse over invention on the cost and pattern questions, pointing at existing research and published work and suggesting people identify the patterns and absorb them rather than starting from scratch. No papers or sources were named.

No audience questions were taken. The session ran roughly ten minutes over and the host closed it, saying chat questions would be answered through the blog and YouTube channel.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.