From AI Assistants to AI Coworkers [Testμ 2026]

Velocity Barely Moved

Glean rolled AI coding tools across its Assistant engineering organisation. He was specific that this was not a pilot with a handful of engineers for a week, but a full-scale rollout with voluntary adoption against real production code.

By his account the tools worked. Code generation got meaningfully faster, engineers went from idea to working pull request in minutes rather than hours, and exploring an approach became cheap enough to try something, discard it, and try again.

At the level of the individual engineer, the productivity gain was obvious. Step back to the level of the system and shipping velocity had barely moved.

He was careful not to present this as a shock. It is a pattern the industry has seen repeatedly, because writing code is only one part of getting something into a customer’s hands.

Eight Stages to Ship One Change

Rather than argue in the abstract, Nilesh laid out the real workflow his team follows to ship a single quality improvement to the Glean Assistant.

• Someone notices a customer problem, whether from an escalation, a downvote, a bad trace, or a quality metric moving the wrong way.
• Assemble an eval set, because one complaint is useful but you need to know whether it represents a repeatable class of failures.
• Run evals, interpret the results, look for patterns, and form a theory of what is going wrong.
• Make a change, which might be code but could equally be a prompt or a tool definition.
• Evaluate the new version and compare it against the baseline.
• Launch through builds, deployments, canaries and experiments, up to a ramp decision.
• Monitor production and respond when real-world outcomes differ from what was predicted.
• Roll back if the change does not hold up.

Exactly one of those eight stages is writing code, and that is the one the coding tools made faster. The other seven still need someone to find context, move information between systems, operate a tool, coordinate with another team, or make a judgment call.

Amdahl’s Law in Engineering

The arithmetic he put on screen is the part worth carrying away. Take a complete unit of engineering work as 100 hours, of which about 20 are spent writing code.

Make coding five times faster and those 20 hours become four. The other 80 are unchanged, so the cycle drops from 100 hours to 84.

A fivefold speedup on the coding step therefore buys a 16 percent shorter cycle overall. He was quick to add that 16 percent is real and worth having, and that none of this is an argument against coding tools, which his team uses and rates.

This is Amdahl’s Law arriving in engineering management. Total improvement is capped by the size of the part you improved, which puts the larger opportunity in the segment nobody has automated.

He was emphatic that the remaining 80 percent is not waste. It is the work that ships the right change safely, and it splits three ways:

• Deciding - combining customer, product and technical insight to define the problem and choose what to build.
• Verifying - reviewing the change, running tests and evals, comparing results, and repeating when quality did not actually improve.
• Shipping and operating - experimenting, building, deploying, monitoring, responding to alerts, and preserving what was learned.

All three are coworking workflows: cross-team, cross-system, cross-permission, and cross-time. Writing code can happen inside one repository in a single focused session, and shipping quality cannot.

Engineers as Human Routers

His diagnosis for why that surrounding work is slow is not technical difficulty. It is that the context needed to solve the problem is scattered.

A single production failure might connect a design decision argued out in chat, an implementation in code, a rationale written in a document, ownership recorded in a ticket, expected behaviour captured in an eval, and the actual outcome sitting in a log.

Every piece exists and nothing is technically missing. An engineer still has to visit each system, reconstruct the timeline, reconcile conflicting information, and decide which source to trust.

The result is that the most experienced engineers become human routers, copying identifiers between tools, tracking down owners, translating between formats, and rebuilding context the organisation already has. A model can be extremely capable and still stall on fragmented context.

Three Layers

Nilesh organises the problem into three layers, each answering a different question about the system.

• Context and tools - the foundation. What the system needs in order to understand the situation, plus the interfaces through which it can do useful work.
• Triggers and judgment - work should begin from the right event rather than when someone remembers to type a prompt, and once begun the system must decide whether to act, continue, ask, stop, or hand off.
• Learning - each run should leave behind verifiable lessons, so the next one starts with more knowledge than the last.

His summary of how they fit: context and tools make the workflow possible, triggers start it, judgment makes it trustworthy, and learning is what compounds the investment over time.

Making Context Usable

Everyone accepts that AI needs context. The more useful question he posed is what work makes organisational context usable in the first place.

• Retooling the manual step - anywhere a human is effectively the API, the agent is blocked. A system reachable only by clicking through a UI, or by asking the one person who understands it, is a system AI cannot use.
• Capturing decision trees - a decision living only in someone’s memory, a meeting, or a long chat thread eventually disappears. The decision and its rationale need a durable, searchable home.
• Turning conversations into searchable text - conversations carry the richest context, and a final document that records only the outcome loses the path taken to reach it.
• Post-processing raw data into intelligence - structured output is more valuable and easier for a model to consume than raw logs.

In practice this meant rewriting a good deal of internal systems and permissioning, so AI could reach the same context engineers already had. They also deployed an internal tool called Scribe that transcribes meetings and makes them indexed and searchable.

A third strand was building pipelines that turn anonymised traces, satisfaction signals, and product feedback into structured intelligence. The stated goal was to give AI the same context an engineer would naturally gather before making a change.

Making the problem legible exposed the next bottleneck. Their eval system had been designed for humans, so a person could pick an eval set, run it, inspect the output, and compare versions, while an assistant could interpret the problem but never advance it.

So they made those operations programmatically callable: selecting an eval set, starting a run, invoking judges, comparing metrics, reading results. That distinction is the one the talk turns on, because participating in the work is what separates a coworker from an assistant.

From Prompt to Event

Callable tools still achieve nothing if a person has to remember to start the work. As Nilesh put it, a workflow that only begins when someone opens a chat pane and writes a prompt may have added a step rather than removed one.

A customer escalation is one of their natural starting events. The escalation itself carries the initial context, so the system can gather similar traces, group the evidence, and assemble the eval set before anyone opens a chat window.

The engineer’s first interaction is then a prepared problem with evidence attached rather than a blank page. The same pattern repeats across the lifecycle: a failed eval starts a diagnosis, an experiment result triggers analysis, a canary alert opens an investigation or a rollback.

Autonomy and Bail Gates

Starting automatically does not mean acting without limits. Nilesh treats autonomy as a progression rather than a switch, with each level carrying more risk than the one below it.

• Understand the escalation by gathering context, identifying complaints, and summarising evidence.
• Recommend a diagnosis, so a person can inspect the theory before anything changes.
• Draft a pull request proposing a concrete change, which still waits for review.
• Evaluate the change and produce evidence on whether the original prediction held.
• Launch or ramp the change, which sits at the highest risk.

Each step demands stronger evidence, appropriate permissions, clear gating metrics, better guardrails, and a credible rollback path. His team started at the lowest level and has expanded gradually, aiming for appropriate autonomy rather than maximum autonomy.

The harder problem, in his framing, is not generating an action but deciding whether to act at all. That requires explicit bail gates through the workflow:

• Vague complaint - ask the customer what they expected rather than manufacturing an answer because the workflow wants one.
• Failure will not reproduce - stop, rather than inventing a diagnosis to keep the process moving.
• Unfamiliar trace pattern - bring in a person, because the system should be most cautious exactly when the evidence is weakest.
• Inconclusive experiment - do not ramp, since a lack of evidence is not evidence of improvement.
• Production metrics deteriorating - roll back or hand off.

Acting, asking, stopping, rolling back and escalating all count as successful outcomes. The reason he insists on that framing is asymmetry: one bad action destroys trust far faster than ten good actions build it.

He drew the comparison to judging a human colleague, where competence is not only about producing the right answer. It also covers knowing when you know enough and when to ask for help.

Closing the Learning Loop

Everything to this point makes one execution faster. The remaining question is how the system gets better next quarter without the workflow being rebuilt from scratch.

His observation is that the workflow already emits learning traces at every stage. The escalation records the production trace and customer feedback, the proposed change records what the system believed would improve quality, and the offline evaluation records whether it did against the test set.

Further along, the experiment records whether the prediction held with real users, a deployment or rollback records the production outcome, and alerts plus human responses record what the workflow missed.

Each stage leaves evidence about what worked, what failed, and where a person intervened. If turning that into a lesson depends on somebody remembering after the incident, the organisation forgets, so it has to be built into the system.

Sometimes those signals contradict the original prediction. A person edits the diagnosis, rejects the pull request, overrides the recommendation, or rolls back the launch, and each of those flows back to update skills, prompts and judges.

Q & A Session

Audience questions were collected during the session for Nilesh to answer separately rather than live on the call. These are the questions the talk itself poses and answers.

• What actually separates an AI assistant from an AI coworker?

  Nilesh: An assistant can tell you what it thinks the issue is and which approach to take, which accelerates only the step that produces a change. A coworker participates in the work and advances the workflow around that change. Reaching the second requires context that makes the problem legible, tools whose actions are callable, triggers that start from the failure, judgment about when not to act, and learning that improves the workflow itself.

• If coding tools work, why has shipping velocity not moved?

  Nilesh: Because coding is roughly a fifth of the work. Speeding that fifth up fivefold takes a 100-hour cycle to 84 hours, a 16 percent gain that is genuine but structurally capped. The other 80 hours go on deciding, verifying, shipping and operating, and those are cross-team and cross-system workflows that a code-editing tool never touches.

• How much autonomy should a team grant such a system?

  Nilesh: Treat it as a progression rather than a switch, starting at the lowest-risk end and expanding only as trust expands. The goal is appropriate autonomy, not maximum autonomy. Every level up needs stronger evidence, the right permissions, clear gating metrics, and a credible rollback path, and stopping or escalating has to count as a successful outcome rather than a failure.

His closing argument was that the winning teams will be the ones that understand how their own organisations work: where context lives, how decisions get made, what evidence is trusted, when a person needs to step in, and what production outcomes should teach the next round.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.