Scaling Test Automation at Microsoft [Testμ 2026]

Five Ways Automation Breaks

Traditional automation treats every test as a handwritten script that engineers author, run and repair whenever the application changes. At organisational scale that produces five failures that arrive together.

• Maintenance cost - selectors break with every UI change, so the suite needs updating continuously.
• Flakiness - timeouts and race conditions make a component unavailable, and the signal erodes until nobody believes the results.
• Slow creation - a comprehensive suite takes weeks or months to write, so coverage permanently trails the product.
• Poor scalability - effort stays linear, meaning more coverage costs proportionally more people.
• Limited intelligence - a test cannot adapt or self-correct, so every failure and every requirement change needs a human.

His framing is that these are connected system problems rather than isolated test problems. They belong to the engineering system, which is why the answer is architectural.

The Five-Stage Journey

He offered his team’s path as one example rather than a roadmap to copy, and its shape is the point.

• Selenium regression - running alongside scripted tests.
• Playwright on the existing runner - because early Playwright shipped without a test runner, so they ran it under Jest.
• Playwright’s native runner - for parallelism, tracing and fixtures.
• Assisted authoring - Playwright codegen and GitHub Copilot generating the boilerplate.
• Agents and Playwright MCP - planning, generation, diagnosis and proposed repairs for failures.

The transferable lesson he drew is evolution rather than replacement. Mature organisations keep the working suites they have and introduce each new capability behind a measurable gate.

100 Packages, 14,000 Tests

The scale is what makes the problem specific. A single monorepo holds more than 100 packages and many production applications, worked on by dozens of partner teams.

A single product change touches multiple owners, because code fixes, test data, dependencies and pipelines all move together. Five capabilities were layered on top of that.

• Test selection - AI maps a code change to its dependencies and risk, with the team approving the selection rules.
• Test maintenance - detecting selector drift and proposing focused patches for the owner to review.
• Test ownership - using code history and coverage data to identify the owning team across 14,000 tests, so a failure routes to someone who can act on it.
• Test repair - combining logs, DOM and screenshot evidence to separate a product defect from a test bug or an infrastructure issue.
• Scenario coverage - because code coverage is straightforward for unit tests and genuinely hard for integration tests.

Probabilistic and Deterministic

The architecture starts by separating probabilistic reasoning from deterministic execution, and he returned to that split as the central design decision of the whole talk.

• Reasoning layer - a frontier LLM interpreting business intent, planning scenarios and forming hypotheses when a test fails.
• Playwright MCP - the browser bridge, turning intent into observable actions such as inspect, click, type and assert.
• Prompt-to-patch flow - parsing a requirement into intent, generating test code, validating it in the browser, and looping on patches until it passes.
• Playwright engine - the deterministic core running Chromium, Firefox and WebKit, covering UI, visual and API testing.
• Reporting and observability - collecting traces, summarising failures, surfacing flakiness and coverage gaps, with digests into Teams or Slack.
• CI/CD layer - selecting impacted tests, sharding them, routing failures and raising draft fix pull requests.

Azure DevOps and GitHub remain the system of record, execution history is retained, and human approval is mandatory for every generated change.

His reasoning for the split is the asymmetry between the layers. Give Playwright the same test and environment and the result is the same every time; give an LLM the same prompt and the output varies, which is why context and history are what make the reasoning layer usable.

Four Agents, One Loop

Requirements arrive from wherever they already live: product requirement documents, work items, test cases in Gherkin or YAML, collaboration threads, screenshots and recordings.

• Planner - combines the written intent with exploration of the application and produces a reviewable markdown test plan.
• Generator - converts an approved plan into Playwright specs, fixtures, page object model classes, test data and utilities.
• Healer - takes failing tests with their traces, DOM snapshots, logs, screenshots and network evidence, and proposes a focused repair that loops back to the generator.
• Reporter - failure analysis, root cause, suggested fixes and results delivered by email.

The design rule is that each agent has one job, inspectable output and a single clear failure boundary. That is what makes a bad result attributable rather than mysterious.

Around them runs an eight-step delivery loop from business intent to a green merged build: define outcomes and guardrails, generate product code, generate the test plan and cases, review both for correctness, security and coverage, execute in CI capturing evidence, classify the failure, route it to the specialised agent for that class, then human review and approval before merge.

Coverage by Intent

Coverage is not a count of test cases, and unit-test code coverage does not answer whether a critical business scenario still works. That gap is what their scenario coverage engine addresses.

It reads the change requirement document, extracts the testable scenarios implied by each business rule, and matches those scenarios to test specs by intent rather than by filename.

Rules with no match become named coverage gaps while the pull request is still open. What surfaces that way is the work nobody remembers: silent server-side validation, forgotten error paths and permission branches.

Generation follows the same discipline. It starts from one requirement or pull request description rather than an unconstrained prompt, extracts the affected form and its field rules, and prefers Playwright’s accessibility-first locators such as getByLabel and getByRole, asserting both visibility and the content announced to screen reader users.

Risk-Based Test Selection

Running every test on every change turns a safety net into a queue. A small bug fix triggered more than a thousand tests, and a pull request took three to four hours to report back.

The replacement reasons about impact instead of running everything. Changed modules are parsed, abstract syntax tree differences identified, and the dependency graph walked so that every affected package is considered rather than only the one that changed.

Failure history and business criticality weight the result, and the highest-risk impacted tests run first under an explicit policy.

He was careful about the boundary. Targeted selection is not a replacement for full regression, so the complete 14,000 still run off-peak, at release gates and in the nightly pipeline, precisely to catch what the impact model missed.

Self-Healing and the Gauntlet

His definition of self-healing is mostly a list of what it is not. It is not a blind retry, and it must not paper over a product regression.

When a locator fails, the system captures test logs, accessibility snapshots and screenshots, then compares them against the last known good baseline. Multiple agents pursue multiple hypotheses rather than one guess.

A candidate fix becomes a git-compatible diff, and the affected tests rerun in isolation in what his team calls the gauntlet pipeline. The fix is run repeatedly there specifically to prove it is not itself flaky, and only then does a draft pull request reach a human.

Underneath sits the classification problem, which he treats as the harder one.

Each hypothesis starts from that evidence plus the code diff, test data and execution history. The output is a classification of test bug, product bug or infrastructure issue, with a confidence score, a root cause narrative and a likely owner so the work routes to the right team.

Governance and Guardrails

His position is that technical capability does not create enterprise adoption. Trust does, and trust comes from controls.

• Human review - on every AI-authored change, whether a generated test, generated product code or a proposed repair.
• Least privilege - read permissions wherever possible, narrow-scope draft pull requests, and no production deployment rights for any agent.
• Auditability - telemetry on every agent and MCP action, covering requirement ID, prompt version, tool action, model output, classification and approval.
• Data privacy - no secrets, certificates, network information or customer data exposed to agents.
• Sandboxing - generation happens in isolated environments that are not dev boxes, test environments or production.

Pilot Results

The numbers he shared came with a caveat he repeated more than once, so it is worth carrying: these are observed results from their own pilot, used to optimise their agents, and not benchmarks anyone should treat as a promise.

• 60 percent faster - regression feedback, with developers seeing useful signal in minutes rather than the previous three to four hours.
• 40 percent reduction - in test script repair effort through self-healing.
• 30 percent reduction - in manual bug triage.
• 90 percent accuracy - classifying a failure as a product bug against a test defect.

A secondary effect matters as much as the headline. Running fewer unrelated tests produces fewer unrelated flaky failures, which makes triage cleaner as well as faster.

Lessons and Pitfalls

The patterns that held up across the pilots start with the taxonomy and end with how the team relates to the tooling.

• Define the failure taxonomy before prompting anything - classifying whether a failure came from the test script, the product, the agent or a hallucinating model.
• Historical data and context beat a bigger model - failure data, syntax trees and accessibility snapshots are what let an agent work properly.
• Specialised agents under a supervisor - decomposing the problem rather than asking one agent to do everything.
• Abstract the model provider - so each specialisation can use a different model and a different prompt.
• Treat AI as a partner rather than a black box - complementing engineers instead of replacing them.

The pitfalls he named are the ones that repeatedly damaged trust, and he was blunt about all of them.

• Never auto-merge AI changes - always a human review, always validated in an isolated environment first.
• No monolithic prompt and no single model - each agent gets its own prompt and a model suited to its task.
• Do not ignore execution history - flakiness and a genuine failure look identical in a single run, so consistent failure is the evidence you need.
• Do not launch every suite at once - promote tests through the gauntlet pipeline and into production only once they pass in isolation.

The Phased Roadmap

He closed by turning the material into an adoption sequence rather than a set of ideas. Start with the foundation of Playwright and the agent tooling as a baseline, then integrate with whichever build infrastructure you already run.

Next come the AI layers that make failures actionable: root cause analysis, scenario coverage and classification, so a failure can be attributed and assigned rather than merely reported.

Scaling follows, across a cross-browser matrix, multiple operating systems and environments, and separate geographies, with flaky detection running throughout. Organisation-wide rollout, analytics, telemetry and cost optimisation come last.

His four takeaways compress the session. Platforms scale teams through shared automation and self-service infrastructure; agents earn their place by removing repetitive authoring, maintenance and first-pass triage; data beats a bigger model, because logs and screenshots are what power flaky detection and risk scoring; and governance is what converts capability into adoption.

This session was part of Testμ Conf 2026, which ran across three days of sessions on agentic engineering and quality. Registrations for the next edition are already open on the Testμ Conference 2027 page.