Spec-Driven Testing: Verify What Your Coding Agent Built Against the Spec

Why Does Spec-Driven Development Need a Testing Step?

Spec-driven development gives a coding agent a written spec, a plan and a task list, then has it build. The spec says what the software should do. The core loop does not run the finished app and check each acceptance criterion against it. Spec-driven testing adds that step: it turns the same spec into checks and a verdict for each criterion.

What the main toolkits verify, as of November 2026:

• GitHub Spec Kit v1.3.1 walks a feature through constitution, specify, plan, tasks, implement and converge (README). When a user asked for a core /speckit.verify command, the replies pointed to the community extensions catalog (issue #1745). The community spec-kit-verify extension is a read-only "post-implementation quality gate" that checks code against the spec artifacts (repo).
• OpenSpec v1.16.0 has an /opsx:verify step to "check the implementation matches the spec" (openspec.dev). Its openspec validate command checks changes and specs for structural issues (CLI docs).
• Kiro extracts properties from EARS requirements and generates random test cases to check them against the implementation (Kiro correctness docs).

These checks read artifacts and code, or test functions with generated inputs. Spec-driven testing drives the running app in a real browser, checks each acceptance criterion against an expected result, and keeps proof of the run.

Spec-Driven Testing vs TDD, BDD, Spec-Driven Development

Spec-driven testing differs from TDD, BDD and spec-driven development in what counts as the source of truth and what decides pass or fail. TDD treats a developer's unit test as the spec. BDD writes examples in Given/When/Then. Spec-driven development uses a spec to steer code generation. Spec-driven testing derives checks from that spec and reports a verdict per criterion.

Most teams combine them: TDD for units, spec-driven tests for acceptance behaviour. For the development side, see our guide to spec-driven development.

How Does Spec-Driven Testing Work?

Spec-driven testing works in five steps: snapshot the spec, extract and approve the use cases and criteria, design scenarios with one test each, run the tests in a real browser, then measure which criteria are proven and reconcile when the spec changes. Kane CLI maps each step to a command group; the full commands are in our Spec Kit walkthrough.

1. 1

   Snapshot the spec. kane-cli context ingest saves the requirement source as a content-hashed snapshot in a local .context/ store, so every later test traces to an exact version (context docs).

   It accepts Markdown and other text files, PDF, Word .docx, images, and Jira, Confluence, Linear or public web page URLs (sources docs).

2. 2

   Extract and review. kane-cli context extract has an agent propose use cases, each quoting the spec text it came from. A person approves, edits or rejects them with kane-cli context review. Conflicting requirements become clarifying questions, each with options and a recommended default.

3. 3

   Design tests. kane-cli design tests turns an approved use case into acceptance criteria with an expected result a machine can check, then scenarios built as a pairwise or 3-wise covering array, then one _test.md per scenario. Each assert step carries an @verifies tag naming the criteria it proves.

   The agent uses 3-wise when money, authentication or data loss is involved. When the spec never states an expected result, the gap is recorded as missing-expected-result with a recommended default instead of an invented answer (design docs).

4. 4

   Run. kane-cli testmd run authors a test in a real browser on its first run. kane-cli testrun run runs saved tests as one suite, and replayed steps have no LLM cost (gate CI guide).

   Every run seals a .evidence file with the steps, screenshots, logs and the verdict.

5. 5

   Measure and maintain. kane-cli cover reports what a run proved and what the design still owes; kane-cli cover gaps lists what is still owed across the requirement set. When the spec changes, kane-cli maintain reconcile plans ADD, MODIFY and ARCHIVE rows for review.

   In a terminal, nothing from the plan commits until you give each row a verdict, and no headless mode archives anything (maintain docs).

AI writes the path; deterministic checks decide the verdict. Each designed test carries a written check whose expected value comes from its acceptance criterion, and design rejects a check that disagrees with its criterion, so no test can assert less than the requirement asks.

When a replayed step no longer matches the page, Kane CLI re-authors the test from the failing step onward, and that heal uses credits.

How Do You Check if a Spec Is Testable?

A spec is testable when each criterion names a trigger or condition and an observable expected result a machine can compare: an exact value, a state, or a message. Paste your spec below. The rule-based checker counts criteria, flags missing expected results and vague words, and tags EARS patterns.

The EARS tags follow the five patterns in EARS notation, with a test for each pattern.

Run Spec-Driven Testing on a GitHub Spec Kit Spec

Kane CLI ingests Markdown and other text files, so the spec your toolkit writes can be the source. This section follows GitHub Spec Kit, the toolkit with a published Kane CLI run.

Spec Kit writes spec.md into a numbered feature folder under specs/. Its template has user stories with Given/When/Then acceptance scenarios, FR- functional requirements and SC- success criteria (template, as of November 2026).

On a terminal, ingest saves the snapshot and goes straight into the extract chat. In CI, add --mode ci to save the snapshot only. Full walkthrough: Spec Kit with Claude Code from the command line.

Compare the toolkits themselves: Spec Kit vs OpenSpec vs Kiro vs BMAD.

Build a Spec Traceability Matrix From Kane CLI Runs

Tag each assertion with the criterion it verifies, then read coverage from real runs. Kane CLI measures coverage on two axes: depth, the criteria a sealed run proved, and completeness, what the design still owes. Together they form a traceability matrix built from evidence rather than from a spreadsheet someone updates by hand.

A spreadsheet matrix says a test exists. A spec-driven one says the test passed against that criterion, with evidence. kane-cli cover reads depth from the .evidence pack and never recomputes it, and coverage records go into the pack whenever the project has a .context/ store (coverage docs).

kane-cli cover reports coverage; it is not a release gate. In our Spec Kit run on kane-cli 0.8.18 it exited 0 with criteria still owed (Spec Kit walkthrough), so to fail a build on missing proof, read the proven figure from kane-cli cover gaps --json, fail the job below your threshold, and treat a missing proven figure, which means nothing has run yet, as a failure. cover gaps works on a local .context/ store: Kane CLI 0.10.0 and later refuse it in a folder linked to a shared project.

To fail on test results, gate on the exit codes of kane-cli testrun run: 0 when all tests passed, 1 when at least one failed, 2 for a usage error, an invalid plan or an auth error, when nothing ran, and 3 when the run was cancelled (gate CI guide).

To build one by hand, see the requirements traceability matrix guide or the free RTM generator.

Property Test or End-to-End Check for Each Spec Criterion

Use a property test when a criterion holds for a whole range of inputs inside one function, such as "for any cart, the total is never negative". Use an end-to-end check when the criterion is about what a user sees across screens, services and state. Most specs need both.

Kiro's own docs call property-based testing "evidence of correctness, not a proof" and say requirements that depend heavily on external services may be better covered by example-based tests (correctness docs).

When Is Spec-Driven Testing the Wrong Tool?

Spec-driven testing is the wrong tool when there is no spec worth trusting, when the question is "is this good?" rather than "does this match?", or when you need load data. It checks agreement with written criteria, so if the spec is wrong, the test faithfully verifies the wrong behaviour.

Exploratory testing

Problems nobody wrote down need a human or an exploratory agent.

UX quality

"Feels clunky" is real feedback with no oracle. Use usability sessions.

Wrong specs

Review the criteria with product before you design tests; that is what context review is for.

Load

A response-time criterion can be checked once; sustained load needs a load-testing tool.

AI output quality

It checks app behaviour and does not replace evals for model output.

Test Your Own Spec With Kane CLI

Point Kane CLI at the spec your agent built from and see which criteria are proven today. Kane CLI is free to install. Runs use KaneAI credits from your TestMu AI account, and you can start free with 10,000 credits, valid for 30 days.

Then sign in and point it at your spec:

Running hundreds of tests? kane-cli testrun run --remote runs the same suite in parallel on HyperExecute runners when your plan includes HyperExecute. A pack sealed on the grid carries no coverage records, so measure coverage from a local run.

For a rollout across several teams, book a demo.