AGENTS.md as a Testing Contract, Not Just Style Notes

What Is AGENTS.md and Which Agents Read It?

AGENTS.md is an open Markdown format that AI coding agents read for project instructions. Codex, Cursor, Copilot, Gemini CLI, and Devin support it across more than 60,000 public repositories.^[2]

The format carries no JSON schema and no YAML frontmatter. It is Markdown in the repository root, versioned with the code, reviewed in the same pull requests. The Agentic AI Foundation stewards it under the Linux Foundation, which is why adoption spread across vendors that compete with each other.

Claude Code is the exception, because it reads CLAUDE.md and not the shared file, so mixed toolchains import or symlink one into the other.^[3] Teams comparing terminal tooling on this basis will find the same split in agentic coding CLI tools.

Do Context Files Improve Coding Agent Results?

No. ETH Zurich researchers evaluating context files across SWE-bench tasks found no general gain in task success and over 20% higher inference cost, though agents did follow explicit instructions.^[1]

The split inside the ETH Zurich result is the useful part. The paper reports that instructions in context files are well followed, while repository overviews are not helpful, and it recommends that human-written files describe only minimal requirements.

The work comes from the SRI Lab at ETH Zurich and was presented at the MemAgents workshop at ICLR 2026.^[4]

The instruction-versus-overview split gives the design rule. An instruction an agent can execute changes behaviour. A paragraph describing the architecture spends context budget and returns nothing measurable, because the agent can read the architecture from the code.

The evidence makes the case for treating the file as a testing contract. Style notes and architecture summaries are overviews. Commands, exit criteria, and prohibitions are instructions, and the ETH Zurich evaluation found only instructions changed agent behaviour.

What Turns Style Notes Into a Testing Contract?

Four clauses do it: the exact commands to run, the definition of done, the protocol for a failing suite, and the prohibitions. Each one states something the agent can execute rather than interpret.

GitHub reached a similar conclusion from the other direction. Reviewing more than 2,500 repositories, its team found the pattern that works is a specific job, exact commands, well-defined boundaries, and clear examples of good output.^[5]

What Does a Complete Testing Contract Look Like?

A testing section of roughly 30 lines: the run commands with flags, the pre-commit checks, the definition of done, the failure protocol, and an explicit list of what the agent must never touch.

## Testing

Run these before reporting a change as finished:

- Unit and integration: pnpm test
- A single file: pnpm test src/billing/invoice.test.ts
- End to end: pnpm test:e2e
- Types and lint: pnpm typecheck && pnpm lint

Done means pnpm test, pnpm typecheck, and pnpm lint all exit 0,
and the e2e suite passes for any change under src/checkout/.

New behaviour needs a new test in the same commit.

## When a test fails

1. Read the failure and fix the source, not the assertion.
2. Re-run the single failing file before re-running the suite.
3. If the same test fails twice for a reason you cannot explain,
   stop and report the failure. Do not continue to the next task.

## Never

- Never edit or weaken an existing assertion to make a suite pass.
- Never add test.skip, .only, or an ignore annotation.
- Never delete a test because it is failing.
- Never mock the payments client in an end-to-end test.
- Never commit secrets or edit files under infra/production/.

Every line in that file is either a command or a rule a reviewer can check against a diff. Nothing in it describes the architecture, names a design pattern, or explains what the project does, because the code already carries all three.

Keep the whole file short. Anthropic targets under 200 lines and states plainly that longer files consume more context and reduce adherence, which is the same pressure the ETH Zurich cost finding measured from the outside.

Where Does the File Go in a Monorepo?

At the repository root, with an optional file inside each package. Agents read the nearest file in the directory tree, so the closest one takes precedence over the root in a monorepo.^[2]

OpenAI Codex walks from the Git root to the current directory and places closer files later in the combined prompt, so they override earlier guidance. Codex stops merging once the combined size reaches its 32 KiB default.^[6]

That 32 KiB cap has a practical consequence teams hit late. A bloated root file can consume the budget before the package-level testing rules are ever reached, so the package that most needed its own commands silently inherits the generic ones.

How Does AGENTS.md Relate to CLAUDE.md and .cursorrules?

Claude Code reads CLAUDE.md, so teams import or symlink AGENTS.md into it. Cursor reads the open format from the root and subdirectories, and Copilot ranks it below its own instruction files.

Anthropic documents the import as a one-line CLAUDE.md that pulls in the shared file with an at-sign reference, with Claude-specific instructions appended below it. A symlink works when no extra content is needed, though on Windows it requires Administrator privileges or Developer Mode, which is why the import is the safer default for mixed teams.

Copilot ranks the file below its own repository files. Path-specific files in .github/instructions rank highest, then repository-wide .github/copilot-instructions.md, then agent instructions, then organization instructions, with personal instructions above all of them.^[7]

Cursor supports the format in the project root and in subdirectories alongside its own project rules, presenting the Markdown file as the option for teams that want readable instructions without structured-rule overhead.^[8]

How to Check the Agent Honored the Contract?

Run the listed commands yourself and compare the output. A context file shapes behaviour without enforcing it, so continuous integration stays the only layer that actually blocks a bad merge.

Anthropic states the limit directly: memory files are treated as context, not enforced configuration, and blocking an action regardless of what the model decides requires a hook. A contract written in Markdown therefore sets an expectation, and it cannot guarantee one.

The specification adds a second limit. Explicit user chat prompts override everything in the file, so a teammate asking an agent to skip the suite wins over the line telling it not to. Both limits point the same way: the file sets defaults, and the pipeline sets rules. That enforcement layer is the subject of pre action checks ai coding agents.

• Loaded-file check: Run /context in Claude Code and read the Memory files list, which shows what actually reached the session rather than what exists on disk.
• Command echo: Ask the agent to state the commands it plans to run before it starts, then compare that list against the file.
• Diff grep: Search each pull request for test.skip, .only, and deleted test files, since these are the prohibitions an agent breaks quietly.
• Required CI status: Make the same commands a required check, because a rule the pipeline enforces survives a distracted reviewer and a direct chat prompt.

A quality layer around agent-written code spans more than one stage, and TestMu AI ships purpose-built AI agents for several of them, including Agent Testing for evaluating conversational agents, a Test Orchestration Agent, and a Root Cause Analysis Agent.

Who Checks the Tests the Agent Writes?

Nothing in the contract does. The file names which suite an agent must run, and it cannot judge whether the assertions inside that suite are meaningful, so a green run can rest on weak coverage.

The coverage-quality gap widens as the agent writes more of the suite. A contract that forbids deleting a failing test does not stop an agent from adding a test that asserts almost nothing, and the pipeline reports both as passing.

KaneAI by TestMu AI works on assertion quality, and its GitHub App already reads the same repository files an agent does. Relevant capabilities:

• Context-aware test generation: Analyzes the pull request diff, title, description, README.md, and optional agent.md to generate end-to-end tests with assertions reflecting both the technical change and the business logic.
• Smart assertions: Proposes assertions where validation matters, covering value checks, element presence, API response shape, and database state, so coverage answers whether the system behaved correctly rather than whether the steps ran.
• Multi-framework export: Exports generated tests to Selenium, Playwright, Cypress, or Appium, so the suite stays in the repository the agent already reads.

The getting started with kane ai documentation covers setup, and the wider question of agents validating their own work is covered in can coding agents test their own code.

Conclusion

Open the AGENTS.md file in your largest repository and delete every paragraph describing the architecture. Replace them with the four commands your agents need, one exit condition, three lines on what to do when a suite fails, and a short list of prohibitions.

The evidence points the same direction as the practice. Instructions get followed and overviews do not, so the file earns its context budget only where it tells an agent something it can run. Everything else is a summary of code the agent can already read.

Then move the parts that must never break out of Markdown and into the pipeline, because a context file sets defaults and only continuous integration sets rules. Teams tightening that loop across a whole workflow can start with vibe coding QA process.

Sources and References Used

• [1] Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? - Gloaguen, Mundler, Muller, Raychev and Vechev, ETH Zurich, arXiv (February 2026)
• [2] AGENTS.md - a simple, open format for guiding coding agents - Agentic AI Foundation, Linux Foundation (accessed August 2026)
• [3] How Claude remembers your project - Claude Code Documentation, Anthropic (accessed August 2026)
• [4] Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? - SRI Lab, ETH Zurich, MemAgents at ICLR 2026
• [5] How to write a great agents.md: Lessons from over 2,500 repositories - The GitHub Blog (November 2025)
• [6] Custom instructions with AGENTS.md - OpenAI Codex Documentation (accessed August 2026)
• [7] Customizing responses for GitHub Copilot - GitHub Docs (accessed August 2026)
• [8] Rules - Cursor Documentation (accessed August 2026)