AI-Generated Code Bugs: What the Data Shows and How to Catch Them

How Buggy Is AI-Generated Code, Really?

Measurably buggier than human-written code in every published dataset, with the reported gap running from 8 percent to 68 percent because the two ends do not measure the same thing. The most widely quoted figure is roughly 1.7 times more issues, and it comes from a single source that is worth understanding before you repeat it.

That figure originates in CodeRabbit's State of AI vs Human Code Generation report, published in December 2025. The study scored 470 open-source GitHub pull requests, 320 of them AI co-authored and 150 written by humans alone.

Across that set, AI-authored changes averaged 10.83 issues per pull request against 6.45 for the human-only group. Two qualifications belong next to the number wherever it is used.

• The word is issues, not bugs - CodeRabbit's taxonomy counts readability findings, which rose more than threefold, and formatting findings alongside genuine logic faults. Secondary coverage swapped in the word bugs, and the number has travelled under that label ever since.
• The counter is an interested party - CodeRabbit sells AI code review, and the issues were identified by its own reviewer rather than by an independent tool or a panel of engineers.

An independent measurement points the same way with a smaller multiplier. Uplevel analysed engineering data from nearly 800 developers, tracking cycle time, pull request throughput, bug rate, and extended working hours rather than self-reported survey answers, and found that the group using GitHub Copilot introduced 41 percent more bugs.

The practical consequence is that the exact multiplier matters far less than where the extra defects land. That is a question about your gates, and it is separate from the question of whether AI review tooling reads a diff well, which we cover in AI code review vs verification.

Why Do the Studies Disagree With Each Other?

Because none of them are counting the same object. One counts review findings on a diff, one counts bugs attributed to a developer cohort, and one counts security issues inside code samples already known to be vulnerable. Those three numbers were never going to converge.

The largest of the four is academic. Cotroneo, Improta and Liguori paired 507,044 code instances drawn from 16,928 open-source repositories and ran identical analysis over both sides, which is the closest anyone has come to a controlled comparison.

Their result is the least dramatic and the most narrowly scoped. In the Python half of the corpus, each vulnerable sample carried 1.62 security issues for human-written code, 1.75 for ChatGPT and DeepSeek-Coder, and 2.10 for Qwen, with the paper reporting Java separately and finding a wider gap there.

That is an 8 percent gap for one model family and 30 percent for another, which says the model you pick moves the number as much as the fact that a model wrote it. It also shows why these figures cannot be stacked: this one counts security issues inside samples already known to be vulnerable, while CodeRabbit counts every category of review finding across whole pull requests.

GitClear's work belongs in a different column entirely. It tracks how code is shaped rather than whether it is correct, and the shift it records over 211 million changed lines is toward duplication and away from refactoring.

What Kinds of Bugs Does AI-Generated Code Actually Produce?

The distribution is skewed rather than uniform. In CodeRabbit's dataset, logic and correctness issues were 75 percent more common in AI pull requests, error-handling gaps were nearly twice as common, and security findings ran up to 2.74 times higher. Five classes account for most of what teams actually hit.

Which Test Gate Catches Which AI Bug?

No single gate catches more than a slice, and the gates most teams already run are concentrated on the classes AI produces least dangerously. Mapping defect class to gate is more useful than adding another scanner.

The scale of the exposure is now industry-wide rather than niche. Google Cloud's 2025 DORA report found that 90 percent of survey respondents use AI at work and that 30 percent report little or no trust in the code it generates, while reporting a positive relationship between AI adoption and delivery throughput alongside a continued negative relationship with delivery stability.

That combination is the whole problem in one sentence: more change arriving faster, with less confidence attached to each unit of it. Closing it means adding a gate the coding agent cannot satisfy by reasoning about its own output.

This is the gap Kane CLI is built for. It drives a real Chrome browser from the terminal or from inside a coding agent's own loop, takes objectives in plain English rather than selectors, and returns a pass or fail anchored to evidence such as DOM state, URL changes, network responses, screenshots, and console logs. Because the verdict comes from the rendered application rather than from the source, the agent that wrote the code cannot talk it into passing.

Can You Trust the Tests the AI Wrote?

Only where the assertion comes from somewhere other than the implementation. When one model writes both the feature and its tests, the suite encodes a single interpretation twice, and agreement between the two proves consistency rather than correctness.

Developers already report the cost of this. In the Stack Overflow Developer Survey 2025, with 84 percent of respondents using or planning to use AI tools, the top frustration was AI solutions that are almost right but not quite, cited by 66 percent, and the second was that debugging AI-generated code is more time-consuming, cited by 45 percent.

The failure is visible in the shape of the assertion rather than in the syntax, which is why it survives review.

// Circular: the expected value is produced by the code under test.
test("applies the discount", () => {
  expect(applyDiscount(100, 20)).toBe(applyDiscount(100, 20));
});

// Independent: the expected value comes from the requirement.
test("a 20 percent discount on 100 leaves 80", () => {
  expect(applyDiscount(100, 20)).toBe(80);
});

Generated end-to-end tests carry a second problem on top of the first. They tend to reach for hard-coded waits and positional selectors, both of which produce suites that pass locally and flake in continuous integration.

• Fixed sleeps instead of explicit waits - a timeout tuned to one machine becomes a race condition on a slower runner.
• Positional XPath - selectors anchored to sibling index rather than to an accessible name break on the next layout change.
• Assertions on internal state - checks against a variable the implementation happens to expose rather than against what the user sees.

Mutation testing is the practical answer to circular coverage, because it changes the implementation deliberately and reports which tests noticed. A suite with high coverage and a low mutation score is measuring itself. The structural version of this argument, and where a second agent does and does not help, is in can coding agents test their own code.

Why Do AI-Generated UI Bugs Survive CI and Break in the Browser?

Because continuous integration usually runs one engine, headless, at one viewport, and a stale-training defect is defined by the engines where the assumption no longer holds. The pipeline is not wrong about what it checked. It checked a surface the bug does not live on.

The shape this takes in practice is familiar. A team merges an agent-written component with passing unit tests and a clean linter, and the support tickets that arrive two days later all come from one browser, because the generated markup reached for a layout property its training data treated as universally supported and Chrome quietly forgave the difference.

The distinction is easy to see once you assert against the rendered page rather than the returned value. The run below is from build 102686377 on the TestMu AI cloud, driving Chrome on Windows 11 against the Simple Form Demo page of the Selenium Playground.

[nav] simple-form-demo loaded in 802 ms
[assert] #message rendered text = "verify rendered output, not source"
[assert] #message boundingBox = {"x":833.84375,"y":427.5,"width":382.15625,"height":24}
[assert] #message computed  = {"display":"block","visibility":"visible","opacity":"1","color":"rgb(74, 74, 74)"}
[result] text present in DOM: true
[result] actually visible to a user: true

Both result lines are true here, which is what a passing check looks like. The reason to capture both is that they come apart independently: text present in the DOM and text visible to a user are two different claims, and a generated CSS change can flip the second while leaving the first untouched.

• A DOM assertion confirms the element exists and holds the right string, which is all most generated tests check.
• A geometry assertion confirms the element occupies real space, which catches zero-height containers and collapsed flex children.
• A computed-style assertion confirms it is actually painted, which catches inherited visibility and opacity changes.
• Running all three across engines is what turns a Chrome-only pass into evidence that the change works where your users are.

Coverage across engines is the part that cannot be simulated locally. TestMu AI's real device cloud provides 10,000+ real Android and iOS devices, alongside 3,000+ browser and operating system combinations, so a generated component can be exercised on the WebKit and older Android builds where stale-training defects surface.

Is It Really 100x More Expensive to Fix a Bug in Production?

That specific figure has no traceable primary source, and it is worth retiring from your slides. The chain runs from modern blog posts back to a citation of an IBM Systems Sciences Institute study that researchers have repeatedly failed to locate, most thoroughly by Laurent Bossavit in The Leprechauns of Software Engineering.

The empirical work that does exist is less convenient. Menzies, Nichols, Shull and Layman examined 171 software projects and found no evidence for the delayed issue effect, reporting that the effort to resolve an issue in a later phase was not consistently or substantially greater.

Two better-grounded numbers do the same rhetorical job honestly:

• NIST Planning Report 02-3 published a relative repair-cost table running from 1X at requirements to 30X after product release, and labelled it an illustrative example rather than a measurement.
• CISQ put the cost of poor software quality in the United States at at least $2.41 trillion in its 2022 report, with accumulated technical debt at roughly $1.52 trillion.

This matters more than pedantry when the argument is about AI-generated code. No published research measures the fix cost of an AI-introduced defect specifically, so any claim that they are more expensive later is an extrapolation. The defensible case for verification is that generated defects escape the gates you already run, not that they cost a mythical multiple to repair.

How Do You Measure Your Own AI Defect Rate?

Every study quoted above measured somebody else's repositories, with somebody else's models and review standards. None of them is published as a method you can run, so the number that applies to your codebase is one you have to generate. Four weeks of tagging is usually enough to see a signal.

• Label authorship at the source. Add a commit trailer or a pull request label whenever an agent wrote a meaningful share of the change, and apply it consistently rather than accurately, since a rule everyone follows beats a rule everyone debates.
• Define an escaped defect before you start counting. A bug found by a gate is a gate working, so count only the ones that reached a user or a production alert.
• Record which gate caught every non-escaped defect. This is the column that tells you where to invest, and it is the one teams skip.
• Normalise by change size rather than by pull request. Agent-authored changes tend to be larger, so a per-pull-request rate flatters human work by comparison.
• Compare the two cohorts after four weeks, then hold the comparison for a second month before acting, because a single sprint of unusual work will move it either way.

The output is a table of defect class against catching gate for your own codebase, which is the version of the earlier table that is actually worth acting on. If the classification step is unfamiliar, our guide to software bugs covers the taxonomy in more depth.

Note: Measuring an escape rate needs the escapes to be visible first, which means running the suite across the browsers and devices your users actually hold rather than the one your pipeline defaults to. TestMu AI runs that coverage on real hardware and keeps the evidence attached to each run. Start free.

Where to Start This Week

Pick the single highest-traffic user flow that an agent has touched in the last month and put one rendered-surface check in front of it. That one change converts the class of defect that currently reaches your users into a failing check, and it takes an afternoon.

• This week - add the rendered check on one flow, and start labelling agent-authored pull requests so the data begins accumulating.
• This month - run a mutation-testing pass on the modules with the most generated tests, and treat a low score as a coverage result rather than a tooling problem.
• This quarter - extend the rendered checks across engines, and review your own defect-class table rather than anyone's published multiplier.

The Kane CLI getting started guide covers the first check end to end, including the headless flags and exit codes a pipeline needs. When you are ready to make it a merge requirement rather than an advisory step, setting up a continuous verification gate walks through the pipeline wiring.

Generated code is not going to stop arriving, and the multiplier in the headline was never the actionable part. Knowing which of your gates is blind to which defect class is.