9 Vibe Coding Risks and How to Test for Them

What Are Vibe Coding Risks?

Vibe coding risks are the defects that reach production when an AI generates the implementation and no human verifies the resulting behavior. The phrase covers the same ground as vibe coding quality issues and problems with vibe coding, and it spans four categories: correctness, security, maintainability, and the review capacity to catch any of them.

The distinguishing feature is not that generated code is sloppy. It is that generated code is fluent.

Veracode's Spring 2026 GenAI Code Security Update reports that syntax correctness now exceeds 95%, while the security pass rate has stayed essentially flat at around 55% since 2023. The surface got polished while what sits underneath did not move.

That gap is what breaks conventional quality signals. A linter, a compiler, and a type checker all read the same closed surface the model wrote against, so they agree with it.

If you want to know whether the software is right, something has to run it. That is the argument behind continuous verification for AI-generated code, and it is what our probe was built to test.

How We Tested Six Vibe-Coded Apps

We tested six live vibe-coded web applications on TestMu AI Browser Cloud, in real Chrome sessions driven by Playwright, on 27 August 2026 under build 102694693. Six is a small sample and we are not going to present it as a survey. Read every number here as "in these six apps", never as "in vibe-coded apps".

Every target had to be attributable. We accepted an app only when the builder was provable from the deploy domain, an intact platform badge in the rendered DOM, or a hackathon gallery entry naming the tool. Four are Bolt projects, one is Lovable, and one is Base44.

We rejected roughly twenty-five other candidates, most of them because they looked like someone's live business rather than a demo.

Each app got the same six checks, in the same order, with a twenty-second per-check timeout:

• Load - navigate and wait for network idle, capturing console errors, page errors, failed requests, and any 4xx or 5xx response.
• Primary control - click the main call to action and confirm the app responded, by a change of view or a triggered action.
• Empty submit - submit the first form with every field blank and record whether validation fired.
• Hostile string - type a string containing quotes and angle brackets into the first text field and record whether it is echoed back as live markup. The string is inert markup rather than an exploit payload, and we sent nothing designed to persist or alter server state.
• Reload - refresh mid-flow and check whether the app returns to a usable state.
• Deep link - request a route that does not exist and record the response status alongside the rendered view.

Getting thirty-six trustworthy results took twenty-one browser sessions. Two operators ran the probe independently and both discarded their early passes, because seven distinct defects in our own harness had produced false verdicts about the apps rather than real findings. The most instructive was a loose call-to-action matcher that clicked a colour swatch named "Indigo" and then reported a dead control.

What We Found

Five of six apps logged an error before anyone touched them. No app failed the primary-control check, so nothing in the sample was broken in the obvious way. Every failure we found was the quiet kind.

The single most useful frame in the study came from the empty-submit check on a Bolt-built financial runway calculator. With both inputs blank, it reports a runway of "Indefinite" under a green tick, labels it "Excellent financial runway", and offers to export a summary of a model containing no data.

Nothing about that screen is broken in a way software can detect. The layout is clean, the arithmetic is internally consistent, and a visual-regression baseline captured from this exact state would pass it forever. It is wrong only if you know what the answer is supposed to be.

The 9 Failure Modes of Vibe-Coded Software

Six of these we observed directly. Three are invisible from a browser and rest on published research, which we flag explicitly rather than implying our probe found them.

3. Silent State Loss on Reload

Two apps threw away in-progress work when the page refreshed. A quiz discarded the entered name and dropped the user out of a started attempt back to the quiz picker; a game discarded the entered name and the round in progress and returned to its setup screen.

Neither warned, and both rendered a perfectly healthy screen afterwards. It was just the wrong screen.

Generated test suites miss this class specifically, because a model writes the flow the feature describes and no feature description mentions refreshing the page.

Which Test Type Catches Which Failure Class

Naming the risks is the easy half. The operational question is which check actually fires, and the honest answer is that several of these classes are caught by nothing in the functional testing pyramid. Those rows matter more than the ones full of green.

Four of nine rows have no automated functional coverage at all. That is the finding to take away, and it is why "we have tests" is not an answer to "is this vibe-coded app safe to ship".

Why Green Builds and Code Review Miss Them

A pipeline reads what the model wrote using tools that operate on the same closed surface the model was optimising against. Compilers, linters, and type checkers all confirm the code is well-formed, which was never in doubt.

Human review is the intended backstop, and it is the part that scales worst.

Enterprise telemetry from Apiiro, published via the Cloud Security Alliance, reports privilege escalation paths up 322% and architectural design flaws up 153% as AI-assisted development scaled, alongside monthly security findings rising from roughly 1,000 to more than 10,000 in six months. The same source reports those developers committing three to four times as much code.

Both numbers deserve care. A tenfold rise in findings against a three-to-fourfold rise in output means volume explains part of the surge and not all of it, and this is vendor telemetry rather than independent research. What it does establish is that the reviewing has to keep pace with the committing.

DORA's 2025 research puts numbers on the resulting posture: 90% of technology professionals now use AI at work while 30% report little or no trust in the code it produces, and higher adoption is associated with a rise in both delivery throughput and delivery instability.

Speed and breakage rise together because generation got cheaper and verification did not.

Do AI-Written Tests Verify Anything?

Agents write competent test code. The weakness is where the expectation comes from. A test derived from the implementation encodes whatever the implementation does, including its mistakes, and then defends them against future change.

Our runway calculator makes the point concrete. An agent asked to test that app would reasonably assert that submitting the form displays a runway figure. It does, the assertion passes, and the number is meaningless.

We looked at this failure mode in more depth in whether coding agents can test their own code.

The practical safeguard is to keep at least one expectation that the implementation cannot influence. Derive it from the ticket, the specification, or a known-correct value computed by hand, so the check is able to disagree with the code it is checking.

Note: Verification only counts when something executes the application and returns evidence a reviewer can open. Kane CLI does that from the terminal and from CI: you describe the outcome in plain language, it drives a real Chrome browser, and it returns a pass or fail anchored to observed DOM state, network responses, console logs, and screenshots rather than to the source. Start verifying with TestMu AI free.

How to Triage a Vibe-Coded App This Week

Everything our probe found is reachable without reading a line of source. Run these four checks first, because they need no knowledge of the codebase and they surfaced defects in five of our six apps.

• Fail the run on console and network errors - not a warning, a failure. This alone caught five of six apps in our sample, before any interaction.
• Assert response status, not rendered text - the soft 404 in our sample looks correct and reports success. Only the status code disagrees.
• Submit every form empty - and check for a real message, not just a blocked submit. A disabled button with no explanation is a defect that passes.
• Reload mid-flow - one refresh, halfway through the primary journey, then assert the user is where they were.

A Playwright probe for the first two is about a dozen lines, and it is the cheapest gate you can put in front of generated code:

import { test, expect } from '@playwright/test';

const ORIGIN = 'https://www.testmuai.com';

test('loads without errors', async ({ page }) => {
  const problems = [];

  // Register listeners before navigating, or anything that fails during load is missed.
  page.on('console', m => m.type() === 'error' && problems.push('console: ' + m.text()));
  page.on('pageerror', e => problems.push('pageerror: ' + e.message));
  page.on('requestfailed', r => {
    const reason = (r.failure() || {}).errorText || '';
    // ERR_ABORTED is normal (prefetch, cancelled navigation). A dead host shows up
    // here as ERR_NAME_NOT_RESOLVED, which is the case worth failing on.
    if (reason !== 'net::ERR_ABORTED') problems.push(reason + ' ' + r.url());
  });
  page.on('response', r => r.status() >= 400 && problems.push(r.status() + ' ' + r.url()));

  // networkidle is deliberate here: the point is to let every request settle so the
  // listeners above see it. Prefer web-first assertions for ordinary functional tests.
  await page.goto(ORIGIN + '/selenium-playground/', { waitUntil: 'networkidle' });
  expect(problems, problems.join('\n')).toHaveLength(0);
});

// A separate test, so the deliberate 404 below never lands in the list above.
test('a route that does not exist returns 404', async ({ page }) => {
  const res = await page.goto(ORIGIN + '/selenium-playground/no-such-route-12345/');
  expect(res, 'navigation produced no response').not.toBeNull();
  expect(res.status()).toBe(404);
});

Expect the first run against your own app to surface noise alongside real defects. Triage it once: move the errors you have consciously accepted into an allowlist, and leave everything else failing. In our sample that would have been one entry for the starter favicon, and the 401 and the dead-host failure would both still have stopped the build.

Point it at your own deployment, run it on every merge, and widen the net from there. Once it passes locally, the same spec runs unchanged across browser and OS combinations on the TestMu AI test automation cloud, which is where the layout and rendering defects our single-browser probe could not see start to show up.

Which Vibe Coding Statistics Do Not Hold Up

Provenance on this topic is unusually bad. Several widely repeated figures failed verification when we went looking for a primary source, and we excluded them rather than pass them along.

• Code-duplication figures from proprietary telemetry - widely quoted, but the underlying corpus is not reproducible and the key metric is defined by the vendor publishing it. We used the peer-reviewed MSR 2026 redundancy measurement instead.
• The "AI makes developers 19% slower" line - a real result from a randomised trial, routinely stripped of its qualification. METR's own 2026 update reports confidence intervals that cross zero in both directions, which means no statistically significant effect, and METR cautions its data is weak evidence. It is not a finding that AI slows developers down.
• Package-hallucination rates quoted without a date - the 19.7% figure measured 2024-era models. Repeating it as the current state of frontier models overstates the rate by roughly three to four times.
• Any figure attributed to a vendor page that will not load - one heavily cited security analysis renders nothing to an automated fetch, and a real cloud browser session returned a stub. Where the numbers appear in a third-party research note, we cite that note and say so.
• Preprints change. The commit-study figures above moved between versions of the paper, so we cite the current version and flag which findings are not yet peer reviewed.

Limits of This Study

The sample is six apps and the builder mix is uneven, with four Bolt projects, one Lovable, and one Base44. There is no Replit or v0 app in the run, and nothing here supports a per-builder ranking.

• Selection bias we chose deliberately - we excluded live production businesses on ethical grounds, which selects for the unmaintained demo end of the distribution. A vibe-coded internal tool with a developer still attached to it is not represented here.
• Everything was measured from outside - no source code, no server logic, no manifests, no secrets. We could not observe a single server-side or security defect and do not claim to have.
• One browser, one platform - Chrome on a single desktop configuration. No mobile, no viewport sweep, no slow-network conditions, so responsive and layout defects are entirely outside this run.
• Six checks are a smoke test - a pass means the app did not fail that specific check, never that it is correct. The deep-link results in particular flatter the apps whose host supplies a default 404 page.
• Measurement hazards are real - one app looked like it had no validation until we sampled repeatedly and found an error message that appears and disappears within seconds. Anyone probing with a single fixed delay will over-report missing validation.
• We do not name the apps - these are small public demos built by individuals, the defect classes are the point rather than their authors, and identifying marks are redacted from the screenshots for that reason. The platform badges are left visible because the builder attribution is part of the finding. This is soft de-identification, not anonymity.

One inference deserves naming plainly. Almost every study cited here measures AI-assisted development inside a review process: merged commits, pull requests that reached human reviewers, models scored on controlled tasks, professional developers surveyed about their working practice. None of them measures what someone ships from a prompt with no review at all.

Carrying those rates over to unreviewed vibe-coded output is an inference, not a measurement. It runs in the direction of this article's argument, since reviewed code should be better than unreviewed code rather than worse, but nobody in this evidence set has measured that gap, and neither have we.

Where to Start

Put the console-and-network gate in front of your next generated change today. It is four lines of listener code, it fails loudly, and in our sample it was the single highest-yield check available.

After that, add the reload step and the status assertion, then decide which of the four uncovered rows in the table above you are accepting as risk and which you are going to cover with static analysis, dependency scanning, or review. Writing that decision down is what separates a quality process from an intention. If you want the fuller pipeline, the vibe coding QA workflow lays out which stages actually block a merge, and what vibe coding is covers the ground beneath all of it.

When you want a check that runs the application rather than reads it, TestMu AI's Kane CLI takes a plain-language objective, drives a real Chrome browser, and returns a verdict backed by DOM state, network responses, and screenshots. It runs the same way on a laptop and in a pipeline, and the Kane CLI documentation covers the CI setup.