Devin Review: What the AI Software Engineer Ships and Misses

What Is Devin?

Devin is an autonomous AI software engineer from Cognition that can write, run, and test code. The Devin intro in Cognition's docs gives a rule of thumb: if you can do a task in three hours, Devin can most likely do it.

Each session runs in Devin's own workspace with a shell, an IDE you can take over, and a browser. You can start that work from several places:

• Web app - an Agent session for the work itself, or Ask Devin to question the codebase and plan first.
• Chat and tickets - tag Devin in a Slack or Microsoft Teams thread, or assign it a Linear or Jira ticket.
• Devin CLI - a local agent in your terminal on macOS, Linux, WSL, and Windows that hands longer tasks to a cloud session with /handoff.
• Devin Desktop - Cognition's IDE and the successor to Windsurf, with an Agent Command Center for local and cloud agents.
• Devin API - programmatic sessions for orchestration and automation.

Cognition's 2025 performance review of Devin reports that 67% of its PRs are now merged, up from 34% a year earlier. The same post says human review is still necessary, because code quality is not straightforwardly verifiable.

Devin Changes in 2026

The Cognition blog announced Devin Desktop on June 2 and the Devin Security upgrade on October 26. This year's Devin launches, in date order:

• Devin Review (January 21) - an AI code reviewer for pull requests inside the Devin web app, free on public GitHub PRs and covered in its own section below.
• Devin 2.2 (February 24) - end-to-end testing with computer use on Devin's own Linux desktop, Devin Review Autofix, and startup that Cognition says is 3x faster, per the Devin 2.2 announcement.
• Devin Desktop (June 2) - the next generation of Windsurf, with the Agent Command Center as the default surface and Agent Client Protocol support for other agents.
• AI Productivity Guarantee (June 4) - for Enterprise customers, Cognition funds usage until Devin delivers the engineering value paid for, up to a stated cap.
• Devin Fusion (June 29) - a hybrid-model harness that pairs a frontier model with sidekick agents and dynamic routing, which Cognition says cuts cost by 60%.
• SWE-1.7 and SWE-2 (July 8 and September 10) - Cognition's own coding models, with SWE-2 described as its most advanced so far.
• FedRAMP High In-Process (July 13) - Cognition's platform listed on the FedRAMP Marketplace for federal engineering teams.
• Devin Security upgrade (October 26) - deep vulnerability scans powered by frontier cyber models, extended two days later when Corridor joined Cognition.

Devin Features Reviewed

Cognition's Computer Use docs say Devin sees the screen as a 1024x768 display and can interact with any application on its desktop, not only a browser.

The rest of Devin's features cover doing the work, understanding the codebase, and remembering what it learned:

Devin's Public Pull Request Record

A GitHub search for pull requests authored by app/devin-ai-integration returns about 327k results.

A logged-out search covers public repositories only, and devin-ai-integration is the GitHub app Devin opens pull requests with, so splitting that search by status shows how Devin's public PRs ended:

Grouping the same search by the quarter each PR was opened shows how the volume and the outcomes changed:

• Volume - public Devin PRs grew from 2,438 in Q4 2024 to 98,943 in Q3 2026, with a nearly fourfold jump from Q1 to Q2 2026.
• Merge share - of the PRs that reached a decision, 52% were merged in Q4 2024, 69% to 73% from Q2 2025 through Q1 2026, and 87% to 89% from Q2 2026 on.
• Unresolved PRs - about one in five PRs opened from Q3 2025 to Q1 2026 is still open months later, so a share of Devin's output sits for months without a merge or close decision.
• Coverage - the quarters start in October 2024 and leave out about 1,000 earlier PRs, which is why they add up to slightly less than the total.

Read the record with these limits in mind:

• Public repositories only - PRs in private repositories do not appear in a logged-out search.
• Bot-authored PRs only - Devin can also open PRs under a user's own GitHub account once that account is linked, and those PRs do not show up under the bot.
• Merged is not the same as correct - a merge in a personal repository can be a self-merge, and the record says nothing about bugs found after merging.
• Open PRs skew recent quarters - PRs opened in the last few months have had less time to be merged or closed.

How Devin Tests Its Own Work

After Devin opens a PR, it offers a Test the app button, or runs testing on its own when Pre-approve testing is turned on. Cognition's testing and recordings docs describe the workflow:

• Setup - Devin reads the PR and codebase, follows any testing skills under .agents/skills/, logs into services, and asks you for missing credentials, which it saves as secrets.
• Test plan - Devin picks the single most important end-to-end flow that proves the feature works and sends you the plan before running it.
• Recording - once CI is green and review comments are addressed, Devin records its screen while it runs the plan, annotates key moments, and sends you the video.

The same docs call the recording a quick sanity check and send you to your existing test suites and CI for more exhaustive coverage. That leaves these checks with you:

• Other flows - regressions in parts of the app the PR touched indirectly, which a one-flow recording does not visit.
• Error paths - invalid input, expired sessions, and failed API calls, unless you ask Devin to test them.
• Repeatable gates - a check that runs on every PR, whoever or whatever wrote it, with a pass or fail result your branch protection can use.

Note: Devin's recording shows one flow passing once. Kane CLI by TestMu AI runs plain-English checks against the PR's preview build on every push and attaches step screenshots, a network log, and console output to each verdict. Try TestMu AI free

Devin vs Devin Review

Devin writes pull requests, and Devin Review reads them.

Devin Review is a code review platform inside the Devin web app that turns large PRs into organized diffs and explanations. For any GitHub.com PR, replacing github.com with devinreview.com in the URL opens it in Devin Review, and public PRs need no Devin account.

How Devin Review Performed on 10 Benchmark PRs

Martian's open-source Code Review Bench holds 50 pull requests from Sentry, Grafana, Cal.com, Discourse, and Keycloak, with 173 human-verified golden comments: the real issues a reviewer should catch.

I opened 10 of those PRs in Devin Review, two per project and weighted toward Critical and High issues, then matched every finding against the golden comments, with Gemini and GPT-5 as a second judge. Martian's published results also include a Devin run from March 2026 on the same PRs:

Severity	Golden issues	Matched by Devin Review	Martian's March 2026 Devin run, same PRs
Critical	9	6	6
High	9	8	3 to 4
Medium	14	9	4 to 6
Low	5	0	1 to 2
Total	37	23	14 to 17

The test turned up these patterns:

• Strong on logic bugs - Devin Review caught unawaited async forEach calls, negative QuerySet slicing, and a null subgroup count, all Critical or High golden issues.
• Security improved most - it matched 5 of the 9 security-tagged golden comments, against 1 or 2 in the March run, and lists each vulnerability with a severity and a CWE ID.
• Flags hide correct findings - 4 of the 23 matches appeared only as investigate flags, which Devin Review does not post to GitHub unless an admin turns that on.
• Critical misses - a case-sensitive blocked-email check, a null check run on the wrong constructor argument, and an SSRF through a feed polling URL.

On the Discourse embed PR, Devin Review listed seven vulnerabilities, six of them Critical, each with a category and CWE ID:

On the Cal.com async import PR, the left rail groups the change into six sections and the Bug Catcher reports the same unawaited forEach bug at four call sites:

Devin Plans and Pricing

Devin has four self-serve plans, Free, Pro, Max, and Teams, and the docs point to Devin's pricing page for authoritative prices.

Each plan includes the following:

Cognition's usage docs explain what consumes quota, credits, or ACUs, and Windows sessions consume about 9% more usage than equivalent Linux sessions. Usage rules that affect a budget:

• What counts - the number and complexity of Devin's actions, plus virtual machine time, which is usually a small fraction of the total.
• What does not count - waiting for your reply, waiting for a test suite, and cloning repositories, apart from the few units that keep the machine running.
• Sleep - an idle session sleeps after 30 minutes by default and uses nothing while asleep.
• Platform - macOS sessions currently cost the same as Linux under promotional launch pricing.

Devin Privacy and Security

Cognition's security page states that Cognition has been SOC 2 Type II certified since September 2024 and that data is encrypted in transit and at rest.

These rules from the same page matter most for code:

• Model training - Cognition may use your data for training by default. Paid-plan users can opt out on the Data Controls page, which also enables Zero Data Retention with model providers, and Enterprise data is not used without written consent.
• Code ownership - output Devin produces is the customer's intellectual property, except that it cannot be used to train a competing model.
• Repository access - the GitHub App installation decides which repositories Devin can reach, and you can change that list later.
• Known limits - the same page warns that Devin can hallucinate, introduce bugs, or suggest insecure code, and recommends code review for its output.
• Local sandbox - the Devin CLI's OS-level sandbox works on macOS and Linux but not on Windows, where sandboxed sessions refuse to start.

Checking Devin's PRs With Kane CLI

Devin's recording proves one flow on Devin's own machine. Kane CLI by TestMu AI checks the PR's preview build from your CI instead: it drives a real Chrome browser from a plain-English objective, streams NDJSON in agent mode, and exits 0 on a pass, 1 on a failed assertion, 2 on an environment error, and 3 on a timeout, so the result can block a merge.

kane-cli run "go to $PREVIEW_URL/login, log in with {{email}} and {{password}}, open the page this PR changed, assert the new behavior is visible, assert no error message is visible" \
  --variables-file ./kane-vars.json \
  --agent --headless --timeout 300 \
  --username "$LT_USERNAME" --access-key "$LT_ACCESS_KEY"

• Devin's recording - one end-to-end flow chosen by Devin, recorded once, sent as a video.
• Kane CLI check - the flows you choose, rerun on every push, with an evidence pack of step screenshots, a HAR network log, and console output.

Devin builds, Kane CLI checks walks through that pairing for Devin's PRs, and AI code review vs verification explains why a reviewer like Devin Review and a runtime check catch different bugs.

Devin Pros and Cons

Devin's strengths come from running in its own machine across many entry points. Its weaknesses sit in task scope, testing depth, and data defaults:

Is Devin Worth It?

Devin is worth it when you can hand it work with a clear finish line.

Cognition's guide on when to use Devin asks three questions before a task: can you describe clear success criteria, is there enough context, and would breaking the task down help.

• Good fit - teams with a backlog of scoped tickets, migrations, dependency upgrades, and CI fixes, where tests and review already gate each merge.
• Weaker fit - open-ended design work, tasks without a test or CI signal, and projects that would take a person days, which Cognition advises splitting into smaller sessions.

The TestMu AI short below explains how Devin differs from a coding assistant, which tasks suit it, and why a generated pull request is not the same as a production-ready change.

If Devin's scope or pricing model does not fit, Devin alternatives compares other coding agents and how to test what each one builds.

Getting Started With Devin

Pick one scoped ticket that already has a test, run it on Devin's Free plan, and watch the recording Devin sends with the PR. Before you merge, open the PR in Devin Review and run your own check against the preview build.

For that last step, the Kane CLI with AI coding agents guide wires Kane CLI into Claude Code, Codex CLI, and Gemini CLI, and the same command runs in any CI job.

Testing from the terminal? Kane CLI by TestMu AI lets SDETs and developers plan, write, run and debug tests from the command line.

To run your first test, start free, or

book a demo

for a full walkthrough of the product.