contenox

How to review AI-written code: what to check, in order

Updated · Facts checked

WORKING WITH AGENTSReview AI-written code

To review AI-written code, ignore the agent’s summary and read the diff. Check, in this order: did it change only what the task needed, did it weaken any tests, do the functions and options it calls really exist, and does it handle errors instead of hiding them. Then run the tests and the feature yourself. Agent code looks polished whether or not it is right, so the review has to look for specific mistakes, not for style.

What it looks like in practice

The agent says: “Fixed the date parsing bug and added tests. All tests pass.”

You run:

git status
git diff --stat
 src/dates/parse.ts          |  6 +++---
 src/dates/parse.test.ts     | 14 +++++++-------
 package.json                |  1 +
 .eslintrc.json              |  2 +-

Four files for a one-function bug. The diff shows a new date library in package.json, a lint rule switched off, and two test assertions changed from exact values to “is defined”. The tests pass because they no longer test anything. That took a minute to find, and the summary mentioned none of it.

Why is reviewing AI code different?

With a human author, you can assume they understood the problem and made choices on purpose. With an agent, you can’t. The code reads well, compiles and is consistently formatted even when it solves a slightly different problem. That fluency makes it easy to skim and approve.

The mistakes are different too. People leave typos and obvious gaps. Agents leave plausible code, quiet workarounds and changes nobody asked for.

Step 1: Read the diff, not the summary

The final message is a claim, not evidence.

git status
git diff --stat
git diff

Start with --stat. If you asked for a one-line fix and see eleven files, you know where to look.

git status matters because git diff doesn’t show new files the agent created. They appear in git status as untracked. To see everything, including new files, stage it and diff the stage:

git add -A
git diff --staged

On a branch per task, compare against where you started:

git diff main...HEAD

If the agent reformatted files, git diff -w hides whitespace-only changes so the real edits stand out. See Git workflow with AI agents for the branch setup.

Step 2: Did it change only what the task needed?

Ask of every file: did this task need it?

  • Drive-by refactors: renamed variables, reordered imports, reformatted blocks. Harmless or not, they bury the real change.
  • New dependencies in package.json, go.mod or requirements.txt. Each is its own decision. Check that the package exists and is the one you think it is.
  • Config changes to CI, linters, tsconfig or build files. These often hide a workaround for something the agent couldn’t fix.
  • Deleted code. Make sure every removal was meant.

Revert what is out of scope instead of keeping it to save time. That is where regressions hide.

git restore path/to/unrelated-file

Step 3: Did it weaken the tests?

This is where agents most often take shortcuts. In the test diff, look for:

  • Assertions loosened, such as toEqual(expected) becoming toBeDefined().
  • Tests skipped, deleted or marked as expected to fail, such as a new test.skip or xit.
  • Mocks that replace the very code being tested.
  • New tests that only try the happy path.
  • Snapshot files regenerated wholesale.

Then run the tests yourself. Don’t rely on the transcript:

npm test

Coverage won’t flag the happy-path problem, and a green run won’t flag a skipped test. In the scenario in When AI Writes Your Code, DevOps Becomes the Last Line of Defense, every generated test for a coupon endpoint used the one valid code WELCOME10, none tried a malformed or oversized input, and coverage looked perfect precisely because the validation code it should have exercised did not exist. Skips hide the same way: Go integration tests gated by if os.Getenv("INTEGRATION_TESTS") == "" { t.Skip(...) } showed as passed in IDE test runners while never running. Read the counts, and in Go run go test -v and look for --- SKIP.

One more check is worth it for any bug fix: undo the fix, keep the new test, and run it. It should fail. A test that passes either way doesn’t test the fix.

Step 4: Do the APIs it calls exist?

Models sometimes call functions, options or flags that don’t exist, or that existed in an older version of a library. Compilers and type checkers catch some of this. Dynamic languages, config files and shell scripts often don’t.

For any call you don’t recognize, jump to its definition or check the docs for the version you actually have installed, not the latest.

Step 5: Does it behave correctly?

Formatting is the linter’s job. Spend your attention here:

  • Error handling: errors swallowed, logged and ignored, or turned into default values that hide the problem.
  • Edge cases: empty input, null, zero, very large values, time zones, two requests at once.
  • Security: input not validated, SQL built from strings, secrets in code or logs, permissions wider than needed.
  • Performance: queries inside loops, loading a whole table to find one row.
  • Fit: does it reuse what the codebase already has, or invent a second way of doing the same thing?

Step 6: Try it

For anything a user would notice, run the program and use the change. An agent can make the tests pass while the feature still doesn’t work end to end.

Early on, testing by hand is often the cheaper choice. The Antigravity build log in Vibecoding: How to get from 0 to SaaS in hours names the trap of automating too soon in one line: “It takes 3 minutes to test, let’s spend 2 hours to automate it! (and another hour every time we introduce a change, updating them)”. The same log shows where that stops holding: once every agent turn meant logging in and clicking through the whole app again, the manual pass took longer than the agent’s work.

The bug no diff shows: a change to a permissions table that an agent had generated months earlier. The table stored each grant by name, so renaming a grant created a new entry instead of updating the old one: new users got the new token allowance, existing users silently kept the old. Tests passed, CI was green, the rollout was green, traffic flowed. The bug surfaced only on logging in as an existing user and seeing the old allowance still granted. The write-up’s verdict: a review of the new feature’s diff could never have caught it, because nobody had thought through the original table when it was generated, so to any reviewer the diff was noise. When the code under a change was generated, the review has to cover the design the change assumes, and the manual test has to run as an existing user, not only a fresh one. The case is in How to kill a Zombie?.

Can I have another AI review it?

Yes, as a first pass, not instead of you. A fresh session that never saw the agent’s reasoning isn’t swayed by it. Claude Code’s /code-review, for example, reviews the diff in a separate subagent. Give the reviewer the original task and tell it to skip style. A reviewer asked to find problems will always find some, and fixing every nitpick adds code you didn’t need.

A reviewer model also reads the comments as intent. In the coupon scenario above, the AI reviewer approved the SQL-injectable endpoint, flagged style, and passed over the injection because a # TODO: Add input validation here comment looked like a human’s deliberate note to handle it later; the post calls this prompt injection via code comments (the # nosec half of the story is on AGENTS.md explained). Tell the reviewer to treat comments as claims to check, not as plans. More review rounds don’t fix the deeper problem either: after the model reviewed its own buildDiff helper and listed four flaws, the verdict in Should We Still Care About Our Code? was “if a human had written this, I would very politely reject both versions.”

Do I have to read every line?

For anything that ships, yes. If a diff is too big to read, the task was too big. Ask the agent to split the change, or start over with smaller tasks. Reading the diff carefully is cheaper than debugging code nobody understood.

What you can skip: lines the formatter changed, if you have checked with git diff -w that nothing else hides there.

Make the next review easier

Most review pain is set up before the agent starts:

  • Smaller tasks, smaller diffs.
  • Say “don’t change the existing tests” in the task when that applies.
  • Put repeated corrections in AGENTS.md so you stop making them. See AGENTS.md explained.
  • Watch the session, not just the result. Which files it read and which commands it ran tell you how far to trust the outcome.
  • Make risky actions ask first. See AI agent permissions.
  • Define the boundaries before the agent fills them in. A generated buildDiff helper, part of about 3,000 generated lines, compiled, passed its tests and read fine, and still called os.ReadFile on a path nobody had defined: relative to what, inside which filesystem, with symlinks allowed or not. The model had guessed, and its guesses compiled. The conclusion in Should We Still Care About Our Code?: review doesn’t scale line by line, constraints do. What a path means, which files are reachable and which errors must be handled belong in AGENTS.md or the task; left undefined, the model decides, and the tests won’t notice.

In the Contenox desktop app, the Diff pane lists the files a session’s edits changed and shows each one’s diff. Changes made by commands the agent ran, such as a formatter or a code generator, don’t appear there, so finish with git status and git diff.

Review checklist

  • git status and git diff --stat read, then the full diff
  • New, untracked files checked
  • Every changed file needed for the task
  • No surprise dependencies, config or CI changes
  • No tests weakened, skipped or deleted
  • New tests fail without the fix
  • Tests run by you
  • Unfamiliar APIs checked against the docs for your version
  • Errors handled, not swallowed
  • Edge cases and security considered
  • Change tried by hand where a user would notice it

Related

Esc to close